Crunchbits - PA: Degraded Performance – Incident details

PA: Degraded Performance

Monitoring
Operational
Started 17 days ago

Affected

Core Network: PA

Partial outage from 7:50 AM to 7:59 PM, Major outage from 7:59 PM to 9:13 PM, Degraded performance from 9:13 PM to 3:22 PM, Operational from 3:22 PM to 12:00 AM

VM Hypervisors - Valley Forge

Operational from 7:50 AM to 7:59 PM, Major outage from 7:59 PM to 9:13 PM, Degraded performance from 9:13 PM to 3:22 PM, Operational from 3:22 PM to 12:00 AM

A-06-U02

Operational from 7:50 AM to 7:59 PM, Major outage from 7:59 PM to 9:13 PM, Degraded performance from 9:13 PM to 3:22 PM, Operational from 3:22 PM to 12:00 AM

A-06-U01

Operational from 7:50 AM to 7:59 PM, Major outage from 7:59 PM to 9:13 PM, Degraded performance from 9:13 PM to 3:22 PM, Operational from 3:22 PM to 12:00 AM

A-06-U07-N2

Operational from 7:50 AM to 7:59 PM, Major outage from 7:59 PM to 9:13 PM, Degraded performance from 9:13 PM to 3:22 PM, Operational from 3:22 PM to 12:00 AM

A-06-U07-N4

Operational from 7:50 AM to 7:59 PM, Major outage from 7:59 PM to 9:13 PM, Degraded performance from 9:13 PM to 3:22 PM, Operational from 3:22 PM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    All our peers seem to have recovered from the recent outages: fiber cuts have been restored and failing hardware replaced. We are back to full capacity but will continue to monitor the situation.

  • Identified
    Identified

    Public internet connectivity has now been restored. Your services should now be accessible, though we're still operating in a degraded state due to one of our circuits still being down. Work continues toward full resolution.

  • Update
    Update

    Our network engineers are now working with on-site remote hands. Investigation continues - we'll post more information as soon as it's available.

  • Update
    Update

    Our backup circuit is currently down. We're still investigating with no ETA at this time. Further updates to follow.

  • Investigating
    Investigating

    We're still investigating this issue. No estimated time for resolution at this time. We'll post updates as soon as we have more information.

  • Update
    Update

    We are continuing to work with our vendors on resolving the situation. In the meantime we have activated a backup circuit to help alleviate potential issues our customers could be facing.

    Arelion just informed us that they are having to "actively chase" their vendors who have gone dark and stopped providing updates. Overall, the severe weather that affected Eastern PA Monday evening has caused widespread damage and the contracted repair crews are currently thinly spread out.

  • Identified
    Identified

    Our PA network is currently operating in a degraded state with each of our diverse circuits being at risk for outage. One is facing a partial fiber cut, while the other is currently up but under a maintenance window.

    We apologize for any inconvenience that this may cause.