Incident: Wolverhampton TE Edge Device

Posted: October 16th, 2017 at 14:00 by Richard Partridge

We have just experience a brief loss of communication to one our edge devices in Wolverhampton TE. This resulted in a brief loss of service for directly terminated leased lines and broadband users. Service appears to have been restored however engineers are currently investigating.

Incident: colindale.core

Posted: October 16th, 2017 at 09:04 by Alistair Patterson

Our core router in Colindale has unexpectedly rebooted. All services appear to have recovered, investigations are underway to determine the cause.

An engineer has been dispatched to site.

Further updates to follow when available.

Incident: Nottingham core

Posted: October 16th, 2017 at 02:13 by David Derrick

Our Nottingham core router has suffered a power outage and one of its blades has failed. Engineers are preparing to head to site with replacement equipment.

  1. David Derrick says:

    The failed blade has been replaced and service has been restored.

Incident: Columbo

Posted: October 15th, 2017 at 19:25 by David Derrick

Our core router in Columbo has unexpectedly rebooted. All services appear to have recovered, we will continue to monitor the situation.

Planned Maintenance: Cardiff

Posted: October 11th, 2017 at 16:34 by Jake Turner

Thursday 19th October 2017 00:01 – 06:00

During the above window we will be replacing equipment at this site.

All services currently terminated in this location will be migrated on to the new hardware. This includes all customer transit, DSL, EWCS, wholesale hand-off and pseudowires terminating at this location.

All other traffic that would normally route through this location will take alternative paths.

  1. Richard Partridge says:

    This work has now been cancelled.

Incident: cardiff.core

Posted: October 10th, 2017 at 18:08 by David Derrick

We have lost connectivity to our Cardiff router. Initial investigation suggests it is rebooting. We will continue to investigate and update when we know more.

  1. David Derrick says:

    The router has finished reloading and services are restored. We will continue to investigate the cause.

  2. Richard Partridge says:

    The cardiff.core router appears to have sustained another unexpected reload. Engineers are currently investigating. Further information will follow shortly.

  3. Richard Partridge says:

    Engineers are being dispatched to Cardiff to continue the investigation. Due to the number of recent failures, we are also planning to replace the core router whilst on site, which was originally planned for next week.

    Once engineers are on site and have assessed the situation, a notification outlining an emergency maintenance window will be published. The site should be considered at-risk until otherwise notified.

  4. Richard Partridge says:

    Engineers are on site and have concluded their initial investigation. Based on our findings we are currently proceeding with our plan to replace the core router this evening. A downtime window will be announced in due course.

  5. Richard Partridge says:

    We are planning to replace the Cardiff core router between 18:00 and 20:00 this evening. The core itself should be back online within 30 minutes during the window, however individual services will take longer to restore as they are migrated in turn from one core to the other.

  6. Jake Turner says:

    Further investigation on site within the maintenance window highlighted a fault with the power infrastructure that will be addressed by our colocation provider. We have taken measures to prevent further outages in the interim. As such the core router was not swapped.

Planned Maintenance: Colindale

Posted: October 9th, 2017 at 13:34 by Jake Turner

Wednesday 18th October 2017 00:01 – 06:00

During the above window we will be replacing colindale.core.enta.net with a higher capacity router.

All services currently terminated in this location will be migrated on to the new hardware. This includes DSL, EWCS, wholesale hand-off and pseudowires terminating at this location.

All other traffic that would normally route through this location will take alternative paths.

  1. Jake Turner says:

    This work is now complete

Planned Maintenance: Columbo

Posted: October 9th, 2017 at 13:27 by Jake Turner

Tuesday 17th October 2017 00:01 – 06:00

During the above window we will be replacing columbo.core.enta.net with a higher capacity router.

All services currently terminated in this location will be migrated on to the new hardware. This includes DSL, EWCS, wholesale hand-off and pseudowires terminating at this location.

All other traffic that would normally route through this location will take alternative paths.

  1. Jake Turner says:

    This work has been cancelled

Incident: Vodafone Circuits

Posted: October 6th, 2017 at 19:34 by Mark Yardley

We are aware of a loss of connectivity for a number of Vodafone Ethernet circuits. Initial diagnostics show the affected circuits are delivered on the same NNI that has recently suffered disruption of service. We are liaising with our suppliers to get the issue investigated further and resolved as quickly as possible. Further updates will be provided as they become available.

  1. David Brewis says:

    Following investigations, Vodafone are arranging to dispatch engineers tonight.

    Further updates to follow once engineers have attended site.

  2. Mark Yardley says:

    Vodafone engineers have attended their site to replace suspect faulty equipment. We currently see the affected services have been restored but should still be considered at risk until our suppliers have advised fault is cleared.

  3. Mark Yardley says:

    These services have again lost connectivity and Vodafone are investigating further. Updates to be provided once available.

  4. David Brewis says:

    Vodafone have escalated the incident and are dispatching engineers to restore service. Further updates to follow as made available.

  5. David Brewis says:

    Connections are restoring to service following hardware replacement, until fault closure consider service at risk.

  6. David Brewis says:

    After 48 hours of monitoring, we have received correspondence from the supplier to show the fault is now closed.

Emergency Work: Sheffield broadband users

Posted: October 5th, 2017 at 13:29 by Steve Lalonde

Friday 6th October 2017 01:00 – 06:00

During the above window we will be performing maintenance on the Sheffield broadband interconnects

We expect a small number of users to be disconnected during the work and should reconnect automatically after a few seconds.

It is possible some users may need to reboot equipment to reconnect.

  1. Steve Lalonde says:

    This work is now complete