Planned Maintenance: Telehouse North

Posted: February 8th, 2010 at 12:35 by Ming-Yu Hsieh

Friday 19th Feb 2010, 00:01hrs – 06:00hrs

During the maintenance window above, we will be working within the cabinet and recovering old equipment. This site must be considered to be at risk during this period.

Essential VoIP Maintenance 9th Feb 2010

Posted: February 8th, 2010 at 11:40 by Jon Farmer

We need to perform some essential maintenance on the main VoIP database. This is scheduled for 4am on 9th February 2010. During the hours of 4am and 5am you may not be able to make or receive calls. We will endeavour to keep the downtime to a minimum.

Outage: Manchester to London

Posted: February 4th, 2010 at 16:15 by David Derrick

We are currently seeing errors on our link between London and Manchester. To maintain network stability, we are taking this link out of service while investigations take place. All traffic will route via alternate paths, some customers may notice a small increase in latency.

  1. David Derrick says:

    Our circuit provider has identified a possible fault with a card in Manchester. Engineers are working to confirm this, there is no ETR at this time. Further updates will follow as we receive them.

  2. Steve Lalonde says:

    Our provider has cleared the fault, we will test the circuit for a errors over the next hour.

  3. Steve Lalonde says:

    We have seen no more errors on the circuit and it is therefore now back in service. We will continue to monitor as the load increases.

Emergency Maintenance: telford-dc3.core.enta.net

Posted: February 1st, 2010 at 12:34 by Steve Lalonde

Tuesday 2nd Feb 2010, 03:00hrs – 04:00hrs

To resolve the intermittent packet loss issue on the Telford to Wolverhampton link it has become necessary to reload telford-dc3.core.enta.net. This emergency work should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

  1. Steve Lalonde says:

    This work was completed successfully. Traffic is now on the preferred paths and full resilience is restored.

At-Risk: Loss of Resilience Telford

Posted: February 1st, 2010 at 11:52 by Steve Lalonde

The Circuit between Telford and Wolverhampton has been taken off-line while we investigate intermittent packet loss issues with the circuit. Traffic usually on this path is using an alternative route. Customers may experience an increase in latency to some destinations.

  1. Steve Lalonde says:

    We have isolated the cause of the intermittent packet loss on this link to a configuration error. Unfortunately to resolve the issue requires a reload of the router. An emergency maintenance notification will follow.

Outage: Telehouse-north0.core

Posted: January 22nd, 2010 at 13:08 by Martyn Haynes

We are seeing issues accross the network that inital investigation shows could be related to a possible fault in Telehouse-north0.core. Our engineers are investigating these issues as a matter of urgency. Further information will be provided when available.

  1. Martyn Haynes says:

    We have stabilised the issue however it is not resolved. An Engineer is tasked to site to investigate further. Further updates will be posted when available.

  2. Steve Lalonde says:

    We need to reload the router to eliminate the IOS version as the cause. the reload will be at 15:55

    All traffic that flows via the router has been rerouted over alternative paths where possible, all directly connected customers will experience an outage of approx 10 minutes.

  3. Steve Lalonde says:

    The router is now operational again we will keep the traffic routed over alternative paths while we monitor the system.

  4. Steve Lalonde says:

    We are now routing approximately 1/3 of the normal traffic back through the router, the current priority is to get all the resiliency that relies on this hardware operating as normal. This should be achieved in the next 30 minutes.

    We will continue to monitor the situation and update.

  5. Steve Lalonde says:

    All resilient links are now operational, we are continuing to route traffic over alternate paths.

  6. Steve Lalonde says:

    After monitoring for more than 24 hours, the router is stable and functioning normally, We have now routed traffic back on to the optimal paths.

  7. Richard Partridge says:

    There is now an outage report available for this issue. It can be found here.

Planned work: telford-dc3.core.enta.net

Posted: January 20th, 2010 at 15:26 by Steve Lalonde

Thursday 28th Jan 2010, 03:30hrs – 04:00hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Steve Lalonde says:

    This work was completed successfully.

Planned work: telford-dc1.core.enta.net

Posted: January 20th, 2010 at 15:22 by Steve Lalonde

Thursday 28th Jan 2010, 03:00hrs – 03:30hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Steve Lalonde says:

    This work was completed successfully.

Emergency work: man-te.core.enta.net

Posted: January 20th, 2010 at 12:33 by Steve Lalonde

Due to router instability we have rebooted this router.

Outage: Birmingham.core

Posted: January 19th, 2010 at 18:16 by Martyn Haynes

Due to advance preparation for tonight’s maintenance work, Birmingham.core has unexpectedly reloaded. Customers connected via this node will have suffered a short (<4 minutes) outage. Please accept our apologies for any inconvenience this may have caused.

As a result of this unexpected reload, tonight’s maintenance window for Birmingham.core is no longer needed.