Archive for January, 2010

Outage: Telehouse-north0.core

Posted: Friday, January 22nd, 2010 at 13:08 by Martyn Haynes

We are seeing issues accross the network that inital investigation shows could be related to a possible fault in Telehouse-north0.core. Our engineers are investigating these issues as a matter of urgency. Further information will be provided when available.

  1. Martyn Haynes says:

    We have stabilised the issue however it is not resolved. An Engineer is tasked to site to investigate further. Further updates will be posted when available.

  2. Steve Lalonde says:

    We need to reload the router to eliminate the IOS version as the cause. the reload will be at 15:55

    All traffic that flows via the router has been rerouted over alternative paths where possible, all directly connected customers will experience an outage of approx 10 minutes.

  3. Steve Lalonde says:

    The router is now operational again we will keep the traffic routed over alternative paths while we monitor the system.

  4. Steve Lalonde says:

    We are now routing approximately 1/3 of the normal traffic back through the router, the current priority is to get all the resiliency that relies on this hardware operating as normal. This should be achieved in the next 30 minutes.

    We will continue to monitor the situation and update.

  5. Steve Lalonde says:

    All resilient links are now operational, we are continuing to route traffic over alternate paths.

  6. Steve Lalonde says:

    After monitoring for more than 24 hours, the router is stable and functioning normally, We have now routed traffic back on to the optimal paths.

  7. Richard Partridge says:

    There is now an outage report available for this issue. It can be found here.

Planned work: telford-dc3.core.enta.net

Posted: Wednesday, January 20th, 2010 at 15:26 by Steve Lalonde

Thursday 28th Jan 2010, 03:30hrs – 04:00hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Steve Lalonde says:

    This work was completed successfully.

Planned work: telford-dc1.core.enta.net

Posted: Wednesday, January 20th, 2010 at 15:22 by Steve Lalonde

Thursday 28th Jan 2010, 03:00hrs – 03:30hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Steve Lalonde says:

    This work was completed successfully.

Emergency work: man-te.core.enta.net

Posted: Wednesday, January 20th, 2010 at 12:33 by Steve Lalonde

Due to router instability we have rebooted this router.

Outage: Birmingham.core

Posted: Tuesday, January 19th, 2010 at 18:16 by Martyn Haynes

Due to advance preparation for tonight’s maintenance work, Birmingham.core has unexpectedly reloaded. Customers connected via this node will have suffered a short (<4 minutes) outage. Please accept our apologies for any inconvenience this may have caused. As a result of this unexpected reload, tonight's maintenance window for Birmingham.core is no longer needed.

Planned work: telehouse-east2.core.enta.net

Posted: Tuesday, January 19th, 2010 at 16:11 by Steve Lalonde

Wednesday 27th Jan 2010, 03:30hrs – 04:00hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Ming-Yu Hsieh says:

    This work was completed successfully.

Planned work: telehouse-east.core.enta.net

Posted: Tuesday, January 19th, 2010 at 16:09 by Steve Lalonde

Wednesday 27th Jan 2010, 03:00hrs – 03:30hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Ming-Yu Hsieh says:

    This work was completed successfully.

Emergency work: noc.enta.net

Posted: Monday, January 18th, 2010 at 14:10 by Steve Lalonde

We will be moving noc.enta.net to a new server in the next few hours as the existing server has become unstable.

  1. Steve Lalonde says:

    We have now finished the move of noc.enta.net to a new server.

Planned work: telecity-hex.core.enta.net

Posted: Monday, January 18th, 2010 at 13:33 by Steve Lalonde

Tuesday 26th Jan 2010, 04:30hrs – 05:00hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Steve Lalonde says:

    This work was completed successfully.

Planned work: redbus-sov.core.enta.net

Posted: Monday, January 18th, 2010 at 13:32 by Steve Lalonde

Tuesday 26th Jan 2010, 04:00hrs – 04:30hrs

During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.

Traffic that would normally be routed through this equipment will route via alternate paths.

  1. Steve Lalonde says:

    The router failed to reboot correctly, an engineer is now onsite.

  2. Steve Lalonde says:

    The router is now operational again. This maintenance work is now closed.