Friday 19th Feb 2010, 00:01hrs – 06:00hrs
During the maintenance window above, we will be working within the cabinet and recovering old equipment. This site must be considered to be at risk during this period.
We need to perform some essential maintenance on the main VoIP database. This is scheduled for 4am on 9th February 2010. During the hours of 4am and 5am you may not be able to make or receive calls. We will endeavour to keep the downtime to a minimum.
We are currently seeing errors on our link between London and Manchester. To maintain network stability, we are taking this link out of service while investigations take place. All traffic will route via alternate paths, some customers may notice a small increase in latency.
Tuesday 2nd Feb 2010, 03:00hrs – 04:00hrs
To resolve the intermittent packet loss issue on the Telford to Wolverhampton link it has become necessary to reload telford-dc3.core.enta.net. This emergency work should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.
Steve Lalonde says: This work was completed successfully. Traffic is now on the preferred paths and full resilience is restored.
The Circuit between Telford and Wolverhampton has been taken off-line while we investigate intermittent packet loss issues with the circuit. Traffic usually on this path is using an alternative route. Customers may experience an increase in latency to some destinations.
Steve Lalonde says: We have isolated the cause of the intermittent packet loss on this link to a configuration error. Unfortunately to resolve the issue requires a reload of the router. An emergency maintenance notification will follow.
We are seeing issues accross the network that inital investigation shows could be related to a possible fault in Telehouse-north0.core. Our engineers are investigating these issues as a matter of urgency. Further information will be provided when available.
Martyn Haynes says: We have stabilised the issue however it is not resolved. An Engineer is tasked to site to investigate further. Further updates will be posted when available.
Steve Lalonde says: We need to reload the router to eliminate the IOS version as the cause. the reload will be at 15:55
All traffic that flows via the router has been rerouted over alternative paths where possible, all directly connected customers will experience an outage of approx 10 minutes.
Steve Lalonde says: The router is now operational again we will keep the traffic routed over alternative paths while we monitor the system.
Steve Lalonde says: We are now routing approximately 1/3 of the normal traffic back through the router, the current priority is to get all the resiliency that relies on this hardware operating as normal. This should be achieved in the next 30 minutes.
We will continue to monitor the situation and update.
Steve Lalonde says: All resilient links are now operational, we are continuing to route traffic over alternate paths.
Steve Lalonde says: After monitoring for more than 24 hours, the router is stable and functioning normally, We have now routed traffic back on to the optimal paths.
Richard Partridge says: There is now an outage report available for this issue. It can be found here.
Thursday 28th Jan 2010, 03:30hrs – 04:00hrs
During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.
Traffic that would normally be routed through this equipment will route via alternate paths.
Steve Lalonde says: This work was completed successfully.
Thursday 28th Jan 2010, 03:00hrs – 03:30hrs
During the maintenance window above the IOS on the router will be upgraded to resolve some configuration issues. This software upgrade will require a reboot of the router which should take approx 10 minutes to complete, during which all directly connected customers will lose connectivity.
Traffic that would normally be routed through this equipment will route via alternate paths.
Steve Lalonde says: This work was completed successfully.
Due to router instability we have rebooted this router.
Due to advance preparation for tonight’s maintenance work, Birmingham.core has unexpectedly reloaded. Customers connected via this node will have suffered a short (<4 minutes) outage. Please accept our apologies for any inconvenience this may have caused.
As a result of this unexpected reload, tonight’s maintenance window for Birmingham.core is no longer needed.
Our circuit provider has identified a possible fault with a card in Manchester. Engineers are working to confirm this, there is no ETR at this time. Further updates will follow as we receive them.
Our provider has cleared the fault, we will test the circuit for a errors over the next hour.
We have seen no more errors on the circuit and it is therefore now back in service. We will continue to monitor as the load increases.