Archive for December, 2013

Preliminary update on power outage Sunday 15th December

Posted: Monday, December 16th, 2013 at 13:24 by Steve Lalonde

Whilst we continue to gather detailed information on the power problems experienced in Telford yesterday to create an outage report, here is a preliminary description of events.

At approximately 10:00am a significant local mains power issue resulted in a loss of power to Entanet’s Telford headquarters and other properties in the area. Our backup generator took over as intended and continued supplying power to our datacentres. All customer services remained uninterrupted.

At approximately between 10.11am and 10.30am local mains power resumed and the generators returned to standby.

Local mains power, although having resumed, was fluctuating following its reinstatement and was lost a second time at approximately 12:00pm. However, the mains supply had experienced a power surge that had tripped the breakers supplying power to the building. This meant that, without manual intervention to reset these breakers, the generator was unable to engage. The UPS systems maintained the load until their batteries were drained. As a consequence of this power interruption, all equipment in our Telford headquarters was affected and customers who are directly connected to our Telford datacentres experienced service interruption from approximately 12:00pm until the power was manually restored to the building at approximately 12.30pm.

From approximately 12:30pm our engineers worked on restoring services in the datacentres on a priority basis (i.e. starting with core services). While the majority of customers’ services were restored from approximately 1:00pm, a small number of customers experienced a longer period of interruption. Our engineers have continued to work on restoring any remaining services.

We continue to gather information from our power suppliers and analyse the sequence of events affecting our generators. We will provide an official outage report once this information has been gathered.

We apologise to customers who have been affected by this issue.

  1. Wayne Hawtin says:

    An incident report for this issue is now available here.

Emergency VoIP Platform Maintenance.

Posted: Monday, December 16th, 2013 at 12:43 by Adam Heath

This is to advise that between 8pm and 10pm on Monday 16th December 2013 we will be performing emergency maintenance on the database servers for the VoIP platform. This work has been planned to not have any effect on the availability of the platform to end users while the work is being undertaken.

 

  1. Jon Farmer says:

    This maintenance is now complete.

Power failure Telford

Posted: Sunday, December 15th, 2013 at 13:37 by Steve Lalonde

We have power problems in Telford

most services are restored

engineers are working on the remaining problems

 

  1. Steve Lalonde says:

    Quick update:

    The phone system is now operational

    a few systems are running at reduced capacity

    engineers are continuing to work on resolving the remaining issues