Archive for the ‘Emergency Work’ Category

Emergency Maintenance: DSL network

Posted: Monday, April 11th, 2016 at 16:12 by Steve Lalonde

Tuesday 12 April 2016 0:00 ~ 8:00

During the above window we will be making some changes to the DSL network to reduce the packet loss some users have been reporting.

The work may cause some users to disconnect/reconnect

No other impact is expected.

  1. Steve Lalonde says:

    This work was completed at approx 8:30

Emergency Supplier Maintenance: Peterborough & Colindale MSIL card changes

Posted: Monday, February 15th, 2016 at 14:25 by Scott Morgan

Tuesday 16th February 2016 00:01 – 06:00

During the above window our suppliers will be upgrading hardware and software for the following exchanges:

(Liverpool Birkenhead, London Stamford Hill, Manchester Denton, Newcastle West, Norwich, Peterborough)

This will impact DSL services routed through our Peterborough and Colindale nodes.
The PW window is from 00:01 until 06:00 and work will start at 00:01 when non-disruptive pre-checks will be performed. There will be no service impact between 00:01 and 01:00. From 02:01 our suppliers will start to re-boot the device to the new version of code. This will cause an outage to all customers of between 10 and 15 minutes whilst the device restarts. However, if roll-back is required,the outage will be extended to 90 minutes. Some customers will experience a second outage of up to 15 minutes while BT upgrade the interface card connecting their circuit. This outage could be extended to 45 minutes if roll back is required. All work will be complete by 06:00.


Emergency Work: VoIP Platform

Posted: Thursday, December 10th, 2015 at 17:09 by Steve Lalonde

Friday 11 December 2015 00:00-06:00 GMT

Following on from this weeks VoIP platform issues we will be replacing the primary Database server.

We expect the work to take approximately 2 hours during which VoIP platform will be unavailable.

The back out plan will be to revert to the existing server.

  1. Ming-Yu Hsieh says:

    we are still working on the database server.

  2. Ming-Yu Hsieh says:

    This work is now complete.

Emergency maintenance: VOIP systems

Posted: Tuesday, December 8th, 2015 at 12:39 by David Derrick

Following on from Saturday’s disruption we have discovered a problem with the VOIP databases which are affecting calls for some customers. Attempts to gracefully recover have failed so we will have to briefly stop and restart the database. This may cause calls to drop. Downtime will be less than 60 seconds and is expected to happen at 12:45.

  1. David Derrick says:

    The database has restarted successfully.

  2. Adam Heath says:

    There seems to have been a continuation of the database issue we previously identified which may be causing issues with inbound calls and amendments to call forwards. We are investigating on our side and will provide further information when it becomes available.

  3. Adam Heath says:

    We have resolved the immediate database issue and services now appear to be working again. Further remedial work is required and will be scheduled in due course.

Emergency maintenance: Sonicwall GMS

Posted: Thursday, November 12th, 2015 at 10:47 by David Derrick

Friday 13 November 2015 11:00-13:00

During the above window we will be applying the latest service pack to our GMS deployment to resolve a number of urgent issues. Customers and support team members will be unable to view reports and make changes although the NOC team can still access appliances directly to make urgent changes.

Please note that the appliances themselves are unaffected by this work and the security of customer networks is not at risk.

  1. David Derrick says:

    This work is complete. Users may need to clear their browser cache if they have trouble logging in.

Emergency Work: Telford UPS

Posted: Thursday, August 13th, 2015 at 13:05 by Mark Yardley

We have experienced a failure of one of the UPS devices that protects telford-dc3.core. We will shortly be undertaking work to replace the failed unit and restore full power redundancy. Services connected to or through telford-dc3.core should be considered at risk until the work is completed.

Further updates will be provided within the hour.

  1. Mark Yardley says:

    Engineers are currently progressing with the work. Further updates will follow shortly.

  2. Mark Yardley says:

    The work is now complete and UPS redundancy has been restored.

Emergency maintenance: Sonicwall GMS

Posted: Wednesday, May 20th, 2015 at 11:50 by David Derrick

Thursday 21 May 2015 11:00-13:00

During the above window we will be replacing the GMS appliance which provides customer access to their managed firewalls. Existing user details will be exported and imported onto the new system however there is a risk that saved reports and passwords may be lost.

Please note that the appliances themselves are unaffected by this work and the security of customer networks is not at risk. Any urgent changes to firewalls can be made via the normal support channels.

Emergency Maintenance: Multiple routers

Posted: Tuesday, March 24th, 2015 at 16:57 by Steve Lalonde

Tuesday 24 March 2015 18:00 ~ 23:59

We will be applying a Hit-Less software update to the following list of routers:


This is the same update applied to that resolved the instability seen last week.

  1. Steve Lalonde says:

    This work is now complete.

Telephone system problems

Posted: Saturday, March 21st, 2015 at 10:56 by Jake Turner

We are currently experiencing problems with our in-office telephone system and are intermittently unable to take support calls. Engineers are working on rectifying the situation as a matter of urgency and will ensure that normal service is resumed as soon as possible. As an alternative method of contact please email

  1. Jake Turner says:

    This issue was resolved at approximately 11:20. After extensive monitoring we can confirm that calls are routing through to the support lines as expected. Please accept our apologies for any inconvenience caused.

Power Maintenance: Interxion, London

Posted: Wednesday, September 17th, 2014 at 11:34 by David Derrick

Saturday 20 September 2014 08:00 to 17:30 BST

Today we have received notification that Interxion will be performing the previously postponed intrusive power upgrade work on both critical and non-critical feeds during the above maintenance window.

The following is the notice received:
“All upgrade works will be undertaken by specialist contractors under the direct supervision of Interxion staff.

The works below are planned for Saturday 20th September. The upgrade works will consist of the non-critical electrical supplies being transferred onto new UPS system (feed ‘B’.) Following satisfactory confirmation of the transfer of the Non critical electrical supplies, we plan to transfer the critical loads onto new UPS system (feed ‘A’). Dual corded equipment if wired correctly should not be impacted. However, single corded equipment WILL be impacted.”

Entanet equipment has already been moved to dual fed supplies so will not be affected although the site should be considered at risk during the maintenance window. Colocated customers with single fed equipment will however be affected.

  1. David Derrick says:

    We have been advised that Interxion are postponing the work planned for this weekend. We will post details of the new schedule when we have more details.