Archive for September, 2019

Incident: Service Outage

Posted: Friday, September 27th, 2019 at 16:09 by Adam Heath

Wednesday 25th September 22:30 – Thursday 26th September 01:04

A change to standardise access lists on some of our core network devices created an unexpected impact whereby some core devices were unable to learn/send routes from our route reflectors.

We have raised a vendor case and updated our regression test procedures to avoid future issues. An RFO will follow in due course.

  1. Incident Management says:

    Impact: Services delivered through some core devices may have experienced a total loss of service.

    Root Cause: In an effort to standardise our management plane access lists across devices through a non-service affecting change, an unforeseen and adverse effect caused the control plane of each device to react to the change. This caused the core devices to lose their established BGP sessions to our route reflectors. Once the issue was identified, roll back was initiated to the pre-change baseline and service restored. We are continuing to liaise with our vendor to gather the root cause of the outage.

    Additional Actions: A review of the failed change has been carried out and actions to prevent recurrence of this unexpected impact and service outage have been identified and will be progressed.

Resolved incident: VoIP outbound calls

Posted: Thursday, September 26th, 2019 at 17:23 by david labouchardiere
We are aware of an incident earlier this morning caused by our anti-fraud measures briefly blocking outbound calls. This issue has been resolved and we are making changes to ensure this is not repeated.
Apologies for any inconvenience caused.

Incident: VoIP calls

Posted: Wednesday, September 25th, 2019 at 13:09 by david labouchardiere

We are currently aware of an issue affecting calls on the VoIP platform. We are investigating the problem and will provide further updates as soon as possible.

Please accept our apologies for any inconvenience caused.

 

  1. david labouchardiere says:

    Our engineers have now confirmed service has been restored. Please ask any affected customers to try rebooting their devices before contacting the support desk.

Incident: Inbound calls

Posted: Friday, September 20th, 2019 at 10:25 by David Brewis

We are aware of an issue affecting inbound calls into the Telford office, all other systems remain operational.  We are currently investigating and will update shortly.

  1. David Brewis says:

    Service has restored, however whilst we investigate please consider this at risk

Emergency Maintenance: slough.core

Posted: Friday, September 13th, 2019 at 16:36 by Robin Posadas

Tuesday 17th September 2019, 00:01 – 03:00

During the above window our supplier will be carrying out essential power works to one of our racks in this location. We expect all services currently terminated in this location to have up to 1 hour downtime. Services Include Leased lines and Pseudowires.

All other traffic that would normally route through this location will take alternative paths.

  1. Gurinder Bansal says:

    This work is now complete.

At Risk: Newcastle POP

Posted: Wednesday, September 4th, 2019 at 17:06 by Iain Beveridge

Wednesday 4th September 2019, 21:00 – 22:00

After a power check within Newcastle POP, BT identified an issue with part of their power infrastructure within one of our racks. BT will undertake emergency work in the above window to swap out a faulty UPS.

No downtime is expected however all directly connected services should be considered at risk.

  1. All the work completed without incident.