Archive for the ‘Incidents’ Category

Resolved Incident: VoIP calls

Posted: Thursday, October 10th, 2019 at 15:35 by david labouchardiere

Good afternoon.

We are aware of an issue that was affecting some users on our VOIP platform. We believe the issue has now been resolved and request any users affected to now retest the service.

Please accept our apologies for any inconvenience caused.

Incident: VoIP calls

Posted: Tuesday, October 1st, 2019 at 20:10 by David Brewis

Good evening.

We are currently aware of an issue affecting calls on the VoIP platform. We are investigating the problem and will provide further updates as soon as possible.

Please accept our apologies for any inconvenience caused.

  1. David Brewis says:

    Engineers continue to investigate with the aim to identify the cause disrupting service across our VoIP platform. We have been able to replicate the issue internally assisting with troubleshooting steps.

    We will continue to update with updates and progress as troubleshooting continues.

  2. Robin Posadas says:

    Engineers are continuing to investigate the problem and we will continue to update as further troubleshooting continues.

  3. Robin Posadas says:

    Engineers have now confirmed service has been restored. Any customers still experiencing issues should reboot their equipment before contacting our support department.

    We apologise for any Inconveniences caused by this outage

Incident: Service Outage

Posted: Friday, September 27th, 2019 at 16:09 by Adam Heath

Wednesday 25th September 22:30 – Thursday 26th September 01:04

A change to standardise access lists on some of our core network devices created an unexpected impact whereby some core devices were unable to learn/send routes from our route reflectors.

We have raised a vendor case and updated our regression test procedures to avoid future issues. An RFO will follow in due course.

  1. Incident Management says:

    Impact: Services delivered through some core devices may have experienced a total loss of service.

    Root Cause: In an effort to standardise our management plane access lists across devices through a non-service affecting change, an unforeseen and adverse effect caused the control plane of each device to react to the change. This caused the core devices to lose their established BGP sessions to our route reflectors. Once the issue was identified, roll back was initiated to the pre-change baseline and service restored. We are continuing to liaise with our vendor to gather the root cause of the outage.

    Additional Actions: A review of the failed change has been carried out and actions to prevent recurrence of this unexpected impact and service outage have been identified and will be progressed.

Incident: VoIP calls

Posted: Wednesday, September 25th, 2019 at 13:09 by david labouchardiere

We are currently aware of an issue affecting calls on the VoIP platform. We are investigating the problem and will provide further updates as soon as possible.

Please accept our apologies for any inconvenience caused.

 

  1. david labouchardiere says:

    Our engineers have now confirmed service has been restored. Please ask any affected customers to try rebooting their devices before contacting the support desk.

Incident: Inbound calls

Posted: Friday, September 20th, 2019 at 10:25 by David Brewis

We are aware of an issue affecting inbound calls into the Telford office, all other systems remain operational.  We are currently investigating and will update shortly.

  1. David Brewis says:

    Service has restored, however whilst we investigate please consider this at risk

Incident: Vodafone Circuits

Posted: Wednesday, August 21st, 2019 at 10:25 by Jamie Smith

We are currently seeing a number of Vodafone circuits a down. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Jamie Smith says:

    Services have now been restored to the affected handoff. We are continuing to liaise with Vodafone to determine the cause of this outage

Incident : VoIP calls

Posted: Friday, August 16th, 2019 at 12:19 by david labouchardiere

We are aware of an outage affecting calls on the VoIP platform at approximately 11:00 today. Service is now restored and our engineers are investigating the underlying cause.

Please ask any affected customers to reboot their equipment if they continue to experience issues.

We apologise for any inconvenience caused.

Edinburgh connectivity

Posted: Friday, August 9th, 2019 at 08:33 by Alistair Patterson

We are currently aware of a number of City Fibre circuits in Edinburgh have lost connectivity. Engineers are currently investigating. Updates will be provided as soon as available.

  1. Jamie Smith says:

    Engineers are currently still Investigating the Issue affecting customers terminating on this Core Device, a reboot of the device has taken place in an effort to restore services. Further updates to follow as and when available.

  2. Jamie Smith says:

    Engineers have now restored connectivity to affected customers, any customers still experiencing issues should perform Initial troubleshooting before contacting our support department.

    We apologise for any Inconveniences caused by this outage.

Incident: PWAN Connectivity

Posted: Wednesday, August 7th, 2019 at 11:14 by Jordan Coleman

We are aware of an incident affecting some PWAN connectivity. Engineers are already engaged and are investigating.

Further updates will be provided as and when they become available.

  1. Jordan Coleman says:

    Connectivity has now been restored. We apologise for any inconvenience caused.

  2. David Brewis says:

    Reason For Outage: PWAN Connectivity

    Start: 7-Aug-2019 10:58
    Finish: 7-Aug-2019 11:13

    Impact: PWAN connectivity experienced a total loss of service (Primary and Backup) due to network loop affecting connectivity. Non PWAN connectivity was not affected.

    Root cause: Standard changes being implemented for a customer PWAN created unexpected impact due to pre-existing, redundant configuration on edge devices, triggering spanning tree to block connectivity to prevent a network loop.

    Additional actions:
    1. Edge devices config audit to check for, and if found remove, any redundant configuration under a maintenance window.
    2. Monitoring system event creation review to improve speed of identification of spanning tree activity.

Incident: Equinix PoP

Posted: Tuesday, August 6th, 2019 at 16:40 by Jordan Coleman
We are currently experiencing a loss of connectivity to our equipment in our Equinix POP. Engineers are currently investigating. Further updates will be provided as they become available.
  1. Jordan Coleman says:

    All services are now restored and we apologise for any inconvenience caused.