Archive for the ‘Incidents’ Category

Incident: Nottingham core

Posted: Monday, October 16th, 2017 at 02:13 by David Derrick

Our Nottingham core router has suffered a power outage and one of its blades has failed. Engineers are preparing to head to site with replacement equipment.

  1. David Derrick says:

    The failed blade has been replaced and service has been restored.

Incident: Columbo

Posted: Sunday, October 15th, 2017 at 19:25 by David Derrick

Our core router in Columbo has unexpectedly rebooted. All services appear to have recovered, we will continue to monitor the situation.

  1. Adam Heath says:

    Our investigations confirm this incident as power related.

    We are working on root cause with our colocation provider, which is responsible for delivering our power, UPS, power monitoring and alerting at these locations.

    This remains at our highest priority and we are monitoring the situation closely.

Incident: Vodafone Circuits

Posted: Friday, October 6th, 2017 at 19:34 by Mark Yardley

We are aware of a loss of connectivity for a number of Vodafone Ethernet circuits. Initial diagnostics show the affected circuits are delivered on the same NNI that has recently suffered disruption of service. We are liaising with our suppliers to get the issue investigated further and resolved as quickly as possible. Further updates will be provided as they become available.

  1. David Brewis says:

    Following investigations, Vodafone are arranging to dispatch engineers tonight.

    Further updates to follow once engineers have attended site.

  2. Mark Yardley says:

    Vodafone engineers have attended their site to replace suspect faulty equipment. We currently see the affected services have been restored but should still be considered at risk until our suppliers have advised fault is cleared.

  3. Mark Yardley says:

    These services have again lost connectivity and Vodafone are investigating further. Updates to be provided once available.

  4. David Brewis says:

    Vodafone have escalated the incident and are dispatching engineers to restore service. Further updates to follow as made available.

  5. David Brewis says:

    Connections are restoring to service following hardware replacement, until fault closure consider service at risk.

  6. David Brewis says:

    After 48 hours of monitoring, we have received correspondence from the supplier to show the fault is now closed.

  7. David Brewis says:

    An incident report for this issue is now available here

Incident: Cardiff

Posted: Sunday, October 1st, 2017 at 10:25 by Jake Turner

Shortly after 10:00 we experienced an unexpected reload of the cardiff.core router which on first examination appears to be the result of a temporary loss of power. Directly connected customers would have experienced a brief drop in connectivity. Traffic passing through this router will have diverted across alternative paths. Connectivity to most services has been restored and engineers are investigating.

Incident: Vodafone Circuits

Posted: Wednesday, September 20th, 2017 at 00:02 by Richard Partridge

We are currently aware of an issue causing a loss of service for a small number of Vodafone based Ethernet circuits. Engineers are investigating and the supplier has also been engaged. Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Connectivity is currently restored. Engagement with Vodafone has yet to provide an explanation for this loss of service, however we are continuing to pursue an answer.

    For the time being Vodafone circuits terminating in redbus-sov.core should be considered at risk.

Peterborough Latency Issues

Posted: Tuesday, September 5th, 2017 at 14:12 by Adam Heath

We are currently aware of an issue affecting some broadband circuits which connect through our Peterborough node. Symptoms appear to be high latency and slow performance. We are investigating internally and have also raised this with BT for review.  We will provide further information when it becomes available.

  1. Adam Heath says:

    Service has been restored and BT have confirmed that this was linked to a wider issue they had on one of their core routers.

Incident: Preston

Posted: Thursday, August 24th, 2017 at 13:54 by Richard Partridge

We have just experienced a brief loss of power to our equipment located in our Preston PoP. Power is currently restored however engineers are investigating with our colocation provider.

Further updates will be provided when available.

  1. Richard Partridge says:

    Our investigation has determined that the problem related to an unexpected reload of the Preston core router and we are continuing to investigate. The core router has shown no signs of any further problems.

  2. David Derrick says:

    The Preston router has again reported a reboot caused by power failure. Services terminating at that POP will have experienced disruption; other services will have had their traffic rerouted automatically.

    Investigations into the power outage on 24th are still ongoing, this incident will be added to the case as a matter of urgency.

  3. Richard Partridge says:

    The core router has shown no further signs of problems. However, we already had plans to replace this core router with a higher specification and capacity device. In light of these recent problems, we’ve accelerated these plans and will be replacing the core as outlined in this emergency maintenance notification:

VoIP Disruption

Posted: Wednesday, August 23rd, 2017 at 10:14 by Adam Heath

We are aware of an issue currently affecting some VoIP services.  We are investigating the issue and will provide further information once it is available.

  1. Adam Heath says:

    Our investigations have led to an issue being identified with a specific upstream carrier. Our engineers are in correspondence with the relevant carrier for more information, which we will make available once obtained.

  2. Adam Heath says:

    Further information attained suggests the root cause may be a power issue with an upstream carrier. It has been advised that there are engineers working to restore service.

  3. Adam Heath says:

    The affected upstream carrier has confirmed that a small number of services have been restored, but there is still a significant amount of non-functional incoming services. Work is continuing towards restoration.

  4. Adam Heath says:

    We have started to see inbound services recover, following a way to reroute numbers in bulk being identified further upstream from ourselves. Internal testing here against numbers previously reported to us, also confirms this to be the case.

  5. Adam Heath says:

    An incident report for this issue is now available here.

Incident: Interxion

Posted: Friday, July 28th, 2017 at 09:42 by Richard Partridge

We are currently experiencing a loss of connectivity to one of our switches in Interxion. Engineers are already investigating and local engineering resources are on the way to site. Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Connectivity has been restored, however we are still investigating the underlying cause.

Incident: VoIP Disruption

Posted: Thursday, July 13th, 2017 at 10:46 by Richard Partridge

Engineers are currently investigating an incident that is causing a loss of service on our VoIP platform. Further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    Service has now been restored. We apologise for any inconvenience.