Archive for the ‘Incidents’ Category

Incident: Vodafone Circuits

Posted: Wednesday, September 20th, 2017 at 00:02 by Richard Partridge

We are currently aware of an issue causing a loss of service for a small number of Vodafone based Ethernet circuits. Engineers are investigating and the supplier has also been engaged. Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Connectivity is currently restored. Engagement with Vodafone has yet to provide an explanation for this loss of service, however we are continuing to pursue an answer.

    For the time being Vodafone circuits terminating in redbus-sov.core should be considered at risk.

Peterborough Latency Issues

Posted: Tuesday, September 5th, 2017 at 14:12 by Adam Heath

We are currently aware of an issue affecting some broadband circuits which connect through our Peterborough node. Symptoms appear to be high latency and slow performance. We are investigating internally and have also raised this with BT for review.  We will provide further information when it becomes available.

  1. Adam Heath says:

    Service has been restored and BT have confirmed that this was linked to a wider issue they had on one of their core routers.

Incident: Preston

Posted: Thursday, August 24th, 2017 at 13:54 by Richard Partridge

We have just experienced a brief loss of power to our equipment located in our Preston PoP. Power is currently restored however engineers are investigating with our colocation provider.

Further updates will be provided when available.

  1. Richard Partridge says:

    Our investigation has determined that the problem related to an unexpected reload of the Preston core router and we are continuing to investigate. The core router has shown no signs of any further problems.

  2. David Derrick says:

    The Preston router has again reported a reboot caused by power failure. Services terminating at that POP will have experienced disruption; other services will have had their traffic rerouted automatically.

    Investigations into the power outage on 24th are still ongoing, this incident will be added to the case as a matter of urgency.

  3. Richard Partridge says:

    The core router has shown no further signs of problems. However, we already had plans to replace this core router with a higher specification and capacity device. In light of these recent problems, we’ve accelerated these plans and will be replacing the core as outlined in this emergency maintenance notification:

    http://noc.enta.net/2017/08/emergency-maintenance-preston/

VoIP Disruption

Posted: Wednesday, August 23rd, 2017 at 10:14 by Adam Heath

We are aware of an issue currently affecting some VoIP services.  We are investigating the issue and will provide further information once it is available.

  1. Adam Heath says:

    Our investigations have led to an issue being identified with a specific upstream carrier. Our engineers are in correspondence with the relevant carrier for more information, which we will make available once obtained.

  2. Adam Heath says:

    Further information attained suggests the root cause may be a power issue with an upstream carrier. It has been advised that there are engineers working to restore service.

  3. Adam Heath says:

    The affected upstream carrier has confirmed that a small number of services have been restored, but there is still a significant amount of non-functional incoming services. Work is continuing towards restoration.

  4. Adam Heath says:

    We have started to see inbound services recover, following a way to reroute numbers in bulk being identified further upstream from ourselves. Internal testing here against numbers previously reported to us, also confirms this to be the case.

  5. Adam Heath says:

    An incident report for this issue is now available here.

Incident: Interxion

Posted: Friday, July 28th, 2017 at 09:42 by Richard Partridge

We are currently experiencing a loss of connectivity to one of our switches in Interxion. Engineers are already investigating and local engineering resources are on the way to site. Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Connectivity has been restored, however we are still investigating the underlying cause.

Incident: VoIP Disruption

Posted: Thursday, July 13th, 2017 at 10:46 by Richard Partridge

Engineers are currently investigating an incident that is causing a loss of service on our VoIP platform. Further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    Service has now been restored. We apologise for any inconvenience.

BT circuit issue

Posted: Thursday, July 6th, 2017 at 10:56 by Adam Heath

We are aware of an issue affecting a number of BT Ethernet circuits which connect to our Columbo core.   Our initial diagnostics indicate the issue to be on BT’s hub and we have this raised with them already for investigation.

We will provide further information once it becomes available.

  1. Adam Heath says:

    We are continuing to correspond and troubleshoot with the suppliers. As a precaution, we have also tasked out an engineer to double check our equipment at the site.

    Further updates will be available shortly.

  2. Adam Heath says:

    We have restored service to affected circuits following an engineer replacing a faulty optic at site. We’re continuing to monitor from our side.

Incident: Viper Hosting Platform

Posted: Monday, June 26th, 2017 at 11:53 by Adam Heath

We are currently investigating an issue affecting some of our Viper hosting platform.

Engineers are currently investigating and further updates will be provided when available.

  1. Adam Heath says:

    Our NOC engineers have started to restore service to the Viper platform which requires a reboot of each server. Current focus is on returning the functionality for users, before reviewing to confirm the root cause.

    Further information will be available on this shortly.

  2. Adam Heath says:

    Our NOC engineers have confirmed that almost all users should now have functionality restored, with the final few affected servers still in the process of being rebooted and brought back online. This is expected to be complete within the next hour.

    Once complete, we’ll be ascertaining the cause and providing the information via this site.

Incident: Bristol.core reload

Posted: Wednesday, June 21st, 2017 at 23:02 by Neil Watson

We are aware that our Bristol.core router has unexpectedly reloaded. Customers routed via this node will have temporarily lost service while the reload completed. Currently, we do not believe that the reload is related to the ongoing planned work. We will investigate the cause and will report as appropriate. Apologies for the inconvenience caused.

Incident: Faraday Power

Posted: Sunday, May 7th, 2017 at 10:54 by Richard Partridge

We are currently experiencing intermittent power loss to some of our equipment located with the Faraday POP. Engineers are already investigating and the issue has been raised to the colocation provider as well.

Further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    Our supplier has advised that the intermittent power failure affecting our equipment should now be resolved. As a precaution, we are considering the site at-risk whilst we monitor the situation for any further disruptions.