Archive for the ‘Incidents’ Category

Incident: Mirada Portal

Posted: Thursday, November 2nd, 2017 at 10:30 by Richard Partridge

Engineers are investigating an issue which is currently rendering the Mirada portal unavailable. Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Service has now been restored to the Mirada portal. Our apologies for any inconvenience.

Incident: Nottingham Core

Posted: Thursday, November 2nd, 2017 at 09:28 by Richard Partridge

We experienced a brief loss of service to our Nottingham Core router between 03:50 and 04:00 this morning. Directly connected services and broadband customers served by the Nottingham node will have experienced a loss of service whilst the core powered back up. Initial investigations indicate that this was a loss of power. This has been raised to our colocation provider for further investigation.

  1. Mark Yardley says:

    We have experienced another loss of service at this location. Engineers are investigating and updates will be provided when available.

  2. Mark Yardley says:

    This loss of service appears to have been due to another power disruption. Service has been restored and we are dispatching an engineer to site to investigate further as well as raising this again with our colocation supplier.

  3. Steve Lalonde says:

    We have just had an additional power issue in Nottingham, A PSU failed while our engineer was onsite mitigating a known power feed issue. The failed PSU has now been replaced and the power feed issue is resolved.

    Service is now restored.

Incident: Newcastle WBC

Posted: Sunday, October 29th, 2017 at 08:17 by Jake Turner

Engineers are currently investigating an issue causing a loss of service for DSL connections served by the Newcastle WBC node.

Further updates will be provided as and when they become available.

  1. Jake Turner says:

    Investigations concluded that there were no issues on the Entanet network and so this incident was raised to BT Wholesale. BT have since advised that there was an incident causing loss of resilience and whilst this shouldn’t have been service affecting, they did see a blip this morning. The majority of sessions reconnected just before 09:00.

Incident: Glasgow POP

Posted: Wednesday, October 25th, 2017 at 13:37 by Gurinder Bansal

We are currently experiencing a brief loss of communication to one our edge devices in Glasgow POP. This has resulted in a brief loss of service for directly terminated leased lines. Initial indications suggest a loss of power. Further updates will be provided shortly.

  1. Richard Partridge says:

    This has now been raised with our colocation provider for further investigation. Updates will be provided as and when they become available.

Incident: Glasgow POP

Posted: Tuesday, October 24th, 2017 at 10:50 by Richard Partridge

We are currently experiencing a loss of connectivity to our Glasgow POP. Initial indications suggest a loss of power. An engineer already in the area has been diverted to site. Further updates will be provided shortly.

  1. Richard Partridge says:

    Our on-site engineer has determined that our 10GB blade has failed as a result of a power outage to our space in the POP. As we already had designs to replace this router with a newer, higher specification device, we are accelerating our plans and will bring this device live as soon as possible. Further updates will be provided as and when they become available.

  2. Richard Partridge says:

    Broadband connectivity has now been restored. Customers may need to reboot their equipment in order to regain connectivity. We are continuing to work through the remainder of services. We will update again in due course.

  3. Mark Yardley says:

    Remaining services have now been restored. Customers are advised to reboot their equipment if they are still experiencing issues prior to contacting us via normal channels.

Incident: Wolverhampton TE Edge Device

Posted: Monday, October 16th, 2017 at 14:00 by Richard Partridge

We have just experience a brief loss of communication to one our edge devices in Wolverhampton TE. This resulted in a brief loss of service for directly terminated leased lines and broadband users. Service appears to have been restored however engineers are currently investigating.

  1. Adam Heath says:

    Our investigations confirm this incident as power related.

    We are working on root cause with our colocation provider, which is responsible for delivering our power, UPS, power monitoring and alerting at these locations.

    This remains at our highest priority and we are monitoring the situation closely.

Incident: Nottingham core

Posted: Monday, October 16th, 2017 at 02:13 by David Derrick

Our Nottingham core router has suffered a power outage and one of its blades has failed. Engineers are preparing to head to site with replacement equipment.

  1. David Derrick says:

    The failed blade has been replaced and service has been restored.

Incident: Columbo

Posted: Sunday, October 15th, 2017 at 19:25 by David Derrick

Our core router in Columbo has unexpectedly rebooted. All services appear to have recovered, we will continue to monitor the situation.

  1. Adam Heath says:

    Our investigations confirm this incident as power related.

    We are working on root cause with our colocation provider, which is responsible for delivering our power, UPS, power monitoring and alerting at these locations.

    This remains at our highest priority and we are monitoring the situation closely.

Incident: Vodafone Circuits

Posted: Friday, October 6th, 2017 at 19:34 by Mark Yardley

We are aware of a loss of connectivity for a number of Vodafone Ethernet circuits. Initial diagnostics show the affected circuits are delivered on the same NNI that has recently suffered disruption of service. We are liaising with our suppliers to get the issue investigated further and resolved as quickly as possible. Further updates will be provided as they become available.

  1. David Brewis says:

    Following investigations, Vodafone are arranging to dispatch engineers tonight.

    Further updates to follow once engineers have attended site.

  2. Mark Yardley says:

    Vodafone engineers have attended their site to replace suspect faulty equipment. We currently see the affected services have been restored but should still be considered at risk until our suppliers have advised fault is cleared.

  3. Mark Yardley says:

    These services have again lost connectivity and Vodafone are investigating further. Updates to be provided once available.

  4. David Brewis says:

    Vodafone have escalated the incident and are dispatching engineers to restore service. Further updates to follow as made available.

  5. David Brewis says:

    Connections are restoring to service following hardware replacement, until fault closure consider service at risk.

  6. David Brewis says:

    After 48 hours of monitoring, we have received correspondence from the supplier to show the fault is now closed.

  7. David Brewis says:

    An incident report for this issue is now available here

Incident: Cardiff

Posted: Sunday, October 1st, 2017 at 10:25 by Jake Turner

Shortly after 10:00 we experienced an unexpected reload of the cardiff.core router which on first examination appears to be the result of a temporary loss of power. Directly connected customers would have experienced a brief drop in connectivity. Traffic passing through this router will have diverted across alternative paths. Connectivity to most services has been restored and engineers are investigating.