Archive for the ‘Incidents’ Category

Incident: Vodafone Circuits

Posted: Sunday, January 6th, 2019 at 11:50 by Matthew Davies

Our monitoring has alerted our service teams to a number of Vodafone circuits that are currently out of service. Our initial diagnostics indicate that the circuits are related to a hub between Vodafone and us. Both we and the supplier are currently investigating to provide a solution as soon as possible. Further updates will be provided via this site.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. Matthew Davies says:

    The fault has been confirmed to a Vodafone hub. Engineers are being dispatched by the supplier to investigate the issue further. We’re expecting they will arrive on site soon, when further investigation will continue.

    A further update will be posted on this post as we receive further information.

  2. Matthew Davies says:

    Engineers are on site investigating the issue and work continues on site and fibre connections are being testing.

    We’re expecting a further update in the next hour and will update here with any further information in the meantime.

  3. Matthew Davies says:

    We believe this issue has now been resolved, engineers have confirmed they are seeing the hub back up and we can now see connections coming back online.

    Please reboot equipment on site before progressing any related issues further.

    Information regarding the fix will be posted here at a later date.

Incident: Virgin Media circuits

Posted: Tuesday, December 11th, 2018 at 13:15 by David Brewis

Our monitoring has alerted our service teams to a number of Virgin Media circuits that are currently out of service. Our initial diagnostics indicate that the circuits are related to a handover between Virgin Media and us, engineers are investigating and are engaged with the supplier. Further updates will be provided via this site.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. David Brewis says:

    We believe the incident to be resolved, during investigation engineers traced the issue to a faulty fibre patch.

    Engineers have replaced the link which has restored service and our engineers will continue to monitor.

    We apologise for the disruption caused by this incident.

VoIP Incident

Posted: Monday, December 10th, 2018 at 12:48 by David Brewis

We are currently aware of an issue affecting some users on our legacy VOIP platform. Engineers are currently investigating and a further update will be provided by 13:50.

  1. David Brewis says:

    We believe the incident to be resolved, during investigation engineers traced to a dns issue hosted on our legacy hosting platform. Our engineers restarted the service which has restored service and our engineers will continue to monitor.

    We apologise for the disruption caused by this incident.

Incident: Legacy VOIP platform

Posted: Friday, December 7th, 2018 at 15:24 by Matthew Davies

We are currently aware of an issue affecting some users on our legacy VOIP platform. Engineers are currently investigating and a further update will be provided by 4:30pm

  1. Matthew Davies says:

    Our engineers have now resolved the issue with our legacy VOIP platform and all services should now be restored. If issues persist please reboot any phones or servers before contacting technical support.

    More information on the cause of the issue will be forthcoming.

Incident: Sheffield Broadband

Posted: Wednesday, November 28th, 2018 at 11:17 by Adam Heath

We are aware of an issue currently affecting a number of broadband services in Sheffield.  We are investigating and will update when there is further information to provide.

  1. Adam Heath says:

    Service seems to be restored for a large number of the affected connections.

    We also are in correspondence with BT where we suspect the issue may have been, due to power on our core router and service to other types of connections such as leased lines, remaining throughout.

Incident: Vodafone circuits

Posted: Wednesday, November 21st, 2018 at 22:25 by Neil Watson

Our monitoring has alerted us to a number of Vodafone circuits that are currently out of service. Our initial diagnostics indicate that the circuits are related to a single handoff point between Vodafone and us. Our engineers are investigating and are engaged with Vodafone. Further updates will be provided via this site. We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. John Benge says:

    Vodafone have confirmed that an engineer is conducting tests on fibres to ascertain the nature of the outage. We will provide further updates as soon as possible.

  2. John Benge says:

    Vodafone have arranged multiple engineer resources to this issue who are working on the fault. We will provide further updates as soon as possible.

  3. David Brewis says:

    Vodafone engineers are located within the data centre and working to troubleshoot and restore service.

    We will provide the next update by 10:45am

  4. David Brewis says:

    Vodafone engineers have identified a fibre break resulting in the loss of service. Appropriate teams have already been engaged to locate the break and resolve service.

    We will provide the next update by 11:45am

  5. David Brewis says:

    Vodafone field engineers continue to locate the fibre break, spare fibres have been identified to restore a level of service whilst a permanent fix is identified.

    We will provide the next update by 12:45pm

  6. Scott Morgan says:

    We are currently seeing a significant number of circuits reconnecting. Further investigation is ongoing to ensure all remaining services are brought back online.

    Further updates will be provided by 14:15.

  7. Scott Morgan says:

    As part of ongoing investigation into this issue, Vodafone are sending out additional engineering resources to their data centre.

    Vodafone believe the break to be located 43 meters into the fibre handoff, which they are working on to resume full service.

    Further updates will be provided at 15:15.

  8. David Brewis says:

    Engineers at the data centre are continuing to work with Vodafone fibre teams and resolve the second identified fibre break.

    Further updates will be provided by 16:15

  9. David Brewis says:

    Vodafone fibre engineers continue to examine the length of the fibre for further issues impacting service. The Entanet service centre are speaking with Vodafone incident managers to understand anticipated clear time.

    Further updates will be provided by 17:30

  10. David Brewis says:

    Vodafone engineers remain on site tracing the underlying infrastructure so that a permanent fix can be put in place for all affected services. We continue to work with the carrier to expedite the resolution with regular communication.

    Currently the agreed next stage is for engineers to work to determining the exact location for the fibre fault.

    Further updates will be provided by 18:45

  11. David Brewis says:

    We believe service to be restored following the identification of a faulty fibre patch. An incident report will be made available within 5 working days.

    We apologise for any disruption caused, if however you continue to be without service, please contact the service desk at your earliest convenience.

  12. David Brewis says:

    An interim report can be found here, whilst we await a full RFO.

Incident: Telford Power Outage – services at risk

Posted: Wednesday, November 14th, 2018 at 03:08 by Neil Watson

There is currently a power outage that is affecting our Telford HQ and our datacentres. As a consequence, we are currently running on our back up generator power. All services that are delivered from this location must, therefore, be considered to be at risk. This a mains power failure that is being worked on by the supplier and they have estimated a return in service at approximately 03:30hrs.

  1. Neil Watson says:

    Power has now been restored and has been stable for a short while. We are no longer running on generator power, and therefore the site is considered to be fully operational again.

Incident: edinburgh.core

Posted: Wednesday, October 17th, 2018 at 17:01 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Edinburgh POP. Engineers are already investigating. Further updates will be provided as and when they become available

  1. Jamie Smith says:

    Connectivity has been restored. Initial investigation indicates a power disruption in one of our cabinets at the site. This has been raised to our supplier for further investigation.

    We apologise for the inconvenience caused by this interruption.

Incident: sheffield.core

Posted: Sunday, October 14th, 2018 at 11:53 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Sheffield POP. Engineers are already investigating. Further updates will be provided as and when they become available

  1. Jamie Smith says:

    Connectivity has been restored. Initial investigation indicates a power disruption in one of our cabinets at the site. This has been raised to our supplier for further investigation.

    We apologise for the inconvenience caused by this interruption.

Incident: Columbo DSL

Posted: Wednesday, September 26th, 2018 at 11:42 by Adam Heath

We are aware of an issue causing loss of service to DSL circuits which connect to our Columbo node.  Early investigations suggest this may be a BT issue, based off reports that other Service Providers may also be affected.  We’re continuing to investigate with the supplier and will provide further information when we have it.

  1. Adam Heath says:

    BT have confirmed they have an open incident and are in the process of completing diagnostics. They haven’t provided a root cause or estimated time of repair as of yet. We’ll continue to update upon receipt of further information.

  2. Adam Heath says:

    We can see that service was restored on the majority of circuits at approximately 1pm. We are still in correspondence with BT in order to obtain information on the cause, which we will then provide here.