Incident: CityFibre network services
Posted: Tuesday, February 16th, 2021 at 18:51 by Adam Heath
We are aware that a number of customers based in the Midlands and M4 corridor are currently experiencing an outage of their On-net service. Our teams are currently investigating and are looking to restore service as soon as possible. Apologies for the impact that this will cause.
Edit: This is now a widespread network issue and potentially causing customers a total loss of service, packet loss or latency.
We are still investigating this issue with our T3 agents within the NOC team and will provide further updates as soon as they are available.
Apologies for any inconvenience and the delay in any substantial update up until now. We endeavour to address this as soon as possible.
Please keep checking here for any updates or any cases you have already raised with us, linked to the parent incident.
Since 20:45PM, we are beginning to see a large number of services beginning to restore.
However, our engineers are currently still investigating the root cause of the outage, as there may still be some services at risk and offline until this has been completely resolved and concluded.
We will continue to provide updates as to when and available.
Following our previous response, our engineers are currently still investigating the root cause of the outage.
We have had confirmation that the majority of services have restored.
Updates will continue to be provided as to when and available.
We are aware of an issue in Peterborough and a large number of services down in the area, we are currently working to fix it and we have scheduled an engineer to go to site to assist us in our investigations.
Specific details regarding this visit will follow when available.
Our engineer is due to arrive on site a 2am to assist investigations and restore the outage to Peterborough services.
Following this visit, we should have a further update available by 3 am.
Updates will continue to be provided as to when and available.
Our engineer is currently on-site performing investigations to restoring service to the Peterborough connections.
Updates will follow after 3 am.
We have had the latest report from our engineer.
Following a re-boot of a core device on-site, services have begun to restore across the network for Peterborough connections.
We now advise to ensure that any services still down, are fully re-booted, and should the issue persist, please contact the service desk to advise the remaining connection issues.
A full report of the root cause is still currently ongoing and to be provided at a later time.
We are currently seeing a recurrence of the issues following the work earlier this morning. Senior network engineers are continuing to investigate the root cause and further updates will be provided as soon as possible
High level investigations into the issue are continuing to take place. At present, the problem will cause packet loss, latency or instances of total loss of service and is affecting multiple tunnels across the CityFibre network.
NOC and architecture & engineering teams are currently working with external vendor support to find the root cause and restore services. Further updates will be provided in the next hour.
Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Major Incident Leader:
Daniel Walker
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
Investigations into the issue are ongoing. Following internal technical calls, the resolver teams are currently focusing efforts on stabilising the impacted services. Once these have been stabilised, they can then focus on finding the root cause and a permanent fix. Another internal technical call is taking place at 12:30 and further updates will be provided following this at 13:15
Next Update
17/02/2021 13:15
Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
Work has been undertaken to return stability which involved shutting down one of our links. Following this, we have seen some services restored however, customers in the Peterborough area may still be impacted. We are currently waiting for engineer resources to attend Peterborough and carry out further troubleshooting.
Once Peterborough services have been restored we will then look to put a permanent fix in place ensuring any changes goes through the change management process due to potential impact to services. Further updates will be provided at 14:15
Next Update:
17/02/2021 14:15
Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
We are seeing the majority of services now restored apart from those directly connected through Peterborough. We are now awaiting an engineer to attend Peterborough sites with an ETA of 16:00, the technical teams will then work with the engineers to restore the services connected to Peterborough sites.
Next Update:
17/02/2021 17:15
Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
Following engineer attendance in Peterborough we can confirm all affected services are now restored. We will now close down the Major Incident and any remedial and root cause efforts will now be managed via the Change & Problem Management processes.
Please accept our apologies for the inconvenience caused by this issue.