Incident: CityFibre network services
Posted: Tuesday, March 23rd, 2021 at 14:25 by david labouchardiere
We are currently aware of an issue affecting CityFibre circuits.
Engineers are investigating and further updates will follow as soon as possible
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net Ethernet Services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net Ethernet customers
Description: A portion of on-net Ethernet customers are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre and Architecture and Engineering
Update 1 –: Our Network Operations Centre are currently investigating a network incident affecting on-net Ethernet customers. Initial indications are that this is not related to a specific geographical location. Teams continue to investigate and further updates will follow at 15:25
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net and off-net Ethernet Services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net and off-net Ethernet customers
Description: A portion of on-net Ethernet customers are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre, Architecture and Engineering
Update 2 –: The incident is related to an issue on a link between Leeds and Sheffield. Third party providers have already been engaged and are investigating. In the meantime, to mitigate the impact we have re-routed traffic over an alternative link. We are now in the process of assessing the impact of the re-routing. A further update will follow at 16:00
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net and off-net Ethernet Services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net and off-net Ethernet customers
Description: A portion of on-net and off-net Ethernet customers are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre, Architecture and Engineering
Update 3 –: After the initial re-routing we are still seeing a portion of customers impacted by the major incident. Further work is being carried out in order to optimise network traffic and alleviate the issues being experienced. We also continue to press our third party supplier for updates on the failure on the link between Leeds and Sheffield. Further update will follow at 17:00
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net Ethernet, Off-net Ethernet and FTTC services
Description: A portion of on-net and off-net Ethernet customers, as well as FTTH customers, are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre, Architecture and Engineering
Update 4 –: After further optimisation of the network we are seeing restoration to the affected customers.
As there is still an open incident on the Leeds to Sheffield link, services should be considered at risk until the link has been brought back into service. We will continue to update this incident until that risk has been fully mitigated.
If any customers are still experiencing an issue we ask them to power-cycle their equipment before contacting the Technical Service Centre.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Update 5 –: Although there is no longer any customer impact, the incident remains open whilst we mitigate the risk caused by the Sheffield to Leeds link failure. Engineer resource has been mobilised to site with an expected time of arrival of 19:50
A further update will be posted by 20:20 or as soon as further information becomes available.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Adam Heath
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Update 6 –: Our TSC have checked in with our engineer resource who advise that they have just arrived on site. We will update upon receipt of further information or at 21:30 regardless of this.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Adam Heath
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Update 7 –: Our engineering resource has located the damaged fibre and are currently performing an assessment with regards to the extent. Again, no services are currently impacted but are at risk. We will update again at 23:00 or upon receipt of further information, whichever comes first.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Adam Heath
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Resolved –: 22:38 We have had confirmation that this issue is now resolved and that services are no longer at risk. Following assessments by our engineering resource at multiple locations, the damage previously mentioned was unrelated and not the cause of the issue on the remaining impacted service. Instead, a fibre microbend was identified and corrected, with testing demonstrating expected results.
An internal post incident review will now follow.