Incident: CityFibre network services
Posted: March 23rd, 2021 at 14:25 by david labouchardiere
We are currently aware of an issue affecting CityFibre circuits.
Engineers are investigating and further updates will follow as soon as possible
Planned Work: man191
Posted: March 17th, 2021 at 16:09 by Alistair Patterson
Tuesday 23rd March 2021, 21:00 – Wednesday 24th March 2021, 02:00
During the above window, we will be completing some none service affecting maintenance following the investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.
Emergency Work: LINX LON1 & DE-CIX Peers
Posted: March 12th, 2021 at 12:25 by Jamie Smith
Friday 12th March 2021, 13:00 – 13:30
During the above window we will be carrying out preventative work on our LINX LON1 & DE-CIX peers. During this window peers will notice sessions going down and there may be a brief interruption to any traffic routing via one of these affected peers before routing via alternative LINX & LONAP peering locations.
We apologize for the short notice and any inconvenience this may cause.
-
Jamie Smith says:
This work is now starting
-
This change has now been completed.
Planned Work: bir16
Posted: March 11th, 2021 at 16:04 by Iain Beveridge
Thursday 11/03/2021 21:00:00 – Friday 12/03/2021 02:00:00
During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only, no services will be affected during the maintenance window.
Planned Work: Peterborough Te
Posted: March 10th, 2021 at 16:10 by Adam Heath
Thursday 25th March 2020 00:01 – Thursday 25th March 2020 03:00
After recently replacing the above router on Tuesday 23rd February 2021 we have identified a need to reload the new equipment as we have an issue with an onboard 10gb port. The reload was recommended after raising a TAC case with hardware vendor support.
The work will take place during the above window and will effect all users connected to PB1244, we do not anticipate downtime for the entire window. We apologise for the inconvenience these planned works will cause and we have exhausted all other possibilities before taking this customer impacting action.
-
Adam Wilson says:
Please be advised, this work has been aborted and will be rescheduled. Further updates will be provided in due course.
-
Jordan Gardner says:
This work has now been rescheduled to Friday 26th March 2021 00:01 – Friday 26th March 2021 03:00. Please see new post https://noc.enta.net/2021/03/emergency-work-peterborough-te/
Planned Work: man191
Posted: March 9th, 2021 at 15:18 by Iain Beveridge Tuesday 9th March 2021, 21:00 – Wednesday 10th March 2021, 02:00
During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.
-
Aiden Spencer says:
All work was successful
Planned Work: man191
Posted: March 8th, 2021 at 16:49 by Iain Beveridge Monday 8th March 2021, 21:00 – Tuesday 9th March 2021, 02:00
During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.
-
The above work was cancelled and rescheduled for this evening, no work took place during the above window
Incident: Peterborough connections
Posted: February 25th, 2021 at 14:13 by Adam Heath We are aware that a number of customers are impacted by an incident in Peterborough. Our teams are investigating the cause and we will provide an update once further information is known.
-
Neil Watson says:
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
-
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
-
Emily Pieniak says:
Engineer resources have been arranged with an ETA of 18:00. We will provide a further update upon the engineers arrival.
-
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
-
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
-
Engineer resources have confirmed that not lights within the fibre is being received between PB1344 and PB1244. They have also proofed that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
-
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
-
Lucy Jordan says:
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
-
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.
Emergency Work: Peterborough (PB1244)
Posted: February 23rd, 2021 at 16:28 by Jamie Smith Tuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59
During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.
-
Robin Posadas says:
Work commencing.
-
Work completed
Incident: Peterborough
Posted: February 23rd, 2021 at 11:02 by david labouchardiere Following the incident yesterday we are still working on restoring service to the portion of customers who remain impacted in Peterborough. Engineers are continuing to troubleshoot and further updates will be provided in the next hour
-
david labouchardiere says:
We are still investigating this issue with our senior agents within the NOC team and will provide further updates in the next hour.
-
Based on NOC troubleshooting we have engaged engineer resources to attend the affected site and carry out further testing. We will provide a further update once we have an ETA for the engineer
-
Engineer resources have been arranged with an ETA of 16:00. We will provide a further update upon the engineers arrival.
-
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
-
Engineer resource is on-site investigating the physical link between PB1344 and PB1244. We will provide further updates in the next hour
-
The link between PB1344 and PB1244 remains down where we are still seeing 1 way MAC betwenPB1344 and PB1244. The SFP and fibre on PB1344 and PB1244 link has been re-seated and cleaned.
We will continue to provide updates in the next hour.
-
A replacement ASR is on-site which at present is being installed where we are awaiting further feedback from the engineers on-site.
We will continue to provide updates in the next hour.
-
SPF swap has taken place at PB1244 where engineers are continuing to work at PB1344 and PB1244.
We will continue to provide updates in the next hour.
-
The link between PB1344 and PB1244 remains down. Engineers have attempted multiple optics on both PB1344 and PB1244 where the port still would not come up.
Next action is for the fault to be logged with Nokia TAC for PB1344.
We will continue to provide updates within the next 2 hours.
-
Our vendors are dispatching an engineer to replace a suspected faulty card in our PB1344-rtr-1 node.
ETA is 02:30. A further update will be provided upon the engineer’s arrival. -
Engineer is on site and about to commence work. Update will follow upon completion of the task.
-
The card has been replaced, and we have collected further diagnostic information. We will confirm operational status once this has been reviewed by our vendor.
-
We believe this issue to now be resolved, and request that any affected users retest their service.
Please accept our apologies for any inconvenice that may have been caused.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net Ethernet Services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net Ethernet customers
Description: A portion of on-net Ethernet customers are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre and Architecture and Engineering
Update 1 –: Our Network Operations Centre are currently investigating a network incident affecting on-net Ethernet customers. Initial indications are that this is not related to a specific geographical location. Teams continue to investigate and further updates will follow at 15:25
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net and off-net Ethernet Services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net and off-net Ethernet customers
Description: A portion of on-net Ethernet customers are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre, Architecture and Engineering
Update 2 –: The incident is related to an issue on a link between Leeds and Sheffield. Third party providers have already been engaged and are investigating. In the meantime, to mitigate the impact we have re-routed traffic over an alternative link. We are now in the process of assessing the impact of the re-routing. A further update will follow at 16:00
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net and off-net Ethernet Services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net and off-net Ethernet customers
Description: A portion of on-net and off-net Ethernet customers are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre, Architecture and Engineering
Update 3 –: After the initial re-routing we are still seeing a portion of customers impacted by the major incident. Further work is being carried out in order to optimise network traffic and alleviate the issues being experienced. We also continue to press our third party supplier for updates on the failure on the link between Leeds and Sheffield. Further update will follow at 17:00
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: Multiple
Customer Impact: Total loss of services for on-net Ethernet, Off-net Ethernet and FTTC services
Description: A portion of on-net and off-net Ethernet customers, as well as FTTH customers, are experiencing a total loss of service.
Team(s) Investigating: Network Operations Centre, Architecture and Engineering
Update 4 –: After further optimisation of the network we are seeing restoration to the affected customers.
As there is still an open incident on the Leeds to Sheffield link, services should be considered at risk until the link has been brought back into service. We will continue to update this incident until that risk has been fully mitigated.
If any customers are still experiencing an issue we ask them to power-cycle their equipment before contacting the Technical Service Centre.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Ryan Berrisford
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Update 5 –: Although there is no longer any customer impact, the incident remains open whilst we mitigate the risk caused by the Sheffield to Leeds link failure. Engineer resource has been mobilised to site with an expected time of arrival of 19:50
A further update will be posted by 20:20 or as soon as further information becomes available.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Adam Heath
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Update 6 –: Our TSC have checked in with our engineer resource who advise that they have just arrived on site. We will update upon receipt of further information or at 21:30 regardless of this.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Adam Heath
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Update 7 –: Our engineering resource has located the damaged fibre and are currently performing an assessment with regards to the extent. Again, no services are currently impacted but are at risk. We will update again at 23:00 or upon receipt of further information, whichever comes first.
Major Incident Raised: 23/03/2021 14:25
Major Incident Leader: Adam Heath
Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services
Service(s) Impacted: No services are currently impacted
Customer Impact: There is currently no impact to customers
Description: Services are at risk until the failed link has been restored
Team(s) Investigating: Technical Service Centre
Resolved –: 22:38 We have had confirmation that this issue is now resolved and that services are no longer at risk. Following assessments by our engineering resource at multiple locations, the damage previously mentioned was unrelated and not the cause of the issue on the remaining impacted service. Instead, a fibre microbend was identified and corrected, with testing demonstrating expected results.
An internal post incident review will now follow.