Incident: Peterborough connections
Posted: Thursday, February 25th, 2021 at 14:13 by Adam Heath
We are aware that a number of customers are impacted by an incident in Peterborough. Our teams are investigating the cause and we will provide an update once further information is known.
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
Engineer resources have been arranged with an ETA of 18:00. We will provide a further update upon the engineers arrival.
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
Engineer resources have confirmed that not lights within the fibre is being received between PB1344 and PB1244. They have also proofed that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.