We are aware that a number of customers are impacted by an incident in Peterborough. Our teams are investigating the cause and we will provide an update once further information is known.
Archive for February, 2021
Emergency Work: Peterborough (PB1244)
Posted: Tuesday, February 23rd, 2021 at 16:28 by Jamie SmithTuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59
During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.
Incident: Peterborough
Posted: Tuesday, February 23rd, 2021 at 11:02 by david labouchardiereFollowing the incident yesterday we are still working on restoring service to the portion of customers who remain impacted in Peterborough. Engineers are continuing to troubleshoot and further updates will be provided in the next hour
Incident: Peterborough
Posted: Monday, February 22nd, 2021 at 16:21 by Jordan GardnerWe are currently aware of an issue affecting services within the Peterborough area. Engineers are already engaged and investigating. A further update will be provided when available.
Incident: CityFibre network services
Posted: Tuesday, February 16th, 2021 at 18:51 by Neil WatsonWe are aware that a number of customers based in the Midlands and M4 corridor are currently experiencing an outage of their On-net service. Our teams are currently investigating and are looking to restore service as soon as possible. Apologies for the impact that this will cause.
Edit: This is now a widespread network issue and potentially causing customers a total loss of service, packet loss or latency.
Incident: CityFibre leased lines
Posted: Monday, February 15th, 2021 at 14:35 by david labouchardiereWe are currently investigating reports of connectivity issues affecting CityFibre leased lines in Scotland. Engineers are investigating and further updates will follow as and when they become available.
Emergency Work: London Interxion Suite (LON44)?
Posted: Wednesday, February 10th, 2021 at 09:47 by Iain BeveridgeWednesday 10th February 2021, 22:00 – Thursday 11th February 2021 05:00
During the above window, we will be working to restore power stability to some customer firewalls within our interxion suite. An ATS that is supplying one of out racks is faulty, we will be moving devices and config one at a time to new infrastructure within another rack in the suite. There will be a small window of downtime as we move these devices one at a time in a staged and controlled manner. This is follows on from last nights work, this window should conclude the work in this rack.
Emergency Work: London Interxion Suite (LON44)
Posted: Tuesday, February 9th, 2021 at 19:45 by Iain BeveridgeTuesday 09th February 2021, 22:00 – Wednesday 10th February 2021 05:00
During the above window, we will be working to restore power stability to some PWAN infrastructure and customer firewalls within our interxion suite. An ATS that is supplying one of out racks is faulty, we will be moving devices and config one at a time to new infrastructure within another rack in the suite. There will be a small window of downtime as we move these devices one at a time in a staged and controlled manner.
Power outage affecting London Interxion Suite (LON44)
Posted: Monday, February 8th, 2021 at 13:43 by mpurcellWe are currently experiencing a major outage. This is under investigation and we will have an update for you as soon as we know anything further. Next update will be within the hour
BT Planned Work: Colindale TE
Posted: Tuesday, February 2nd, 2021 at 14:40 by Iain BeveridgeThursday 04th March 2021, 21:00 – Friday 05th March 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above site requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service it should still be considered at risk.
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
Engineer resources have been arranged with an ETA of 18:00. We will provide a further update upon the engineers arrival.
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
Engineer resources have confirmed that not lights within the fibre is being received between PB1344 and PB1244. They have also proofed that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.