We are currently aware of an incident affecting Vodafone leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided soon.
Archive for the ‘Incidents’ Category
Incident: Vodafone leased linesPosted: Tuesday, April 26th, 2022 at 22:38 by Teodor Pletosu
Incident: Peterborough connectionsPosted: Thursday, February 25th, 2021 at 14:13 by Adam Heath
We are aware that a number of customers are impacted by an incident in Peterborough. Our teams are investigating the cause and we will provide an update once further information is known.
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
Engineer resources have been arranged with an ETA of 18:00. We will provide a further update upon the engineers arrival.
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
Engineer resources have confirmed that not lights within the fibre is being received between PB1344 and PB1244. They have also proofed that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.
Incident: CityFibre network servicesPosted: Tuesday, February 16th, 2021 at 18:51 by Adam Heath
We are aware that a number of customers based in the Midlands and M4 corridor are currently experiencing an outage of their On-net service. Our teams are currently investigating and are looking to restore service as soon as possible. Apologies for the impact that this will cause.
Edit: This is now a widespread network issue and potentially causing customers a total loss of service, packet loss or latency.
We are still investigating this issue with our T3 agents within the NOC team and will provide further updates as soon as they are available.
Apologies for any inconvenience and the delay in any substantial update up until now. We endeavour to address this as soon as possible.
Please keep checking here for any updates or any cases you have already raised with us, linked to the parent incident.
Incident: CityFibre leased linesPosted: Monday, February 15th, 2021 at 14:35 by david labouchardiere
We are currently investigating reports of connectivity issues affecting CityFibre leased lines in Scotland. Engineers are investigating and further updates will follow as and when they become available.
Our engineering team reported a large number of MPLS tunnels showing as down. These have been re-established and believe that services should begin to be restoring.
We are aware that some customers are still reporting that services have not been restored, as such we have further investigated the issue. We believe that we have identified the cause, which is related to MTU, and the teams are working to correct the issue now. Next update will be in approximately 1hr or sooner, when we believe the service will be restored.
Our teams have now restored service for all impacted customers. The incident was related to MTU configuration within the core network, which has now been corrected. We do not anticipate a repeat of the incident and will undertake a review of the root cause. Apoligies to all customers that were affected.
Power outage affecting London Interxion Suite (LON44)Posted: Monday, February 8th, 2021 at 13:43 by Adam Heath
We are currently experiencing a major outage. This is under investigation and we will have an update for you as soon as we know anything further. Next update will be within the hour
Initial reports are coming in that a power outage is causing this disruption. We appear to have lost a large amount of Edge devices. Engineers are on route to site.
A further update will be made available within the hour.
Engineers are now on site and investigations have begun.
Further updates within the hour
Engineers have reported power is now restored. We will now begin checking for service restoration.
Interxion Engineers are currently making their way around the suites in the datacentre, but as yet we have no ETA for when they will get to our equipment.
We are chasing every 15 minutes for an update and once we have more information we will be in touch. Next update due in 1 hour.
This matter has been escalated to Datacentre Management and continue to chase for further updates. However, we are beginning to see power restoration to our suite as well as some services being restored.
Further updates in 1 hour
PWAN services are now mostly restored. Please re-boot any services before contacting the Service Desk.
However, there are a few stragglers that are still showing as down and further work is still ongoing to restore services for all customers.
Further updates to follow.
Incident: Interxion.edgePosted: Tuesday, November 24th, 2020 at 13:09 by Adam Heath
We have just observed a temporary loss of service on our Interxion.edge device which has interrupted internet breakout for some PWAN customers.
Services are now restored and engineers are investigating the root cause. We apologise for any inconvenience this has caused.
We have received further reports of interruption to PWAN services which we are currently investigating. We will provide a further update when available.
Initial information received earlier this afternoon indicated an on-going issue with Interxion.edge, however, further investigations showed the issue to be specific to some PWAN DSL services. We have taken measures to mitigate the root cause of this interruption and we will continue to closely monitor.
Incident: CityFibre circuitsPosted: Wednesday, October 14th, 2020 at 13:09 by david labouchardiere
We are currently aware of an issue affecting CityFibre provided pseudowire circuits through Telehouse North. Engineers are investigating and further updates will be provided as soon as possible.
Incident: Leased lines in Sheffield, Peterborough and NottinghamPosted: Tuesday, September 15th, 2020 at 12:03 by david labouchardiere
We are aware of an issue affecting leased lines in and around Sheffied, Peterborough and Nottingham. Our networking team are currently investigating the issue and further updates will be provided as soon as possible.
Sheffield and Nottingham core routers are back online but we still have an issue in Peterborough. Engineers are being dispatched.
Engineers are expected on site at 1500
The Engineer is due to arrive on site imminently.
Engineer is now on site and investigations have begun
NOC have now gained access to the core router and have begun troubleshooting the issue.
Further updates to be published at 1730
Services have been restored. A reboot was required which has resulted in the data necessary to carry out further root cause analysis to be lost. A reboot in this instance was the last available option in order for us to restore service.
Due to this we will continue to monitor for the next 48hours.
Incident: BT EdinburghPosted: Wednesday, August 12th, 2020 at 08:57 by David Brewis
A major incident has been declared by BT for an exchange in Edinburgh impacting broadband and ethernet services.
Loss of service is a result of water ingress into the location. The emergency command centre is onsite with majority of water removed, with engineers ensuring safe to enter to continue investigations and restore services as soon as possible.
Further updates will be provided.
The supplier is currently investigating and confirms their hardware at the affected site is currently unreachable. We will provide further updates in the next hour.
We are observing Vodafone Off-net services now starting to restore.
Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We will be reaching out to our 3rd party supplier for RFO once investigation of the incident will be completed on their end.
We apologies for any inconvenience caused.