We are currently aware of an incident affecting Vodafone leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided soon.
Archive for the ‘Incidents’ Category
Incident: Peterborough connections
Posted: Thursday, February 25th, 2021 at 14:13 by Adam Heath
We are aware that a number of customers are impacted by an incident in Peterborough. Our teams are investigating the cause and we will provide an update once further information is known.
-
Neil Watson says:
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
-
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
-
Emily Pieniak says:
Engineer resources have been arranged with an ETA of 18:00. We will provide a further update upon the engineers arrival.
-
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
-
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
-
Engineer resources have confirmed that not lights within the fibre is being received between PB1344 and PB1244. They have also proofed that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
-
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
-
Lucy Jordan says:
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
-
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.
Incident: CityFibre network services
Posted: Tuesday, February 16th, 2021 at 18:51 by Adam Heath
We are aware that a number of customers based in the Midlands and M4 corridor are currently experiencing an outage of their On-net service. Our teams are currently investigating and are looking to restore service as soon as possible. Apologies for the impact that this will cause.
Edit: This is now a widespread network issue and potentially causing customers a total loss of service, packet loss or latency.
-
We are still investigating this issue with our T3 agents within the NOC team and will provide further updates as soon as they are available.
Apologies for any inconvenience and the delay in any substantial update up until now. We endeavour to address this as soon as possible.
Please keep checking here for any updates or any cases you have already raised with us, linked to the parent incident.
-
jbiddulph says:
Since 20:45PM, we are beginning to see a large number of services beginning to restore.
However, our engineers are currently still investigating the root cause of the outage, as there may still be some services at risk and offline until this has been completely resolved and concluded.
We will continue to provide updates as to when and available.
-
Following our previous response, our engineers are currently still investigating the root cause of the outage.
We have had confirmation that the majority of services have restored.
Updates will continue to be provided as to when and available.
-
We are aware of an issue in Peterborough and a large number of services down in the area, we are currently working to fix it and we have scheduled an engineer to go to site to assist us in our investigations.
Specific details regarding this visit will follow when available.
-
Our engineer is due to arrive on site a 2am to assist investigations and restore the outage to Peterborough services.
Following this visit, we should have a further update available by 3 am.
Updates will continue to be provided as to when and available.
-
Our engineer is currently on-site performing investigations to restoring service to the Peterborough connections.
Updates will follow after 3 am.
-
We have had the latest report from our engineer.
Following a re-boot of a core device on-site, services have begun to restore across the network for Peterborough connections.
We now advise to ensure that any services still down, are fully re-booted, and should the issue persist, please contact the service desk to advise the remaining connection issues.
A full report of the root cause is still currently ongoing and to be provided at a later time.
-
david labouchardiere says:
We are currently seeing a recurrence of the issues following the work earlier this morning. Senior network engineers are continuing to investigate the root cause and further updates will be provided as soon as possible
-
High level investigations into the issue are continuing to take place. At present, the problem will cause packet loss, latency or instances of total loss of service and is affecting multiple tunnels across the CityFibre network.
-
NOC and architecture & engineering teams are currently working with external vendor support to find the root cause and restore services. Further updates will be provided in the next hour.
-
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Major Incident Leader:
Daniel WalkerService (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
Investigations into the issue are ongoing. Following internal technical calls, the resolver teams are currently focusing efforts on stabilising the impacted services. Once these have been stabilised, they can then focus on finding the root cause and a permanent fix. Another internal technical call is taking place at 12:30 and further updates will be provided following this at 13:15Next Update
17/02/2021 13:15
-
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Service (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
Work has been undertaken to return stability which involved shutting down one of our links. Following this, we have seen some services restored however, customers in the Peterborough area may still be impacted. We are currently waiting for engineer resources to attend Peterborough and carry out further troubleshooting.Once Peterborough services have been restored we will then look to put a permanent fix in place ensuring any changes goes through the change management process due to potential impact to services. Further updates will be provided at 14:15
Next Update:
17/02/2021 14:15 -
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Service (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
We are seeing the majority of services now restored apart from those directly connected through Peterborough. We are now awaiting an engineer to attend Peterborough sites with an ETA of 16:00, the technical teams will then work with the engineers to restore the services connected to Peterborough sites.Next Update:
17/02/2021 17:15 -
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Service (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
Following engineer attendance in Peterborough we can confirm all affected services are now restored. We will now close down the Major Incident and any remedial and root cause efforts will now be managed via the Change & Problem Management processes.Please accept our apologies for the inconvenience caused by this issue.
Incident: CityFibre leased lines
Posted: Monday, February 15th, 2021 at 14:35 by david labouchardiere
We are currently investigating reports of connectivity issues affecting CityFibre leased lines in Scotland. Engineers are investigating and further updates will follow as and when they become available.
-
mpurcell says:
Our engineering team reported a large number of MPLS tunnels showing as down. These have been re-established and believe that services should begin to be restoring.
-
We are aware that some customers are still reporting that services have not been restored, as such we have further investigated the issue. We believe that we have identified the cause, which is related to MTU, and the teams are working to correct the issue now. Next update will be in approximately 1hr or sooner, when we believe the service will be restored.
-
Our teams have now restored service for all impacted customers. The incident was related to MTU configuration within the core network, which has now been corrected. We do not anticipate a repeat of the incident and will undertake a review of the root cause. Apoligies to all customers that were affected.
Power outage affecting London Interxion Suite (LON44)
Posted: Monday, February 8th, 2021 at 13:43 by Adam Heath
We are currently experiencing a major outage. This is under investigation and we will have an update for you as soon as we know anything further. Next update will be within the hour
-
Initial reports are coming in that a power outage is causing this disruption. We appear to have lost a large amount of Edge devices. Engineers are on route to site.
A further update will be made available within the hour.
-
Engineers are now on site and investigations have begun.
Further updates within the hour
-
Engineers have reported power is now restored. We will now begin checking for service restoration.
-
Interxion Engineers are currently making their way around the suites in the datacentre, but as yet we have no ETA for when they will get to our equipment.
We are chasing every 15 minutes for an update and once we have more information we will be in touch. Next update due in 1 hour.
-
This matter has been escalated to Datacentre Management and continue to chase for further updates. However, we are beginning to see power restoration to our suite as well as some services being restored.
Further updates in 1 hour
-
Jonathan Clarke says:
Engineers remain at the Datacentre working to restore all power.
We will have a further update in 1 hour.
-
We are continually seeing more services restore, however, currently our engineers are still on-site working on restoring all services to the currently downed devices.
A further update will be provided in 1 hour.
-
PWAN services are now mostly restored. Please re-boot any services before contacting the Service Desk.
However, there are a few stragglers that are still showing as down and further work is still ongoing to restore services for all customers.
Further updates to follow.
Incident: Interxion.edge
Posted: Tuesday, November 24th, 2020 at 13:09 by Adam Heath
We have just observed a temporary loss of service on our Interxion.edge device which has interrupted internet breakout for some PWAN customers.
Services are now restored and engineers are investigating the root cause. We apologise for any inconvenience this has caused.
-
Jordan Gardner says:
We have received further reports of interruption to PWAN services which we are currently investigating. We will provide a further update when available.
-
Initial information received earlier this afternoon indicated an on-going issue with Interxion.edge, however, further investigations showed the issue to be specific to some PWAN DSL services. We have taken measures to mitigate the root cause of this interruption and we will continue to closely monitor.
Incident: CityFibre circuits
Posted: Wednesday, October 14th, 2020 at 13:09 by david labouchardiere
We are currently aware of an issue affecting CityFibre provided pseudowire circuits through Telehouse North. Engineers are investigating and further updates will be provided as soon as possible.
-
Engineers have investigated and believe the issue to now be resolved. Please ask any customers still having issues to reboot their router before contacting the service desk. We apologise for any inconvenience caused.
Incident: Leased lines in Sheffield, Peterborough and Nottingham
Posted: Tuesday, September 15th, 2020 at 12:03 by david labouchardiere
We are aware of an issue affecting leased lines in and around Sheffied, Peterborough and Nottingham. Our networking team are currently investigating the issue and further updates will be provided as soon as possible.
-
Sheffield and Nottingham core routers are back online but we still have an issue in Peterborough. Engineers are being dispatched.
-
Engineers are expected on site at 1500
-
The Engineer is due to arrive on site imminently.
-
Engineer is now on site and investigations have begun
-
NOC have now gained access to the core router and have begun troubleshooting the issue.
Further updates to be published at 1730
-
Services have been restored. A reboot was required which has resulted in the data necessary to carry out further root cause analysis to be lost. A reboot in this instance was the last available option in order for us to restore service.
Due to this we will continue to monitor for the next 48hours.
Incident: BT Edinburgh
Posted: Wednesday, August 12th, 2020 at 08:57 by David Brewis
A major incident has been declared by BT for an exchange in Edinburgh impacting broadband and ethernet services.
Loss of service is a result of water ingress into the location. The emergency command centre is onsite with majority of water removed, with engineers ensuring safe to enter to continue investigations and restore services as soon as possible.
Further updates will be provided.
-
David Brewis says:
Engineers remain on site with a plan in place, working to attempt to restore service. Equipment continues to remove remaining water with spare hardware and cabling on site and on route.
Next update to follow by 13:30
-
Water has been removed from the building and clean up is in progress. Cables have been reconnected providing restoration steps with testing being carried out.
Works will continue on site ensuring the location is secure and no further risk.
Next update to follow by 15:30
-
Sessions appear to have returned from around 13:30 and we will continue to monitor our network equipment and the progress of clean up activities.
No further updates to follow unless further disruption is caused by these activities as the incident is now resolved. If you experience a loss of service please call into the helpdesk following a reboot of your equipment.
Incident: Goswell Core
Posted: Friday, June 12th, 2020 at 04:58 by Matthew Davies
We have experienced a loss of connectivity to our Goswell core. This has been raised with our colocation providers, and their initial diagnostics suggest a wider power issue to be the cause. Investigations are ongoing, and further updates will be provided when available.
-
Matthew Davies says:
Our colocation providers have confirmed a power outage at our Goswell POP. Engineers are working to restore power and a further update will be provided when available.
-
Our colocation providers have restored power at our Goswell POP. We apologise for any inconvenience caused.
The supplier is currently investigating and confirms their hardware at the affected site is currently unreachable. We will provide further updates in the next hour.
We are observing Vodafone Off-net services now starting to restore.
Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We will be reaching out to our 3rd party supplier for RFO once investigation of the incident will be completed on their end.
We apologies for any inconvenience caused.