We are currently aware of an issue affecting services within the Peterborough area. Engineers are already engaged and investigating. A further update will be provided when available.
We are aware that a number of customers based in the Midlands and M4 corridor are currently experiencing an outage of their On-net service. Our teams are currently investigating and are looking to restore service as soon as possible. Apologies for the impact that this will cause.
Edit: This is now a widespread network issue and potentially causing customers a total loss of service, packet loss or latency.
Lucy Jordan says: We are still investigating this issue with our T3 agents within the NOC team and will provide further updates as soon as they are available.
Apologies for any inconvenience and the delay in any substantial update up until now. We endeavour to address this as soon as possible.
Please keep checking here for any updates or any cases you have already raised with us, linked to the parent incident.
jbiddulph says: Since 20:45PM, we are beginning to see a large number of services beginning to restore.
However, our engineers are currently still investigating the root cause of the outage, as there may still be some services at risk and offline until this has been completely resolved and concluded.
We will continue to provide updates as to when and available.
jbiddulph says: Following our previous response, our engineers are currently still investigating the root cause of the outage.
We have had confirmation that the majority of services have restored.
Updates will continue to be provided as to when and available.
jbiddulph says: We are aware of an issue in Peterborough and a large number of services down in the area, we are currently working to fix it and we have scheduled an engineer to go to site to assist us in our investigations.
Specific details regarding this visit will follow when available.
jbiddulph says: Our engineer is due to arrive on site a 2am to assist investigations and restore the outage to Peterborough services.
Following this visit, we should have a further update available by 3 am.
Updates will continue to be provided as to when and available.
jbiddulph says: Our engineer is currently on-site performing investigations to restoring service to the Peterborough connections.
Updates will follow after 3 am.
jbiddulph says: We have had the latest report from our engineer.
Following a re-boot of a core device on-site, services have begun to restore across the network for Peterborough connections.
We now advise to ensure that any services still down, are fully re-booted, and should the issue persist, please contact the service desk to advise the remaining connection issues.
A full report of the root cause is still currently ongoing and to be provided at a later time.
david labouchardiere says: We are currently seeing a recurrence of the issues following the work earlier this morning. Senior network engineers are continuing to investigate the root cause and further updates will be provided as soon as possible
david labouchardiere says: High level investigations into the issue are continuing to take place. At present, the problem will cause packet loss, latency or instances of total loss of service and is affecting multiple tunnels across the CityFibre network.
david labouchardiere says: NOC and architecture & engineering teams are currently working with external vendor support to find the root cause and restore services. Further updates will be provided in the next hour.
david labouchardiere says: Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Major Incident Leader:
Daniel Walker
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
Investigations into the issue are ongoing. Following internal technical calls, the resolver teams are currently focusing efforts on stabilising the impacted services. Once these have been stabilised, they can then focus on finding the root cause and a permanent fix. Another internal technical call is taking place at 12:30 and further updates will be provided following this at 13:15
Next Update
17/02/2021 13:15
david labouchardiere says: Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
Work has been undertaken to return stability which involved shutting down one of our links. Following this, we have seen some services restored however, customers in the Peterborough area may still be impacted. We are currently waiting for engineer resources to attend Peterborough and carry out further troubleshooting.
Once Peterborough services have been restored we will then look to put a permanent fix in place ensuring any changes goes through the change management process due to potential impact to services. Further updates will be provided at 14:15
Next Update:
17/02/2021 14:15
david labouchardiere says: Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
We are seeing the majority of services now restored apart from those directly connected through Peterborough. We are now awaiting an engineer to attend Peterborough sites with an ETA of 16:00, the technical teams will then work with the engineers to restore the services connected to Peterborough sites.
Next Update:
17/02/2021 17:15
david labouchardiere says: Please find the incident update below;
CityFibre Incident Number:
INC0080597
Major Incident Raised:
17/02/2021 – 10:00
Service (product) Type:
Various
Service(s) Impacted:
All traffic/services on Entanet network potentially impacted
Configuration Item(s) Impacted:
NA
Customer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.
Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.
Team(s) Investigating:
NOC & A&E
Update:
Following engineer attendance in Peterborough we can confirm all affected services are now restored. We will now close down the Major Incident and any remedial and root cause efforts will now be managed via the Change & Problem Management processes.
Please accept our apologies for the inconvenience caused by this issue.
We are currently investigating reports of connectivity issues affecting CityFibre leased lines in Scotland. Engineers are investigating and further updates will follow as and when they become available.
mpurcell says: Our engineering team reported a large number of MPLS tunnels showing as down. These have been re-established and believe that services should begin to be restoring.
Neil Watson says: We are aware that some customers are still reporting that services have not been restored, as such we have further investigated the issue. We believe that we have identified the cause, which is related to MTU, and the teams are working to correct the issue now. Next update will be in approximately 1hr or sooner, when we believe the service will be restored.
Neil Watson says: Our teams have now restored service for all impacted customers. The incident was related to MTU configuration within the core network, which has now been corrected. We do not anticipate a repeat of the incident and will undertake a review of the root cause. Apoligies to all customers that were affected.
Wednesday 10th February 2021, 22:00 – Thursday 11th February 2021 05:00
During the above window, we will be working to restore power stability to some customer firewalls within our interxion suite. An ATS that is supplying one of out racks is faulty, we will be moving devices and config one at a time to new infrastructure within another rack in the suite. There will be a small window of downtime as we move these devices one at a time in a staged and controlled manner. This is follows on from last nights work, this window should conclude the work in this rack.
Bogdan Voda says: This work has now been completed.
Apologies for any inconvenience this may have caused.
Tuesday 09th February 2021, 22:00 – Wednesday 10th February 2021 05:00
During the above window, we will be working to restore power stability to some PWAN infrastructure and customer firewalls within our interxion suite. An ATS that is supplying one of out racks is faulty, we will be moving devices and config one at a time to new infrastructure within another rack in the suite. There will be a small window of downtime as we move these devices one at a time in a staged and controlled manner.
Bogdan Voda says: Please be aware the above works is now about to commence.
Bogdan Voda says: This has now been completed for tonight.
Further work will need to be completed out of hours and will be communicated in due course.
Apologies for the short notice and any inconvenience this may cause.
We are currently experiencing a major outage. This is under investigation and we will have an update for you as soon as we know anything further. Next update will be within the hour
mpurcell says: Initial reports are coming in that a power outage is causing this disruption. We appear to have lost a large amount of Edge devices. Engineers are on route to site.
A further update will be made available within the hour.
mpurcell says: Engineers are now on site and investigations have begun.
Further updates within the hour
mpurcell says: Engineers have reported power is now restored. We will now begin checking for service restoration.
mpurcell says: Interxion Engineers are currently making their way around the suites in the datacentre, but as yet we have no ETA for when they will get to our equipment.
We are chasing every 15 minutes for an update and once we have more information we will be in touch. Next update due in 1 hour.
mpurcell says: This matter has been escalated to Datacentre Management and continue to chase for further updates. However, we are beginning to see power restoration to our suite as well as some services being restored.
Further updates in 1 hour
Jonathan Clarke says: Engineers remain at the Datacentre working to restore all power.
We will have a further update in 1 hour.
jbiddulph says: We are continually seeing more services restore, however, currently our engineers are still on-site working on restoring all services to the currently downed devices.
A further update will be provided in 1 hour.
Lucy Jordan says: PWAN services are now mostly restored. Please re-boot any services before contacting the Service Desk.
However, there are a few stragglers that are still showing as down and further work is still ongoing to restore services for all customers.
Further updates to follow.
Thursday 04th March 2021, 21:00 – Friday 05th March 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above site requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service it should still be considered at risk.
Thursday 25th February 2021, 21:00 – Friday 26th February 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above site requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
Thursday 18th February 2021, 21:00 – Friday 19th February 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above site requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
Thursday 11th February 2021, 21:00 – Friday 12th February 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above site requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
Engineer resources have been engaged to attend the affected site with an ETA of 19:00. Further updates will be posted upon the engineer’s arrival.
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 2 -: Further engineer resource is required at the location for investigation and restoration to commence. We do not have an ETA as of yet for their arrival but will provide this as soon as we are in receipt of it.
Update 1 –: 17:22 At 16:00 connection to PB1244 dropped causing a total loss of service. PB1244 remains unreachable and an FLM engineer has been dispatched to site to carry out investigations. ETA for engineer to arrive onsite is currently 19:00.
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 3 – Further engineer resource is expected to arrive at site at 21:35
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 4 -: The additional engineer resource has arrived onsite and working with our NOC, investigation has commenced.
Next Update: 23:00
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 5 -: Our NOC have confirmed that the core device is now reachable following resource at the location rebooting it. They are just performing some final checks before we mark this as resolved.
Next Update: 23:30
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Resolved -: 23:37 Our NOC have confirmed from their checks following the device being rebooted, the majority of impacted services have been restored and the device is operational. Therefore, we will mark the Major Incident as resolved at this point.
They are aware of a very small number of services still offline. It is advised that the equipment at the customer’s side should be rebooted if this is the case and should the service still remain offline, to contact our Technical Service Centre.
There will be a follow up internal post incident review scheduled, with any further required actions being managed by the Change and Problem Management processes.