We are aware that a number of customers are impacted by an incident in Peterborough. Our teams are investigating the cause and we will provide an update once further information is known.
Archive for February, 2021
Emergency Work: Peterborough (PB1244)
Posted: Tuesday, February 23rd, 2021 at 16:28 by Jamie Smith
Tuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59
During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.
-
Robin Posadas says:
Work commencing.
-
Work completed
Incident: Peterborough
Posted: Tuesday, February 23rd, 2021 at 11:02 by david labouchardiere
Following the incident yesterday we are still working on restoring service to the portion of customers who remain impacted in Peterborough. Engineers are continuing to troubleshoot and further updates will be provided in the next hour
-
david labouchardiere says:
We are still investigating this issue with our senior agents within the NOC team and will provide further updates in the next hour.
-
Based on NOC troubleshooting we have engaged engineer resources to attend the affected site and carry out further testing. We will provide a further update once we have an ETA for the engineer
-
Engineer resources have been arranged with an ETA of 16:00. We will provide a further update upon the engineers arrival.
-
Emily Pieniak says:
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
-
Engineer resource is on-site investigating the physical link between PB1344 and PB1244. We will provide further updates in the next hour
-
The link between PB1344 and PB1244 remains down where we are still seeing 1 way MAC betwenPB1344 and PB1244. The SFP and fibre on PB1344 and PB1244 link has been re-seated and cleaned.
We will continue to provide updates in the next hour.
-
A replacement ASR is on-site which at present is being installed where we are awaiting further feedback from the engineers on-site.
We will continue to provide updates in the next hour.
-
SPF swap has taken place at PB1244 where engineers are continuing to work at PB1344 and PB1244.
We will continue to provide updates in the next hour.
-
The link between PB1344 and PB1244 remains down. Engineers have attempted multiple optics on both PB1344 and PB1244 where the port still would not come up.
Next action is for the fault to be logged with Nokia TAC for PB1344.
We will continue to provide updates within the next 2 hours.
-
Adam Wilson says:
Our vendors are dispatching an engineer to replace a suspected faulty card in our PB1344-rtr-1 node.
ETA is 02:30. A further update will be provided upon the engineer’s arrival. -
Engineer is on site and about to commence work. Update will follow upon completion of the task.
-
The card has been replaced, and we have collected further diagnostic information. We will confirm operational status once this has been reviewed by our vendor.
-
We believe this issue to now be resolved, and request that any affected users retest their service.
Please accept our apologies for any inconvenice that may have been caused.
Incident: Peterborough
Posted: Monday, February 22nd, 2021 at 16:21 by Adam Heath
We are currently aware of an issue affecting services within the Peterborough area. Engineers are already engaged and investigating. A further update will be provided when available.
-
Engineer resources have been engaged to attend the affected site with an ETA of 19:00. Further updates will be posted upon the engineer’s arrival.
-
Adam Heath says:
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 2 -: Further engineer resource is required at the location for investigation and restoration to commence. We do not have an ETA as of yet for their arrival but will provide this as soon as we are in receipt of it.
Update 1 –: 17:22 At 16:00 connection to PB1244 dropped causing a total loss of service. PB1244 remains unreachable and an FLM engineer has been dispatched to site to carry out investigations. ETA for engineer to arrive onsite is currently 19:00.
-
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 3 – Further engineer resource is expected to arrive at site at 21:35
-
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 4 -: The additional engineer resource has arrived onsite and working with our NOC, investigation has commenced.
Next Update: 23:00
-
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 5 -: Our NOC have confirmed that the core device is now reachable following resource at the location rebooting it. They are just performing some final checks before we mark this as resolved.
Next Update: 23:30
-
CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Resolved -: 23:37 Our NOC have confirmed from their checks following the device being rebooted, the majority of impacted services have been restored and the device is operational. Therefore, we will mark the Major Incident as resolved at this point.
They are aware of a very small number of services still offline. It is advised that the equipment at the customer’s side should be rebooted if this is the case and should the service still remain offline, to contact our Technical Service Centre.
There will be a follow up internal post incident review scheduled, with any further required actions being managed by the Change and Problem Management processes.
Incident: CityFibre network services
Posted: Tuesday, February 16th, 2021 at 18:51 by Adam Heath
We are aware that a number of customers based in the Midlands and M4 corridor are currently experiencing an outage of their On-net service. Our teams are currently investigating and are looking to restore service as soon as possible. Apologies for the impact that this will cause.
Edit: This is now a widespread network issue and potentially causing customers a total loss of service, packet loss or latency.
-
Lucy Jordan says:
We are still investigating this issue with our T3 agents within the NOC team and will provide further updates as soon as they are available.
Apologies for any inconvenience and the delay in any substantial update up until now. We endeavour to address this as soon as possible.
Please keep checking here for any updates or any cases you have already raised with us, linked to the parent incident.
-
jbiddulph says:
Since 20:45PM, we are beginning to see a large number of services beginning to restore.
However, our engineers are currently still investigating the root cause of the outage, as there may still be some services at risk and offline until this has been completely resolved and concluded.
We will continue to provide updates as to when and available.
-
Following our previous response, our engineers are currently still investigating the root cause of the outage.
We have had confirmation that the majority of services have restored.
Updates will continue to be provided as to when and available.
-
We are aware of an issue in Peterborough and a large number of services down in the area, we are currently working to fix it and we have scheduled an engineer to go to site to assist us in our investigations.
Specific details regarding this visit will follow when available.
-
Our engineer is due to arrive on site a 2am to assist investigations and restore the outage to Peterborough services.
Following this visit, we should have a further update available by 3 am.
Updates will continue to be provided as to when and available.
-
Our engineer is currently on-site performing investigations to restoring service to the Peterborough connections.
Updates will follow after 3 am.
-
We have had the latest report from our engineer.
Following a re-boot of a core device on-site, services have begun to restore across the network for Peterborough connections.
We now advise to ensure that any services still down, are fully re-booted, and should the issue persist, please contact the service desk to advise the remaining connection issues.
A full report of the root cause is still currently ongoing and to be provided at a later time.
-
We are currently seeing a recurrence of the issues following the work earlier this morning. Senior network engineers are continuing to investigate the root cause and further updates will be provided as soon as possible
-
High level investigations into the issue are continuing to take place. At present, the problem will cause packet loss, latency or instances of total loss of service and is affecting multiple tunnels across the CityFibre network.
-
NOC and architecture & engineering teams are currently working with external vendor support to find the root cause and restore services. Further updates will be provided in the next hour.
-
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Major Incident Leader:
Daniel WalkerService (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
Investigations into the issue are ongoing. Following internal technical calls, the resolver teams are currently focusing efforts on stabilising the impacted services. Once these have been stabilised, they can then focus on finding the root cause and a permanent fix. Another internal technical call is taking place at 12:30 and further updates will be provided following this at 13:15Next Update
17/02/2021 13:15
-
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Service (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
Work has been undertaken to return stability which involved shutting down one of our links. Following this, we have seen some services restored however, customers in the Peterborough area may still be impacted. We are currently waiting for engineer resources to attend Peterborough and carry out further troubleshooting.Once Peterborough services have been restored we will then look to put a permanent fix in place ensuring any changes goes through the change management process due to potential impact to services. Further updates will be provided at 14:15
Next Update:
17/02/2021 14:15 -
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Service (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
We are seeing the majority of services now restored apart from those directly connected through Peterborough. We are now awaiting an engineer to attend Peterborough sites with an ETA of 16:00, the technical teams will then work with the engineers to restore the services connected to Peterborough sites.Next Update:
17/02/2021 17:15 -
Please find the incident update below;
CityFibre Incident Number:
INC0080597Major Incident Raised:
17/02/2021 – 10:00Service (product) Type:
VariousService(s) Impacted:
All traffic/services on Entanet network potentially impactedConfiguration Item(s) Impacted:
NACustomer Impact:
Impacting all customers across the network using the MPLS TE Tunnels – total loss of service, packet loss or latency.Description:
We are currently aware of an issue impacting any traffic over the MPLS TE tunnels and are currently investigating the issue and looking to restore services ASAP.Team(s) Investigating:
NOC & A&EUpdate:
Following engineer attendance in Peterborough we can confirm all affected services are now restored. We will now close down the Major Incident and any remedial and root cause efforts will now be managed via the Change & Problem Management processes.Please accept our apologies for the inconvenience caused by this issue.
Incident: CityFibre leased lines
Posted: Monday, February 15th, 2021 at 14:35 by david labouchardiere
We are currently investigating reports of connectivity issues affecting CityFibre leased lines in Scotland. Engineers are investigating and further updates will follow as and when they become available.
-
mpurcell says:
Our engineering team reported a large number of MPLS tunnels showing as down. These have been re-established and believe that services should begin to be restoring.
-
Neil Watson says:
We are aware that some customers are still reporting that services have not been restored, as such we have further investigated the issue. We believe that we have identified the cause, which is related to MTU, and the teams are working to correct the issue now. Next update will be in approximately 1hr or sooner, when we believe the service will be restored.
-
Our teams have now restored service for all impacted customers. The incident was related to MTU configuration within the core network, which has now been corrected. We do not anticipate a repeat of the incident and will undertake a review of the root cause. Apoligies to all customers that were affected.
Emergency Work: London Interxion Suite (LON44)?
Posted: Wednesday, February 10th, 2021 at 09:47 by Iain Beveridge
Wednesday 10th February 2021, 22:00 – Thursday 11th February 2021 05:00
During the above window, we will be working to restore power stability to some customer firewalls within our interxion suite. An ATS that is supplying one of out racks is faulty, we will be moving devices and config one at a time to new infrastructure within another rack in the suite. There will be a small window of downtime as we move these devices one at a time in a staged and controlled manner. This is follows on from last nights work, this window should conclude the work in this rack.
-
Bogdan Voda says:
This work has now been completed.
Apologies for any inconvenience this may have caused.
Emergency Work: London Interxion Suite (LON44)
Posted: Tuesday, February 9th, 2021 at 19:45 by Iain Beveridge
Tuesday 09th February 2021, 22:00 – Wednesday 10th February 2021 05:00
During the above window, we will be working to restore power stability to some PWAN infrastructure and customer firewalls within our interxion suite. An ATS that is supplying one of out racks is faulty, we will be moving devices and config one at a time to new infrastructure within another rack in the suite. There will be a small window of downtime as we move these devices one at a time in a staged and controlled manner.
-
Please be aware the above works is now about to commence.
-
This has now been completed for tonight.
Further work will need to be completed out of hours and will be communicated in due course.Apologies for the short notice and any inconvenience this may cause.
Power outage affecting London Interxion Suite (LON44)
Posted: Monday, February 8th, 2021 at 13:43 by Adam Heath
We are currently experiencing a major outage. This is under investigation and we will have an update for you as soon as we know anything further. Next update will be within the hour
-
Initial reports are coming in that a power outage is causing this disruption. We appear to have lost a large amount of Edge devices. Engineers are on route to site.
A further update will be made available within the hour.
-
Engineers are now on site and investigations have begun.
Further updates within the hour
-
Engineers have reported power is now restored. We will now begin checking for service restoration.
-
Interxion Engineers are currently making their way around the suites in the datacentre, but as yet we have no ETA for when they will get to our equipment.
We are chasing every 15 minutes for an update and once we have more information we will be in touch. Next update due in 1 hour.
-
This matter has been escalated to Datacentre Management and continue to chase for further updates. However, we are beginning to see power restoration to our suite as well as some services being restored.
Further updates in 1 hour
-
Jonathan Clarke says:
Engineers remain at the Datacentre working to restore all power.
We will have a further update in 1 hour.
-
We are continually seeing more services restore, however, currently our engineers are still on-site working on restoring all services to the currently downed devices.
A further update will be provided in 1 hour.
-
PWAN services are now mostly restored. Please re-boot any services before contacting the Service Desk.
However, there are a few stragglers that are still showing as down and further work is still ongoing to restore services for all customers.
Further updates to follow.
BT Planned Work: Colindale TE
Posted: Tuesday, February 2nd, 2021 at 14:40 by Iain Beveridge
Thursday 04th March 2021, 21:00 – Friday 05th March 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above site requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service it should still be considered at risk.
Our Team have advised that it is the same link at fault as earlier in the week. However, the fault conditions are different, so it does not appear to be a repeat fault. We have undertaken several steps to clear any issues with our hardware and cards, and we can see that other ports etc. are up and running. We’ve interrogated the logs and events and can see nothing untoward other than a loss of light from one side. Therefore, we are arranging engineers to attend the site and will update with and ETA as soon as we are able.
Our NOC team have continued to investigate and believe the issue is fibre related rather than hardware. They have spoken with our ERS contractor in advance of them attending site and to ensure that they are aware of the actions needed. We have also engaged with our first line maintenance teams in case anything additional is needed and have them on standby. As soon as we are able to confirm planned arrival on site, we will update.
Engineer resources have been arranged with an ETA of 18:00. We will provide a further update upon the engineers arrival.
Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
Engineer resources are on-site who continue to investigate the issue. We currently have fibre engineers and hardware teams present working alongside our NOC team.
We will provide a further updates in the next hour.
Engineer resources have confirmed that not lights within the fibre is being received between PB1344 and PB1244. They have also proofed that the port is up which suggests the SFP is not related to the issue and lies within the fibre.
Further updates will follow once the engineer resources have reached PB1244.
Replacement optics have been sourced that are compatible with both Cisco and Nokia (MSA compatible). The link from PB1244 – PB1344 is 12km, these optics are 80km so have requested attenuators to align the link up with the required light level.
Engineer resources have been arranged to collect parts and attend both sites.
Further updates will follow once parts and engineer resources have arrived on-site.
Engineer parts have arrived on site and further work is ongoing, alongside our NOC team.
Further updates will be provided, once we have any new information from the engineers.
Engineers have restored customer services although this is only a temporary solution at the moment as a workaround is in place. Further investigation work will be undertaken and a permanent solution will be introduced through our change management process. We will update when this will happen. Apologies for the issue and the disruption this will have caused.