Tuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59
During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.
Following the incident yesterday we are still working on restoring service to the portion of customers who remain impacted in Peterborough. Engineers are continuing to troubleshoot and further updates will be provided in the next hour
david labouchardiere says: We are still investigating this issue with our senior agents within the NOC team and will provide further updates in the next hour.
david labouchardiere says: Based on NOC troubleshooting we have engaged engineer resources to attend the affected site and carry out further testing. We will provide a further update once we have an ETA for the engineer
david labouchardiere says: Engineer resources have been arranged with an ETA of 16:00. We will provide a further update upon the engineers arrival.
Emily Pieniak says: Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.
Emily Pieniak says: Engineer resource is on-site investigating the physical link between PB1344 and PB1244. We will provide further updates in the next hour
Emily Pieniak says: The link between PB1344 and PB1244 remains down where we are still seeing 1 way MAC betwenPB1344 and PB1244. The SFP and fibre on PB1344 and PB1244 link has been re-seated and cleaned.
We will continue to provide updates in the next hour.
Emily Pieniak says: A replacement ASR is on-site which at present is being installed where we are awaiting further feedback from the engineers on-site.
We will continue to provide updates in the next hour.
Emily Pieniak says: SPF swap has taken place at PB1244 where engineers are continuing to work at PB1344 and PB1244.
We will continue to provide updates in the next hour.
Emily Pieniak says: The link between PB1344 and PB1244 remains down. Engineers have attempted multiple optics on both PB1344 and PB1244 where the port still would not come up.
Next action is for the fault to be logged with Nokia TAC for PB1344.
We will continue to provide updates within the next 2 hours.
Adam Wilson says: Our vendors are dispatching an engineer to replace a suspected faulty card in our PB1344-rtr-1 node.
ETA is 02:30. A further update will be provided upon the engineer’s arrival.
Adam Wilson says: Engineer is on site and about to commence work. Update will follow upon completion of the task.
Adam Wilson says: The card has been replaced, and we have collected further diagnostic information. We will confirm operational status once this has been reviewed by our vendor.
Adam Wilson says: We believe this issue to now be resolved, and request that any affected users retest their service.
Please accept our apologies for any inconvenice that may have been caused.
We are currently aware of an issue affecting services within the Peterborough area. Engineers are already engaged and investigating. A further update will be provided when available.
david labouchardiere says: Engineer resources have been engaged to attend the affected site with an ETA of 19:00. Further updates will be posted upon the engineer’s arrival.
Adam Heath says: CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 2 -: Further engineer resource is required at the location for investigation and restoration to commence. We do not have an ETA as of yet for their arrival but will provide this as soon as we are in receipt of it.
Update 1 –: 17:22 At 16:00 connection to PB1244 dropped causing a total loss of service. PB1244 remains unreachable and an FLM engineer has been dispatched to site to carry out investigations. ETA for engineer to arrive onsite is currently 19:00.
Adam Heath says: CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 3 – Further engineer resource is expected to arrive at site at 21:35
Adam Heath says: CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 4 -: The additional engineer resource has arrived onsite and working with our NOC, investigation has commenced.
Next Update: 23:00
Adam Heath says: CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Update 5 -: Our NOC have confirmed that the core device is now reachable following resource at the location rebooting it. They are just performing some final checks before we mark this as resolved.
Next Update: 23:30
Adam Heath says: CityFibre Incident Number: INC0080756
Major Incident Raised: 22/02/2021 – 16:00
Major Incident Leader: Adam Heath
Service (product) Type: Off-net Ethernet & On-net Ethernet
Service(s) Impacted: Multiple
Configuration Item(s) Impacted: NA
Customer Impact: Total loss of services for customers connected to PB1244.
Description: Core device has become unreachable.
Team(s) Investigating: NOC and Field Engineers
Resolved -: 23:37 Our NOC have confirmed from their checks following the device being rebooted, the majority of impacted services have been restored and the device is operational. Therefore, we will mark the Major Incident as resolved at this point.
They are aware of a very small number of services still offline. It is advised that the equipment at the customer’s side should be rebooted if this is the case and should the service still remain offline, to contact our Technical Service Centre.
There will be a follow up internal post incident review scheduled, with any further required actions being managed by the Change and Problem Management processes.
We are currently aware of an issue affecting CityFibre provided leased lines in the Bristol area. Engineers are on site investigating and working to restore service as soon as possible
david labouchardiere says: We have had a confirmation service should now be restored. Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We apologise for any inconvenience caused by this incident.
Thursday 17th December 2020, 13:30 – 14:00
During the above window we will be carrying out emergency maintenance to the traffic shapers within the DSL platform. This is necessary work due to the ongoing issues documented at https://noc.enta.net/2020/12/incident-dsl-packet-loss/
No downtime is expected to services however broadband services should be considered at risk for the duration of this work. Further work will need to be completed out of hours after this initial maintenance and will be communicated in due course.
Apologies for the short notice and any inconvenience this may cause.
Jamie Smith says: Please be aware that between 18:00 – 21:00 tonight (17/12/2020) we will be carrying out further remedial work as mentioned previously on the traffic shapers within the Broadband Network.
No downtime is expected to services however broadband services should be considered at risk for the duration of this work.
We apologize for the short notice and any inconvenience this may have caused.
Jamie Smith says: Please be aware the above works is now about to commence
Jamie Smith says: Work has now been completed
We are currently investigating reports of packet loss affecting broadband customers.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
david labouchardiere says: We are continuing to investigate reports of packet loss, however we would request any affected customers to provide the service desk with supporting evidence in the form of pathpings or winMTR data to allow further troubleshooting
david labouchardiere says: Investigations into the packet loss reports are ongoing. We continue to ask that any affected circuits provide us with supporting evidence using winMTR or pathpings to assist in troubleshooting
david labouchardiere says: Investigations are ongoing into the reports of packet loss. We are in talks with the supplier requesting they troubleshoot on their site based off the evidence supplied so far. We will update once we have further information from them.
david labouchardiere says: The supplier is currently investigating the reported issue and has raised the case to their networking team for further assistance. We will continue to push for further updates. At present, no further evidence is required from affected customers.
david labouchardiere says: The supplier is continuing to investigate the issue on their side and further updates are expected in the next hour.
peter butler says: Investigations with our supplier are continuing but, unfortunately, we do not have any solid updates or timescales for resolution at this point.
We shall continue to chase this over the course of the evening and post updates when they become available.
Jonathan Clarke says: Investigations into the packet loss reports are ongoing, BT advise they are chasing their 2nd line team for updates. We are continuing to chase for updates, as soon as we anything further, we will update you again.
Jonathan Clarke says: BT advise that packet loss was observed when testing. BT advise that the there is a potential issue on their core network. Further troubleshooting will be required, however, the issue is only present within core hours.
As of this, BT are going to be continuing testing throughout the day. BT’s 2nd line team are engaging with their 3rd line team for further assistance today.
We will update once we have further information from them.
Peter Butler says: BT advise that as the packet loss does not appear to occur overnight this has been passed to their day shift to investigate today. Intermittent loss has been seen again this morning so they are continuing to trace the root cause of the issue.
Our escalation point is aiming to get an update to us by 1200 today, so we will aim to update via this post shortly thereafter.
Peter Butler says: BT have advised that the intermittent packet loss that was seen at 0800 this morning has not been present since, which has hindered progress with their fault finding.
Investigations continue and BT expect to be able to provide a further update by around 1600 today.
Adam Wilson says: We have provided further evidence of packet loss to BT, and this is currently being reviewed. Further updates will be provided as soon as they become available.
Jonathan Clarke says: BT have advised that their 3rd line team are not observing any packet loss whilst testing. We have been advised that they will continue to troubleshoot out of hours, however, mentioned that they believe that the packet loss will not present itself until Monday morning in core hours.
Our escalation point within BT has stressed to them that if this can be resolved before Monday and ensure us all efforts to resolve this before then are being made.
A Further update will be provided as they become available.
Jonathan Clarke says: BT advise that 2nd line and 3rd line team have been monitoring the BTW network for signs of packet loss. In the last 48 hours no packet loss has been observed. As of this, BT have been unable to replicate the issue out of core hours.
BT’s 3rd line team are monitoring the network this morning for when the packet loss reappears, to ascertain where in the network the issue lies to resolve.
A Further update will be provided following BT’s troubleshooting.
david labouchardiere says: BT advise they have seen no further packet loss today and our monitoring confirms this. BT are continuing to investigate and have setup additional monitors to assist this. We will provide a further update once we have any more information.
Peter Butler says: We have been receiving further reports of packet loss coming into the desk as of around 1430 today. We have alerted BT to this and they have confirmed that they are looking into this.
We shall provide further updates once they are received.
david labouchardiere says: We are continuing to push BT for further updates on their investigation into this issue and provided additional details to them to assist in their testing. We will provide a further update once we have any more information.
Peter Butler says: We are continuing to provide examples of circuits affected by this issue along with evidence to BT to try and push this to resolution, we are also engaging escalation points within BT to ensure this is treated at the highest priority on their side.
We shall continue to chase and provide updates when available.
Peter Butler says: Investigations are still ongoing with BT with our NOC team also assisting and investigating the traffic as it enters our network to ensure we are looking at the whole picture.
Unfortunately as the loss seems to occur predominantly during core hours it is likely to be the morning before any further update can be provided, so a further post will be made tomorrow morning or upon the receipt of further information.
Peter Butler says: Unfortunately no issues could be found overnight whilst the issue was not manifesting itself so this has now been picked back up by the day shifts on both BTs and our side.
We have multiple circuits being monitored which allows us to see when the issue is present so will update further when more information is available.
No further evidence of the issue is required from customers at this point.
Peter Butler says: An emergency change is currently being planned in order to assist with troubleshooting the issues curently being seen. This is being discussed at present so that it can be decided when best to carry this out.
A further update will be posted at 1500 or upon receipt of further information.
david labouchardiere says: We are seeing no further signs of loss following the emergency work carried out yesterday and believe this issue to now be resolved. Any remaining customers that are affected are advised to reboot their router before calling into the Service Desk. We apologise for any inconveniences caused by this outage
We are currently aware of an incident affecting leased line connections in the Portsmouth area. This is being looked into by our NOC team and engineers are on site to try and resolve this as soon as possible.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
Lucy Jordan says: This is still being investigated. However, we expect a Nokia engineer on site at approx 18:45 this evening and a replacement card will hopefully help to resolve the issue. Further work is ongoing with the current engineer on site, to try and restore links in the meantime.
jbiddulph says: We are currently still awaiting a part to be delivered so we can restore complete service. This was unfortunately delayed and is due to arrive for 9:30PM. Since the previous engineer has had to be relieved due to working on this issue since 8AM, we are also dispatching a new engineer to help fix the fault.
In the meantime, we are continuously working on trying to restore this remotely. A further update will follow promptly when available.
jbiddulph says: All Services should now be restored. We are seeing the core device that was offline for this to now be online.
A full RFO will be provided at a later time to advise of the fault cause, and fix. If you have any further issues, we advise to fully re-boot all services.
We are currently aware of an incident affecting Virgin Media leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
Jonathan Clarke says: The supplier is currently investigating the issue. They advised that there appears to be an issue at Telehouse exchange. We are now awaiting a further update.
As soon as we have anything further, we will update you again.
mpurcell says: In relation to the loss of service you are currently experiencing. Our supplier is still working hard to resolve these issues as a priority.
They have engineers on site continuing investigations into the loss and intermittent connections that are being experienced.
We will update you again in one hour
mpurcell says: In relation to the loss of service you are currently experiencing: our supplier is still working hard to resolve these issues as a priority. Their engineers are on site continuing their investigations in to the intermittent loss of service and packet loss that you may be experiencing. We have chased for further details but unfortunately at present there is no further update.
We will update you again in one hour. Thank you for your continued patience.
mpurcell says: Report received from our supplier that they have reconnected a patch in their head end and are seeing our service come back online.
If you are still being impacted please contact us as soon as possible and confirm site checks.
We are currently aware of an incident affecting Vodafone leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
david labouchardiere says: The supplier is currently investigating and confirms their hardware at the affected site is currently unreachable. We will provide further updates in the next hour.
david labouchardiere says: The supplier has arranged engineer resources to attend the affected site. Current ETA for the engineer is 15:00
david labouchardiere says: Engineers have arrived on site and are accessing the data centre. Further updates will be provided in the next hour.
mpurcell says: Apologies for the delays being experienced here. Our suppliers have experienced some difficulty acquiring replacement hardware but are now en route back to site with the new hardware and have an ETA of 1700. We will provide a further update in one hour.
Matthew Davies says: Apologies for the further delays. The suppliers engineer is on site and currently working on the issue, with the additional spare equipment required.
We’ll have a further update for you within the hour.
Matthew Davies says: Our supplier has rebooted the device that is at cause for the outage, which will have restored most services.
The device is being replaced however, so a further outage is expected. We’ll let you know as work progresses.
Matthew Davies says: Our supplier is onsite currently working on loading new config onto the replacement device. We’re expecting the device to be replaced soon.
We’ll provide further updates as they come.
Jonathan Clarke says: Our supplier has advised that the spare equipment at site that they were installing appears to be faulty, therefore, suppliers are ordering new hardware to configure and reinstall. We have been advised and observed ourselves that the services that were affected are up and stable post reboot.
Due to the above, another Field Engineer will be scheduled for the new equipment install. VF has advised they aim to have this swap completed ASAP. VF’s NOC team is in contact with vendors to give us a timeline of the work.
We’ll have a further update for you as soon as possible and will update you again.
Jonathan Clarke says: Our suppliers are still working on timescales for the hardware replacement, we are keeping in regular contact and as soon as we have any further information, we will update you again.
Jonathan Clarke says: Suppliers advise engineer has been dispatched back to site to replace hardware. ETA is 23:15 GMT
We’ll provide further updates as we receive them.
Jonathan Clarke says: Suppliers advise that the hardware was due to arrive at 23:30 GMT, so as of this, should have now arrived. We have been advised that the VF field engineer and VF NOC will configure and install.
We’ll have a further update for you within the hour.
Jonathan Clarke says: Supplier engineer and their NOC team continue to work on the issue with replacement part being configured.
We expect a further update within the hour.
Jonathan Clarke says: Suppliers advise Vodafone 2nd line is coordinating with field engineers to replace the Hardware now, we expect to see the affected circuits drop whilst this replacement takes place.
We expect a further update within the hour.
Jonathan Clarke says: Suppliers advise that the hardware swap will occur now, we expect circuits to remain down for 45 minutes to an hour.
We will continue to monitor the network and advise once this change has been completed.
Jonathan Clarke says: We observed circuits connected to suppliers hub disconnect around 03:07. We are seeing the circuits reconnect now.
We will await a response from suppliers and advise once the circuits are no longer considered to be at risk
Jonathan Clarke says: Suppliers has advised that the swap of the affected hardware has been completed. However, whilst trying to upgrade the IOS of the new hardware, this has so far been unsuccessful.
VF’s 2nd and 3rd line team are coordinating with each other to have this resolved, therefore, further interruptions to the service may be observed. As of this, circuits are still at risk.
As soon as we have a further update, we will let you know.
Jonathan Clarke says: Suppliers advise that the IOS firmware upgrade has been successful and was has confirmed by their 3rd line team. Suppliers have informed us that services are up and observe good traffic over the network. I can confirm the same from our monitoring.
VF has now stood their team down with this Major Incident. VF’s service desk team will send us the RFO once completed.
Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We apologies to you and your customers for the inconvenience caused.
We are aware of a wide-spread packet loss issue following an outage instance that occurred last night. This is being thoroughly investigated for a resolution for as soon as possible.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
jbiddulph says: Investigating this further we have confirmed the issue is nationwide.
Unfortunately, we have no further specific details on this, however, at this moment in time we are continuously working towards the incident resolution.
Further updates will follow when available.
mpurcell says: Further to our last update.
Due to the unplanned outage last night we are experiencing issues. As a result all traffic has been routed through an alternative BGP route and is subsequently causing congestion on this secondary route. We are currently in conversation with a supplier to establish whether or not we can bring this primary route back online. Despite this we are seeing that services are restored, but with issues.
Further updates will be made available as soon as we have them.
jbiddulph says: At this moment in time, contact with the supplier is still currently underway to establish whether we can bring the primary route back online.
We treating this with the highest priority that we can to restore complete service to all services affected, and apologise for the inconvenience caused so far.
Further updates will follow as soon as they are available.
Jamie Smith says: Emergency Work
14:00 – 16:00 6/11/2020
Due to the ongoing issues being observed on the DSL platform causing heavy packet loss, NOC engineers will soon be carrying out remedial work.
All broadband users should be considered at risk for the duration of this work.
We apologize for any inconvenience and further updates to be provided.
Jamie Smith says: This work has now been completed. We will continue to monitor the DSL platform.
Any users still experiencing residual issues are advised to reboot the equipment prior to contacting the support desk.
Once again we apologize for any inconvenience caused by the issue.
Work commencing.
Work completed