We are currently aware of an issue affecting CityFibre provided leased lines in the Bristol area. Engineers are on site investigating and working to restore service as soon as possible
Thursday 14th January 2021, 20:30 – Friday 14th January 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above TE requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service it should still be considered at risk.
The above works have been cancelled due to adverse weather conditions in the local area. These works will be rescheduled with the new date communicated out accordingly.
Thursday 17th December 2020, 13:30 – 14:00
During the above window we will be carrying out emergency maintenance to the traffic shapers within the DSL platform. This is necessary work due to the ongoing issues documented at https://noc.enta.net/2020/12/incident-dsl-packet-loss/
No downtime is expected to services however broadband services should be considered at risk for the duration of this work. Further work will need to be completed out of hours after this initial maintenance and will be communicated in due course.
Apologies for the short notice and any inconvenience this may cause.
Jamie Smith says: Please be aware that between 18:00 – 21:00 tonight (17/12/2020) we will be carrying out further remedial work as mentioned previously on the traffic shapers within the Broadband Network.
No downtime is expected to services however broadband services should be considered at risk for the duration of this work.
We apologize for the short notice and any inconvenience this may have caused.
Jamie Smith says: Please be aware the above works is now about to commence
Jamie Smith says: Work has now been completed
Thursday 7th January 2021, 20:30 – Friday 8th January 2021 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, Kingston TE requires both the ATS and UPS to be refreshed. Although only Kingston is directly effected due to network design customers with services on Colindale core should also be considered at risk.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
Naz says: work is now commencing
Naz says: Work has been completed successfully.
We are currently investigating reports of packet loss affecting broadband customers.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
david labouchardiere says: We are continuing to investigate reports of packet loss, however we would request any affected customers to provide the service desk with supporting evidence in the form of pathpings or winMTR data to allow further troubleshooting
david labouchardiere says: Investigations into the packet loss reports are ongoing. We continue to ask that any affected circuits provide us with supporting evidence using winMTR or pathpings to assist in troubleshooting
david labouchardiere says: Investigations are ongoing into the reports of packet loss. We are in talks with the supplier requesting they troubleshoot on their site based off the evidence supplied so far. We will update once we have further information from them.
david labouchardiere says: The supplier is currently investigating the reported issue and has raised the case to their networking team for further assistance. We will continue to push for further updates. At present, no further evidence is required from affected customers.
david labouchardiere says: The supplier is continuing to investigate the issue on their side and further updates are expected in the next hour.
peter butler says: Investigations with our supplier are continuing but, unfortunately, we do not have any solid updates or timescales for resolution at this point.
We shall continue to chase this over the course of the evening and post updates when they become available.
Jonathan Clarke says: Investigations into the packet loss reports are ongoing, BT advise they are chasing their 2nd line team for updates. We are continuing to chase for updates, as soon as we anything further, we will update you again.
Jonathan Clarke says: BT advise that packet loss was observed when testing. BT advise that the there is a potential issue on their core network. Further troubleshooting will be required, however, the issue is only present within core hours.
As of this, BT are going to be continuing testing throughout the day. BT’s 2nd line team are engaging with their 3rd line team for further assistance today.
We will update once we have further information from them.
Peter Butler says: BT advise that as the packet loss does not appear to occur overnight this has been passed to their day shift to investigate today. Intermittent loss has been seen again this morning so they are continuing to trace the root cause of the issue.
Our escalation point is aiming to get an update to us by 1200 today, so we will aim to update via this post shortly thereafter.
Peter Butler says: BT have advised that the intermittent packet loss that was seen at 0800 this morning has not been present since, which has hindered progress with their fault finding.
Investigations continue and BT expect to be able to provide a further update by around 1600 today.
Adam Wilson says: We have provided further evidence of packet loss to BT, and this is currently being reviewed. Further updates will be provided as soon as they become available.
Jonathan Clarke says: BT have advised that their 3rd line team are not observing any packet loss whilst testing. We have been advised that they will continue to troubleshoot out of hours, however, mentioned that they believe that the packet loss will not present itself until Monday morning in core hours.
Our escalation point within BT has stressed to them that if this can be resolved before Monday and ensure us all efforts to resolve this before then are being made.
A Further update will be provided as they become available.
Jonathan Clarke says: BT advise that 2nd line and 3rd line team have been monitoring the BTW network for signs of packet loss. In the last 48 hours no packet loss has been observed. As of this, BT have been unable to replicate the issue out of core hours.
BT’s 3rd line team are monitoring the network this morning for when the packet loss reappears, to ascertain where in the network the issue lies to resolve.
A Further update will be provided following BT’s troubleshooting.
david labouchardiere says: BT advise they have seen no further packet loss today and our monitoring confirms this. BT are continuing to investigate and have setup additional monitors to assist this. We will provide a further update once we have any more information.
Peter Butler says: We have been receiving further reports of packet loss coming into the desk as of around 1430 today. We have alerted BT to this and they have confirmed that they are looking into this.
We shall provide further updates once they are received.
david labouchardiere says: We are continuing to push BT for further updates on their investigation into this issue and provided additional details to them to assist in their testing. We will provide a further update once we have any more information.
Peter Butler says: We are continuing to provide examples of circuits affected by this issue along with evidence to BT to try and push this to resolution, we are also engaging escalation points within BT to ensure this is treated at the highest priority on their side.
We shall continue to chase and provide updates when available.
Peter Butler says: Investigations are still ongoing with BT with our NOC team also assisting and investigating the traffic as it enters our network to ensure we are looking at the whole picture.
Unfortunately as the loss seems to occur predominantly during core hours it is likely to be the morning before any further update can be provided, so a further post will be made tomorrow morning or upon the receipt of further information.
Peter Butler says: Unfortunately no issues could be found overnight whilst the issue was not manifesting itself so this has now been picked back up by the day shifts on both BTs and our side.
We have multiple circuits being monitored which allows us to see when the issue is present so will update further when more information is available.
No further evidence of the issue is required from customers at this point.
Peter Butler says: An emergency change is currently being planned in order to assist with troubleshooting the issues curently being seen. This is being discussed at present so that it can be decided when best to carry this out.
A further update will be posted at 1500 or upon receipt of further information.
david labouchardiere says: We are seeing no further signs of loss following the emergency work carried out yesterday and believe this issue to now be resolved. Any remaining customers that are affected are advised to reboot their router before calling into the Service Desk. We apologise for any inconveniences caused by this outage
Thursday 8th December 2020, 10:00 – 17:30
Due to on going improvements to the power infrastructure within the BT POP’s, we are preforming exploratory work before refreshing the ATS and UPS for this site.
BT will be undertaking maintenance work within the above window which will require the tracing of some cables on site. This will involve the removal of a panel on a DC inverter to visually inspect the connectivity within.
We will not be attempting anything other than a lid removal and visual inspection and the risk to impact during this window is considered low.
We are currently aware of an incident affecting leased line connections in the Portsmouth area. This is being looked into by our NOC team and engineers are on site to try and resolve this as soon as possible.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
Lucy Jordan says: This is still being investigated. However, we expect a Nokia engineer on site at approx 18:45 this evening and a replacement card will hopefully help to resolve the issue. Further work is ongoing with the current engineer on site, to try and restore links in the meantime.
jbiddulph says: We are currently still awaiting a part to be delivered so we can restore complete service. This was unfortunately delayed and is due to arrive for 9:30PM. Since the previous engineer has had to be relieved due to working on this issue since 8AM, we are also dispatching a new engineer to help fix the fault.
In the meantime, we are continuously working on trying to restore this remotely. A further update will follow promptly when available.
jbiddulph says: All Services should now be restored. We are seeing the core device that was offline for this to now be online.
A full RFO will be provided at a later time to advise of the fault cause, and fix. If you have any further issues, we advise to fully re-boot all services.
We are currently aware of an incident affecting Virgin Media leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
Jonathan Clarke says: The supplier is currently investigating the issue. They advised that there appears to be an issue at Telehouse exchange. We are now awaiting a further update.
As soon as we have anything further, we will update you again.
mpurcell says: In relation to the loss of service you are currently experiencing. Our supplier is still working hard to resolve these issues as a priority.
They have engineers on site continuing investigations into the loss and intermittent connections that are being experienced.
We will update you again in one hour
mpurcell says: In relation to the loss of service you are currently experiencing: our supplier is still working hard to resolve these issues as a priority. Their engineers are on site continuing their investigations in to the intermittent loss of service and packet loss that you may be experiencing. We have chased for further details but unfortunately at present there is no further update.
We will update you again in one hour. Thank you for your continued patience.
mpurcell says: Report received from our supplier that they have reconnected a patch in their head end and are seeing our service come back online.
If you are still being impacted please contact us as soon as possible and confirm site checks.
Thursday 10th December 2020, 21:00 – Friday 11th December 2020 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above TE requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
We have just observed a temporary loss of service on our Interxion.edge device which has interrupted internet breakout for some PWAN customers.
Services are now restored and engineers are investigating the root cause. We apologise for any inconvenience this has caused.
Jordan Gardner says: We have received further reports of interruption to PWAN services which we are currently investigating. We will provide a further update when available.
Jordan Gardner says: Initial information received earlier this afternoon indicated an on-going issue with Interxion.edge, however, further investigations showed the issue to be specific to some PWAN DSL services. We have taken measures to mitigate the root cause of this interruption and we will continue to closely monitor.
We have had a confirmation service should now be restored. Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We apologise for any inconvenience caused by this incident.