Incident: DSL packet loss
Posted: Thursday, December 10th, 2020 at 15:24 by david labouchardiere
We are currently investigating reports of packet loss affecting broadband customers.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
We are continuing to investigate reports of packet loss, however we would request any affected customers to provide the service desk with supporting evidence in the form of pathpings or winMTR data to allow further troubleshooting
Investigations into the packet loss reports are ongoing. We continue to ask that any affected circuits provide us with supporting evidence using winMTR or pathpings to assist in troubleshooting
Investigations are ongoing into the reports of packet loss. We are in talks with the supplier requesting they troubleshoot on their site based off the evidence supplied so far. We will update once we have further information from them.
The supplier is currently investigating the reported issue and has raised the case to their networking team for further assistance. We will continue to push for further updates. At present, no further evidence is required from affected customers.
The supplier is continuing to investigate the issue on their side and further updates are expected in the next hour.
Investigations with our supplier are continuing but, unfortunately, we do not have any solid updates or timescales for resolution at this point.
We shall continue to chase this over the course of the evening and post updates when they become available.
Investigations into the packet loss reports are ongoing, BT advise they are chasing their 2nd line team for updates. We are continuing to chase for updates, as soon as we anything further, we will update you again.
BT advise that packet loss was observed when testing. BT advise that the there is a potential issue on their core network. Further troubleshooting will be required, however, the issue is only present within core hours.
As of this, BT are going to be continuing testing throughout the day. BT’s 2nd line team are engaging with their 3rd line team for further assistance today.
We will update once we have further information from them.
BT advise that as the packet loss does not appear to occur overnight this has been passed to their day shift to investigate today. Intermittent loss has been seen again this morning so they are continuing to trace the root cause of the issue.
Our escalation point is aiming to get an update to us by 1200 today, so we will aim to update via this post shortly thereafter.
BT have advised that the intermittent packet loss that was seen at 0800 this morning has not been present since, which has hindered progress with their fault finding.
Investigations continue and BT expect to be able to provide a further update by around 1600 today.
We have provided further evidence of packet loss to BT, and this is currently being reviewed. Further updates will be provided as soon as they become available.
BT have advised that their 3rd line team are not observing any packet loss whilst testing. We have been advised that they will continue to troubleshoot out of hours, however, mentioned that they believe that the packet loss will not present itself until Monday morning in core hours.
Our escalation point within BT has stressed to them that if this can be resolved before Monday and ensure us all efforts to resolve this before then are being made.
A Further update will be provided as they become available.
BT advise that 2nd line and 3rd line team have been monitoring the BTW network for signs of packet loss. In the last 48 hours no packet loss has been observed. As of this, BT have been unable to replicate the issue out of core hours.
BT’s 3rd line team are monitoring the network this morning for when the packet loss reappears, to ascertain where in the network the issue lies to resolve.
A Further update will be provided following BT’s troubleshooting.
BT advise they have seen no further packet loss today and our monitoring confirms this. BT are continuing to investigate and have setup additional monitors to assist this. We will provide a further update once we have any more information.
We have been receiving further reports of packet loss coming into the desk as of around 1430 today. We have alerted BT to this and they have confirmed that they are looking into this.
We shall provide further updates once they are received.
We are continuing to push BT for further updates on their investigation into this issue and provided additional details to them to assist in their testing. We will provide a further update once we have any more information.
We are continuing to provide examples of circuits affected by this issue along with evidence to BT to try and push this to resolution, we are also engaging escalation points within BT to ensure this is treated at the highest priority on their side.
We shall continue to chase and provide updates when available.
Investigations are still ongoing with BT with our NOC team also assisting and investigating the traffic as it enters our network to ensure we are looking at the whole picture.
Unfortunately as the loss seems to occur predominantly during core hours it is likely to be the morning before any further update can be provided, so a further post will be made tomorrow morning or upon the receipt of further information.
Unfortunately no issues could be found overnight whilst the issue was not manifesting itself so this has now been picked back up by the day shifts on both BTs and our side.
We have multiple circuits being monitored which allows us to see when the issue is present so will update further when more information is available.
No further evidence of the issue is required from customers at this point.
An emergency change is currently being planned in order to assist with troubleshooting the issues curently being seen. This is being discussed at present so that it can be decided when best to carry this out.
A further update will be posted at 1500 or upon receipt of further information.
We are seeing no further signs of loss following the emergency work carried out yesterday and believe this issue to now be resolved. Any remaining customers that are affected are advised to reboot their router before calling into the Service Desk. We apologise for any inconveniences caused by this outage