We are currently investigating reports of packet loss affecting broadband customers.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
Archive for the ‘Uncategorized’ Category
Incident: DSL packet loss
Posted: Thursday, December 10th, 2020 at 15:24 by david labouchardiere
Portsmouth outage – 28/11/2020
Posted: Saturday, November 28th, 2020 at 16:17 by Adam Heath
We are currently aware of an incident affecting leased line connections in the Portsmouth area. This is being looked into by our NOC team and engineers are on site to try and resolve this as soon as possible.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
-
Lucy Jordan says:
This is still being investigated. However, we expect a Nokia engineer on site at approx 18:45 this evening and a replacement card will hopefully help to resolve the issue. Further work is ongoing with the current engineer on site, to try and restore links in the meantime.
-
jbiddulph says:
We are currently still awaiting a part to be delivered so we can restore complete service. This was unfortunately delayed and is due to arrive for 9:30PM. Since the previous engineer has had to be relieved due to working on this issue since 8AM, we are also dispatching a new engineer to help fix the fault.
In the meantime, we are continuously working on trying to restore this remotely. A further update will follow promptly when available.
-
All Services should now be restored. We are seeing the core device that was offline for this to now be online.
A full RFO will be provided at a later time to advise of the fault cause, and fix. If you have any further issues, we advise to fully re-boot all services.
Incident: Virgin Media leased lines
Posted: Thursday, November 26th, 2020 at 07:12 by Jonathan Clarke
We are currently aware of an incident affecting Virgin Media leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
-
Jonathan Clarke says:
The supplier is currently investigating the issue. They advised that there appears to be an issue at Telehouse exchange. We are now awaiting a further update.
As soon as we have anything further, we will update you again.
-
mpurcell says:
In relation to the loss of service you are currently experiencing. Our supplier is still working hard to resolve these issues as a priority.
They have engineers on site continuing investigations into the loss and intermittent connections that are being experienced.
We will update you again in one hour
-
In relation to the loss of service you are currently experiencing: our supplier is still working hard to resolve these issues as a priority. Their engineers are on site continuing their investigations in to the intermittent loss of service and packet loss that you may be experiencing. We have chased for further details but unfortunately at present there is no further update.
We will update you again in one hour. Thank you for your continued patience.
-
Report received from our supplier that they have reconnected a patch in their head end and are seeing our service come back online.
If you are still being impacted please contact us as soon as possible and confirm site checks.
Incident: Vodafone leased lines
Posted: Wednesday, November 11th, 2020 at 11:52 by david labouchardiere
We are currently aware of an incident affecting Vodafone leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
-
david labouchardiere says:
The supplier is currently investigating and confirms their hardware at the affected site is currently unreachable. We will provide further updates in the next hour.
-
The supplier has arranged engineer resources to attend the affected site. Current ETA for the engineer is 15:00
-
Engineers have arrived on site and are accessing the data centre. Further updates will be provided in the next hour.
-
Apologies for the delays being experienced here. Our suppliers have experienced some difficulty acquiring replacement hardware but are now en route back to site with the new hardware and have an ETA of 1700. We will provide a further update in one hour.
-
Matthew Davies says:
Apologies for the further delays. The suppliers engineer is on site and currently working on the issue, with the additional spare equipment required.
We’ll have a further update for you within the hour.
-
Our supplier has rebooted the device that is at cause for the outage, which will have restored most services.
The device is being replaced however, so a further outage is expected. We’ll let you know as work progresses.
-
Our supplier is onsite currently working on loading new config onto the replacement device. We’re expecting the device to be replaced soon.
We’ll provide further updates as they come.
-
Our supplier has advised that the spare equipment at site that they were installing appears to be faulty, therefore, suppliers are ordering new hardware to configure and reinstall. We have been advised and observed ourselves that the services that were affected are up and stable post reboot.
Due to the above, another Field Engineer will be scheduled for the new equipment install. VF has advised they aim to have this swap completed ASAP. VF’s NOC team is in contact with vendors to give us a timeline of the work.
We’ll have a further update for you as soon as possible and will update you again.
-
Our suppliers are still working on timescales for the hardware replacement, we are keeping in regular contact and as soon as we have any further information, we will update you again.
-
Suppliers advise engineer has been dispatched back to site to replace hardware. ETA is 23:15 GMT
We’ll provide further updates as we receive them.
-
Suppliers advise that the hardware was due to arrive at 23:30 GMT, so as of this, should have now arrived. We have been advised that the VF field engineer and VF NOC will configure and install.
We’ll have a further update for you within the hour.
-
Supplier engineer and their NOC team continue to work on the issue with replacement part being configured.
We expect a further update within the hour.
-
Suppliers advise Vodafone 2nd line is coordinating with field engineers to replace the Hardware now, we expect to see the affected circuits drop whilst this replacement takes place.
We expect a further update within the hour.
-
Suppliers advise that the hardware swap will occur now, we expect circuits to remain down for 45 minutes to an hour.
We will continue to monitor the network and advise once this change has been completed.
-
We observed circuits connected to suppliers hub disconnect around 03:07. We are seeing the circuits reconnect now.
We will await a response from suppliers and advise once the circuits are no longer considered to be at risk
-
Suppliers has advised that the swap of the affected hardware has been completed. However, whilst trying to upgrade the IOS of the new hardware, this has so far been unsuccessful.
VF’s 2nd and 3rd line team are coordinating with each other to have this resolved, therefore, further interruptions to the service may be observed. As of this, circuits are still at risk.
As soon as we have a further update, we will let you know.
-
Suppliers advise that the IOS firmware upgrade has been successful and was has confirmed by their 3rd line team. Suppliers have informed us that services are up and observe good traffic over the network. I can confirm the same from our monitoring.
VF has now stood their team down with this Major Incident. VF’s service desk team will send us the RFO once completed.
Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We apologies to you and your customers for the inconvenience caused.
DSL Packet Loss
Posted: Friday, November 6th, 2020 at 10:55 by jbiddulph
We are aware of a wide-spread packet loss issue following an outage instance that occurred last night. This is being thoroughly investigated for a resolution for as soon as possible.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
-
Investigating this further we have confirmed the issue is nationwide.
Unfortunately, we have no further specific details on this, however, at this moment in time we are continuously working towards the incident resolution.
Further updates will follow when available.
-
Further to our last update.
Due to the unplanned outage last night we are experiencing issues. As a result all traffic has been routed through an alternative BGP route and is subsequently causing congestion on this secondary route. We are currently in conversation with a supplier to establish whether or not we can bring this primary route back online. Despite this we are seeing that services are restored, but with issues.
Further updates will be made available as soon as we have them.
-
At this moment in time, contact with the supplier is still currently underway to establish whether we can bring the primary route back online.
We treating this with the highest priority that we can to restore complete service to all services affected, and apologise for the inconvenience caused so far.
Further updates will follow as soon as they are available.
-
Jamie Smith says:
Emergency Work
14:00 – 16:00 6/11/2020
Due to the ongoing issues being observed on the DSL platform causing heavy packet loss, NOC engineers will soon be carrying out remedial work.
All broadband users should be considered at risk for the duration of this work.
We apologize for any inconvenience and further updates to be provided.
-
This work has now been completed. We will continue to monitor the DSL platform.
Any users still experiencing residual issues are advised to reboot the equipment prior to contacting the support desk.
Once again we apologize for any inconvenience caused by the issue.
DSL Connectivity
Posted: Thursday, November 5th, 2020 at 23:35 by Ryan Berrisford
We are currently aware of an issue affecting DSL connectivity. Initial indications suggest services have synchronisation but no PPP authentication. Technical escalation points have been engaged and an investigation is underway.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
-
Ryan Berrisford says:
We can confirm that this incident is in relation to a planned change within the network that has had an unexpected impact on DSL customers. Engineers remain engaged and updates will be provided until incident resolution has been reached.
-
Service has been now been restored to the vast majority of customers and the major incident is declared as resolved. Any customers that still have DSL connectivity issues are advised to power-cycle their router before contacting the Technical Service Centre.
We apologise for any inconvience caused by this incident.
Incident: Vodafone leased lines
Posted: Tuesday, November 3rd, 2020 at 10:24 by david labouchardiere
We are currently aware of an incident affecting Vodafone leased lines. We are engaging with the supplier to investigate and further updates will be provided ASAP
-
The supplier is currently investigating the issue but confirm they are seeing their hardware at the affected hub as down. We will provide a further update in the next hour.
-
The supplier is still investigating this issue, they are now in the process of raising a field engineer to troubleshoot the affected hardware. We will provide a further update in the next hour
-
Engineer resources are en route to the affected POP with an ETA of 14:00
-
Engineer is now on site and investigations have begun
-
Supplier engineer is continuing to work on the issue with replacement parts expected to arrive on site at 15:30
-
We are seeing services coming back up following a reload of the affected hardware on site. Please ask any affected customers to power cycle their hardware on site if they continue to experience issues.
Incident – SKY Leased Line Outage – 30-10-2020
Posted: Friday, October 30th, 2020 at 01:42 by jbiddulph
Good Morning.
We have had a large number of SKY circuits go offline for a period of 30 minutes early this morning. This has been due to unplanned works performed by SKY, but as of 00:55AM. All services should be restored.
We apologise in advance for the inconvenience caused. Given this happened yesterday and again this morning, we are taking further measures to ensure with the supplier there are no further unexpected works.
If you are experiencing any further issues since this morning’s works, please re-boot your service and proceed to call into the support desk if the issue remains.
-
The above incident is part of a 5 day planned works, starting from 28/10/2020. The details of this disruption are below:-
Change Summary: Wholesale Ethernet Planned Works Full Outage multiple circuits
Type of Change: Planned
Impact: Full outage to service
Start Date: 26/10/2020 00:00:00 GMT
End Date: 30/10/2020 06:00:00 GMTReason for work: Maintenance-Third party work
Apologies for any disruption this may case in the meantime.
Incident – SKY Leased lined Outage
Posted: Thursday, October 29th, 2020 at 02:45 by jbiddulph
Good Morning.
We are currently seeing a large number of SKY leased line circuits offline. At this moment in time, this is under investigation and we are currently in correspondence with suppliers for diagnostics.
Further updates will be provided when available.
-
Good Morning.
The majority of the circuits appear to now be restored.
Investigations are currently still underway to identify the cause of this outage.
Further updates will follow when available.
-
Good Morning.
We have had confirmation from the supplier that this was part of a maintenance change on the SKY side of an ENNI. All services on this ENNI were unexpectedly taken offline as a result.
We are now pushing for the specific details and RFO for the unplanned outage.
Further updates will follow when available.
-
Good Morning.
The supplier has confirmed the outage that took place was due to the movement of circuits on their side to a Device Card. This was to enable further services for their hardware, but unfortunately due to the lack of awareness made to ourselves, this was an unexpected outage.
We have full confirmation that all services involved are now back online and restored.
If you have any services that you believe to be related to this, please fully re-boot the service and re-test the service. If you are having continued issues after this, please contact the Technical Support Department.
Incident: Virgin Media Leased Lines
Posted: Thursday, October 15th, 2020 at 18:05 by Jonathan Clarke
We are currently seeing a number of Virgin Media leased line circuits as down. These circuits appear to be in the Birmingham area at present. We are currently in correspondence with suppliers for diagnostics.
Further updates will be provided when available.
-
Suppliers has advised that this is already a known issue. Engineers are currently on route to site and their current ETA is 18:15, for further investigation.
Virgin media are proactively updating us again at 18:45.
Further updates will be provided when available.
-
Suppliers has advised that the engineer is now on site.
ETA is 19:30
Further updates will be provided when available.
-
Suppliers are now seeing services restored. Their engineer is still on site working on this issue.
When we have confirmation from them that that the issue is fully resolved we will update you again.
Services should still be considered at risk until we have had confirmation.
-
Our suppliers have now advised that work has now been completed, and these circuits should no longer be at risk.
They advised the below work has been carried out:
A high level transmission card was reseated which restored services.
If you have a service that is still down, please reboot the equipment on site and contact the support department after this has been done.
We are continuing to investigate reports of packet loss, however we would request any affected customers to provide the service desk with supporting evidence in the form of pathpings or winMTR data to allow further troubleshooting
Investigations into the packet loss reports are ongoing. We continue to ask that any affected circuits provide us with supporting evidence using winMTR or pathpings to assist in troubleshooting
Investigations are ongoing into the reports of packet loss. We are in talks with the supplier requesting they troubleshoot on their site based off the evidence supplied so far. We will update once we have further information from them.
The supplier is currently investigating the reported issue and has raised the case to their networking team for further assistance. We will continue to push for further updates. At present, no further evidence is required from affected customers.
The supplier is continuing to investigate the issue on their side and further updates are expected in the next hour.
Investigations with our supplier are continuing but, unfortunately, we do not have any solid updates or timescales for resolution at this point.
We shall continue to chase this over the course of the evening and post updates when they become available.
Investigations into the packet loss reports are ongoing, BT advise they are chasing their 2nd line team for updates. We are continuing to chase for updates, as soon as we anything further, we will update you again.
BT advise that packet loss was observed when testing. BT advise that the there is a potential issue on their core network. Further troubleshooting will be required, however, the issue is only present within core hours.
As of this, BT are going to be continuing testing throughout the day. BT’s 2nd line team are engaging with their 3rd line team for further assistance today.
We will update once we have further information from them.
BT advise that as the packet loss does not appear to occur overnight this has been passed to their day shift to investigate today. Intermittent loss has been seen again this morning so they are continuing to trace the root cause of the issue.
Our escalation point is aiming to get an update to us by 1200 today, so we will aim to update via this post shortly thereafter.
BT have advised that the intermittent packet loss that was seen at 0800 this morning has not been present since, which has hindered progress with their fault finding.
Investigations continue and BT expect to be able to provide a further update by around 1600 today.
We have provided further evidence of packet loss to BT, and this is currently being reviewed. Further updates will be provided as soon as they become available.
BT have advised that their 3rd line team are not observing any packet loss whilst testing. We have been advised that they will continue to troubleshoot out of hours, however, mentioned that they believe that the packet loss will not present itself until Monday morning in core hours.
Our escalation point within BT has stressed to them that if this can be resolved before Monday and ensure us all efforts to resolve this before then are being made.
A Further update will be provided as they become available.
BT advise that 2nd line and 3rd line team have been monitoring the BTW network for signs of packet loss. In the last 48 hours no packet loss has been observed. As of this, BT have been unable to replicate the issue out of core hours.
BT’s 3rd line team are monitoring the network this morning for when the packet loss reappears, to ascertain where in the network the issue lies to resolve.
A Further update will be provided following BT’s troubleshooting.
BT advise they have seen no further packet loss today and our monitoring confirms this. BT are continuing to investigate and have setup additional monitors to assist this. We will provide a further update once we have any more information.
We have been receiving further reports of packet loss coming into the desk as of around 1430 today. We have alerted BT to this and they have confirmed that they are looking into this.
We shall provide further updates once they are received.
We are continuing to push BT for further updates on their investigation into this issue and provided additional details to them to assist in their testing. We will provide a further update once we have any more information.
We are continuing to provide examples of circuits affected by this issue along with evidence to BT to try and push this to resolution, we are also engaging escalation points within BT to ensure this is treated at the highest priority on their side.
We shall continue to chase and provide updates when available.
Investigations are still ongoing with BT with our NOC team also assisting and investigating the traffic as it enters our network to ensure we are looking at the whole picture.
Unfortunately as the loss seems to occur predominantly during core hours it is likely to be the morning before any further update can be provided, so a further post will be made tomorrow morning or upon the receipt of further information.
Unfortunately no issues could be found overnight whilst the issue was not manifesting itself so this has now been picked back up by the day shifts on both BTs and our side.
We have multiple circuits being monitored which allows us to see when the issue is present so will update further when more information is available.
No further evidence of the issue is required from customers at this point.
An emergency change is currently being planned in order to assist with troubleshooting the issues curently being seen. This is being discussed at present so that it can be decided when best to carry this out.
A further update will be posted at 1500 or upon receipt of further information.
We are seeing no further signs of loss following the emergency work carried out yesterday and believe this issue to now be resolved. Any remaining customers that are affected are advised to reboot their router before calling into the Service Desk. We apologise for any inconveniences caused by this outage