We are currently aware of an incident affecting Virgin Media leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
Thursday 10th December 2020, 21:00 – Friday 11th December 2020 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above TE requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
We have just observed a temporary loss of service on our Interxion.edge device which has interrupted internet breakout for some PWAN customers.
Services are now restored and engineers are investigating the root cause. We apologise for any inconvenience this has caused.
Jordan Gardner says: We have received further reports of interruption to PWAN services which we are currently investigating. We will provide a further update when available.
Jordan Gardner says: Initial information received earlier this afternoon indicated an on-going issue with Interxion.edge, however, further investigations showed the issue to be specific to some PWAN DSL services. We have taken measures to mitigate the root cause of this interruption and we will continue to closely monitor.
Thursday 3rd December 2020, 21:00 – Friday 4th December 2020 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above TE requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
Thursday 26th November 2020, 21:00 – Friday 27th November 2020 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above TE requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
Thursday 19th November 2020, 21:00 – Friday 20th November 2020 05:00
Due to on going improvements to the power infrastructure within the BT POP’s, The above TE requires both the ATS and UPS to be refreshed.
Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment may experience a small period of downtime.
We are currently aware of an incident affecting Vodafone leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
david labouchardiere says: The supplier is currently investigating and confirms their hardware at the affected site is currently unreachable. We will provide further updates in the next hour.
david labouchardiere says: The supplier has arranged engineer resources to attend the affected site. Current ETA for the engineer is 15:00
david labouchardiere says: Engineers have arrived on site and are accessing the data centre. Further updates will be provided in the next hour.
mpurcell says: Apologies for the delays being experienced here. Our suppliers have experienced some difficulty acquiring replacement hardware but are now en route back to site with the new hardware and have an ETA of 1700. We will provide a further update in one hour.
Matthew Davies says: Apologies for the further delays. The suppliers engineer is on site and currently working on the issue, with the additional spare equipment required.
We’ll have a further update for you within the hour.
Matthew Davies says: Our supplier has rebooted the device that is at cause for the outage, which will have restored most services.
The device is being replaced however, so a further outage is expected. We’ll let you know as work progresses.
Matthew Davies says: Our supplier is onsite currently working on loading new config onto the replacement device. We’re expecting the device to be replaced soon.
We’ll provide further updates as they come.
Jonathan Clarke says: Our supplier has advised that the spare equipment at site that they were installing appears to be faulty, therefore, suppliers are ordering new hardware to configure and reinstall. We have been advised and observed ourselves that the services that were affected are up and stable post reboot.
Due to the above, another Field Engineer will be scheduled for the new equipment install. VF has advised they aim to have this swap completed ASAP. VF’s NOC team is in contact with vendors to give us a timeline of the work.
We’ll have a further update for you as soon as possible and will update you again.
Jonathan Clarke says: Our suppliers are still working on timescales for the hardware replacement, we are keeping in regular contact and as soon as we have any further information, we will update you again.
Jonathan Clarke says: Suppliers advise engineer has been dispatched back to site to replace hardware. ETA is 23:15 GMT
We’ll provide further updates as we receive them.
Jonathan Clarke says: Suppliers advise that the hardware was due to arrive at 23:30 GMT, so as of this, should have now arrived. We have been advised that the VF field engineer and VF NOC will configure and install.
We’ll have a further update for you within the hour.
Jonathan Clarke says: Supplier engineer and their NOC team continue to work on the issue with replacement part being configured.
We expect a further update within the hour.
Jonathan Clarke says: Suppliers advise Vodafone 2nd line is coordinating with field engineers to replace the Hardware now, we expect to see the affected circuits drop whilst this replacement takes place.
We expect a further update within the hour.
Jonathan Clarke says: Suppliers advise that the hardware swap will occur now, we expect circuits to remain down for 45 minutes to an hour.
We will continue to monitor the network and advise once this change has been completed.
Jonathan Clarke says: We observed circuits connected to suppliers hub disconnect around 03:07. We are seeing the circuits reconnect now.
We will await a response from suppliers and advise once the circuits are no longer considered to be at risk
Jonathan Clarke says: Suppliers has advised that the swap of the affected hardware has been completed. However, whilst trying to upgrade the IOS of the new hardware, this has so far been unsuccessful.
VF’s 2nd and 3rd line team are coordinating with each other to have this resolved, therefore, further interruptions to the service may be observed. As of this, circuits are still at risk.
As soon as we have a further update, we will let you know.
Jonathan Clarke says: Suppliers advise that the IOS firmware upgrade has been successful and was has confirmed by their 3rd line team. Suppliers have informed us that services are up and observe good traffic over the network. I can confirm the same from our monitoring.
VF has now stood their team down with this Major Incident. VF’s service desk team will send us the RFO once completed.
Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We apologies to you and your customers for the inconvenience caused.
We are aware of a wide-spread packet loss issue following an outage instance that occurred last night. This is being thoroughly investigated for a resolution for as soon as possible.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
jbiddulph says: Investigating this further we have confirmed the issue is nationwide.
Unfortunately, we have no further specific details on this, however, at this moment in time we are continuously working towards the incident resolution.
Further updates will follow when available.
mpurcell says: Further to our last update.
Due to the unplanned outage last night we are experiencing issues. As a result all traffic has been routed through an alternative BGP route and is subsequently causing congestion on this secondary route. We are currently in conversation with a supplier to establish whether or not we can bring this primary route back online. Despite this we are seeing that services are restored, but with issues.
Further updates will be made available as soon as we have them.
jbiddulph says: At this moment in time, contact with the supplier is still currently underway to establish whether we can bring the primary route back online.
We treating this with the highest priority that we can to restore complete service to all services affected, and apologise for the inconvenience caused so far.
Further updates will follow as soon as they are available.
Jamie Smith says: Emergency Work
14:00 – 16:00 6/11/2020
Due to the ongoing issues being observed on the DSL platform causing heavy packet loss, NOC engineers will soon be carrying out remedial work.
All broadband users should be considered at risk for the duration of this work.
We apologize for any inconvenience and further updates to be provided.
Jamie Smith says: This work has now been completed. We will continue to monitor the DSL platform.
Any users still experiencing residual issues are advised to reboot the equipment prior to contacting the support desk.
Once again we apologize for any inconvenience caused by the issue.
We are currently aware of an issue affecting DSL connectivity. Initial indications suggest services have synchronisation but no PPP authentication. Technical escalation points have been engaged and an investigation is underway.
Further updates will follow as and when they become available and we apologise for any inconvenience caused.
Ryan Berrisford says: We can confirm that this incident is in relation to a planned change within the network that has had an unexpected impact on DSL customers. Engineers remain engaged and updates will be provided until incident resolution has been reached.
Ryan Berrisford says: Service has been now been restored to the vast majority of customers and the major incident is declared as resolved. Any customers that still have DSL connectivity issues are advised to power-cycle their router before contacting the Technical Service Centre.
We apologise for any inconvience caused by this incident.
Friday 6th of November 20:00 – Monday 9th November 06:00
During the above window as small number of circuits provided to us by Virgin Media are at risk while Virgin conduct work to resplice a section of fibre due to a location closure. No circuit will be down for the entirety of the window and any that are effected are estimated to only be down for 30 minutes to a maximum of 4 hours.
The supplier is currently investigating the issue. They advised that there appears to be an issue at Telehouse exchange. We are now awaiting a further update.
As soon as we have anything further, we will update you again.
In relation to the loss of service you are currently experiencing. Our supplier is still working hard to resolve these issues as a priority.
They have engineers on site continuing investigations into the loss and intermittent connections that are being experienced.
We will update you again in one hour
In relation to the loss of service you are currently experiencing: our supplier is still working hard to resolve these issues as a priority. Their engineers are on site continuing their investigations in to the intermittent loss of service and packet loss that you may be experiencing. We have chased for further details but unfortunately at present there is no further update.
We will update you again in one hour. Thank you for your continued patience.
Report received from our supplier that they have reconnected a patch in their head end and are seeing our service come back online.
If you are still being impacted please contact us as soon as possible and confirm site checks.