Incident: Vodafone leased lines
Posted: Wednesday, November 11th, 2020 at 11:52 by david labouchardiere
We are currently aware of an incident affecting Vodafone leased line circuits. We are engaging with the supplier to investigate this issue and further updates will be provided in the next hour.
The supplier is currently investigating and confirms their hardware at the affected site is currently unreachable. We will provide further updates in the next hour.
The supplier has arranged engineer resources to attend the affected site. Current ETA for the engineer is 15:00
Engineers have arrived on site and are accessing the data centre. Further updates will be provided in the next hour.
Apologies for the delays being experienced here. Our suppliers have experienced some difficulty acquiring replacement hardware but are now en route back to site with the new hardware and have an ETA of 1700. We will provide a further update in one hour.
Apologies for the further delays. The suppliers engineer is on site and currently working on the issue, with the additional spare equipment required.
We’ll have a further update for you within the hour.
Our supplier has rebooted the device that is at cause for the outage, which will have restored most services.
The device is being replaced however, so a further outage is expected. We’ll let you know as work progresses.
Our supplier is onsite currently working on loading new config onto the replacement device. We’re expecting the device to be replaced soon.
We’ll provide further updates as they come.
Our supplier has advised that the spare equipment at site that they were installing appears to be faulty, therefore, suppliers are ordering new hardware to configure and reinstall. We have been advised and observed ourselves that the services that were affected are up and stable post reboot.
Due to the above, another Field Engineer will be scheduled for the new equipment install. VF has advised they aim to have this swap completed ASAP. VF’s NOC team is in contact with vendors to give us a timeline of the work.
We’ll have a further update for you as soon as possible and will update you again.
Our suppliers are still working on timescales for the hardware replacement, we are keeping in regular contact and as soon as we have any further information, we will update you again.
Suppliers advise engineer has been dispatched back to site to replace hardware. ETA is 23:15 GMT
We’ll provide further updates as we receive them.
Suppliers advise that the hardware was due to arrive at 23:30 GMT, so as of this, should have now arrived. We have been advised that the VF field engineer and VF NOC will configure and install.
We’ll have a further update for you within the hour.
Supplier engineer and their NOC team continue to work on the issue with replacement part being configured.
We expect a further update within the hour.
Suppliers advise Vodafone 2nd line is coordinating with field engineers to replace the Hardware now, we expect to see the affected circuits drop whilst this replacement takes place.
We expect a further update within the hour.
Suppliers advise that the hardware swap will occur now, we expect circuits to remain down for 45 minutes to an hour.
We will continue to monitor the network and advise once this change has been completed.
We observed circuits connected to suppliers hub disconnect around 03:07. We are seeing the circuits reconnect now.
We will await a response from suppliers and advise once the circuits are no longer considered to be at risk
Suppliers has advised that the swap of the affected hardware has been completed. However, whilst trying to upgrade the IOS of the new hardware, this has so far been unsuccessful.
VF’s 2nd and 3rd line team are coordinating with each other to have this resolved, therefore, further interruptions to the service may be observed. As of this, circuits are still at risk.
As soon as we have a further update, we will let you know.
Suppliers advise that the IOS firmware upgrade has been successful and was has confirmed by their 3rd line team. Suppliers have informed us that services are up and observe good traffic over the network. I can confirm the same from our monitoring.
VF has now stood their team down with this Major Incident. VF’s service desk team will send us the RFO once completed.
Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.
We apologies to you and your customers for the inconvenience caused.