Telehouse Circuit Migration

Posted: July 28th, 2020 at 15:45 by Iain Beveridge

From Friday 31st July 2020 19:00 – 02:00, Nightly until Wednesday 12th August 2020.

Due to the Telehouse Metro data centre closing, there is an ongoing project to migrate all Vodafone provided services to a new handoff location at Equinix Slough.

A maintenance window is scheduled nightly from 19:00 until 02:00 the following morning, customers will experience a single 30 minute outage whilst their individual circuit’s configuration is migrated and tested.

If you require any additional information, please contact the Service Desk at Service.desk@cityfibre.com

Incident: Vodafone leased lines

Posted: July 24th, 2020 at 16:19 by david labouchardiere

We are currently seeing a number of Vodafone leased line circuits as down. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Jonathan Clarke says:

    Suppliers have advised that diagnostics show their management kit unreachable due to which services are impacted. Vodafone have proactively escalated to Level 1 to expedite the fault and restoration of the services.

    Further updates will be provided when available.

  2. Jonathan Clarke says:

    Suppliers have advised that a field engineer has been dispatched to investigate, the ETA for the engineer is 18:30.

    Further updates will be provided when available.

  3. Jonathan Clarke says:

    Suppliers have advised that the field engineer recently dispatched to investigate has been delayed and therefore, the ETA is now 19:30

    We apologise for the delay and we will give further updates when provided.

  4. Jonathan Clarke says:

    Suppliers have advised that the field engineer has arrived on site to complete investigations.

    Further updates will be provided when available.

  5. Jonathan Clarke says:

    Suppliers advise investigations are still ongoing and we are awaiting further updates.

    Thank you for you patience.

  6. Jonathan Clarke says:

    Suppliers advise that the field engineer has been investigating with a 2nd Line Engineer. We have been advised that their equipment is unable to pass traffic and they suspect a line card issue. Suppliers transmission team has been engaged and a resolution is being attempted.

  7. Jonathan Clarke says:

    Suppliers advise that the transmission team has ordered a spare line card for replacement. ETA for the spare card to arrive at the site is 60 minutes. We are proactively monitoring for further updates. When a further update is obtained, we will update you again.

    We appreciate your patience regarding the incident.

  8. Jonathan Clarke says:

    Engineers are continuing to work on this incident, we are awaiting a further update. Once obtained, we will update you again.

  9. Jonathan Clarke says:

    Suppliers have replaced the line card and confirmed that the issue is resolved. We are now seeing our managed routers back online.

    If you have any further issues, please reboot the equipment and if after, the issue persists please call us.

    We appreciate your patience and apologise for any inconvenience caused.

Telehouse Metro Circuit Migration

Posted: July 17th, 2020 at 11:01 by Iain Beveridge

From Monday 20th July 2020 19:00 – 02:00, Nightly until Friday 30th July 2020.

Due to the Telehouse Metro data centre closing, there is an ongoing project to migrate all Vodafone provided services to a new handoff location at Equinix Slough.

A maintenance window is scheduled nightly from 19:00 until 02:00 the following morning, customers will experience a single 30 minute outage whilst their individual circuit’s configuration is migrated and tested.

If you require any additional information, please contact the Service Desk at Service.desk@cityfibre.com

At risk: bristol.core / bri165-rtr-1

Posted: July 15th, 2020 at 15:51 by Robin Posadas

Thursday 16th July 2020, 22:00 – 23:59

After a power check within Bristol POP, BT identified an issue with part of the power infrastructure within one of our racks.

Therefore, BT will be undertaking emergency maintenance work in the above window which may require the power feeds to be moved, whilst a UPS is swapped within the rack. The core network router is not expected to experience any loss of service.

  1. Naz says:

    work is now commencing

  2. Naz says:

    work now completed

Planned Work: Birmingham TE.

Posted: July 14th, 2020 at 16:31 by Iain Beveridge

Wednesday 29th July 2020, 21:00 – Thursday 30th July 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Birmingham TE requires both the ATS and UPS to be refreshed.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

  1. All work was completed without incident.

At risk: bristol.core / bri165-rtr-1

Posted: July 3rd, 2020 at 14:43 by Adam Wilson

Tuesday 7th July 2020, 22:00 – 23:59

After a power check within Bristol POP, BT identified an issue with part of the power infrastructure within one of our racks.

Therefore, BT will be undertaking emergency maintenance work in the above window which may require the power feeds to be moved, whilst a UPS is swapped within the rack. The core network router is not expected to experience any loss of service.

  1. This work was postponed due to third party resourcing issues, revised date will be republished shortly.

Incident: Bristol POP

Posted: June 25th, 2020 at 03:56 by Robin Posadas

We have experienced a loss of connectivity to our equipment in our Bristol POP. Between 02:50 – 03:00. Currently all services have been restored. Engineers are investigating. Further updates will be provided as and when they become available.

  1. Alistair Patterson says:

    This incident was due to a power issue within the pop. This was raised to our supplier at the time who are continuing to investigate the root cause. A further update will be provided when available.

Emergency Work: interxion2.edge switch stack reboot

Posted: June 23rd, 2020 at 17:48 by Richard Blunt

Tuesday 23 June 23:00 – 23:15

Memory fragmentation has been detected in one of the Interxion stack switches, This is causing intermittent connectivity to some customer firewalls.

The device will have an emergency reboot at 23:00 to clear the issue. This will cause a brief outage with some none HA devices.

 

An update will be posted on completion of the work.

  1. Richard Blunt says:

    Scheduled reboot completed successfully.

At Risk: goswell.core

Posted: June 18th, 2020 at 11:51 by Iain Beveridge

Friday 19th June 2020  00:00 – 02:00

During the above window our colocation provider will be undertaking work to one of the power feeds that supplies our suite. During this window the core will be running on one power feed, no outage is anticipated due to work undertaken in an earlier window to provide better power resilience but it should be considered at risk.

At Risk: goswell.core

Posted: June 18th, 2020 at 11:41 by Iain Beveridge

Thursday 18th June 2020  12:00 – 14:00

During the above window our colocation provider will be swapping one of the power feeds to the suite to a new source. This is to prevent a future total outage while work to increase reliability of the power to the suite is undertaken. During this window the core will be running on one power feed no outage is anticipated but it should be considered at risk.