Archive for March, 2021

CityFibre Network Status page for Major Incidents

Posted: Wednesday, March 31st, 2021 at 08:00 by Iain Beveridge

All Major Incident notifications and updates will be posted within the unify portal here. https://support.cityfibre.com/networks

This will replace our current Entanet NOC page for Major Incidents ONLY. All Change Management activities, such as planned maintenance will continue to be updated on the NOC site.

Please ensure you visit the new site and sign up to the unify portal receive any communications related to Major Incidents.

Emergency Work: Peterborough TE

Posted: Thursday, March 25th, 2021 at 12:33 by Adam Heath

Friday 26th March 2021 00:01 – Friday 26th March 2021 03:00

Rescheduled from original window of Thursday 25th March 2021 00:01 – Thursday 25th March 2021 03:00

After recently replacing the above router on Tuesday 23rd February 2021 we have identified a need to reload the new equipment as we have an issue with an onboard 10gb port. The reload was recommended after raising a TAC case with hardware vendor support.

The work will take place during the above window and will effect all users connected to PB1244. We do not anticipate downtime for the entire window. We apologise for the inconvenience these planned works will cause and we have exhausted all other possibilities before taking this customer impacting action.

  1. Adam Wilson says:

    Work is about to commence.

  2. Adam Wilson says:

    This work is now complete.

Incident: CityFibre network services

Posted: Tuesday, March 23rd, 2021 at 14:25 by david labouchardiere

We are currently aware of an issue affecting CityFibre circuits.
Engineers are investigating and further updates will follow as soon as possible

  1. Ryan Berrisford says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net Ethernet Services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net Ethernet customers

    Description: A portion of on-net Ethernet customers are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre and Architecture and Engineering

    Update 1 –: Our Network Operations Centre are currently investigating a network incident affecting on-net Ethernet customers. Initial indications are that this is not related to a specific geographical location. Teams continue to investigate and further updates will follow at 15:25

  2. Ryan Berrisford says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net and off-net Ethernet Services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net and off-net Ethernet customers

    Description: A portion of on-net Ethernet customers are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre, Architecture and Engineering

    Update 2 –: The incident is related to an issue on a link between Leeds and Sheffield. Third party providers have already been engaged and are investigating. In the meantime, to mitigate the impact we have re-routed traffic over an alternative link. We are now in the process of assessing the impact of the re-routing. A further update will follow at 16:00

  3. Ryan Berrisford says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net and off-net Ethernet Services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net and off-net Ethernet customers

    Description: A portion of on-net and off-net Ethernet customers are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre, Architecture and Engineering

    Update 3 –: After the initial re-routing we are still seeing a portion of customers impacted by the major incident. Further work is being carried out in order to optimise network traffic and alleviate the issues being experienced. We also continue to press our third party supplier for updates on the failure on the link between Leeds and Sheffield. Further update will follow at 17:00

  4. Ryan Berrisford says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net Ethernet, Off-net Ethernet and FTTC services

    Description: A portion of on-net and off-net Ethernet customers, as well as FTTH customers, are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre, Architecture and Engineering

    Update 4 –: After further optimisation of the network we are seeing restoration to the affected customers.

    As there is still an open incident on the Leeds to Sheffield link, services should be considered at risk until the link has been brought back into service. We will continue to update this incident until that risk has been fully mitigated.

    If any customers are still experiencing an issue we ask them to power-cycle their equipment before contacting the Technical Service Centre.

  5. Ryan Berrisford says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Update 5 –: Although there is no longer any customer impact, the incident remains open whilst we mitigate the risk caused by the Sheffield to Leeds link failure. Engineer resource has been mobilised to site with an expected time of arrival of 19:50

    A further update will be posted by 20:20 or as soon as further information becomes available.

  6. Adam Heath says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Adam Heath

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Update 6 –: Our TSC have checked in with our engineer resource who advise that they have just arrived on site.  We will update upon receipt of further information or at 21:30 regardless of this.

  7. Adam Heath says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Adam Heath

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Update 7 –: Our engineering resource has located the damaged fibre and are currently performing an assessment with regards to the extent. Again, no services are currently impacted but are at risk. We will update again at 23:00 or upon receipt of further information, whichever comes first.

  8. Adam Heath says:

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Adam Heath

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Resolved –: 22:38 We have had confirmation that this issue is now resolved and that services are no longer at risk. Following assessments by our engineering resource at multiple locations, the damage previously mentioned was unrelated and not the cause of the issue on the remaining impacted service. Instead, a fibre microbend was identified and corrected, with testing demonstrating expected results.

    An internal post incident review will now follow.

Planned Work: man191

Posted: Wednesday, March 17th, 2021 at 16:09 by Alistair Patterson

Tuesday 23rd March 2021, 21:00 – Wednesday 24th March 2021, 02:00

During the above window, we will be completing some none service affecting maintenance following the investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.

Emergency Work: LINX LON1 & DE-CIX Peers

Posted: Friday, March 12th, 2021 at 12:25 by Jamie Smith

Friday 12th March 2021, 13:00 – 13:30

During the above window we will be carrying out preventative work on our LINX LON1 & DE-CIX peers. During this window peers will notice sessions going down and there may be a brief interruption to any traffic routing via one of these affected peers before routing via alternative LINX & LONAP peering locations.

We apologize for the short notice and any inconvenience this may cause.

  1. Jamie Smith says:

    This work is now starting

  2. Jamie Smith says:

    This change has now been completed.

Planned Work: bir16

Posted: Thursday, March 11th, 2021 at 16:04 by Iain Beveridge

Thursday 11/03/2021 21:00:00 – Friday 12/03/2021 02:00:00

During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only, no services will be affected during the maintenance window.

Planned Work: Peterborough Te

Posted: Wednesday, March 10th, 2021 at 16:10 by Adam Heath

Thursday 25th March 2020 00:01 – Thursday 25th March 2020 03:00

After recently replacing the above router on Tuesday 23rd February 2021 we have identified a need to reload the new equipment as we have an issue with an onboard 10gb port. The reload was recommended after raising a TAC case with hardware vendor support.

The work will take place during the above window and will effect all users connected to PB1244, we do not anticipate downtime for the entire window. We apologise for the inconvenience these planned works will cause and we have exhausted all other possibilities before taking this customer impacting action.

  1. Adam Wilson says:

    Please be advised, this work has been aborted and will be rescheduled. Further updates will be provided in due course.

  2. Jordan Gardner says:

    This work has now been rescheduled to Friday 26th March 2021 00:01 – Friday 26th March 2021 03:00. Please see new post https://noc.enta.net/2021/03/emergency-work-peterborough-te/

Planned Work: man191

Posted: Tuesday, March 9th, 2021 at 15:18 by Iain Beveridge

Tuesday 9th March 2021, 21:00 – Wednesday 10th March 2021, 02:00

During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.

  1. Aiden Spencer says:

    All work was successful

Planned Work: man191

Posted: Monday, March 8th, 2021 at 16:49 by Iain Beveridge

Monday 8th March 2021, 21:00 – Tuesday 9th March 2021, 02:00

During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.

  1. The above work was cancelled and rescheduled for this evening, no work took place during the above window