Archive for the ‘Uncategorized’ Category

« Older Entries
Newer Entries »

Emergency Work: Peterborough TE

Posted: Thursday, March 25th, 2021 at 12:33 by Adam Heath

Friday 26th March 2021 00:01 – Friday 26th March 2021 03:00

Rescheduled from original window of Thursday 25th March 2021 00:01 – Thursday 25th March 2021 03:00

After recently replacing the above router on Tuesday 23rd February 2021 we have identified a need to reload the new equipment as we have an issue with an onboard 10gb port. The reload was recommended after raising a TAC case with hardware vendor support.

The work will take place during the above window and will effect all users connected to PB1244. We do not anticipate downtime for the entire window. We apologise for the inconvenience these planned works will cause and we have exhausted all other possibilities before taking this customer impacting action.

Tags:
Posted in Uncategorized | 2 Comments »

  1. Adam Wilson says:
    March 26, 2021 at 00:09

    Work is about to commence.

  2. Adam Wilson says:
    March 26, 2021 at 01:32

    This work is now complete.

Incident: CityFibre network services

Posted: Tuesday, March 23rd, 2021 at 14:25 by david labouchardiere

We are currently aware of an issue affecting CityFibre circuits.
Engineers are investigating and further updates will follow as soon as possible

Posted in Uncategorized | 8 Comments »

  1. Ryan Berrisford says:
    March 23, 2021 at 15:06

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net Ethernet Services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net Ethernet customers

    Description: A portion of on-net Ethernet customers are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre and Architecture and Engineering

    Update 1 –: Our Network Operations Centre are currently investigating a network incident affecting on-net Ethernet customers. Initial indications are that this is not related to a specific geographical location. Teams continue to investigate and further updates will follow at 15:25

  2. Ryan Berrisford says:
    March 23, 2021 at 15:32

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net and off-net Ethernet Services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net and off-net Ethernet customers

    Description: A portion of on-net Ethernet customers are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre, Architecture and Engineering

    Update 2 –: The incident is related to an issue on a link between Leeds and Sheffield. Third party providers have already been engaged and are investigating. In the meantime, to mitigate the impact we have re-routed traffic over an alternative link. We are now in the process of assessing the impact of the re-routing. A further update will follow at 16:00

  3. Ryan Berrisford says:
    March 23, 2021 at 16:12

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net and off-net Ethernet Services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net and off-net Ethernet customers

    Description: A portion of on-net and off-net Ethernet customers are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre, Architecture and Engineering

    Update 3 –: After the initial re-routing we are still seeing a portion of customers impacted by the major incident. Further work is being carried out in order to optimise network traffic and alleviate the issues being experienced. We also continue to press our third party supplier for updates on the failure on the link between Leeds and Sheffield. Further update will follow at 17:00

  4. Ryan Berrisford says:
    March 23, 2021 at 17:12

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: Multiple

    Customer Impact: Total loss of services for on-net Ethernet, Off-net Ethernet and FTTC services

    Description: A portion of on-net and off-net Ethernet customers, as well as FTTH customers, are experiencing a total loss of service.

    Team(s) Investigating: Network Operations Centre, Architecture and Engineering

    Update 4 –: After further optimisation of the network we are seeing restoration to the affected customers.

    As there is still an open incident on the Leeds to Sheffield link, services should be considered at risk until the link has been brought back into service. We will continue to update this incident until that risk has been fully mitigated.

    If any customers are still experiencing an issue we ask them to power-cycle their equipment before contacting the Technical Service Centre.

  5. Ryan Berrisford says:
    March 23, 2021 at 18:23

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Ryan Berrisford

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Update 5 –: Although there is no longer any customer impact, the incident remains open whilst we mitigate the risk caused by the Sheffield to Leeds link failure. Engineer resource has been mobilised to site with an expected time of arrival of 19:50

    A further update will be posted by 20:20 or as soon as further information becomes available.

  6. Adam Heath says:
    March 23, 2021 at 20:29

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Adam Heath

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Update 6 –: Our TSC have checked in with our engineer resource who advise that they have just arrived on site.  We will update upon receipt of further information or at 21:30 regardless of this.

  7. Adam Heath says:
    March 23, 2021 at 21:40

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Adam Heath

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Update 7 –: Our engineering resource has located the damaged fibre and are currently performing an assessment with regards to the extent. Again, no services are currently impacted but are at risk. We will update again at 23:00 or upon receipt of further information, whichever comes first.

  8. Adam Heath says:
    March 23, 2021 at 23:01

    Major Incident Raised: 23/03/2021 14:25

    Major Incident Leader: Adam Heath

    Service (product) Type: On-net Ethernet, Off-net Ethernet and FTTC services

    Service(s) Impacted: No services are currently impacted

    Customer Impact: There is currently no impact to customers

    Description: Services are at risk until the failed link has been restored

    Team(s) Investigating: Technical Service Centre

    Resolved –: 22:38 We have had confirmation that this issue is now resolved and that services are no longer at risk. Following assessments by our engineering resource at multiple locations, the damage previously mentioned was unrelated and not the cause of the issue on the remaining impacted service. Instead, a fibre microbend was identified and corrected, with testing demonstrating expected results.

    An internal post incident review will now follow.

Emergency Work: LINX LON1 & DE-CIX Peers

Posted: Friday, March 12th, 2021 at 12:25 by Jamie Smith

Friday 12th March 2021, 13:00 – 13:30

During the above window we will be carrying out preventative work on our LINX LON1 & DE-CIX peers. During this window peers will notice sessions going down and there may be a brief interruption to any traffic routing via one of these affected peers before routing via alternative LINX & LONAP peering locations.

We apologize for the short notice and any inconvenience this may cause.

Tags: ,
Posted in Uncategorized | 2 Comments »

  1. Jamie Smith says:
    March 12, 2021 at 13:00

    This work is now starting

  2. Jamie Smith says:
    March 12, 2021 at 13:52

    This change has now been completed.

Planned Work: bir16

Posted: Thursday, March 11th, 2021 at 16:04 by Iain Beveridge

Thursday 11/03/2021 21:00:00 – Friday 12/03/2021 02:00:00

During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only, no services will be affected during the maintenance window.

Posted in Uncategorized | No Comments »

Planned Work: man191

Posted: Tuesday, March 9th, 2021 at 15:18 by Iain Beveridge

Tuesday 9th March 2021, 21:00 – Wednesday 10th March 2021, 02:00

During the above window, we will be completing some none service affecting maintenance as part of the on going investigation of the network issues we experienced in February. This is for information only and no services will be affected during the maintenance window.

Posted in Uncategorized | 1 Comment »

  1. Aiden Spencer says:
    March 10, 2021 at 08:29

    All work was successful

Emergency Work: Peterborough (PB1244)

Posted: Tuesday, February 23rd, 2021 at 16:28 by Jamie Smith

Tuesday 23rd February 2021, 19:00 – Tuesday 23rd February 2021, 23:59

During the above window, we will be completing emergency maintenance to upgrade and replace our core router at the above location. This is required in order to resolve the issues we have been experiencing in Peterborough. There will be a small window of downtime as we replace the chassis and migrate services across.

Tags: ,
Posted in Uncategorized | 2 Comments »

  1. Robin Posadas says:
    February 23, 2021 at 19:08

    Work commencing.

  2. Robin Posadas says:
    February 23, 2021 at 21:18

    Work completed

Incident: Peterborough

Posted: Tuesday, February 23rd, 2021 at 11:02 by david labouchardiere

Following the incident yesterday we are still working on restoring service to the portion of customers who remain impacted in Peterborough. Engineers are continuing to troubleshoot and further updates will be provided in the next hour

Posted in Uncategorized | 13 Comments »

  1. david labouchardiere says:
    February 23, 2021 at 12:10

    We are still investigating this issue with our senior agents within the NOC team and will provide further updates in the next hour.

  2. david labouchardiere says:
    February 23, 2021 at 13:22

    Based on NOC troubleshooting we have engaged engineer resources to attend the affected site and carry out further testing. We will provide a further update once we have an ETA for the engineer

  3. david labouchardiere says:
    February 23, 2021 at 13:52

    Engineer resources have been arranged with an ETA of 16:00. We will provide a further update upon the engineers arrival.

  4. Emily Pieniak says:
    February 23, 2021 at 15:59

    Engineer resources have now arrived on-site where investigations have commenced. We will provide further updates in the next hour.

  5. Emily Pieniak says:
    February 23, 2021 at 17:00

    Engineer resource is on-site investigating the physical link between PB1344 and PB1244. We will provide further updates in the next hour

  6. Emily Pieniak says:
    February 23, 2021 at 19:19

    The link between PB1344 and PB1244 remains down where we are still seeing 1 way MAC betwenPB1344 and PB1244. The SFP and fibre on PB1344 and PB1244 link has been re-seated and cleaned.

    We will continue to provide updates in the next hour.

  7. Emily Pieniak says:
    February 23, 2021 at 20:02

    A replacement ASR is on-site which at present is being installed where we are awaiting further feedback from the engineers on-site.

    We will continue to provide updates in the next hour.

  8. Emily Pieniak says:
    February 23, 2021 at 21:01

    SPF swap has taken place at PB1244 where engineers are continuing to work at PB1344 and PB1244.

    We will continue to provide updates in the next hour.

  9. Emily Pieniak says:
    February 23, 2021 at 21:42

    The link between PB1344 and PB1244 remains down. Engineers have attempted multiple optics on both PB1344 and PB1244 where the port still would not come up.

    Next action is for the fault to be logged with Nokia TAC for PB1344.

    We will continue to provide updates within the next 2 hours.

  10. Adam Wilson says:
    February 24, 2021 at 00:12

    Our vendors are dispatching an engineer to replace a suspected faulty card in our PB1344-rtr-1 node.
    ETA is 02:30. A further update will be provided upon the engineer’s arrival.

  11. Adam Wilson says:
    February 24, 2021 at 03:23

    Engineer is on site and about to commence work. Update will follow upon completion of the task.

  12. Adam Wilson says:
    February 24, 2021 at 04:55

    The card has been replaced, and we have collected further diagnostic information. We will confirm operational status once this has been reviewed by our vendor.

  13. Adam Wilson says:
    February 24, 2021 at 05:25

    We believe this issue to now be resolved, and request that any affected users retest their service.
    Please accept our apologies for any inconvenice that may have been caused.

Incident: Peterborough

Posted: Monday, February 22nd, 2021 at 16:21 by Adam Heath

We are currently aware of an issue affecting services within the Peterborough area. Engineers are already engaged and investigating. A further update will be provided when available.

Tags:
Posted in Uncategorized | 6 Comments »

  1. david labouchardiere says:
    February 22, 2021 at 17:11

    Engineer resources have been engaged to attend the affected site with an ETA of 19:00. Further updates will be posted upon the engineer’s arrival.

  2. Adam Heath says:
    February 22, 2021 at 20:12

    CityFibre Incident Number: INC0080756

    Major Incident Raised: 22/02/2021 – 16:00

    Major Incident Leader: Adam Heath

    Service (product) Type: Off-net Ethernet & On-net Ethernet

    Service(s) Impacted: Multiple

    Configuration Item(s) Impacted: NA

    Customer Impact: Total loss of services for customers connected to PB1244.

    Description: Core device has become unreachable.

    Team(s) Investigating: NOC and Field Engineers

    Update 2 -: Further engineer resource is required at the location for investigation and restoration to commence. We do not have an ETA as of yet for their arrival but will provide this as soon as we are in receipt of it.

    Update 1 –: 17:22 At 16:00 connection to PB1244 dropped causing a total loss of service. PB1244 remains unreachable and an FLM engineer has been dispatched to site to carry out investigations. ETA for engineer to arrive onsite is currently 19:00.

  3. Adam Heath says:
    February 22, 2021 at 21:10

    CityFibre Incident Number: INC0080756

    Major Incident Raised: 22/02/2021 – 16:00

    Major Incident Leader: Adam Heath

    Service (product) Type: Off-net Ethernet & On-net Ethernet

    Service(s) Impacted: Multiple

    Configuration Item(s) Impacted: NA

    Customer Impact: Total loss of services for customers connected to PB1244.

    Description: Core device has become unreachable.

    Team(s) Investigating: NOC and Field Engineers

    Update 3 – Further engineer resource is expected to arrive at site at 21:35

  4. Adam Heath says:
    February 22, 2021 at 22:10

    CityFibre Incident Number: INC0080756

    Major Incident Raised: 22/02/2021 – 16:00

    Major Incident Leader: Adam Heath

    Service (product) Type: Off-net Ethernet & On-net Ethernet

    Service(s) Impacted: Multiple

    Configuration Item(s) Impacted: NA

    Customer Impact: Total loss of services for customers connected to PB1244.

    Description: Core device has become unreachable.

    Team(s) Investigating: NOC and Field Engineers

    Update 4 -: The additional engineer resource has arrived onsite and working with our NOC, investigation has commenced.

    Next Update: 23:00

  5. Adam Heath says:
    February 22, 2021 at 23:01

    CityFibre Incident Number: INC0080756

    Major Incident Raised: 22/02/2021 – 16:00

    Major Incident Leader: Adam Heath

    Service (product) Type: Off-net Ethernet & On-net Ethernet

    Service(s) Impacted: Multiple

    Configuration Item(s) Impacted: NA

    Customer Impact: Total loss of services for customers connected to PB1244.

    Description: Core device has become unreachable.

    Team(s) Investigating: NOC and Field Engineers

    Update 5 -: Our NOC have confirmed that the core device is now reachable following resource at the location rebooting it. They are just performing some final checks before we mark this as resolved.

    Next Update: 23:30

  6. Adam Heath says:
    February 22, 2021 at 23:38

    CityFibre Incident Number: INC0080756

    Major Incident Raised: 22/02/2021 – 16:00

    Major Incident Leader: Adam Heath

    Service (product) Type: Off-net Ethernet & On-net Ethernet

    Service(s) Impacted: Multiple

    Configuration Item(s) Impacted: NA

    Customer Impact: Total loss of services for customers connected to PB1244.

    Description: Core device has become unreachable.

    Team(s) Investigating: NOC and Field Engineers

    Resolved -: 23:37 Our NOC have confirmed from their checks following the device being rebooted, the majority of impacted services have been restored and the device is operational. Therefore, we will mark the Major Incident as resolved at this point.

    They are aware of a very small number of services still offline. It is advised that the equipment at the customer’s side should be rebooted if this is the case and should the service still remain offline, to contact our Technical Service Centre.

    There will be a follow up internal post incident review scheduled, with any further required actions being managed by the Change and Problem Management processes.

Incident: CityFibre leased lines in Bristol

Posted: Thursday, January 14th, 2021 at 13:41 by david labouchardiere

We are currently aware of an issue affecting CityFibre provided leased lines in the Bristol area. Engineers are on site investigating and working to restore service as soon as possible

Posted in Uncategorized | 1 Comment »

  1. david labouchardiere says:
    January 14, 2021 at 14:01

    We have had a confirmation service should now be restored. Please ask any affected customers to power cycle their hardware on-site if they continue to experience issues.

    We apologise for any inconvenience caused by this incident.

Emergency Work: DSL Platform

Posted: Thursday, December 17th, 2020 at 13:19 by Jamie Smith

Thursday 17th December 2020, 13:30 – 14:00

During the above window we will be carrying out emergency maintenance to the traffic shapers within the DSL platform. This is necessary work due to the ongoing issues documented at https://noc.enta.net/2020/12/incident-dsl-packet-loss/

No downtime is expected to services however broadband services should be considered at risk for the duration of this work. Further work will need to be completed out of hours after this initial maintenance and will be communicated in due course.

Apologies for the short notice and any inconvenience this may cause.

Tags:
Posted in Uncategorized | 3 Comments »

  1. Jamie Smith says:
    December 17, 2020 at 16:01

    Please be aware that between 18:00 – 21:00 tonight (17/12/2020) we will be carrying out further remedial work as mentioned previously on the traffic shapers within the Broadband Network.

    No downtime is expected to services however broadband services should be considered at risk for the duration of this work.

    We apologize for the short notice and any inconvenience this may have caused.

  2. Jamie Smith says:
    December 17, 2020 at 18:00

    Please be aware the above works is now about to commence

  3. Jamie Smith says:
    December 17, 2020 at 19:55

    Work has now been completed

« Older Entries
Newer Entries »