Archive for the ‘Uncategorized’ Category

« Older Entries
Newer Entries »

Planned Work: BT Slough, kingston.core & colindale.core

Posted: Tuesday, October 6th, 2020 at 14:33 by Iain Beveridge

Thursday 29th October 2020, 21:00 – Friday 30th October 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Slough TE requires both the ATS and UPS to be refreshed. Although only Slough is directly effected due to network design customers with services on Kingston & Colindale core’s should also be considered at risk.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

Posted in Uncategorized | No Comments »

Planned Work: Update to firmware openstack routes

Posted: Tuesday, September 15th, 2020 at 15:11 by Iain Beveridge

Monday 21st September 2020 22:00:00 – 23:00:00

During the above window we are carrying out some essential maintenance on our network hardware to improve its reliability and performance. During this window our Mirada platform will be unavailable.

Posted in Uncategorized | No Comments »

Planned Work: Peterborough TE

Posted: Wednesday, September 9th, 2020 at 08:42 by Iain Beveridge

Thursday 15th October 2020, 21:00 – Friday 16th October 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Peterborough TE requires both the ATS and UPS to be refreshed.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

Posted in Uncategorized | No Comments »

Planned Work: Milton Keynes POP

Posted: Wednesday, August 26th, 2020 at 10:01 by Iain Beveridge

Thursday 1st October 2020, 21:00 – Friday 2nd October 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Milton Keynes TE requires both the ATS and UPS to be refreshed.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

Posted in Uncategorized | No Comments »

Emergency Work: Cardiff POP

Posted: Tuesday, August 25th, 2020 at 15:08 by Iain Beveridge

Friday 28th August 2020, 00:00 – 02:00

After a power check within Cardiff POP, BT identified an issue with part of the power infrastructure within one of our racks.

Therefore, BT will be undertaking emergency maintenance work in the above window whilst a UPS is swapped within the rack. The core network router is not in this rack and not expected to experience any loss of service, customers with circuits terminating in Cardiff BT POP should be considered at risk only as the work should be able to be completed with no disruption to service.

Posted in Uncategorized | 2 Comments »

  1. Robin Posadas says:
    August 28, 2020 at 00:00

    Work now commencing.

  2. Robin Posadas says:
    August 28, 2020 at 02:43

    Work now completed.

Planned Work: Wolverhampton POP

Posted: Wednesday, August 19th, 2020 at 16:01 by Iain Beveridge

Thursday 17th September 2020, 21:00 – Friday 18th September 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Wolverhampton TE requires both the ATS and UPS to be refreshed.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

Posted in Uncategorized | 2 Comments »

  1. Naz says:
    September 17, 2020 at 20:59

    work is now commencing

  2. Naz says:
    September 17, 2020 at 23:45

    Work is now completed

Planned Work: Wolverhampton POP

Posted: Wednesday, August 19th, 2020 at 15:47 by Iain Beveridge

Thursday 10th September 2020, 21:00 – Friday 11th September 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Wolverhampton TE requires both the ATS and UPS to be refreshed.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

Posted in At-Risk, Planned Work, Uncategorized | No Comments »

Planned Work: Birmingham TE.

Posted: Wednesday, August 19th, 2020 at 15:45 by Iain Beveridge

Thursday 3rd September 2020, 21:00 – Friday 4th September 2020 05:00

Due to on going improvements to the power infrastructure within the BT POP’s, Birmingham TE requires both the ATS and UPS to be refreshed.

Therefore, BT will be undertaking maintenance work within the above window which will require the power feeds to be moved, whilst an ATS & UPS are swapped within the rack. Although the core network router is not expected to experience any loss of service, customers connecting into the adjacent equipment will experience a small period of downtime.

Posted in At-Risk, Planned Work, Uncategorized | 2 Comments »

  1. Naz says:
    September 3, 2020 at 21:15

    Work is now commencing

  2. Naz says:
    September 3, 2020 at 23:47

    Work has been completed

Emergency Work: Peterborough BT POP.

Posted: Tuesday, August 18th, 2020 at 17:27 by Iain Beveridge

Wednesday 19th August 2020, 00:00 – 03:00

After a power check within Peterborough POP, BT identified an issue with part of the power infrastructure within one of our racks.

Therefore, BT will be undertaking emergency maintenance work in the above window whilst a UPS is swapped within the rack. The core network router is not in this rack and not expected to experience any loss of service, customers with circuits terminating in Peterborough BT POP should be considered at risk only as the work should be able to be completed with no disruption.

Posted in Uncategorized | 2 Comments »

  1. Naz says:
    August 19, 2020 at 00:04

    work is now commencing

  2. Naz says:
    August 19, 2020 at 00:36

    work is now completed

Lon50/Telecity-hex89.core device outage

Posted: Tuesday, August 18th, 2020 at 05:20 by Adam Heath

Our Co-Location provider for this site has advised of a power outage which is effecting multiple floors within the London Harbour Exchange building. Investigations are being carried out to determine an RCA. At the time of this post the building has been evacuated due to the fire alarm triggering.

More information to be provided as and when we have updates from our Co-Location provider

Posted in Uncategorized | 16 Comments »

  1. ljordan says:
    August 18, 2020 at 06:16

    Further updates pending but Talk Talk are currently invesigating further. As soon as they are made available we will advise further.

  2. ljordan says:
    August 18, 2020 at 07:39

    We are still awaiting further updates from the suppliers on this outage. The London building is also still under evacuation. The engineers are on site to investigate as soon as possible. As soon as we have any news, we will advise further.

  3. ljordan says:
    August 18, 2020 at 07:55

    It been advised by Equinix that there is multiple core links experiencing an outage and 36 network to network interface ports causing Business customers connected to the London Harbour and Harbour exchanges to experience a Total Loss of Service. Outage start time was 04:23 (18/08). Full Impact assessment is still ongoing.

    Latest Update
    Equinix have advised that a fire alarm was triggered by the failure of output static switch from their Galaxy UPS system. This has resulted in a loss of power for multiple customers and Equinix IBX Engineers are working to resolve the issue

  4. mpurcell says:
    August 18, 2020 at 09:21

    Latest Update:

    Engineers from Equinix continue to work on restoring power at the datacentre to restore service.

    Escalations have been made to suppliers who hand over services to us at this location with a view of getting more detail and timescales from Equinix. Equinix IBX site staff reports that IBX engineers and the specialist vendor have begun restoring services to customers by migrating to newly installed and commissioned infrastructure. IBX engineers continue to work towards restoring service to all customers

    Root cause has not currently been identified.

    Next update expected at 1000

  5. mpurcell says:
    August 18, 2020 at 10:04

    Update is due shortly. Will update here as soon as it is received.

  6. mpurcell says:
    August 18, 2020 at 10:18

    Latest Update:

    Talk Talk Support Teams have confirmed that services remain impacted but IBX staff are being allowed back on site and have confirmed that the fire detection system has been fully restored. The Galaxy UPS failure remains an ongoing incident and is currently under investigation

  7. mpurcell says:
    August 18, 2020 at 11:00

    Latest Update:

    Equinix Engineers have advised that their IBX team have begun restoring power to affected devices. Unfortunately, at present there remains no estimate resolution time.

    We will continue to provide updates as soon as they come through.

  8. mpurcell says:
    August 18, 2020 at 12:03

    Latest Update:

    Equinix IBX staff have reported that some services have been reinstated and they continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure. Talk Talk Network teams have also confirmed that some 3rd party services are coming back online.

  9. mpurcell says:
    August 18, 2020 at 12:52

    Latest Update:

    Equinix IBX Site Staff reports that services have been further restored for several more customers and IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure

  10. mpurcell says:
    August 18, 2020 at 14:03

    Apologies, no further updates at present. As soon as one comes through we will update here.

  11. mpurcell says:
    August 18, 2020 at 14:08

    Latest Update:

    Equinix IBX Site Staff reports that services have been further restored to more customers and increasing numbers of those affected are now operational along with the majority of Equinix Network Services. IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure.

    No root cause has yet been identified.

  12. mpurcell says:
    August 18, 2020 at 15:21

    Latest Update:

    Internally we are beginning to see an increasing number of our services coming back online.

  13. mpurcell says:
    August 18, 2020 at 16:29

    Latest Update:

    Talk Talk reports that Service was restored at 15:36 to the TalkTalk switch, in HEX, on the Legacy TUK Network.

    IBX Engineers continue to work towards restoring services to all customers by migrating to the newly installed and commissioned infrastructure and estimate full restoration by or before 21:00 GMT/BST.

  14. mpurcell says:
    August 18, 2020 at 16:33

    Additional to last:

    Equinix have advised that electrical work is being carried out at the data centre whereby services (under floor sockets) require migration to a new distribution board as one has failed. There are 8 floors that require this work to be carried out. Floors one and two have been completed. As a result we still consider the services to be at risk, despite them being restored currently. Further outages may be seen until the expected completion time of 21:00hrs tonight.

  15. ljordan says:
    August 18, 2020 at 22:07

    Latest Update
    Equinix IBX Site Staff reports that restoration of services continues. The remaining affected customers are in the process of being migrated to the newly installed and commissioned infrastructure. A revised resolution time has not yet been provided.

  16. ljordan says:
    August 18, 2020 at 22:56

    Equinix advised that all services are now restored. All customers are now fed from the new infrastructure and at full redundancy. UPS failure was deemed the cause. The incident will now be resolved as all systems are available with no further issues reported. Monitoring will remain in place to ensure stability.

« Older Entries
Newer Entries »