Archive for May, 2019

Incident: Talk Talk Circuits

Posted: Friday, May 24th, 2019 at 13:29 by Jonathan Clarke

We have been advised by Talk Talk that a number of their circuits are down in the Northern Ireland area. We are currently speaking with suppliers for further investigation. Further updates will be provided on an hourly basis.

  1. Jonathan Clarke says:

    Suppliers have declared the incident as a major service outage. Talk Talk’s incidents team are still working towards a resolution and we will update upon receipt of further updates.

  2. Jonathan Clarke says:

    Suppliers have advised an engineer was dispatched to the affected network to investigate further. The engineer who attended identified the reason for the disruption to service. BT Openreach has now marked this major service outage as an MBORC (Matters Beyond Our Reasonable Control). This is due to a third party damaging fibre cables. Replacement cabling has been ordered by Openreach, they have requested authorities for traffic management to permit them to dig.

    Further updates will be provided as they are available.

  3. John Benge says:

    Our supplier has confirmed that services were fully restored overnight. The disruption to service was caused by a 3rd party breaking BT Openreach fibre on the A6 near Antrim. The fibre was pulled and respliced. All affected services should now be restored.

Planned Maintenance: Telehouse-north.core

Posted: Thursday, May 23rd, 2019 at 17:47 by Gurinder Bansal

Friday 31st May 2019 00:01 – 04:00

During the above window we will completing a planned maintenance to enhance the security on our aggregation devices on telehouse-north.core.

We expect all services to have a maximum downtime of 1 hour. Services include Transit, EWCS, Leased lines, Wholesale handoff and Pseudowires terminating at this location. Whilst actual downtime is expected to be no more than 1 hour, the full 4 hours may be required to account for any unforeseen issues.

  1. Gurinder Bansal says:

    This work is now commencing.

  2. Gurinder Bansal says:

    This work is now complete.

Incident: Virgin Media outage

Posted: Tuesday, May 21st, 2019 at 14:11 by david labouchardiere

We are currently aware of a number of Virgin Media circuits which have lost connectivity. We have raised this with the supplier to investigate as a matter of urgency. Further updates will be provided on an hourly basis.

  1. David Brewis says:

    Following investigations we believe this incident to be resolved. If you are still experiencing a loss of service following rebooting your devices please call into the helpdesk.

    We apologise for any disruption caused by this incident.

Planned Maintenance: Glasgow

Posted: Tuesday, May 21st, 2019 at 11:18 by Richard Partridge

Wednesday 29th May 01:00 – 06:00

During the above period we need to carry out some planned maintenance on
our network.

We’re letting you know because this work may impact your active CityFibre
service(s) and result in a loss of service.

The work is estimated to take 1 – 4 hours but the full duration of the
maintenance window may be needed for maintenance to be completed.

Planned Maintenance: Aberdeen

Posted: Tuesday, May 21st, 2019 at 11:18 by Richard Partridge

Wednesday 29th May 01:00 – 06:00

During the above period we need to carry out some planned maintenance on
our network.

We’re letting you know because this work may impact your active CityFibre
service(s) and result in a loss of service.

The work is estimated to take 1 – 4 hours but the full duration of the
maintenance window may be needed for maintenance to be completed.

Planned Maintenance: Edinburgh

Posted: Tuesday, May 21st, 2019 at 11:18 by Richard Partridge

Wednesday 29th May 01:00 – 06:00

During the above period we need to carry out some planned maintenance on
our network.

We’re letting you know because this work may impact your active CityFibre
service(s) and result in a loss of service.

The work is estimated to take 1 – 4 hours but the full duration of the
maintenance window may be needed for maintenance to be completed.

Incident: Virgin Media circuits

Posted: Tuesday, May 21st, 2019 at 11:01 by david labouchardiere

We are currently aware of an issue affecting Virgin Media circuits causing slow speeds. This has been raised to Virgin and their engineers are currently investigating. Further updates will be provided in the next hour.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. david labouchardiere says:

    Suppliers have advised an engineer is currently being dispatched to the affected POP to investigate further. Further updates will be provided in the next hour.

  2. david labouchardiere says:

    We have noticed a number of additional Virgin Media circuits in the affected POP appear to have lost connectivity. We are requesting the supplier investigate further as a matter of urgency.

  3. David Brewis says:

    Following discussions with Virgin Media Service Management we understand the two incidents to be unrelated at this time.

    A new post will be created to provide updates for the outage and this will remain for performance related.

  4. David Brewis says:

    Virgin Media have confirmed this is part of a major service outage affecting many customers.

    An agreed action plan is underway with support from hardware vendors and engineers on site with spare hardware available should issues arise. A further update will be provided by 1700.

  5. David Brewis says:

    We now believe this incident to be restored with all alarms cleared, a 24 hour period of monitoring will follow to ensure stability.

    We apologise for any disruption caused by this.

Incident: Vodafone Hub Outage

Posted: Monday, May 20th, 2019 at 03:33 by Adam Heath

Following on from a Vodafone maintenance our monitoring has alerted us to a number of Vodafone circuits that are currently out of service. Our initial diagnostics indicate that the circuits are related to a single handoff point between ourselves and Vodafone. Our engineers are investigating and are engaged with the supplier and further updates will be provided via this site.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. William George says:

    Vodafone have confirmed they see the issue and are in the process of arranging a resource to go to site to address it. Further updates will be provided as they are available.

  2. David Brewis says:

    Engineers from the supplier are already on site investigating cause for disruption to service and will update with their findings. As this progresses updates will be posted for your notice. Please expect the next update by 10:20.

  3. David Brewis says:

    Investigations from field engineers have identified an equipment fault, work has begun with hardware vendors to replace and resolve service as soon as possible.

    We apologise for the ongoing disruption whilst this incident progresses.

    Please expect the next update by 11:20

  4. David Brewis says:

    Following contact by Vodafone to their hardware vendor, a replacement component has been shipped to site with the expected arrival time of 11:00. We are awaiting confirmation of this delivery to begin works for replacement to restore service.

    Please expect the next update by 12:20.

  5. David Brewis says:

    The replacement unit has been installed and initial troubleshooting by Vodafone demonstrates the incident has been resolved.

    We are completing our own triage before confirming the engineer can leave site.

    Please expect a further update by 12:30.

  6. David Brewis says:

    We can confirm the handoff to Vodafone has returned to working service and the engineer stood down.

    If your service remains offline please contact the helpdesk following a reboot of your devices.

    We apologise for any disruption caused by this incident and will continue discussions with Vodafone for root cause analysis.

Incident: CityFibre circuits in Aberdeen

Posted: Tuesday, May 14th, 2019 at 13:01 by david labouchardiere

We are currently aware of an outage affecting CityFibre circuits in the Aberdeen area. We are in the process of engaging engineer resources to investigate the issue. Further updates will be provided in the next hour.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. david labouchardiere says:

    Engineers confirm services have been restored.

    If any further issues are experienced please contact our Support Desk.

    We apologise for the disruption caused by this incident.

Planned Maintenance: Telecity-Hex89.core

Posted: Wednesday, May 8th, 2019 at 10:11 by Jamie Smith

Thursday 23rd May 2019 00:00– 06:00

During the above period we will be carrying out essential service affecting works within our core rack at Telecity-Hex8/9

All services currently terminated in this location will be subject to downtime. Services Include Leased lines, Wholesale handoff, Transit, EWCS & Psuedowires.

We anticipate the full duration of the maintenance window will be needed for works to be completed.

All other traffic that would normally route through this location will take alternative paths.

  1. Jamie Smith says:

    Please be aware this work is now starting.

  2. Jamie Smith says:

    This maintenance has now been completed.

    Customers are advised to reboot their routers in the first instance before contacting our tech support team, should there be any residual effects following on from this planned work.