Archive for the ‘Incidents’ Category

Incident: nottingham.core

Posted: Sunday, June 3rd, 2018 at 11:17 by Mark Yardley

We have experienced an unscheduled reload of our Nottingham core router. All directly attached services will have experienced a brief interruption. Initial investigation suggests this was due to loss of power in the location and engineers are investigating further with our co-location provider. All services are now restored and we apologise for any inconvenience caused.

Incident: Talk Talk Circuits

Posted: Friday, June 1st, 2018 at 13:59 by Adam Heath

We are aware that a number of Talk Talk circuits are currently experiencing a disruption to service.  Talk Talk have acknowledged an issue and are currently investigating.  We will provide further information when it becomes available.

  1. Adam Heath says:

    Talk Talk have confirmed that the disruption has been caused by core network circuit flaps connecting into their Milton Keynes core device. This is affecting circuits nationwide.

    Talk Talk have engineering resource investigating a root cause but are unable to provide us with an ERT at this time. We will continue to update upon receipt of further information.

At Risk: Telford-Dc2.core

Posted: Thursday, April 12th, 2018 at 00:46 by Jamie Smith

Telford-dc2.core is currently operating with reduced resilience, due to the failure of a core network link. The issue has been raised with our suppliers for further investigation. Further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Our supplier have advised that this is in fact planned engineering work , however no notification had been received to advise of this.

    Work is expected to be completed by 06:00.

    Apologies for the inconvenience.

  2. Jamie Smith says:

    Further to the most recent update;

    VM have come back to advise that this has now been escalated as a fault due to potential faulty fibre patches and possible faulty amp related to medium power work.

    VM have despatched engineers to site to Investigate the issue further and work towards a resolve.

    I apologise for any confusion as the previous update had been incorrectly advised by the supplier.

    Updates will follow in due course

  3. Jamie Smith says:

    The affected core link is operational again however we have yet to receive an RFO from our supplier. As such should still be considered at risk until further notice

Incident: Vodafone Circuits

Posted: Tuesday, March 27th, 2018 at 00:42 by Adam Heath

We are aware of an issue currently affecting some Vodafone circuits.  This has been raised with the supplier who are currently investigating.  We’ll provide further information once it becomes available.

  1. Adam Heath says:

    Vodafone have identified an issue and are in the process of dispatching engineering resource to the relevant location. Further updates will be available in due course.

  2. Jamie Smith says:

    Latest supplier update:

    Vodafone engineers have identified a fibre break and are engaging the relevant fibre team to site. Further updates will be available in due course.

  3. Jamie Smith says:

    We have been advised by our supplier that fibre engineers are due on site between 7.30 – 7.40am to begin working towards a fix.

    Further updates to be provided as and when they become available

  4. Adam Heath says:

    Vodafone have confirmed the incident is due to a multiple fibre break. They have engineers present on site and are currently exploring other suitable service options.

  5. David Brewis says:

    Service has been restored following a repair on the Vodafone fibre break and confirmation received at 10:33.

    We are liaising with Vodafone post incident and a full RFO will be provided.

  6. David Brewis says:

    An interim report can be found here, whilst we await a full RFO.

  7. David Brewis says:

    A full incident report can be found here.

Incident:Vodafone Circuits

Posted: Friday, March 23rd, 2018 at 14:50 by Jamie Smith

We have experienced a brief loss of service to a number of Vodafone circuits  Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier.

The circuits in question are now showing as back up, further updates to be provided.

  1. Scott Morgan says:

    After further discussion with Vodafone, they have confirmed the brief drop of connection was caused by human error.

    Whilst conducting diagnostics on a leased line fault, a Vodafone engineer inadvertently carried out loop testing to a Vodafone hub serving Entanet customers. As a result, this caused disruption between 14:43:45 until service was restored at 14:48:09 as the engineer rectified their mistake.

    We apologise for the inconvenience caused by this. We are in talks with our suppliers to see how this can be avoided in future.

Incident: GMS 8.2

Posted: Wednesday, March 7th, 2018 at 15:52 by Jamie Smith

We are currently Investigating an issue that is causing an outage to our GMS 8.2 platform.

  1. Jamie Smith says:

    Engineers are still investigating the root cause of this issue, we are aiming to get this service live again as quick as possible.

    We apologise for any inconvenience

  2. Jamie Smith says:

    This issue has now been fixed, GMS is once again accessible. It may take up to 1 hour for Information to populate

  3. Mark Yardley says:

    We are experiencing residual issues with the GMS platform. Engineers are continuing to investigate.

  4. Jamie Smith says:

    Engineers are still working to resolve the issues currently being experienced with the GMS platform. Updates will be provided when they become available

Incident: telford-dc1.core

Posted: Tuesday, March 6th, 2018 at 13:29 by Jamie Smith

We are currently experiencing a loss of service to telford-dc1.core. Engineers are currently investigating, further updates to follow.

  1. Richard Partridge says:

    Service has now been restored. This incident was caused by failed PSUs which have now been replaced. We apologise for the inconvenience caused.

  2. David Brewis says:

    A report is now available for this incident, which can be found here.

Incident: Mirada Portal

Posted: Monday, March 5th, 2018 at 11:33 by Jamie Smith

Engineers are investigating an issue which is currently rendering the Mirada portal unavailable. Further updates will be provided as and when they become available.

Mirada Monitoring

Posted: Wednesday, February 21st, 2018 at 14:46 by Jamie Smith

We are currently aware of an issue affecting our Mirada monitoring platform, this will be causing Mirada to falsely show all devices as down.

Engineers are currently investigating.

  1. Jake Turner says:

    Engineers have resolved the problems with the Mirada monitoring platform.

    Apologies for any inconvenience caused.

Incident: Interxion2

Posted: Tuesday, February 20th, 2018 at 13:40 by Richard Partridge

We have just experienced a brief period of routing instability to our interxion2.core router. Directly connected leased lines, PWANs and broadband customers would have experienced disrupted service for several minutes. Engineers are currently investigating and further updates will be provided as and when they become available.