Archive for the ‘Incidents’ Category

At Risk: Telford-Dc2.core

Posted: Thursday, April 12th, 2018 at 00:46 by Jamie Smith

Telford-dc2.core is currently operating with reduced resilience, due to the failure of a core network link. The issue has been raised with our suppliers for further investigation. Further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Our supplier have advised that this is in fact planned engineering work , however no notification had been received to advise of this.

    Work is expected to be completed by 06:00.

    Apologies for the inconvenience.

  2. Jamie Smith says:

    Further to the most recent update;

    VM have come back to advise that this has now been escalated as a fault due to potential faulty fibre patches and possible faulty amp related to medium power work.

    VM have despatched engineers to site to Investigate the issue further and work towards a resolve.

    I apologise for any confusion as the previous update had been incorrectly advised by the supplier.

    Updates will follow in due course

  3. Jamie Smith says:

    The affected core link is operational again however we have yet to receive an RFO from our supplier. As such should still be considered at risk until further notice

Incident: Vodafone Circuits

Posted: Tuesday, March 27th, 2018 at 00:42 by Adam Heath

We are aware of an issue currently affecting some Vodafone circuits.  This has been raised with the supplier who are currently investigating.  We’ll provide further information once it becomes available.

  1. Adam Heath says:

    Vodafone have identified an issue and are in the process of dispatching engineering resource to the relevant location. Further updates will be available in due course.

  2. Jamie Smith says:

    Latest supplier update:

    Vodafone engineers have identified a fibre break and are engaging the relevant fibre team to site. Further updates will be available in due course.

  3. Jamie Smith says:

    We have been advised by our supplier that fibre engineers are due on site between 7.30 – 7.40am to begin working towards a fix.

    Further updates to be provided as and when they become available

  4. Adam Heath says:

    Vodafone have confirmed the incident is due to a multiple fibre break. They have engineers present on site and are currently exploring other suitable service options.

  5. David Brewis says:

    Service has been restored following a repair on the Vodafone fibre break and confirmation received at 10:33.

    We are liaising with Vodafone post incident and a full RFO will be provided.

  6. David Brewis says:

    An interim report can be found here, whilst we await a full RFO.

Incident:Vodafone Circuits

Posted: Friday, March 23rd, 2018 at 14:50 by Jamie Smith

We have experienced a brief loss of service to a number of Vodafone circuits  Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier.

The circuits in question are now showing as back up, further updates to be provided.

  1. Scott Morgan says:

    After further discussion with Vodafone, they have confirmed the brief drop of connection was caused by human error.

    Whilst conducting diagnostics on a leased line fault, a Vodafone engineer inadvertently carried out loop testing to a Vodafone hub serving Entanet customers. As a result, this caused disruption between 14:43:45 until service was restored at 14:48:09 as the engineer rectified their mistake.

    We apologise for the inconvenience caused by this. We are in talks with our suppliers to see how this can be avoided in future.

Incident: GMS 8.2

Posted: Wednesday, March 7th, 2018 at 15:52 by Jamie Smith

We are currently Investigating an issue that is causing an outage to our GMS 8.2 platform.

  1. Jamie Smith says:

    Engineers are still investigating the root cause of this issue, we are aiming to get this service live again as quick as possible.

    We apologise for any inconvenience

  2. Jamie Smith says:

    This issue has now been fixed, GMS is once again accessible. It may take up to 1 hour for Information to populate

  3. Mark Yardley says:

    We are experiencing residual issues with the GMS platform. Engineers are continuing to investigate.

  4. Jamie Smith says:

    Engineers are still working to resolve the issues currently being experienced with the GMS platform. Updates will be provided when they become available

Incident: telford-dc1.core

Posted: Tuesday, March 6th, 2018 at 13:29 by Jamie Smith

We are currently experiencing a loss of service to telford-dc1.core. Engineers are currently investigating, further updates to follow.

  1. Richard Partridge says:

    Service has now been restored. This incident was caused by failed PSUs which have now been replaced. We apologise for the inconvenience caused.

  2. David Brewis says:

    A report is now available for this incident, which can be found here.

Incident: Mirada Portal

Posted: Monday, March 5th, 2018 at 11:33 by Jamie Smith

Engineers are investigating an issue which is currently rendering the Mirada portal unavailable. Further updates will be provided as and when they become available.

Mirada Monitoring

Posted: Wednesday, February 21st, 2018 at 14:46 by Jamie Smith

We are currently aware of an issue affecting our Mirada monitoring platform, this will be causing Mirada to falsely show all devices as down.

Engineers are currently investigating.

  1. Jake Turner says:

    Engineers have resolved the problems with the Mirada monitoring platform.

    Apologies for any inconvenience caused.

Incident: Interxion2

Posted: Tuesday, February 20th, 2018 at 13:40 by Richard Partridge

We have just experienced a brief period of routing instability to our interxion2.core router. Directly connected leased lines, PWANs and broadband customers would have experienced disrupted service for several minutes. Engineers are currently investigating and further updates will be provided as and when they become available.

Incident: Openreach circuits

Posted: Wednesday, February 14th, 2018 at 12:05 by David Brewis

We are currently experiencing a loss of connectivity for a number of Openreach circuits. We are seeing most services restore and engineers are investigating. Further updates will be provided when available.

  1. David Brewis says:

    An engineer is currently on route to our Birmingham POP to carry out further investigations at site, as we are observing continued loss of some Openreach services.

    The engineer’s ETA is 13:30. Further updates to follow.

  2. David Brewis says:

    The engineer arrived on site ahead of ETA and service resumed promptly for most services. Engineers continue to investigate until full service is resumed and cause identified.

    Further updates to follow.

  3. David Brewis says:

    Connectivity to the affected services has now been restored and engineers are continuing to investigate the underlying cause. We apologise for any inconvenience.

Incident: Vodafone circuits

Posted: Thursday, January 25th, 2018 at 18:50 by David Derrick

We have suffered a recurrence of previous incidents related to one of our Vodafone interconnects. Staff are already on the case with Vodafone.

  1. David Brewis says:

    Vodafone have advised that a field engineer has been assigned and is on route with an eta of 20:00

  2. David Brewis says:

    Investigations have discovered a fibre break, fibre engineers are currently working to restore service

  3. Adam Heath says:

    Vodafone engineers have now re-spliced a snapped fibre. Once the fibre was re-spliced and connected, alarms cleared and services have been restored. Vodafone are continuing to monitor the links.

  4. David Brewis says:

    A report is now available for this incident, which can be found here.