Posts Tagged ‘Incidents’

At Risk: Telford-Dc2.core

Posted: Thursday, April 12th, 2018 at 00:46 by Jamie Smith

Telford-dc2.core is currently operating with reduced resilience, due to the failure of a core network link. The issue has been raised with our suppliers for further investigation. Further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Our supplier have advised that this is in fact planned engineering work , however no notification had been received to advise of this.

    Work is expected to be completed by 06:00.

    Apologies for the inconvenience.

  2. Jamie Smith says:

    Further to the most recent update;

    VM have come back to advise that this has now been escalated as a fault due to potential faulty fibre patches and possible faulty amp related to medium power work.

    VM have despatched engineers to site to Investigate the issue further and work towards a resolve.

    I apologise for any confusion as the previous update had been incorrectly advised by the supplier.

    Updates will follow in due course

  3. Jamie Smith says:

    The affected core link is operational again however we have yet to receive an RFO from our supplier. As such should still be considered at risk until further notice

Incident: telford-dc1.core

Posted: Tuesday, March 6th, 2018 at 13:29 by Jamie Smith

We are currently experiencing a loss of service to telford-dc1.core. Engineers are currently investigating, further updates to follow.

  1. Richard Partridge says:

    Service has now been restored. This incident was caused by failed PSUs which have now been replaced. We apologise for the inconvenience caused.

  2. David Brewis says:

    A report is now available for this incident, which can be found here.

Incident: Mirada Portal

Posted: Monday, March 5th, 2018 at 11:33 by Jamie Smith

Engineers are investigating an issue which is currently rendering the Mirada portal unavailable. Further updates will be provided as and when they become available.

Mirada Monitoring

Posted: Wednesday, February 21st, 2018 at 14:46 by Jamie Smith

We are currently aware of an issue affecting our Mirada monitoring platform, this will be causing Mirada to falsely show all devices as down.

Engineers are currently investigating.

  1. Jake Turner says:

    Engineers have resolved the problems with the Mirada monitoring platform.

    Apologies for any inconvenience caused.

Incident: Vodafone Circuits

Posted: Wednesday, January 24th, 2018 at 00:19 by Mark Yardley

We are currently experiencing a loss of connectivity for a number of Vodafone circuits. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Richard Partridge says:

    Service was restored at approximately 02:30 this morning. It appears that this incident related to planned maintenance being carried out by Vodafone. We do not appear to have received pro-active notification for the work being carried out. This is something that will be raised with Vodafone for further investigation.

    We apologise for the inconvenience caused by this.

Incident: Bristol

Posted: Monday, January 22nd, 2018 at 10:12 by Jamie Smith

We have just experienced a loss of service to our Bristol node. Engineers are currently investigating and further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Service has been restored. Engineers are still investigating to determine a cause.

Incident: Vodafone Circuits

Posted: Thursday, January 18th, 2018 at 12:30 by Mark Yardley

We are currently experiencing a loss of connectivity for a number of Vodafone circuits. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Mark Yardley says:

    This incident has been escalated with Vodafone and their engineers are being dispatched to site to investigate. Further updates to follow. This incident has been raised to senior management at Vodafone due to this handoff being affected by previous issues.

  2. Scott Morgan says:

    We have had confirmation that Vodafone field engineers will be onsite between 15:15 and 15:30. This is following extensive diagnostics indicating a potential fault with the Vodafone network terminating equipment.

    We are in constant communication with Vodafone and further updates will follow as and when they become available.

  3. Adam Heath says:

    We have just received confirmation from the Vodafone engineer that they have reached the site and are about to begin their investigations. Further updates will follow once available.

  4. Scott Morgan says:

    We have been notified by the onsite engineer that he is currently in the process of replacing the Vodafone network termination hardware.

    We have been advised that this should be completed by approximately 17:15.

    We will continue to update as necessary.

  5. Scott Morgan says:

    Service has been restored since the Vodafone network termination hardware was replaced at 17:00.

    We will be liaising with Vodafone post incident and a full RFO will be provided.

  6. David Brewis says:

    A report is now available for this incident, which can be found here.

Incident: Outgoing calls

Posted: Tuesday, January 16th, 2018 at 09:38 by David Brewis

We are currently experiencing disruption to a small number of customers for outbound calls on the VoIP platform. Engineers are currently investigating for a cause and updates will be provided as and when they become available.

  1. David Brewis says:

    Service has now been restored to outgoing calls, if customers continue to experience issues please call into the help desk whilst we will continue to monitor.

    We apologise for any inconvenience caused by this incident.

Incident: TalkTalk Circuits

Posted: Friday, January 12th, 2018 at 06:47 by Richard Partridge

We are currently experiencing a loss of connectivity to a small number of TalkTalk leased line circuits. Engineers are currently investigating and updates will be provided as and when they become available.

  1. Richard Partridge says:

    Engineers have identified and resolved the problem. Customers still experiencing issues are advised to reboot their routers before contacting the help desk.

    We apologise for any inconvenience caused by this incident.

  2. David Brewis says:

    A report is now available for this incident which can be found here.

Incident: Cardiff, Glasgow & Bristol Edge Devices

Posted: Monday, November 27th, 2017 at 09:32 by Jamie Smith

We have just experienced an unexpected simultaneous reload of our edge devices in Cardiff, Glasgow & Bristol. This resulted in a brief loss of service for directly terminated leased line customers.

Previous investigations suggest this was due to a bug in the IOS software on these devices. We had preemptively loaded new software onto these devices in preparation for a controlled and scheduled reload. These devices are now running the alternative software following this unexpected reload. We will continue to monitor these devices. All services are now restored.

We apologise for any inconvenience this may have caused.