Posts Tagged ‘Incidents’

Mirada Monitoring

Posted: Wednesday, February 21st, 2018 at 14:46 by Jamie Smith

We are currently aware of an issue affecting our Mirada monitoring platform, this will be causing Mirada to falsely show all devices as down.

Engineers are currently investigating.

  1. Jake Turner says:

    Engineers have resolved the problems with the Mirada monitoring platform.

    Apologies for any inconvenience caused.

Incident: Vodafone Circuits

Posted: Wednesday, January 24th, 2018 at 00:19 by Mark Yardley

We are currently experiencing a loss of connectivity for a number of Vodafone circuits. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Richard Partridge says:

    Service was restored at approximately 02:30 this morning. It appears that this incident related to planned maintenance being carried out by Vodafone. We do not appear to have received pro-active notification for the work being carried out. This is something that will be raised with Vodafone for further investigation.

    We apologise for the inconvenience caused by this.

Incident: Bristol

Posted: Monday, January 22nd, 2018 at 10:12 by Jamie Smith

We have just experienced a loss of service to our Bristol node. Engineers are currently investigating and further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Service has been restored. Engineers are still investigating to determine a cause.

Incident: Vodafone Circuits

Posted: Thursday, January 18th, 2018 at 12:30 by Mark Yardley

We are currently experiencing a loss of connectivity for a number of Vodafone circuits. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Mark Yardley says:

    This incident has been escalated with Vodafone and their engineers are being dispatched to site to investigate. Further updates to follow. This incident has been raised to senior management at Vodafone due to this handoff being affected by previous issues.

  2. Scott Morgan says:

    We have had confirmation that Vodafone field engineers will be onsite between 15:15 and 15:30. This is following extensive diagnostics indicating a potential fault with the Vodafone network terminating equipment.

    We are in constant communication with Vodafone and further updates will follow as and when they become available.

  3. Adam Heath says:

    We have just received confirmation from the Vodafone engineer that they have reached the site and are about to begin their investigations. Further updates will follow once available.

  4. Scott Morgan says:

    We have been notified by the onsite engineer that he is currently in the process of replacing the Vodafone network termination hardware.

    We have been advised that this should be completed by approximately 17:15.

    We will continue to update as necessary.

  5. Scott Morgan says:

    Service has been restored since the Vodafone network termination hardware was replaced at 17:00.

    We will be liaising with Vodafone post incident and a full RFO will be provided.

  6. David Brewis says:

    A report is now available for this incident, which can be found here.

Incident: Outgoing calls

Posted: Tuesday, January 16th, 2018 at 09:38 by David Brewis

We are currently experiencing disruption to a small number of customers for outbound calls on the VoIP platform. Engineers are currently investigating for a cause and updates will be provided as and when they become available.

  1. David Brewis says:

    Service has now been restored to outgoing calls, if customers continue to experience issues please call into the help desk whilst we will continue to monitor.

    We apologise for any inconvenience caused by this incident.

Incident: TalkTalk Circuits

Posted: Friday, January 12th, 2018 at 06:47 by Richard Partridge

We are currently experiencing a loss of connectivity to a small number of TalkTalk leased line circuits. Engineers are currently investigating and updates will be provided as and when they become available.

  1. Richard Partridge says:

    Engineers have identified and resolved the problem. Customers still experiencing issues are advised to reboot their routers before contacting the help desk.

    We apologise for any inconvenience caused by this incident.

  2. David Brewis says:

    A report is now available for this incident which can be found here.

Incident: Cardiff, Glasgow & Bristol Edge Devices

Posted: Monday, November 27th, 2017 at 09:32 by Jamie Smith

We have just experienced an unexpected simultaneous reload of our edge devices in Cardiff, Glasgow & Bristol. This resulted in a brief loss of service for directly terminated leased line customers.

Previous investigations suggest this was due to a bug in the IOS software on these devices. We had preemptively loaded new software onto these devices in preparation for a controlled and scheduled reload. These devices are now running the alternative software following this unexpected reload. We will continue to monitor these devices. All services are now restored.

We apologise for any inconvenience this may have caused.

Incident: Vodafone Circuits

Posted: Thursday, November 9th, 2017 at 16:14 by Mark Yardley

We are currently experiencing a loss of connectivity for a number of Vodafone circuits. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Adam Heath says:

    Our supplier has confirmed that they have engineering resource currently investigating the issue. We have also reached out to our Service Managers to ensure this is resolved at the earliest opportunity. Further updates will follow, once they are available.

  2. Adam Heath says:

    Vodafone identified a potential issue with some of their equipment a short while ago. Engineering resource is being arranged to attend the location. Further updates will follow.

  3. Richard Partridge says:

    Service appears to be restored as of approximately 01:30 this morning. We’re still waiting for a confirmation that the fault has been rectified and an RFO from Vodafone.

  4. Adam Heath says:

    A report is now available for this incident which can be found here.

Incident: Vodafone Circuits

Posted: Friday, October 6th, 2017 at 19:34 by Mark Yardley

We are aware of a loss of connectivity for a number of Vodafone Ethernet circuits. Initial diagnostics show the affected circuits are delivered on the same NNI that has recently suffered disruption of service. We are liaising with our suppliers to get the issue investigated further and resolved as quickly as possible. Further updates will be provided as they become available.

  1. David Brewis says:

    Following investigations, Vodafone are arranging to dispatch engineers tonight.

    Further updates to follow once engineers have attended site.

  2. Mark Yardley says:

    Vodafone engineers have attended their site to replace suspect faulty equipment. We currently see the affected services have been restored but should still be considered at risk until our suppliers have advised fault is cleared.

  3. Mark Yardley says:

    These services have again lost connectivity and Vodafone are investigating further. Updates to be provided once available.

  4. David Brewis says:

    Vodafone have escalated the incident and are dispatching engineers to restore service. Further updates to follow as made available.

  5. David Brewis says:

    Connections are restoring to service following hardware replacement, until fault closure consider service at risk.

  6. David Brewis says:

    After 48 hours of monitoring, we have received correspondence from the supplier to show the fault is now closed.

  7. David Brewis says:

    An incident report for this issue is now available here

Incident: Cardiff

Posted: Sunday, October 1st, 2017 at 10:25 by Jake Turner

Shortly after 10:00 we experienced an unexpected reload of the cardiff.core router which on first examination appears to be the result of a temporary loss of power. Directly connected customers would have experienced a brief drop in connectivity. Traffic passing through this router will have diverted across alternative paths. Connectivity to most services has been restored and engineers are investigating.