Archive for the ‘Uncategorized’ Category

Incident: cardiff.core

Posted: Tuesday, October 10th, 2017 at 18:08 by David Derrick

We have lost connectivity to our Cardiff router. Initial investigation suggests it is rebooting. We will continue to investigate and update when we know more.

  1. David Derrick says:

    The router has finished reloading and services are restored. We will continue to investigate the cause.

  2. Richard Partridge says:

    The cardiff.core router appears to have sustained another unexpected reload. Engineers are currently investigating. Further information will follow shortly.

  3. Richard Partridge says:

    Engineers are being dispatched to Cardiff to continue the investigation. Due to the number of recent failures, we are also planning to replace the core router whilst on site, which was originally planned for next week.

    Once engineers are on site and have assessed the situation, a notification outlining an emergency maintenance window will be published. The site should be considered at-risk until otherwise notified.

  4. Richard Partridge says:

    Engineers are on site and have concluded their initial investigation. Based on our findings we are currently proceeding with our plan to replace the core router this evening. A downtime window will be announced in due course.

  5. Richard Partridge says:

    We are planning to replace the Cardiff core router between 18:00 and 20:00 this evening. The core itself should be back online within 30 minutes during the window, however individual services will take longer to restore as they are migrated in turn from one core to the other.

  6. Jake Turner says:

    Further investigation on site within the maintenance window highlighted a fault with the power infrastructure that will be addressed by our colocation provider. We have taken measures to prevent further outages in the interim. As such the core router was not swapped.

Kingston Node

Posted: Thursday, September 28th, 2017 at 13:59 by David Brewis

We are currently noticing intermittent periods of packet loss on traffic passing through our Kingston node, affecting only ADSL services. This has been raised with BT for further investigation.

We will provide further information when it becomes available.

  1. David Brewis says:

    We have seen the levels return to normal but continue to discuss the case with BT

  2. David Brewis says:

    Following investigations an issue has been identified and steps have begun to to further identify the cause allowing for resolution.

  3. David Brewis says:

    Following changes on the incoming connection made by the supplier, we have not seen a repeat of the issue. Monitoring and investigations continue with updates published as made available.

Incident: Vodafone Circuits

Posted: Thursday, September 28th, 2017 at 08:32 by Alistair Patterson

We are currently aware of an issue causing a loss of connectivity for a number of Vodafone based Ethernet circuits. Our initial diagnostics indicate the issue to be on one of Vodafone’s hubs. A fault has already been raised with Vodafone and is under investigation.

Further updates will follow as they become available.

  1. Alistair Patterson says:

    Vodafone have confirmed engineers are expected to arrive at the fault site shortly to continue investigations.

    Further updates will be posted as soon as available.

  2. Alistair Patterson says:

    Vodafone engineers have resolved the issue and all services have been restored. Circuits should still be considered at risk until Vodafone clear the incident.

    Further information on the root cause and fix will be made available when received from Vodafone.

  3. Alistair Patterson says:

    This incident has now been cleared. Vodafone confirm an issue with a line card was resolved by their engineers.

    We apologise for the inconvenience caused during this incident.

At-Risk: Kingston to Slough fibre link maintenance

Posted: Wednesday, August 16th, 2017 at 07:38 by Iain Beveridge

21-Aug-2017 02:00 GMT to 21-Aug-2017 06:00 GMT

During the above window one of our fibre suppliers will be carrying out maintenance on one of our links between Kingston and Slough. Whilst the actual downtime is expected to be less, the full 4 hrs may be required to account for any unforeseen issues. This work is not expected to be service affecting, however, you may see unusual paths and increase in latency while traffic is automatically rerouted via alternate routes.

Email Disruption

Posted: Thursday, July 13th, 2017 at 10:15 by Richard Partridge

We are currently receiving reports that a small number of DSL customers are experiencing difficulty when trying to send or receive emails. Our engineers are currently investigating and further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    Engineers are still investigating the email issue at this time. Further updates will be provided over the course of the day.

  2. Richard Partridge says:

    Engineers are continuing their investigation. Further updates will be provided over the course of the day

  3. Richard Partridge says:

    Engineers have now been able to recreate the symptoms being reported and are investigating a possible root cause of the problem. Further updates will follow in due course.

  4. Richard Partridge says:

    Adjustments have been made to try and resolve this problem, however we will need to return to normal bandwidth levels in order to determine their effectiveness. Further updates will be provided tomorrow morning once the impact of these changes have been ascertained.

  5. Richard Partridge says:

    “Yesterday’s adjustments have had a positive affect, but haven’t entirely resolved the problem. We’re continuing to work on this and will provide further updates over the course of the day.”

  6. Richard Partridge says:

    No further issues have been reported over the course of the afternoon. We will closely monitor the situation over the course of the weekend and advise of any further occurrences of this issue accordingly.

Incident: kingston.core

Posted: Friday, March 31st, 2017 at 13:05 by Richard Partridge

We are currently investigating a loss of connectivity to kingston.core. Engineers are already investigating. Further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    We’ve received confirmation from the power provider in the region that there is a power problem affecting the area. Power is expected to be restored between 14:30 and 15:30.

    Further updates will follow.

  2. Neil Watson says:

    Details of the local power outage can be seen here. The reference is INCD-314299-U.

    This issue will be affecting all customers that connect via our Kingston node, including both Ethernet and Broadband services. Apologies for the disruption, we will update further as soon as more information is available.

  3. Richard Partridge says:

    Connectivity has been restored however the site should still be considered at-risk until we receive official confirmation that the power outage has been resolved.

At-Risk: Cambridge to Telehouse fibre link maintenance

Posted: Thursday, March 16th, 2017 at 14:20 by Iain Beveridge
30-Mar-2017 02:00 GMT to 30-Mar-2017 06:00 GMT
During the above window one of our fibre suppliers will be carrying out maintenance on one of our links between Cambridge and Telehouse East. Whilst the actual downtime is expected to be less, the full 4 hrs may be required to account for any unforeseen issues. This work is not expected to be service affecting, however, you may see unusual paths and increase in latency while traffic is automatically rerouted via alternate routes.

Incident: Telford Power Loss

Posted: Wednesday, March 15th, 2017 at 08:42 by Richard Partridge

We have just experienced a loss of power to the Telford Data Centre. Generator and UPS protection kicked in as expected however some internal systems remain offline. Engineers are currently investigating.

  1. Richard Partridge says:

    Mains power and the affected internal systems have now been restored. The power loss resulted from an issue with the power supplier in the local area.

  2. Richard Partridge says:

    We’ve sustained a further loss of mains power to the local area and are currently running on generator backup. Further updates will be provided as and when they become available.

  3. Richard Partridge says:

    Mains power has once again been restored. The loss of power to the local area was another supplier failure which has now been rectified.

Telephone System Issue

Posted: Tuesday, February 7th, 2017 at 11:16 by Adam Heath

We are aware of a problem with our telephone system and are currently experiencing difficulties making and receiving calls.  We are working with our provider to ensure functionality is restored at the earliest opportunity.

  1. Adam Heath says:

    Functionality to our telephone system was restored a short while ago. We are continuing to monitor from our side.

Incident: Vodafone Circuits

Posted: Friday, January 13th, 2017 at 18:51 by Richard Partridge

We are currently aware of and investigating an issue causing a loss of connectivity for a small number of Vodafone based connections. Engineers are working on the problem and further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Initial investigations suggest this incident may reside within the Vodafone network, so we have raised this issue to them for investigation. The circuits themselves appear to have returned to service, however as no cause has been identified at this time, the affected circuits should be considered at risk for the time being.

  2. Richard Partridge says:

    Vodafone have confirmed that the incident related to failed hardware within their network. A solution has been put in place and normal service should now be considered restored.

    We apologise for any inconvenience caused by this incident.