Archive for the ‘Incidents’ Category

Incident: DSL authentication

Posted: Friday, February 7th, 2020 at 11:23 by david labouchardiere

We are currently investigating a problem with authentication on DSL sessions. The issue will be preventing any new sessions from establishing but any circuits that are currently connected will remain online unless they are disconnected or rebooted.

We will provide further updates as soon as they are available

  1. david labouchardiere says:

    We have had confirmation that this issue is now resolved. Please ask any affected customers to reboot their router and contact the service desk should they have any further problems.

    We apologise for any inconvenience caused by this incident.

Incident: Kirklees

Posted: Tuesday, January 28th, 2020 at 11:00 by Daniel Jones

We are currently aware of an issue affecting circuits within the Kirklees area. There are ongoing investigations in relation to this issue and we will keep you informed of progress.

  1. Daniel Jones says:

    Services have now been restored. We are still investigating the root cause.

Incident: VoIP calls

Posted: Thursday, January 23rd, 2020 at 11:43 by david labouchardiere

We have been made aware that one of our VoIP upstream carriers has seen a drop in call volumes and has raised an incident to investigate further. This will affect calls on hosted and sip only accounts. We will provide further updates once they become available

  1. david labouchardiere says:

    Our upstream carrier has implemented a fix in relation to this incident and are monitoring to confirm if the issue is resolved.

Incident: Edinburgh

Posted: Tuesday, January 14th, 2020 at 09:45 by David Brewis
We are currently aware of an issue affecting circuits within the Edinburgh area. There are ongoing investigations in relation to this issue and we will keep you informed of progress.
  1. David Brewis says:

    Investigations highlight this only impacted leased line services in Edinburgh. We are starting to see services restore as we continue investigations to find the root cause.

  2. David Brewis says:

    After a state of monitoring all services restored and remained stable by 10:15. We apologise for any inconvenience this caused.

Incident: camb.core

Posted: Thursday, January 9th, 2020 at 10:20 by Jordan Coleman

We have experienced a brief loss of connectivity to our equipment at our Cambridge PoP. Engineers are currently investigating, updates to follow as soon as available.

Incident: Mirada Portal

Posted: Friday, December 27th, 2019 at 11:20 by David Brewis

Engineers are investigating an issue which is currently rendering the Mirada portal unavailable. Further updates will be provided as and when they become available.

Incident: PWAN DSL Connectivity

Posted: Monday, November 18th, 2019 at 23:34 by Jordan Coleman

We are aware of an incident affecting some PWAN ADSL/VDSL services. Engineers are already engaged and are investigating.

Further updates will be provided as and when they become available.

  1. Jordan Coleman says:

    We now believe this issue to be resolved. Any remaining customers that are affected are advised to reboot their router before calling into the Service Desk.

Resolved Incident: VoIP calls

Posted: Thursday, October 10th, 2019 at 15:35 by david labouchardiere

Good afternoon.

We are aware of an issue that was affecting some users on our VOIP platform. We believe the issue has now been resolved and request any users affected to now retest the service.

Please accept our apologies for any inconvenience caused.

Incident: VoIP calls

Posted: Tuesday, October 1st, 2019 at 20:10 by David Brewis

Good evening.

We are currently aware of an issue affecting calls on the VoIP platform. We are investigating the problem and will provide further updates as soon as possible.

Please accept our apologies for any inconvenience caused.

  1. David Brewis says:

    Engineers continue to investigate with the aim to identify the cause disrupting service across our VoIP platform. We have been able to replicate the issue internally assisting with troubleshooting steps.

    We will continue to update with updates and progress as troubleshooting continues.

  2. Robin Posadas says:

    Engineers are continuing to investigate the problem and we will continue to update as further troubleshooting continues.

  3. Robin Posadas says:

    Engineers have now confirmed service has been restored. Any customers still experiencing issues should reboot their equipment before contacting our support department.

    We apologise for any Inconveniences caused by this outage

Incident: Service Outage

Posted: Friday, September 27th, 2019 at 16:09 by Adam Heath

Wednesday 25th September 22:30 – Thursday 26th September 01:04

A change to standardise access lists on some of our core network devices created an unexpected impact whereby some core devices were unable to learn/send routes from our route reflectors.

We have raised a vendor case and updated our regression test procedures to avoid future issues. An RFO will follow in due course.

  1. Incident Management says:

    Impact: Services delivered through some core devices may have experienced a total loss of service.

    Root Cause: In an effort to standardise our management plane access lists across devices through a non-service affecting change, an unforeseen and adverse effect caused the control plane of each device to react to the change. This caused the core devices to lose their established BGP sessions to our route reflectors. Once the issue was identified, roll back was initiated to the pre-change baseline and service restored. We are continuing to liaise with our vendor to gather the root cause of the outage.

    Additional Actions: A review of the failed change has been carried out and actions to prevent recurrence of this unexpected impact and service outage have been identified and will be progressed.