Archive for the ‘Incidents’ Category

Incident: Inbound calls

Posted: Friday, September 20th, 2019 at 10:25 by David Brewis

We are aware of an issue affecting inbound calls into the Telford office, all other systems remain operational.  We are currently investigating and will update shortly.

  1. David Brewis says:

    Service has restored, however whilst we investigate please consider this at risk

Incident: Vodafone Circuits

Posted: Wednesday, August 21st, 2019 at 10:25 by Jamie Smith

We are currently seeing a number of Vodafone circuits a down. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Jamie Smith says:

    Services have now been restored to the affected handoff. We are continuing to liaise with Vodafone to determine the cause of this outage

Incident : VoIP calls

Posted: Friday, August 16th, 2019 at 12:19 by david labouchardiere

We are aware of an outage affecting calls on the VoIP platform at approximately 11:00 today. Service is now restored and our engineers are investigating the underlying cause.

Please ask any affected customers to reboot their equipment if they continue to experience issues.

We apologise for any inconvenience caused.

Edinburgh connectivity

Posted: Friday, August 9th, 2019 at 08:33 by Alistair Patterson

We are currently aware of a number of City Fibre circuits in Edinburgh have lost connectivity. Engineers are currently investigating. Updates will be provided as soon as available.

  1. Jamie Smith says:

    Engineers are currently still Investigating the Issue affecting customers terminating on this Core Device, a reboot of the device has taken place in an effort to restore services. Further updates to follow as and when available.

  2. Jamie Smith says:

    Engineers have now restored connectivity to affected customers, any customers still experiencing issues should perform Initial troubleshooting before contacting our support department.

    We apologise for any Inconveniences caused by this outage.

Incident: PWAN Connectivity

Posted: Wednesday, August 7th, 2019 at 11:14 by Jordan Coleman

We are aware of an incident affecting some PWAN connectivity. Engineers are already engaged and are investigating.

Further updates will be provided as and when they become available.

  1. Jordan Coleman says:

    Connectivity has now been restored. We apologise for any inconvenience caused.

  2. David Brewis says:

    Reason For Outage: PWAN Connectivity

    Start: 7-Aug-2019 10:58
    Finish: 7-Aug-2019 11:13

    Impact: PWAN connectivity experienced a total loss of service (Primary and Backup) due to network loop affecting connectivity. Non PWAN connectivity was not affected.

    Root cause: Standard changes being implemented for a customer PWAN created unexpected impact due to pre-existing, redundant configuration on edge devices, triggering spanning tree to block connectivity to prevent a network loop.

    Additional actions:
    1. Edge devices config audit to check for, and if found remove, any redundant configuration under a maintenance window.
    2. Monitoring system event creation review to improve speed of identification of spanning tree activity.

Incident: Equinix PoP

Posted: Tuesday, August 6th, 2019 at 16:40 by Jordan Coleman
We are currently experiencing a loss of connectivity to our equipment in our Equinix POP. Engineers are currently investigating. Further updates will be provided as they become available.
  1. Jordan Coleman says:

    All services are now restored and we apologise for any inconvenience caused.

Incident: slough.core

Posted: Wednesday, July 31st, 2019 at 10:49 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Slough POP. Engineers are already investigating. Further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Initial indications believe this to be a power related issue within the PoP, we have raised this with the supplier and currently awaiting a response.

    Further updates to be provided as and when they are available

Incident: wolv6.core

Posted: Thursday, July 25th, 2019 at 12:15 by Richard Partridge

We have experienced a brief loss of connectivity to wolv6.core. Engineers are currently investigating and further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    This incident is still under investigation. Further information will be provided as and when it becomes available.

  2. This outage has been attributed to our third party supplier, we are still awaiting an update as to what caused the drop. We will try and provide further information by 16:30 today.

  3. We have experienced another brief loss of connectivity to wolv6.core. This will be investigated as part of the previous outage and updates provided as soon as we have information back from the third party supplier.

  4. Our supplier has identified a fibre disturbance on their network between Bristol & London. The signal degrade has cleared at this time & a full investigation is under way. We are awaiting further details to form a more detailed RFO.

  5. Upon escalation of this issue with our supplier we have not received a detailed explanation for this outage and therefore a sufficient RFO can not currently published. We have asked for further investigation and hope to receive a more detailed explanation in the coming days.

  6. Our provider has come back to say this blip was due to a 3rd party contractor working on some fibre in the same pit as ours and inadvertently knocked / bent the bundle were on causing signal loss momentarily. They sent out an engineer to make sure the fibre wasn’t damaged and have confirmed it wasn’t.

Incident: VoIP registration

Posted: Tuesday, June 11th, 2019 at 09:34 by David Brewis

We are experiencing an issue with phone registration on SIP and Hosted accounts ordered from myuserportal, which is under investigation. Further updates will be provided by 10:30.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. David Brewis says:

    Service has been restored to users, if you continue to experience a problem we ask affected handsets are rebooted prior to calling.

    We apologise for any inconvenience caused.

Incident: Vodafone Hub Outage

Posted: Monday, May 20th, 2019 at 03:33 by William George

Following on from a Vodafone maintenance our monitoring has alerted us to a number of Vodafone circuits that are currently out of service. Our initial diagnostics indicate that the circuits are related to a single handoff point between ourselves and Vodafone. Our engineers are investigating and are engaged with the supplier and further updates will be provided via this site.

We apologise to any customers that are impacted and want to reassure you that we are working to return full service as soon as possible.

  1. William George says:

    Vodafone have confirmed they see the issue and are in the process of arranging a resource to go to site to address it. Further updates will be provided as they are available.

  2. David Brewis says:

    Engineers from the supplier are already on site investigating cause for disruption to service and will update with their findings. As this progresses updates will be posted for your notice. Please expect the next update by 10:20.

  3. David Brewis says:

    Investigations from field engineers have identified an equipment fault, work has begun with hardware vendors to replace and resolve service as soon as possible.

    We apologise for the ongoing disruption whilst this incident progresses.

    Please expect the next update by 11:20

  4. David Brewis says:

    Following contact by Vodafone to their hardware vendor, a replacement component has been shipped to site with the expected arrival time of 11:00. We are awaiting confirmation of this delivery to begin works for replacement to restore service.

    Please expect the next update by 12:20.

  5. David Brewis says:

    The replacement unit has been installed and initial troubleshooting by Vodafone demonstrates the incident has been resolved.

    We are completing our own triage before confirming the engineer can leave site.

    Please expect a further update by 12:30.

  6. David Brewis says:

    We can confirm the handoff to Vodafone has returned to working service and the engineer stood down.

    If your service remains offline please contact the helpdesk following a reboot of your devices.

    We apologise for any disruption caused by this incident and will continue discussions with Vodafone for root cause analysis.