Archive for the ‘Incidents’ Category

Incident: Telford Power Outage – services at risk

Posted: Wednesday, November 14th, 2018 at 03:08 by Neil Watson

There is currently a power outage that is affecting our Telford HQ and our datacentres. As a consequence, we are currently running on our back up generator power. All services that are delivered from this location must, therefore, be considered to be at risk. This a mains power failure that is being worked on by the supplier and they have estimated a return in service at approximately 03:30hrs.

  1. Neil Watson says:

    Power has now been restored and has been stable for a short while. We are no longer running on generator power, and therefore the site is considered to be fully operational again.

Incident: edinburgh.core

Posted: Wednesday, October 17th, 2018 at 17:01 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Edinburgh POP. Engineers are already investigating. Further updates will be provided as and when they become available

  1. Jamie Smith says:

    Connectivity has been restored. Initial investigation indicates a power disruption in one of our cabinets at the site. This has been raised to our supplier for further investigation.

    We apologise for the inconvenience caused by this interruption.

Incident: sheffield.core

Posted: Sunday, October 14th, 2018 at 11:53 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Sheffield POP. Engineers are already investigating. Further updates will be provided as and when they become available

  1. Jamie Smith says:

    Connectivity has been restored. Initial investigation indicates a power disruption in one of our cabinets at the site. This has been raised to our supplier for further investigation.

    We apologise for the inconvenience caused by this interruption.

Incident: Columbo DSL

Posted: Wednesday, September 26th, 2018 at 11:42 by Adam Heath

We are aware of an issue causing loss of service to DSL circuits which connect to our Columbo node.  Early investigations suggest this may be a BT issue, based off reports that other Service Providers may also be affected.  We’re continuing to investigate with the supplier and will provide further information when we have it.

  1. Adam Heath says:

    BT have confirmed they have an open incident and are in the process of completing diagnostics. They haven’t provided a root cause or estimated time of repair as of yet. We’ll continue to update upon receipt of further information.

  2. Adam Heath says:

    We can see that service was restored on the majority of circuits at approximately 1pm. We are still in correspondence with BT in order to obtain information on the cause, which we will then provide here.

Incident: Vodafone Circuits

Posted: Thursday, September 13th, 2018 at 14:37 by Jamie Smith

We are currently seeing a number of Vodafone circuits a down. Initial diagnostics indicate the circuits are related to a single Vodafone handoff. Engineers are investigating and engaging with our supplier. Further updates will be provided when available.

  1. Jamie Smith says:

    Services have now been restored to the affected handoff. We are continuing to liaise with Vodafone to determine the cause of this outage

  2. David Brewis says:

    In speaking with Vodafone service teams, we have received confirmation emergency proactive work was conducted to replace a faulty 10G line card to ensure the node was stabilised.

    We do not expect to see further disruption following this work.

Incident: VoIP

Posted: Wednesday, September 12th, 2018 at 09:46 by David Brewis

We are aware of an interruption to service affecting inbound calls on our Enrich platform. Our teams are investigating and will update hourly.

We apologise for any inconvenience during this time.

  1. David Brewis says:

    Whilst investigations continue, a small set of users appear to be working. We are asking for users to retest.

  2. David Brewis says:

    Service has been restored to users on the Enrich platform, if you continue to experience a problem we ask affected handsets are rebooted prior to calling.

    We apologise for any inconvenience caused.

Incident: VoIP

Posted: Tuesday, September 4th, 2018 at 14:16 by Adam Heath

We are aware of a brief interruption to both SIP and Hosted VoIP services at approximately 14:00.  Services appear to be restored but our teams are aware and currently investigating.  We’ll provide further information when this becomes available.

  1. David Brewis says:

    On 04 September 2018 at 13:55 our monitoring systems identified an issue affecting one our media servers resulting in intermittent issues with calls terminating on this node.

    The issue was caused by a loop in logging due to an issue with a call queue’s configuration, this resulted in the media servers logging engine crashing and consuming all threads preventing the ability to serve calls.

    A full review of all other queues has been conducted and no further issues have been identified.

    We apologise for any inconvenience caused by this interruption.

Incident: Cardiff

Posted: Monday, September 3rd, 2018 at 13:39 by Adam Heath

We are aware of an issue affecting services on our Cardiff node.  Our NOC are aware and currently investigating.  We’ll provide updates when further information is available.

  1. Adam Heath says:

    We can see that services have now been restored. Our investigations suggest a power disruption in one of our cabinets at the location. This will be raised to our supplier for further analysis and an explanation.

    We apologise for any inconvenience caused by this interruption.

Incident: sheffield.core

Posted: Tuesday, August 21st, 2018 at 14:49 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Sheffield POP. Engineers are already investigating. Further updates will be provided as and when they become available.

  1. Gurinder Bansal says:

    Connectivity has been restored. Initial investigation indicates a power disruption in one of our cabinets at the site. This has been raised to our supplier for further investigation.

    We apologise for the inconvenience caused by this interruption.

Incident: cardiff.core

Posted: Monday, July 30th, 2018 at 22:20 by Jamie Smith

We are currently experiencing a loss of connectivity to our equipment in our Cardiff POP. Engineers are already investigating. Further updates will be provided as and when they become available.

  1. Jamie Smith says:

    Connectivity has been restored. Engineers are still continuing to Investigate, updates will be communicated as and when they become available.

    We apologise for any Inconvenience this may have caused