Posts Tagged ‘Incidents’

Incident: Glasgow PoP

Posted: Thursday, December 31st, 2015 at 04:15 by Mark Yardley

We are currently experiencing and issue at our Glasgow PoP. Engineers are investigating, further information to follow when available.

  1. Mark Yardley says:

    The cause of this incident has been identified as power loss to the location. However, power has now been restored and services provided by this location are again operational. Further investigation will be undertaken to identify the reason for power loss with our suppliers.

  2. Mark Yardley says:

    Power to this location has been lost again. We are still speaking to suppliers to investigate the cause.

  3. Mark Yardley says:

    Suppliers have now confirmed that there is a known power issue at this location and they currently have a specialist team on site to rectify the problem. Further updates will be provided as more information is available.

  4. Mark Yardley says:

    Our suppliers have advised that engineering work resolved the power issue earlier this morning.

VoIP Disruption

Posted: Saturday, December 5th, 2015 at 16:08 by Mark Yardley

We are currently aware of an issue affecting some users of our VoIP platform. Engineers are investigating and a further update will be provided when available.

  1. Mark Yardley says:

    Engineers have now rectified the issue and can see calls are establishing normally. We will continue to monitor to ensure there are no further problems.

Incident: VoIP Disruption

Posted: Tuesday, November 17th, 2015 at 15:01 by Ben Bliss

We are currently aware of an issue which is affecting outbound calls across our VoIP platform.
Initial investigation shows this to be down to one of our upstream providers. We are now routing calls to alternate providers while this is being investigated.
Further information will be provided as and when it becomes available.

  1. Ben Bliss says:

    We are currently awaiting further information from our affected upstream provider.

    While currently not proven, indications are that the problems being experienced are as a direct result of the power issues currently being experienced within Telecity.

    We are continuing to monitor the situation, and will provide a further update as and when additional information is available.

  2. Ben Bliss says:

    Further to the last post, connectivity to the affected upstream provider was re-established shortly before 5pm.

    However, testing has indicated there are still issues with their systems, so we have not re-established call routing to them at this time.

    We will continue to monitor the situation, and as soon as we receive information to advise the issues within their network are resolved, we will provide a further update.

  3. Ben Bliss says:

    Following further investigation and the resolution of various issues within the upstream suppliers network caused by the earlier power failure in Telecity, we are satisfied that we can now start sending calls via the affected carrier.

    As such, we have recommenced sending calls to the affected provider, and all appears to be working correctly.

    We will continue to monitor the situation and provide further updates if necessary.

Incident: Vodafone Circuits

Posted: Tuesday, November 17th, 2015 at 13:47 by Mark Yardley

We are currently aware of an issue affecting a number of Vodafone based circuits. We are working with our supplier who are currently investigating.

We will continue to work with our suppliers and publish updates as they become available.

  1. Mark Yardley says:

    The affected services have been now been restored. This incident initially appears to be a power issue at another location within Telecity Sov, however we are continuing to investigate with our suppliers.

  2. Mark Yardley says:

    Telecity have confirmed that this incident was caused by an isolated power issue. Telecity engineers are working to ensure the underlying issue is fully resolved. Until we receive confirmation from Telecity that all work has completed, previously affected services should be considered as at risk.

    Further updates will be provided as they are made available from our suppliers.

  3. Mark Yardley says:

    We are still currently awaiting an update from Telecity to confirm work has been completed and subsequently for an RFO report once that has been advised. For clarity, this power issue did not directly affect our equipment or location within Telecity, however it did impact a number of suppliers and customers who connect to us at this location.

Incident: Viper Hosting Platform

Posted: Friday, September 4th, 2015 at 08:54 by Richard Partridge

We are currently investigating an issue causing a loss of service affecting our Viper hosting platform.

Engineers are currently investigating and further updates will be provided within the hour.

  1. Richard Partridge says:

    Work has begun to restore service to the Vipers, and some Vipers are now operating normally. Further updates will follow shortly.

  2. Ryan Berrisford says:

    Engineers are continuing to work on the Viper platform, although the majority of the Viper servers are back in service.

    Further updates will follow shortly and we apologise for the inconvenience caused.

  3. Ryan Berrisford says:

    All services are now restored for Viper customers.

    Customers with an entadsl email address will see some missing historical email for the time being. This is being worked on and should be restored over the course of the afternoon.

    We apologise for the inconvenience caused.

Emergency Work: Telford UPS

Posted: Thursday, August 13th, 2015 at 13:05 by Mark Yardley

We have experienced a failure of one of the UPS devices that protects telford-dc3.core. We will shortly be undertaking work to replace the failed unit and restore full power redundancy. Services connected to or through telford-dc3.core should be considered at risk until the work is completed.

Further updates will be provided within the hour.

  1. Mark Yardley says:

    Engineers are currently progressing with the work. Further updates will follow shortly.

  2. Mark Yardley says:

    The work is now complete and UPS redundancy has been restored.

Phone System Disruption

Posted: Wednesday, May 6th, 2015 at 12:11 by Richard Partridge

We have become aware that one of our upstream carriers has just encountered an issue which resulted in intermittent disruption when trying to call our contact numbers. Our supplier has indicated that the problem should now be resolved.

Our apologies for any inconvenience caused.

Incident: Cardiff to Birmingham

Posted: Tuesday, March 31st, 2015 at 14:35 by Richard Partridge

We are aware of a fibre break affecting our Cardiff to Birmingham core link. Traffic is already re-routing around alternative paths. Our supplier is already working to resolve the problem however no ETR is available at this time.

Further updates will be provided as and when they become available.

  1. Phil Davies says:

    Our supplier has confirmed that the Fibre beaks in the Gloucester area which are affecting Virgin Media provided Circuits are also affecting the Cardiff to Birmingham core Link. Engineers are currently on site working on the repair, it is a major incident and we have been advised that work will continue through the night with no current estimated time of repair. Traffic is still re-routing around alternative paths.

  2. Phil Davies says:

    Service was restored at approximately 02.40am and remains stable. We will continue to monitor.

Incident: LONAP Peering

Posted: Wednesday, November 19th, 2014 at 15:11 by Richard Partridge

We’re aware of an issue within LONAP. We’ve temporarily disabled our peering to LONAP for the time being.

Incident: Interxion Blade Failure

Posted: Wednesday, November 12th, 2014 at 08:41 by Richard Partridge

We are currently investigating what appears to be a blade failure within our interxion.core.enta.net router.

Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Entanet engineers have been dispatched to Interxion with replacement hardware. We will update again once our engineers have arrived..

  2. Richard Partridge says:

    DSL customers affected by this incident are being re-routed however we’re receiving reports of packet loss. This is currently under investigation.

  3. Richard Partridge says:

    Further investigation has shown that DSL traffic is re-routing around the blade failure in Interxion. However, this is causing increased strain on equipment forming parts of the alternative paths currently being used to bypass the failed blade.

    Engineers are still en-route to Interxion with the aim of replacing the failed hardware, however we’re continuing to determine if there is anything that can be done in the meantime to alleviate the packet loss.

  4. Richard Partridge says:

    Engineers are still en-route and are expected to arrive on site shortly.

  5. Richard Partridge says:

    Our engineers have now arrived on site.

  6. Richard Partridge says:

    Work to replace the blade is currently ongoing.

  7. Richard Partridge says:

    The blade replacement hasn’t stabilised the issue, so we will be rebooting the Interxion core immediately.

  8. Richard Partridge says:

    Service appears to have been restored. We will be monitoring the situation closely.

  9. Richard Partridge says:

    An incident report on this event is currently being produced and will be made available as soon as possible.

  10. Neil Watson says:

    An incident report is now available and can be found here