Posts Tagged ‘Incidents’

Incident: VoIP Disruption

Posted: Tuesday, November 17th, 2015 at 15:01 by Ben Bliss

We are currently aware of an issue which is affecting outbound calls across our VoIP platform.
Initial investigation shows this to be down to one of our upstream providers. We are now routing calls to alternate providers while this is being investigated.
Further information will be provided as and when it becomes available.

  1. Ben Bliss says:

    We are currently awaiting further information from our affected upstream provider.

    While currently not proven, indications are that the problems being experienced are as a direct result of the power issues currently being experienced within Telecity.

    We are continuing to monitor the situation, and will provide a further update as and when additional information is available.

  2. Ben Bliss says:

    Further to the last post, connectivity to the affected upstream provider was re-established shortly before 5pm.

    However, testing has indicated there are still issues with their systems, so we have not re-established call routing to them at this time.

    We will continue to monitor the situation, and as soon as we receive information to advise the issues within their network are resolved, we will provide a further update.

  3. Ben Bliss says:

    Following further investigation and the resolution of various issues within the upstream suppliers network caused by the earlier power failure in Telecity, we are satisfied that we can now start sending calls via the affected carrier.

    As such, we have recommenced sending calls to the affected provider, and all appears to be working correctly.

    We will continue to monitor the situation and provide further updates if necessary.

Incident: Vodafone Circuits

Posted: Tuesday, November 17th, 2015 at 13:47 by Mark Yardley

We are currently aware of an issue affecting a number of Vodafone based circuits. We are working with our supplier who are currently investigating.

We will continue to work with our suppliers and publish updates as they become available.

  1. Mark Yardley says:

    The affected services have been now been restored. This incident initially appears to be a power issue at another location within Telecity Sov, however we are continuing to investigate with our suppliers.

  2. Mark Yardley says:

    Telecity have confirmed that this incident was caused by an isolated power issue. Telecity engineers are working to ensure the underlying issue is fully resolved. Until we receive confirmation from Telecity that all work has completed, previously affected services should be considered as at risk.

    Further updates will be provided as they are made available from our suppliers.

  3. Mark Yardley says:

    We are still currently awaiting an update from Telecity to confirm work has been completed and subsequently for an RFO report once that has been advised. For clarity, this power issue did not directly affect our equipment or location within Telecity, however it did impact a number of suppliers and customers who connect to us at this location.

Incident: Viper Hosting Platform

Posted: Friday, September 4th, 2015 at 08:54 by Richard Partridge

We are currently investigating an issue causing a loss of service affecting our Viper hosting platform.

Engineers are currently investigating and further updates will be provided within the hour.

  1. Richard Partridge says:

    Work has begun to restore service to the Vipers, and some Vipers are now operating normally. Further updates will follow shortly.

  2. Ryan Berrisford says:

    Engineers are continuing to work on the Viper platform, although the majority of the Viper servers are back in service.

    Further updates will follow shortly and we apologise for the inconvenience caused.

  3. Ryan Berrisford says:

    All services are now restored for Viper customers.

    Customers with an entadsl email address will see some missing historical email for the time being. This is being worked on and should be restored over the course of the afternoon.

    We apologise for the inconvenience caused.

Emergency Work: Telford UPS

Posted: Thursday, August 13th, 2015 at 13:05 by Mark Yardley

We have experienced a failure of one of the UPS devices that protects telford-dc3.core. We will shortly be undertaking work to replace the failed unit and restore full power redundancy. Services connected to or through telford-dc3.core should be considered at risk until the work is completed.

Further updates will be provided within the hour.

  1. Mark Yardley says:

    Engineers are currently progressing with the work. Further updates will follow shortly.

  2. Mark Yardley says:

    The work is now complete and UPS redundancy has been restored.

Phone System Disruption

Posted: Wednesday, May 6th, 2015 at 12:11 by Richard Partridge

We have become aware that one of our upstream carriers has just encountered an issue which resulted in intermittent disruption when trying to call our contact numbers. Our supplier has indicated that the problem should now be resolved.

Our apologies for any inconvenience caused.

Incident: Cardiff to Birmingham

Posted: Tuesday, March 31st, 2015 at 14:35 by Richard Partridge

We are aware of a fibre break affecting our Cardiff to Birmingham core link. Traffic is already re-routing around alternative paths. Our supplier is already working to resolve the problem however no ETR is available at this time.

Further updates will be provided as and when they become available.

  1. Phil Davies says:

    Our supplier has confirmed that the Fibre beaks in the Gloucester area which are affecting Virgin Media provided Circuits are also affecting the Cardiff to Birmingham core Link. Engineers are currently on site working on the repair, it is a major incident and we have been advised that work will continue through the night with no current estimated time of repair. Traffic is still re-routing around alternative paths.

  2. Phil Davies says:

    Service was restored at approximately 02.40am and remains stable. We will continue to monitor.

Incident: LONAP Peering

Posted: Wednesday, November 19th, 2014 at 15:11 by Richard Partridge

We’re aware of an issue within LONAP. We’ve temporarily disabled our peering to LONAP for the time being.

Incident: Interxion Blade Failure

Posted: Wednesday, November 12th, 2014 at 08:41 by Richard Partridge

We are currently investigating what appears to be a blade failure within our router.

Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    Entanet engineers have been dispatched to Interxion with replacement hardware. We will update again once our engineers have arrived..

  2. Richard Partridge says:

    DSL customers affected by this incident are being re-routed however we’re receiving reports of packet loss. This is currently under investigation.

  3. Richard Partridge says:

    Further investigation has shown that DSL traffic is re-routing around the blade failure in Interxion. However, this is causing increased strain on equipment forming parts of the alternative paths currently being used to bypass the failed blade.

    Engineers are still en-route to Interxion with the aim of replacing the failed hardware, however we’re continuing to determine if there is anything that can be done in the meantime to alleviate the packet loss.

  4. Richard Partridge says:

    Engineers are still en-route and are expected to arrive on site shortly.

  5. Richard Partridge says:

    Our engineers have now arrived on site.

  6. Richard Partridge says:

    Work to replace the blade is currently ongoing.

  7. Richard Partridge says:

    The blade replacement hasn’t stabilised the issue, so we will be rebooting the Interxion core immediately.

  8. Richard Partridge says:

    Service appears to have been restored. We will be monitoring the situation closely.

  9. Richard Partridge says:

    An incident report on this event is currently being produced and will be made available as soon as possible.

  10. Neil Watson says:

    An incident report is now available and can be found here

Virgin Media Leased Lines

Posted: Wednesday, August 27th, 2014 at 10:06 by Richard Partridge

We are currently aware of an incident on Virgin Media’s network which is causing a loss of service to several Virgin Media provided leased lines.

Virgin Media are currently investigating and updates will be provided as and when they become available.

  1. Richard Partridge says:

    Virgin Media have identified a problem with one of their core routers in Telehouse and they are continuing their investigation although no ETR is available as of yet.

  2. Richard Partridge says:

    Service appears to have been restored and stable for approximately 10 minutes now however no further information is available from Virgin Media as of yet. The affected circuits should be considered at risk until further information on this incident is available.

  3. Richard Partridge says:

    Service has remained stable and the incident is now considered closed. We are awaiting an incident report from Virgin Media which should be made available to us next week.


Posted: Wednesday, May 21st, 2014 at 16:39 by Richard Partridge

We’re currently investigating a problem with our Telehouse Metro core router, causing packet loss and difficulty accessing external networks.

Traffic has been re-routed in an effort to minimise the impact of the problem, however customers directly connected into Telehouse Metro will be affected.

  1. Richard Partridge says:

    Our engineers have isolated the problem however the resolution requires a reboot. This will be happening in the next few minutes. The reboot process will take approximately 15 minutes.

    This reboot will only affect directly connected customers.

  2. Richard Partridge says:

    A workaround has now been put in place. A permanent fix will be deployed as part of upcoming planned/emergency maintenance to be announced shortly.

  3. Richard Partridge says:

    An incident report has been made available for this issue. It can be found here.