Incident: Virgin Media Based Circuits

Posted: March 31st, 2015 at 20:26 by Phil Davies

We are currently investigating an issue affecting some customers with Virgin Media provided circuits.
A fault is been raised with the supplier and we will work with them closely, until resolution. Further updates will be provided, as and when they become available.

  1. Phil Davies says:

    Our supplier has confirmed there are multiple Fibre breaks in the Gloucester area which will be affecting many of our Virgin Media provided circuits. Engineers are currently on site working on the repair, it is a major incident and we have been advised that work will continue through the night with no current estimated time of repair.

  2. Phil Davies says:

    Our Supplier has informed us that work continued through the night with most circuits having a restored service at 22:30 due to re-routing within their network however the physical work to is due to complete around 10.30 today so further downtime may occur.

Incident: Talk Talk Based Circuits

Posted: March 31st, 2015 at 16:14 by Ryan Berrisford

We are currently aware of an issue affecting some customers with Talk Talk provided circuits. Initial investigations indicate the affected circuits are all based in the Midlands area.

A fault has been raised with the supplier and we will work with them closely, until resolution. Further updates will be provided, as and when they become available.

  1. Ryan Berrisford says:

    All affected circuits were restored by approximately 23:30 last night. This incident was caused by a failed network card in our suppliers network at Wolverhampton, which has now been removed from service.

    No further impacts are expected by this incident.

Incident: Cardiff to Birmingham

Posted: March 31st, 2015 at 14:35 by Richard Partridge

We are aware of a fibre break affecting our Cardiff to Birmingham core link. Traffic is already re-routing around alternative paths. Our supplier is already working to resolve the problem however no ETR is available at this time.

Further updates will be provided as and when they become available.

  1. Phil Davies says:

    Our supplier has confirmed that the Fibre beaks in the Gloucester area which are affecting Virgin Media provided Circuits are also affecting the Cardiff to Birmingham core Link. Engineers are currently on site working on the repair, it is a major incident and we have been advised that work will continue through the night with no current estimated time of repair. Traffic is still re-routing around alternative paths.

  2. Phil Davies says:

    Service was restored at approximately 02.40am and remains stable. We will continue to monitor.

At Risk: Redbus POPs

Posted: March 31st, 2015 at 10:32 by Richard Partridge

Thursday 9th April 2015 00:01 – 06:00

The following sites will be at risk while we perform power upgrades.

  • redbus-sov.core.enta.net
  • redbus-hex.core.enta.net
  • redbus-mer.core.enta.net

All Vodafone based connections on redbus-sov.core.enta.net will experience approximately 15 minutes downtime whilst the Vodafone termination equipment reloads.

Emergency Maintenance: Multiple routers

Posted: March 24th, 2015 at 16:57 by Steve Lalonde

Tuesday 24 March 2015 18:00 ~ 23:59

We will be applying a Hit-Less software update to the following list of routers:

  • equinix2.core.enta.net
  • interxion3.core.enta.net
  • telehouse-east4.core.enta.net
  • telehouse-metro2.core.enta.net
  • telford-dc2.core.enta.net

This is the same update applied to telehouse-north.core.enta.net that resolved the instability seen last week.

  1. Steve Lalonde says:

    This work is now complete.

Incident: Incoming VOIP calls

Posted: March 23rd, 2015 at 10:15 by David Derrick

One of our suppliers has advised that they are currently experiencing an outage on their incoming call handling systems. Some customers may experience problems receiving calls from external phone networks.

  1. David Derrick says:

    Our supplier reports their incident is resolved and initial tests indicate that inbound calls are working correctly again.

Telephone system problems

Posted: March 21st, 2015 at 10:56 by Jake Turner

We are currently experiencing problems with our in-office telephone system and are intermittently unable to take support calls. Engineers are working on rectifying the situation as a matter of urgency and will ensure that normal service is resumed as soon as possible. As an alternative method of contact please email support@enta.net.

  1. Jake Turner says:

    This issue was resolved at approximately 11:20. After extensive monitoring we can confirm that calls are routing through to the support lines as expected. Please accept our apologies for any inconvenience caused.

Planned Maintenance: Telehouse North – Phase 2

Posted: March 17th, 2015 at 10:18 by Ming-Yu Hsieh

Thursday 26th March 2015  0:01 ~ 06:00

During the above window we will be migrating remaining customers from telehouse-north0.core.enta.net to telehouse-north.core.enta.net, each connection will have downtime of up to 30 minutes during the window.

Affected services are:

  • All customer interconnects ( Transit / EWCS / Wholesale handoff / Pseudowire ) terminating at telehouse-north0.core.enta.net

Traffic that would normally be routed through this equipment will be routed via alternate paths.

  1. Ming-Yu Hsieh says:

    This work is now complete.

Talk Talk Based Circuits

Posted: March 12th, 2015 at 08:22 by Wayne Hawtin

We are currently aware of an issue affecting some customers with Talk Talk provided circuits.

Talk Talk are currently investigating and updates will be provided as soon as they become available.

  1. Wayne Hawtin says:

    Talk Talk advise the issue is affecting customers in East Midland and East Anglia regions and although their Network Support Engineers have restored service to some sites, corrective work is continuing with additional engineering resource being tasked to assist in restoring remaining sites.

    We expect further updates by 09:45.

  2. Wayne Hawtin says:

    Further updates from Talk Talk advise that corrective work to restore all impacted services remains on-going and they have identified a further number of southern regions impacted by the issue. The Talk Talk Incident Management team are continuing to liaise with their NOC and Network Support Engineers and further updates will be provided upon receipt.

    There is currently no ETR for the issue although further updates are expected at approximately 11:00.

  3. Wayne Hawtin says:

    Talk Talk NOC Engineers have confirmed that although further restorative steps have been taken they can see circuits continuing to experience issues. Their IP Engineers are currently investigating further and Talk Talk Incident Management will continue to liaise with their NOC Engineers.

    There is still no ETR and further updates are expected around 12:00.

  4. Ryan Berrisford says:

    Talk Talk have advised that a fibre break has been identified between Telehouse North and Hemel Hempstead. Engineers are working on the break and we have seen the restoration of some services, although these should be considered at risk until the work is complete.

    Talk Talk are working on restoring the remaining affected circuits with the next update expected at approximately 13:30.

  5. Alistair Patterson says:

    We are now seeing all services back up. Talk Talk engineers are still working on site so affected circuits should be considered at risk until the incident is cleared.

  6. Wayne Hawtin says:

    Talk Talk have now confirmed that service has been restored and this incident has been closed.

    We apologise for any inconvenience caused by these events.

Incident: Telehouse North

Posted: March 11th, 2015 at 16:52 by Jake Turner

Our telehouse-north.core router has unexpectedly reloaded. Engineers are investigating.

  1. Jake Turner says:

    Service has been restored. Engineers are continuing to monitor.

  2. Steve Lalonde says:

    Router is currently stable, we have noticed some unusual log messages since the reboot.

    We have rerouted most of the traffic away from the router as a precaution.

    A TAC case is open with the vendor.

  3. Steve Lalonde says:

    We are seeing further instability with this router. All possible traffic is being rerouted while we escalate with TAC.

  4. Steve Lalonde says:

    We have dispatched engineers to the site with replacement hardware, It may still be possible to resolve the problem remotely.

  5. Steve Lalonde says:

    We are continuing to work with the TAC team, our engineers are still on route to site.

  6. Steve Lalonde says:

    Latest from TAC, they believe this is a software issue and not a hardware fault.

  7. Steve Lalonde says:

    We have received a SMU from TAC that should fix the fault, it is a hitless fix and is currently installing.

  8. Steve Lalonde says:

    The fix has been installed, we will monitor the router overnight.

    Directly connected customers should see stable service resumed.

    We will keep through traffic on alternative paths as a precaution.

  9. Steve Lalonde says:

    The router has now been stable for 5 hours, we will restore traffic back to optimal paths and continue monitoring.