Incident: BT Wholesale Circuits

Posted: June 25th, 2018 at 11:03 by Mark Yardley

We are aware of an issue affecting a number of BT Wholesale leased line circuits. The issue has been raised with our suppliers for investigation. At the time of posting we are starting to see services restoring, however services should still be considered at risk. More information will be provided when available.

  1. David Brewis says:

    BT have identified an issue in the London Stepney Green exchange with a task raised for an engineer to attend and resolve today. We expect the next update at 17:00.

  2. Scott Morgan says:

    Our suppliers have attended site to investigate the issue.

    The engineer has conducted some preventative maintenance on the equipment and users may have seen another brief interruption to service during this work. However, we have not as yet been provided with a root cause for the previous disruption. We are continuing to escalate this with our supplier for investigation and RFO.

  3. Mark Yardley says:

    As part of ongoing investigation into this issue we are dispatching an engineer to perform pre-emptive maintenance on our equipment and to co-op with a BT Wholesale engineer. This work is scheduled for between 15:00 and 16:00 today. During the window services delivered via this supplier hand-off will experience a planned disruption to connectivity. We anticipate this should be no more than 10 minutes.

  4. Mark Yardley says:

    This work was completed successfully by our engineer in the previously mentioned window. However we are still experiencing intermittent problems with the hand-off. This issue is being escalated again with our supplier for further investigation. Updates to be provided when available.

  5. Gurinder Bansal says:

    Our suppliers have attended site and fixed a fibre break. Connectivity has been restored however services should still be considered at risk. We will continue to monitoring overnight and an update will be provided when available.

  6. Mark Yardley says:

    We have seen a recurrence of this issue again this morning. We are escalating the case further with our supplier. Updates will be provided once available.

  7. David Brewis says:

    With the recurrence of disruption to service, despite locating a suppliers faulty fibre last night, investigations continue to identify the cause. This has been further escalated in BT and we are currently still waiting on an update which will be provided as soon as it comes available.

  8. David Brewis says:

    Following hardware replacements and no further disruption experienced overnight we will continue to monitor service throughout today.

    Should a repeat of disruptions continue, further work involving BT and their vendors will be carried out moving service to new a configuration.

  9. David Brewis says:

    We continue to monitor the status of connections and observe no further drops at the exchange since 16:44 on 27th June 2018.

    Entanet will continue to monitor for an extended period whilst working with the supplier to ensure service returns to normal operation.

  10. Jamie Smith says:

    We have seen a recurrence of this again this morning. This will be re-escalated with out supplier Updates to follow once available

  11. Scott Morgan says:

    Our suppliers have advised that an engineer is currently onsite as part of this major service incident. Spare parts should be with the engineer shortly so that hardware changes can be made to help rectify the issue.

    We will update further once we have received confirmation that this work has been carried out.

  12. Scott Morgan says:

    We have received confirmation from our suppliers that a suspected faulty line card has now been replaced.

    We apologise for any inconvenience caused by this incident and we will continue to monitor.

    Please consider this as still at risk, until further monitoring allows us absolute confirmation that this is now stable.

Emergency Maintenance: DSL

Posted: June 21st, 2018 at 14:26 by David Brewis

Thursday 21st June 2018 18:30 – 1900

During the above window we will be completing configuration work on an interconnect to correct throughput issues for a small set of broadband users. As the configuration is applied, users are likely to experience disruption to service for up to one minute.

We apologise for the short notice of this works. Should disruption continue past this window please reboot your router before contacting support.

  1. Steve Lalonde says:

    This work is now complete

Incident: Milton Keynes Broadband

Posted: June 21st, 2018 at 11:16 by David Brewis

We have experienced an issue with our DSL platform where broadband users in Milton Keynes experienced slow speeds with packet loss.

Whilst normal service has now been restored we reaching out to the supplier to investigate and understand cause.

  1. David Brewis says:

    Our suppliers continue to investigate, and we expect a response within 10 working days. Please find an interim report here.

Emergency Maintenance – DSL platform

Posted: June 18th, 2018 at 17:54 by Neil Watson

We are aware BT are performing emergency works affecting a limited number of service aggregators spread across several nodes on our network. A small percentage of customers on each node will be affected and will experience an outage of up to ten minutes.

The works begin tonight at 23:00 until 06:00 and are scheduled to run for the next two nights from 00:01 until 06:00 until Thursday morning. The work is essential for ongoing and improved capacity management on interconnects delivered to Entanet. Apologies for the short notice in relation to the work. Any affected customer should reboot their equipment before calling support as this should resolve any unanticipated residual issue.

Planned Maintenance: Telford

Posted: June 18th, 2018 at 11:36 by David Brewis

Thursday 21st June 2018 08:00 – 09:00

During the above window, Western Distribution will be conducting maintenance works to service the electricity substation supplying the local area. During these works, external mains supply will be disconnected to allow safe operation to all involved, in doing so our external generators will supply the Telford data centre allowing for continued and uninterrupted service.

No down time is expected and all precautions will be taken to avoid any disruption, however during this window the Telford data centre should be considered at risk.

  1. Mark Yardley says:

    This work is now complete and normal supply has been restored.

Emergency Maintenance: Stepney Green – Faraday – Columbo

Posted: June 15th, 2018 at 16:36 by Gurinder Bansal

Monday 18th June 2018 20:00 – Tuesday 19th June 2018 03:00

During the above maintenance window, we will be undertaking work on core links between the above locations as part of ongoing network expansion.

No down time is expected and all precautions will be taken to avoid any disruption, however during this window all three sites should be considered at risk.

  1. Gurinder Bansal says:

    This work is now complete.

Emergency Maintenance: Nottingham

Posted: June 12th, 2018 at 17:55 by David Brewis

Wednesday 13th June 2018 20:00 – 22:00

During the above window we will be investigating a potential issue with the power infrastructure at this site. No downtime is expected however all directly connected services should be considered at risk for the duration of this maintenance window.

  1. Our investigation is now complete, services should no longer be considered at risk.

Emergency Maintenance: Bristol

Posted: June 12th, 2018 at 17:43 by David Brewis

Thursday 14th June 2018 00:01 – 02:00

During the above window we will be investigating a potential issue with the power infrastructure at this site. No downtime is expected however all directly connected services should be considered at risk for the duration of this maintenance window.

  1. Gurinder Bansal says:

    Our investigation is now complete, services should no longer be considered at risk.

Planned Maintenance: DSL Platform

Posted: June 12th, 2018 at 16:39 by David Brewis

Wednesday 20th June 2018 01:00 – 06:00

During the above maintenance window we will be completing configuration changes identified during a previous maintenance. This work will allow for metrics to be placed on the broadband traffic facilitating the direction of traffic to be steered dependent on where best suited.

Once these works are completed, broadband traffic will be steered away from Entanet’s DSL platform in Telehouse to Interxion, with sessions monitored to evaluate the failover. After confirmation sessions have fully migrated, the DSL platform will be steered away from Interxion to Telehouse allowing to test the other direction. Once sessions have fully migrated over, resiliency will be restored across both locations.

No downtime is expected however DSL customers should be considered at risk for the duration of the window.

  1. Ming-Yu Hsieh says:

    This work is now complete.

Incident: nottingham.core

Posted: June 3rd, 2018 at 11:17 by Mark Yardley

We have experienced an unscheduled reload of our Nottingham core router. All directly attached services will have experienced a brief interruption. Initial investigation suggests this was due to loss of power in the location and engineers are investigating further with our co-location provider. All services are now restored and we apologise for any inconvenience caused.