Archive for the ‘Incidents’ Category

Incident: VoIP Disruption

Posted: Wednesday, November 2nd, 2016 at 15:45 by Richard Partridge

One of our upstream carriers is experiencing difficulties which is having an adverse affect on outbound call setups. Engineers are working to bypass this problem and direct outbound calls towards other carriers.

Further updates will be provided as and when they become available.

  1. Richard Partridge says:

    The affected carrier has now been bypassed.

  2. Richard Partridge says:

    The upstream carrier has acknowledged the problem and is working on a resolution. We will continue to re-route calls around this problem until a solution has been put in place.

  3. David Derrick says:

    The affected carrier resolved their problem and has now been brought back into service.

Incident: Broadband Connections

Posted: Saturday, October 22nd, 2016 at 13:36 by Neil Watson

We are aware of an incident earlier today that affected a proportion of our broadband base which prevented them from being able to connect to our service for around 15 – 30 minutes. All services have been restored and we continue to monitor to ensure that all customers can connect as expected. We will investigate further and will post a report in due course. Apologies to any customer that was impacted.

  1. Adam Heath says:

    An incident report for this issue is now available here

Incident: Vodafone Circuits

Posted: Monday, October 17th, 2016 at 16:47 by Adam Heath

We are currently aware of an issue affecting a number of Vodafone based circuits.  We are working with our supplier who are currently investigating.

We will publish further updates as they become available.

  1. Adam Heath says:

    After initial testing Vodafone have acknowledged the issue on their side. Further updates will be available shortly.

  2. Ryan Berrisford says:

    Vodafone have confirmed that following diagnostics, engineers have been dispatched to Sovereign House as an issue is suspected within that domain. Estimated time of arrival is currently 21:00.

    Further updates will follow once the issue has been further isolated or resolved.

  3. Ryan Berrisford says:

    Vodafone have confirmed that although engineering resource has reached site, there was a delay in arrival due to the black cab protest taking place in Central London. Fault resolution has therefore been delayed slightly but further updates will be made available shortly.

  4. Ryan Berrisford says:

    Vodafone have advised that the fault is still in hand with engineering resource at Telecity, Soveriegn House. We are pressing for resolution and updates and will post further information as it becomes available.

  5. Ryan Berrisford says:

    Service has now been restored to the affected circuits. We are awaiting full resolution details and will update this post in due course.

    We apologise for the inconvenience caused by this incident.

  6. Ryan Berrisford says:

    Vodafone have confirmed the incident was caused by an issue with the line card at Telecity, Sovereign House. The issue is confirmed as resolved by Vodafone.

    We apologise for the inconvenience caused during this incident.

Incident: Faraday

Posted: Wednesday, October 12th, 2016 at 12:56 by Jake Turner

We are aware of an issue affecting customers on our Faraday core. Engineers are investigating and further updates will be provided when available.

  1. Jake Turner says:

    Service has been restored. Engineers will continue to monitor and provide further information when available.

  2. Richard Partridge says:

    It appears that there are still residual problems affecting service for a small number of customers. These are being investigated on a case by case basis.

    We apologise for the inconvenience caused and are working as quickly as possible to resolve the issue. Further updates will be provided as and when they become available.

  3. Richard Partridge says:

    Engineers are continuing to work on the problem and the affected circuits are being brought back into service one at a time.

    Further updates will be provided shortly.

  4. Phil Davies says:

    More circuits have now been successfully restored and engineers are continuing to work on the few remaining which are down.

    Further updates to follow.

  5. Richard Partridge says:

    With only a few circuits remaining with issues, customers affected will now be contacted directly with any further updates.

    Thank you for your patience.

Incident: Sheffield

Posted: Friday, September 9th, 2016 at 17:12 by Mark Yardley

We are aware of an issue affecting customers on our Sheffield core. Engineers are investigating and further updates will be provided when available.

  1. Jake Turner says:

    Service has been restored. Engineers will continue to monitor and provide further information when available.

Ongoing VoIP traffic issues

Posted: Tuesday, August 16th, 2016 at 17:52 by Neil Watson

The monitoring as advised previously below has identified specific actions for us to take to overcome the issues impacting a relatively small number of customers using VoIP on a subset of the six nodes previously identified. We will implement these actions during the course of the evening and the pre-published maintenance window tonight. Users may see their service reconnect during the window. We anticipate that this will be an instant reconnect, however we would ask customers to reboot their equipment should subsequent connection problems be experienced. We will provide an update once normal daytime-business traffic levels are reached.

  1. Neil Watson says:

    As a result of the work carried out last night, we are seeing greatly improved performance on a number nodes. Work continues to correct the problem for the small amount of other customers that remain affected. As well as aiming to correct the root cause, we are also attempting to put a work-around in place to alleviate the symptoms. We anticipate that this will be in place today, however will confirm when complete.

  2. Neil Watson says:

    We have continued to work on the traffic management platform throughout the day with the assistance of the manufacturer to resolve the issues that continue to be seen. This work has so far not been successful; however we have implemented a workaround which has significantly improved the performance of VoIP on one test node.

    We have also been monitoring the changes made last night to a number of nodes and have seen no further issues reported today. We will therefore replicate those actions tonight during the pre-published maintenance window on other nodes. Users may see their service reconnect during the window. We anticipate that this will be an instant reconnect, however we would ask customers to reboot their equipment should subsequent connection problems be experienced. We will continue to investigate the root cause of the issue and will make further updates.

  3. Neil Watson says:

    Monitoring this morning has proved positive and we believe that last night’s actions have restored service quality to customers using VoIP. We are continuing to investigate the root cause and will update with more information as soon as we are able. If any customer experiences further issues please contact our support team.

  4. Neil Watson says:

    Following further final work last night and a period of monitoring this morning, we are now able to confirm that the VoIP issues have been resolved for all customers. We apologise for the extended problems that have been experienced.

managed-lns01.inx.dsl Incident

Posted: Friday, August 12th, 2016 at 13:18 by Adam Heath

We recently experienced an unexpected reload of the above device which will have caused a disconnection of broadband services for the customers connected to it.  Services appear to have reconnected over the alternative LNS and the reloaded device is now back online.

We apologise for the inconvenience this may have caused.

Virgin Media Based Circuits

Posted: Thursday, August 4th, 2016 at 12:49 by Richard Partridge

We are currently aware of an issue affecting a number of Virgin Media based circuits delivered into our Interxion POP. Engineers are currently investigating and further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    It appears that the outage relates to an unexpected reload of a Virgin Media NTE. Service has now returned to normal.

Virgin Media Based Circuits

Posted: Wednesday, August 3rd, 2016 at 08:29 by Alistair Patterson

We are currently aware of an issue affecting a number of Virgin Media based circuits. This has been confirmed as a major fibre break on Virgin Media’s network. Engineers are currently in attendance but no ETR is available at the moment.

 

Further updates to follow when available.

  1. Alistair Patterson says:

    Virgin Media have advised that the location of the break has been identified as 7.5km away from a local Virgin Media hub. Splicing engineers are currently attending the closest joint to determine the exact location of the break.

    Further updates to follow.

  2. Ryan Berrisford says:

    We have had further contact with Virgin Media. Following our previous post, several engineers are now on site identifying the exact location of the fibre break. We are expecting more specific details on the break in due course.

    We will keep in contact with the relevant escalation points within Virgin Media to obtain the latest information, as it becomes available.

  3. Alistair Patterson says:

    Virgin Media have advised engineers have attended the next joint along from the break and completed Optical time-domain reflectometer (OTDR) testing between the joints. Red light testing is currently underway to confirm the precise location of the break. Virgin Media have also confirmed splicing teams are on site to begin work as soon as the break location is confirmed.

    We are in regular contact with Virgin Media and will post updates as soon as they become available.

  4. Alistair Patterson says:

    Virgin Media engineers have located the break between Corbet Place and Brick Lane in London. Engineers are currently accessing the pits this cable passes through to visually inspect the cable and identify if any spare fibres can be used. If any spare fibres are identified they will be prioritised for our services.

    Further updates will be posted as soon as available.

  5. Alistair Patterson says:

    Virgin Media have advised that spare fibres have been located. Splicing will begin on the first end of the cable shortly and should be complete within 30 minutes. Splicing at the second end of the cable involves more work and is expected to take up to a further 60 minutes to complete.

    Further updates will be posted when available.

  6. Alistair Patterson says:

    Splicing work on the cables that carry our services has been completed and connectivity restored to all affected circuits. Virgin Media advised work is still ongoing at the site of the break so circuits should still be considered at risk.

    We will advise once Virgin Media confirm all repair work complete.

  7. Alistair Patterson says:

    Virgin Media have advised that all repair work is now complete and all services restored. Circuits should no longer be considered at risk.

    We apologise the inconvenience caused during the course of this incident.

  8. Alistair Patterson says:

    An incident report for this issue is now available here.

Incident: Telecity Hex8/9

Posted: Tuesday, August 2nd, 2016 at 02:29 by David Derrick

We have lost connectivity to all our devices in Telecity Hex 8/9. This is affecting some leased line customers and customers with directly connected services at that site. Engineers are investigating.

  1. David Derrick says:

    This was caused by a power outage to our equipment which is now resolved. All services appear to be back up.

  2. Adam Heath says:

    An incident report for this issue is now available here.