At Risk: Cambridge POP

Posted: August 19th, 2016 at 16:21 by Richard Partridge

Monday 22nd August – 10:00 to 16:00

During the above window we will be replacing the UPS protecting camb.core. No downtime is expected however the site should be considered “at-risk” for the duration of the window.

  1. Richard Partridge says:

    This work has now been completed.

Ongoing VoIP traffic issues

Posted: August 16th, 2016 at 17:52 by Neil Watson

The monitoring as advised previously below has identified specific actions for us to take to overcome the issues impacting a relatively small number of customers using VoIP on a subset of the six nodes previously identified. We will implement these actions during the course of the evening and the pre-published maintenance window tonight. Users may see their service reconnect during the window. We anticipate that this will be an instant reconnect, however we would ask customers to reboot their equipment should subsequent connection problems be experienced. We will provide an update once normal daytime-business traffic levels are reached.

  1. Neil Watson says:

    As a result of the work carried out last night, we are seeing greatly improved performance on a number nodes. Work continues to correct the problem for the small amount of other customers that remain affected. As well as aiming to correct the root cause, we are also attempting to put a work-around in place to alleviate the symptoms. We anticipate that this will be in place today, however will confirm when complete.

  2. Neil Watson says:

    We have continued to work on the traffic management platform throughout the day with the assistance of the manufacturer to resolve the issues that continue to be seen. This work has so far not been successful; however we have implemented a workaround which has significantly improved the performance of VoIP on one test node.

    We have also been monitoring the changes made last night to a number of nodes and have seen no further issues reported today. We will therefore replicate those actions tonight during the pre-published maintenance window on other nodes. Users may see their service reconnect during the window. We anticipate that this will be an instant reconnect, however we would ask customers to reboot their equipment should subsequent connection problems be experienced. We will continue to investigate the root cause of the issue and will make further updates.

  3. Neil Watson says:

    Monitoring this morning has proved positive and we believe that last night’s actions have restored service quality to customers using VoIP. We are continuing to investigate the root cause and will update with more information as soon as we are able. If any customer experiences further issues please contact our support team.

  4. Neil Watson says:

    Following further final work last night and a period of monitoring this morning, we are now able to confirm that the VoIP issues have been resolved for all customers. We apologise for the extended problems that have been experienced.

managed-lns01.inx.dsl Incident

Posted: August 12th, 2016 at 13:18 by Adam Heath

We recently experienced an unexpected reload of the above device which will have caused a disconnection of broadband services for the customers connected to it.  Services appear to have reconnected over the alternative LNS and the reloaded device is now back online.

We apologise for the inconvenience this may have caused.

Emergency maintenance – 10/08/2016 17:30hrs – IPStream Connect and WBC connections

Posted: August 10th, 2016 at 12:01 by Neil Watson

We are currently aware that some customers are experiencing a level of packet loss on IPSC and a small number of WBC nodes. As well as investigating internally, we have also engaged with 3rd party suppliers who have identified a configuration problem that will lead to the issues being experienced. In order to correct this, changes need to be made, however in doing so, this will cause a port flap on our equipment which will effectively disconnect all DSL customers on the following nodes:

All IPSC nodes and the following WBC nodes
Peterborough
Preston
Sheffield
Slough
Stepney Green
Wolverhampton

We will therefore undertake this work at approximately 17:30hrs this evening to restore full service to all customers. It is anticipated that all customer will reconnect automatically, however we would ask anyone that does not connect to power cycle their on site equipment as a first step.

We wish to apologise for the short notice of this emergency maintenance, but feel that it is necessary in the interest of the affected customers. We’d also like to apologise for any inconvenience that this may cause and wish to reassure you that we are working hard to minimise any impact.

  1. Neil Watson says:

    We have been advised that the 3rd party supplier is not currently in a position to undertake the changes and is working on an alternative method of resolving the issue. We will therefore cancel the maintenance at 17:30 today. Should any further work be needed we will look to roll this into the pre-planned maintenance window from 23:00 tonight. Apologies for the short notice. We will continue to work to get the issue resolved and will update further on this post.

  2. Neil Watson says:

    Work was undertaken during the maintenance window last night to correct the packet loss issue that some customers were seeing. This work did not disconnect users and did not impact users’ services negatively. Initial reports this morning indicate that the levels of packet loss are greatly diminished or are not present. We are however seeing some disruption to VoIP services which are impacting a minority of users. We are currently trying further diagnose. We apologise for the ongoing nature and would like to reassure you that the issue has high level visibility within Entanet and we are working to resolve as quickly as possible.

  3. Neil Watson says:

    Investigations are continuing and further escalations have been made with 3rd party suppliers. We are expecting a further update before the close of play today and will update as soon as we have further information.

  4. Neil Watson says:

    We have received a further update from our suppliers. They are now planning on introducing further monitoring and diagnostics to confirm the suspected cause of the VoIP issues being seen. Once confirmed, there is an anticipated root to fix which will be implemented as soon as possible. More info will be posted as soon as we have it.

  5. Neil Watson says:

    Further escalations have been made to push this issue as we’re keen to get a resolution as soon as possible. We apologise for the delays being experienced in returning full service to all customers. We will update with more concrete information once we have it.

  6. Neil Watson says:

    Investigation work has continued throughout the day and a potential problem with mis-categorised flows has been identified. Work is underway to determine that this is indeed the cause, after which plans for rectification will be implemented. We will continue to update until resolved.

  7. Neil Watson says:

    Further work has continued over the weekend and changes have been agreed with 3rd party suppliers which have been implemented. These changes appear to have resolved the issue with VoIP and our monitoring has been positive since the changes were made. Unfortunately we will only be able to confirm once we have seen the normal weekday load return. We will continue to monitor closely and would encourage partners and customers to report further issues should they be seen. Once again apologies for the incident and the time taken to resolve.

  8. Adam Heath says:

    Following the previous update, early reports from users inform us that there is still disruption being experienced on some VoIP services. We can assure you this is being handled as the highest priority and we are continuing to work with the suppliers for a resolution.

  9. Neil Watson says:

    Following on from this morning’s reports of further VoIP disruption we have, at the manufacturer’s request, applied additional configuration changes to correct the miscategorised traffic flows. This action was determined based on the traffic dumps and monitoring that took place this morning as soon as the reports of further issues were received. Initial reports following the change suggest that this has had a positive effect. We will continue to monitor and ask for partners to contact the support team should they experience further problems.

  10. Neil Watson says:

    We are receiving reports that the issue has not been resolved and that some customers are still experiencing VoIP issues. We have already passed the issue back to the 3rd party suppliers, with whom we are closely engaged – a further update is expected soon. We are also looking to make changes within the Entanet network to try to further alleviate the problem. We anticipate those changes being made shortly, after which we will re-test again.

  11. Neil Watson says:

    Since our last post we have made further changes to try to alleviate the symptoms of the issues seen. Whilst these changes will improve the quality of some calls, it is not the full solution. We are continuing to work on the root cause of the issue and will update further as soon as we are able. Once again apologies for the extended time to fix for this issue.

  12. Neil Watson says:

    Further configuration changes were made late last night. We are currently assessing the impact of those changes and will advise soon if improvement is seen. We’ll continue to work today to get to a point of resolution.

  13. Neil Watson says:

    As a result of further diagnostics and investigations we have just made further changes to the platform to alleviate the problems being experienced by some VoIP customers. We will now monitor the support calls being received for any VoIP issues that occur after this post and the interfaces of the relevant systems for any loss or drops. Based on the results of this monitoring we will then advise next steps. Additional updates will be made with the results of that monitoring.

Virgin Media Based Circuits

Posted: August 4th, 2016 at 12:49 by Richard Partridge

We are currently aware of an issue affecting a number of Virgin Media based circuits delivered into our Interxion POP. Engineers are currently investigating and further information will be provided as and when it becomes available.

  1. Richard Partridge says:

    It appears that the outage relates to an unexpected reload of a Virgin Media NTE. Service has now returned to normal.

Virgin Media Based Circuits

Posted: August 3rd, 2016 at 08:29 by Alistair Patterson

We are currently aware of an issue affecting a number of Virgin Media based circuits. This has been confirmed as a major fibre break on Virgin Media’s network. Engineers are currently in attendance but no ETR is available at the moment.

 

Further updates to follow when available.

  1. Alistair Patterson says:

    Virgin Media have advised that the location of the break has been identified as 7.5km away from a local Virgin Media hub. Splicing engineers are currently attending the closest joint to determine the exact location of the break.

    Further updates to follow.

  2. Ryan Berrisford says:

    We have had further contact with Virgin Media. Following our previous post, several engineers are now on site identifying the exact location of the fibre break. We are expecting more specific details on the break in due course.

    We will keep in contact with the relevant escalation points within Virgin Media to obtain the latest information, as it becomes available.

  3. Alistair Patterson says:

    Virgin Media have advised engineers have attended the next joint along from the break and completed Optical time-domain reflectometer (OTDR) testing between the joints. Red light testing is currently underway to confirm the precise location of the break. Virgin Media have also confirmed splicing teams are on site to begin work as soon as the break location is confirmed.

    We are in regular contact with Virgin Media and will post updates as soon as they become available.

  4. Alistair Patterson says:

    Virgin Media engineers have located the break between Corbet Place and Brick Lane in London. Engineers are currently accessing the pits this cable passes through to visually inspect the cable and identify if any spare fibres can be used. If any spare fibres are identified they will be prioritised for our services.

    Further updates will be posted as soon as available.

  5. Alistair Patterson says:

    Virgin Media have advised that spare fibres have been located. Splicing will begin on the first end of the cable shortly and should be complete within 30 minutes. Splicing at the second end of the cable involves more work and is expected to take up to a further 60 minutes to complete.

    Further updates will be posted when available.

  6. Alistair Patterson says:

    Splicing work on the cables that carry our services has been completed and connectivity restored to all affected circuits. Virgin Media advised work is still ongoing at the site of the break so circuits should still be considered at risk.

    We will advise once Virgin Media confirm all repair work complete.

  7. Alistair Patterson says:

    Virgin Media have advised that all repair work is now complete and all services restored. Circuits should no longer be considered at risk.

    We apologise the inconvenience caused during the course of this incident.

  8. Alistair Patterson says:

    An incident report for this issue is now available here.

Planned Work: Broadband (DSL) platform

Posted: August 2nd, 2016 at 16:27 by Steve Lalonde

We will be migrating broadband customers to new equipment during the following 2 week period.

Friday 5th August 2016 – Friday 19th August 2016 23:00 – 07:00

Customers will be disconnected from the old equipment to reconnect to the new equipment

Most customers will see downtime of approx 5 minutes, however some customers may need to reboot their routers to reconnect.

 

Incident: Telecity Hex8/9

Posted: August 2nd, 2016 at 02:29 by David Derrick

We have lost connectivity to all our devices in Telecity Hex 8/9. This is affecting some leased line customers and customers with directly connected services at that site. Engineers are investigating.

  1. David Derrick says:

    This was caused by a power outage to our equipment which is now resolved. All services appear to be back up.

  2. Adam Heath says:

    An incident report for this issue is now available here.

VoIP Enrich Issue

Posted: July 21st, 2016 at 10:00 by Adam Heath

We are currently aware of an issue affecting a portion of our Enrich users.  Initial reports highlight a problem with both inbound and outbound calls.  We suspect this is being caused by a 3rd party outage elsewhere, however are currently investigating further.  We will provide further information as it becomes available.

  1. Adam Heath says:

    The issue appears to be resolved following engineers restarting a process on one of our media servers. Initial suspicions of an outage elsewhere being the cause were incorrect, but considered due to timing of the events correlating.

Broadband Problems

Posted: July 20th, 2016 at 09:04 by Adam Heath

We are currently aware of problems affecting the broadband platform with reports of numerous services being offline. Sources suggest this may be an issue on the BT network rather than our own, but we are continuing to investigate and gather information. Unfortunately, bt.com and btwholesale.com are also currently offline, meaning we have limited resources available to investigate at present.

We will update when further information becomes available.

  1. Adam Heath says:

    The problem has been confirmed on BT’s side and they are currently investigating. We haven’t been provided anything relating to the cause as of yet, but we will update upon receipt of this information.

    Unfortunately the BT systems remain down preventing us from looking into individual line issues.

  2. Adam Heath says:

    BT released information a short while ago advising that the problems are ‘due to power issues at one of our internet peering partners’ sites in London’. Work continues to be undertaken on BT’s side to restore service and we will update in due course.

  3. Adam Heath says:

    BT have advised that they are restoring services now. We also currently have access to some of the BT systems which were earlier unavailable. We are treating these as still at risk however until confirmation of restoration comes from BT.

  4. Adam Heath says:

    We have received confirmation from BT regarding the restoration of their systems which we access. Reports of users having no connectivity have also ceased coming into our desk, in relation to the earlier outage.

    We will be following up with BT to obtain any further formal information they release.