Archive for January, 2009

Outage: Amsterdam to Telehouse North

Posted: Thursday, January 29th, 2009 at 11:58 by Neil Watson

We are currently experiencing an outage on our Amsterdam to Telehouse North connection. Traffic is currently routing via alternative paths on the protected ring. Traffic from Amsterdam to London may experience slightly higher than normal latency. We will advise once service is returned to normal.

  1. Ming-Yu Hsieh says:

    Our carrier has identified problem near Rotterdam area, and investigating what was the cause.

  2. Steve Lalonde says:

    Outage is due to fibre cut isolated in Aalsmeer, the Netherlands.
    Our carrier is dispatching technicians to the cut location.
    No ETR yet.

  3. Steve Lalonde says:

    Our carrier is on site at the damaged cable.
    No ETR yet.

  4. Steve Lalonde says:

    We now have ETR of 19:45 UTC.

  5. Steve Lalonde says:

    Circuit is still down. Our Carriers engineers are having trouble at the fault site.
    Technicians have tested all nine spare ducts on the segment and all of them appear to be broken. They are
    going to attempt an excavation at the mid-point.

  6. Steve Lalonde says:

    Our carriers technical team is currently at the suspected cut location. The excavation of the duct is being continued by them and it will be approximately another hour before the 5 meter length is completely exposed to allow access and testing of the entire duct.
    Unfortunately, there is currently no ETR for this fibre cut.

  7. Steve Lalonde says:

    There is a gas main in the common trench and the field technicians have stopped excavating and are currently seeking a temporary restoration plan. A segment of the existing fibre duct will be used and then re-routed above ground through a temporary duct which is being conducted now. The estimated restoration time is currently at 5 hours.
    The root cause of the damaged fibres has not been determined.

  8. Steve Lalonde says:

    Service was restored at approx 04:00. All traffic has returned to the normal path.

Planned Maintenance: Interxion – 10th Feb 2009 12:00am – 4:00am

Posted: Tuesday, January 27th, 2009 at 10:25 by Martyn Haynes

Engineers will be in attendance at Interxion (London) in the early hours of Tuesday 10th February 2009.  Directly affected customers will be informed of any planned outages prior to this time, however as work is being carried out in and around core services, all services at this site should be considered at risk during this window.

Telecity Hex 6/7 Power

Posted: Tuesday, January 27th, 2009 at 09:54 by Steve Lalonde

We have been informed by Telecity that  Hex6/7 is running on generator power. All services connected to this site should be considered at risk.

  1. Steve Lalonde says:

    We have now been informed that the site is running on mains power again and has been stable for more than 2 hours.

Network Issues: Cardiff, Kingston and Peterborough Pop

Posted: Sunday, January 25th, 2009 at 00:15 by Ming-Yu Hsieh

We are currently experiencing issues at Cardiff, Kingston and Peterborough pops. This is due to a fault with NTL / Virgin Media in Telehouse, they are currently working to fix this.

Further reports will be made as soon as we get an update.

  1. Neil Watson says:

    An outage report is now available for this incident:

    Outage Report

Network Issues: Sheffield & Nottingham POPs

Posted: Friday, January 23rd, 2009 at 14:35 by Neil Watson

Subsequent to our earlier notification regarding the network issues effecting the Sheffield and Nottingham POPs, we have received information from our suppliers that they have tasked engineers to go to site. Unfortunately the estimate time to resolve is currently 6 hours. We are pushing for a speedier resolution, and will update as soon as possible.

  1. Neil Watson says:

    We have had further information that the location of the issue is in the Rotherham area, engineers are being directed to there.

  2. Neil Watson says:

    We’re happy to inform that the service has now been restored from the two affected POPs. We are currently awaiting official confirmation that all services have been restored, hence the circuit is still considered to be at risk as engineers may still be working on other fibres within the vicinity. We will post more information when an outage report is received from our supplier.

  3. Steve Lalonde says:

    Unfortunately we have lost service the Sheffield and Nottingham PoPs again, this is probably due to the ongoing repair work.

  4. Neil Watson says:

    We’ve received further information that the cause of the outage is a cable that has been dug up during rail works. This work exposed two joint 100m apart which has required a full cable removal and insertion. The engineers are currently splicing fibres and the estimated time of completion is 21.00hrs.

  5. Neil Watson says:

    We are currently awaiting further information and are chasing for an update. We will update as soon as we have more information.

  6. Neil Watson says:

    Indications are that the link has been restored. Traffic is now passing over the affected links and has been doing so for a short while, suggesting that all splicing work has been completed. We have not yet had official confirmation that the fault has been cleared, so the links must be considered to still be at risk. A further update will be made when more concrete information is available.

  7. Neil Watson says:

    An outage report is now available for this incident:

    Outage Report

Outage: London to Manchester

Posted: Friday, January 23rd, 2009 at 12:44 by Martyn Haynes

We are currently aware of an issue with the link between London and Manchester.  We are working to correct this as soon as possible, traffic is routing via alternative paths  however this may affect latency or speed.

Further reports will be made as appropriate.

  1. Neil Watson says:

    Connectivitiy has now been restored. We are investigating the cause and will report back as soon as possible.

  2. Martyn Haynes says:

    We have been informed that there is also a connectivity issue in the Leeds to Sheffield area that is affecting multiple suppliers, our supplier is investigating this fault as a matter of urgency.

    This outage is likely to affect some xDSL customers in this area.

    Further updates will be supplied as soon as they are available.

  3. Neil Watson says:

    An outage report is now available for this incident:

    Outage Report

Central Pipe Decommissioning

Posted: Tuesday, January 20th, 2009 at 15:20 by Neil Watson

As part of the ongoing move to IPStream Connect, we are requesting that all users currently connected to Centrals 1-4 re-establish connection prior to the end of Sunday 25th January 2009. This reconnection is necessary as during the course of Monday 26th January 2009 the four centrals will be taken out of service. Any users that remain connected on Monday will be disconnected and will be migrated on to alternative infrastructure. This notice affects all DSL customers including both L2TP and LSS customers.

IPStream Connect Status Tool

Posted: Tuesday, January 20th, 2009 at 14:45 by Neil Watson

Whilst network changes are taking place as previously advised on this site, it has become necessary to temporarily remove access to the IPStream Connect status tool. This is because the tool could present incorrect information at times when necessary changes to bandwidth availability are being made on an IPSC interconnect by interconnect basis. We aim to restore access to the tool as soon as possible and apologise for any inconvenience that this restriction may cause you.

Factors Affecting ADSL Performance

Posted: Monday, January 19th, 2009 at 20:05 by Neil Watson

Further to our communication on this site (http://noc.enta.net/?p=381) we write to explain the reasons for reduced performance over the past few days and to apologise for any inconvenience caused to customers. Please be assured that we are working to ensure normal service levels are resumed as soon as possible.

In our previous communication we advised that, as users re-authenticate in order to connect to the IPSC interconnect serving them, we expect user volumes and bandwidth demand to vary among the interconnects and to be unrepresentative of performance expected when all moved users are connected to IPSC. It is therefore possible that variable performance may be experienced, especially during off-peak hours, until such time that normal volumes are established.

The process of moving connections from IPStream Centrals to IPSC is being executed without issue. However, at the point of executing the move, it is not possible for Entanet to know which IPSC interconnect individual users will be connected to, as this information is not available from BT Wholesale. Furthermore, as complete execution of the move relies on users re-authenticating to establish their connection on the new interconnect, it has not been possible for us to estimate the total bandwidth required to serve them. It is not feasible for us to set an artificially high bandwidth level, as doing so commits Entanet to that level with BTW for a minimum 3 month period. Bandwidth usage is therefore being continuously monitored and adjusted as users re-authenticate.

Whilst managing an increase in bandwidth availability per interconnect it has also been necessary for Entanet to operate the Anti-Loss Tool (ALT) on IPSC as it does on IPStream Centrals, in order that all users receive service. However, in this unsettled period, at times the application of the ALT has uncharacteristically affected performance to a greater degree. Further incremental increases in the bandwidth level proved to successfully suspend the ALT, albeit only temporarily.

These affects are currently being experienced mainly in off-peak hours only. During peak daytime hours the available bandwidth results in some minor activation of the ALT. Nonetheless we regret that off-peak users have experienced differing performance levels than they are accustomed to and will continue to actively manage bandwidth to a successful conclusion.

DSL transfer to IPStream Connect update

Posted: Wednesday, January 14th, 2009 at 18:00 by Steve Lalonde

Further to the previous post, we have now moved the bulk of the remaining realms to IPStream Connect. Please be aware that end users need to re-authenticate in order to connect to the IPSC interconnect serving them (note that dropping the session for more than 30 seconds will be required to ensure the session is re-established on the new interconnects).  As users do so over the coming days, we expect user volumes and bandwidth demand to vary among the interconnects and to be unrepresentative of performance expected when all moved users are connected to IPSC. It is therefore possible that variable availability may be experienced, especially during off-peak hours, until such time that normal volumes are established. The ALT is operational on IPSC and will take effect whenever necessary as normal.

The interconnect graph will be adjusted to reflect the available bandwidth at each handoff (instead of a percentage of the maximum capable on the interconnect). Please note that, in the short term, the illustrated results are unlikely to be indicative of available capacity.