Archive for February, 2008

« Older Entries

Planned Maintenance: Telehouse North Core Router

Posted: Thursday, February 28th, 2008 at 16:58 by Neil Watson

Wednesday 05th March 2008: 00.00 – 06.00

We will be undertaking scheduled maintenance work on our telehouse-north.core.enta.net router during the period above. This work will involve the replacement of Gigabit Ethernet blades with higher performance blades. Anyone that is directly connected will experience a short period of downtime whilst the blades are installed and the router rebooted. It is anticipated however that the total period of downtime will be no longer than 30 minutes.

This work will not affect our general DSL pool, however some L2TP customers connected to this router will be affected by the work.

We wish to assure you that we will keep any downtime to a minimum and naturally apologise for any inconvenience that the work will cause.

Posted in Incidents, Planned Work | 1 Comment »

  1. Ming-Yu Hsieh says:
    March 5, 2008 at 00:40

    This maintenance work is now complete.

Network Monitoring and Traffic Graphs

Posted: Tuesday, February 26th, 2008 at 19:28 by Steve Lalonde

We have been experiencing some technical problems with our monitoring system for the last few days, the problems have been showing up as huge spikes and troughs on graphs on some systems including the xDSL central capacity graphs.

We have identified the problem and have taken steps to resolve, all monitoring systems appear to be fully operational again.

Posted in Emergency Work | 4 Comments »

  1. Steve Lalonde says:
    February 27, 2008 at 18:01

    Looks like we have not got to the bottom of the monitoring issues, as we have just had a re-occurrence.

    We have applied a temporary fix for tonight and will be replacing the affected server tomorrow.

  2. Steve Lalonde says:
    February 28, 2008 at 09:50

    We are about to start work on the monitoring servers, so expect the graphs to be unstable for the next few hours.

  3. Steve Lalonde says:
    March 1, 2008 at 13:04

    The monitoring system is having a bad day again, looks to be an NFS file locking problem where some of the RRD files are left in a locked state so further updates can not happen.

    We now have to wait to see if the changes just made will help.

    Note: this is just the monitoring system affected, the network is operating as expected.

  4. Steve Lalonde says:
    March 1, 2008 at 15:32

    system seems to have settled back to its normal working order now.

Outage: Wolverhampton to Birmingham Link

Posted: Sunday, February 24th, 2008 at 10:48 by Neil Watson

It appears that the Wolverhampton to Birmingham link has been flapping this morning causing some problems with routing in the area. Our engineers have now shut down the link and traffic is being re-routed successfully. We will look to re-introduce the link at an appropriate time.

Please note that the monitoring and reporting of the DSL Central pipes has been affected by these problems and it appears as though large loss of traffic has taken place. It is however monitoring that has been affected and we do not have Central Pipes out of service.

We apologise for the problems and will advise on any progress.

Posted in Incidents | Comments Off

Additional Capacity Live

Posted: Thursday, February 14th, 2008 at 13:31 by Steve Lalonde

Good News!

BT have enabled the next 155 megabits of capacity on central 5 a day early.

Posted in News | Comments Off

noc.enta.net problems

Posted: Thursday, February 14th, 2008 at 00:17 by Steve Lalonde

As some of you will have noticed the site disappeared tonight, this was down to a server fault.

From the info I have it looks like 1 of the scripts that feeds data to noc.enta.net got stuck in a loop and used up all the memory and all the swap space, that made everything grind to a halt and the database server crashed.

I’m going to put a wrapper round the script that looks to be the culprit to stop it running more than 1 copy at a time, that should prevent it happening again.

The server had been up for 119 days  (thats not a long time for a FreeBSD box)

Posted in Incidents, News | Comments Off

Central Pipe Balance

Posted: Wednesday, February 13th, 2008 at 11:03 by Neil Watson

Following on from last night’s maintenance work carried out by BT, we are now seeing an imbalance in the utilisation of the pipes. Unfortunately as BT will not supply a method of session steering we are unable to direct traffic to the under utilised pipe. We recommend that any users experiencing packet loss, reconnect to achieve better performance. We will also undertake some rebalancing work which may involve users being disconnected for a very short period of time. Please note that customers that take our PWAN service will not be disconnected. We apologise for any problems that this imbalance will cause.

Posted in Emergency Work, News | Comments Off

Planned Maintenance: Interxion

Posted: Monday, February 4th, 2008 at 13:51 by Steve Lalonde

 

Sunday 10th February 2008 05:00

We will be upgrading the IOS on interxion.core.enta.net, all customers directly connected to this router will see approximately 10 minutes down time. We apologise for any inconvenience caused.

Posted in Incidents, Planned Work | 1 Comment »

  1. Steve Lalonde says:
    February 10, 2008 at 06:38

    This work is now finished, router was down for under 5 minutes.

Planned Maintenance: Telehouse East

Posted: Monday, February 4th, 2008 at 11:50 by Steve Lalonde

Monday 11th February 2008 00:01 – 04:00

We will be upgrading elements of our core network in Telehouse East during the maintenance window specified above. The work will include an upgrade to IOS and upgrading the 10Gbps blades which, as well as ensuring the latest stable revision of IOS is installed, will also increase the forwarding performance of the routers. During the work the following downtime is to be expected:

telehouse-east.core.enta.net – 30 minutes maximum
telehouse-east2.core.enta.net – 30 minutes maximum

There will also be some routing instability during the maintenance window as the network will re-converge across alternative paths, thereby avoiding the affected routers. Note that the routers will be upgraded sequentially, and will not be down at the same time. Any customers connected directly to either of these routers will experience downtime. Our general DSL pool should not be affected by this work, however L2TP customers may be affected by the route re-convergence.

We apologise for the inconvenience that this maintenance will cause and wish to reassure you that we will keep the required downtime to an absolute minimum.

Posted in Incidents, Planned Work | 4 Comments »

  1. Steve Lalonde says:
    February 11, 2008 at 00:10

    OK time to get started on the maintenance, going to start with telehouse-east2.core.enta.net.
    All the DSL traffic has been moved away from the router so just connected customers should be affected.

  2. Steve Lalonde says:
    February 11, 2008 at 01:12

    telehouse-east2.core.enta.net is now back in service.

    Now going to move all dsl traffic so work can start on telehouse-east.core.enta.net

  3. Steve Lalonde says:
    February 11, 2008 at 01:46

    Work on telehouse-east.core.enta.net will be limited to IOS upgrade only, this is due to one of the blades for the hardware upgrade being faulty.

    The hardware upgrade will be re-scheduled.

  4. Steve Lalonde says:
    February 11, 2008 at 02:20

    This maintenance work is now finished.

Planned Maintenance: Frankfurt

Posted: Monday, February 4th, 2008 at 10:19 by Steve Lalonde

Saturday 9th February 2008 05:00

We will be upgrading the IOS on frankfurt.core.enta.net, all customers directly connected to this router will see approximately 10 minutes down time. We apologise for any inconvenience caused.

Posted in Incidents, Planned Work | 1 Comment »

  1. Steve Lalonde says:
    February 9, 2008 at 05:26

    This work has now completed, again router was off line for less than 5 minutes.

Planned Maintenance: Amsterdam

Posted: Monday, February 4th, 2008 at 10:17 by Steve Lalonde

Friday 8th February 2008 05:00

We will be upgrading the IOS on amsterdam.core.enta.net, all customers directly connected to this router will see approximately 10 minutes down time. We apologise for any inconvenience caused.

Posted in Incidents, Planned Work | 1 Comment »

  1. Steve Lalonde says:
    February 8, 2008 at 06:25

    Upgrade work is now complete, router was off line for approximately 2 minutes.

« Older Entries