BY Tibus
Things have returned to normal after a long night of restoration work. Calls from BT Senior Management to us last night were appreciated: thank you.
Ultrafast / Connected Cities / Connectivity: UP
Virtual Hosting: UP
Shared Hosting: UP
Co-Lo Hosting: UP (one customer still affected, we're working with them now)
Dedicated Hosting / Private Cloud: UP
Business email: UP. Some individual customers experiencing connection issues related to their phones and laptops. We'll put up a blog piece about that shortly.
Attention now shifts to the power provision by BT as the data centre is still on mains power (as we understand it). That will need moved back to UPS power which could be impactful. We'll be talking to them today to work out what the plans are and how that will affect us.
As feared, the return of power without notification has damaged two of our PDUs. These are pieces of equipment that servers plug into. They are designed to regulate or 'groom' power that goes into servers, as servers are delicate. However they are subject to tripping when excessive or sudden power is offered to them.
We will re-power the servers concerned. The customers affected have been informed.
Email continues to settle down, but connection attempts (computers are often set to automatically check for email) are maxing this server out. This should be resolved soon.
We will continue to work on the sweep-up issues overnight.
We will update you tomorrow morning, unless something unexpected occurs between now and 08.00 Thursday.
Next update Thursday 25th June 08.30
We are getting physical access to the data centre now. By fluke of geography, our HQ is only a 5 minute power walk from the BT Data Centre.
We know of two customer systems that are still unresponsive after power pulled / restored, and will attend to those by hand (engineer en route).
Email: if we provide Anti-Virus scanning and you are an SMTP customer, your email services are unavailable as that system is not responding as yet. If it continues to prove troublesome we will invoke a contingency. No email will be lost. No other email services are affected, have returned to normal.
We are still running some core systems on single power feeds and will have to do so until BT present dual power feeds again. No ETA for that as yet. It is clearly a risk.
All other services have returned to normal.
We will continue to restore the last remaining systems tonight.
We still cannot gain access to the data centre and we cannot get any information from the BT phone number.
One of our ESXi hosts (these are the large servers that host many individual systems) seems to be unresponsive, due to power being removed and / or being resupplied inelegantly. We are now working on that, to try and get those workloads shifted to other machines.
Systems affected (Belfast only)
Email: outgoing services degraded
Virtual servers: services down
Other services are unaffected.
Some of our core equipment is running on one power feed only whilst full / dual power is reestablished by BT.
Next update 16.45
Power: We think that only the A feed or the B feed is operating in the BT data centre. We observed both coming up initially but one seems to have disappeared again. Most of our equipment is dual powered but obviously we're concerned at this state.
We've raised calls to ask the specific question about power provision and will update when we know.
If there any other service providers in that data centre that can help us understand what the actual power provision is to their estate, it would be good to compare notes. Please contact us privately via Twitter.
Email services have almost returned to normal now.
We're seeing power re-established to most of our systems now. We're relying on our monitoring systems telling us which if them have had power re-presented to them. A schedule of power re-provisioning from BT would have been extremely useful here so that we could make sure we do this in some sort of priority order.
The following services pertain ONLY to to Belfast-based services, all London and Dublin services remain unaffected.
Ultrafast / Connected cities: up.
Co-Located Hosting: up.
Shared Hosting: down.
Dedicated Hosting: down
Email services: up (backlog clearing).
More at 14.45
Some of our systems are now visible to our monitoring systems. Presuming power has been reestablished, at least partially.
Still no update from BT and the NOC number diverts to a number that doesn't exist.
We've been told that the damage to the data centre is isolated to one room on the 4th floor. We understand that this room houses the connectivity equipment that feeds the main data centre space.
Power is still off completely for the floor affected at the data centre and we understand that the BT technicians will gain access to the building at around 12.00.
We are hoping that the underlying issue is isolated to power provision and there isn't any underlying damage to fibre connections.
Our biggest concern is around the re-application of power to our own cabinets (these physically house servers). An ungraceful power re-provision process can cause damage to the delicate hosting gear that we (and all service providers) use to provide services. We are trying to get assurances from BT on their timescales and their plans for power re-supply.
We have developed a plan to re-establish email services but this is a non-trivial task, so the important piece of information is an estimate from BT as to when they will re-power our equipment. We do not have that ETD at present (as of 11.47).
We will further update you via this blog and via Twitter https://twitter.com/tibus
Next update: 12.45 BST.
All services provided from our Dublin data centres and our London data centres are unaffected.
BT have confirmed that there is NO fire damage to the floor that houses our equipment. If you have a server co-located with us, or have a private server with us in NI, it has not been damaged, according to BT.
There is "almost certainly" damage to the area of the BT building that houses the various fibre links into that building. We understand that thousands of customers across NI are affected by this.
We are not sure which of our various fibre links traverse the physical area that has been damaged by the fire.
The power to the whole data centre was (quite correctly) cut by the fire service upon their arrival at the scene. It remains off.
Tibus email services are not functioning, but email is being queued. Emails to our business customers will not be lost.
Our next step is to ascertain what physical damage there is to our fibre connections.
We will make a decision as to whether to move to service provision from one of our Dublin data centres, and recover from back-ups. We do not want to do this if BT can recover our fibre connections in a relatively short time-frame. We will try and get an ETA from BT on that now.
Next update 11.45 BST.
===
BT has confirmed there is no damage to the floor where our - and other service providers' - servers are located.
This corroborates earlier reports that it is damage to the wiring and fibre to the servers within the data centre that is causing problems for some Tibus customers.
====
Fire extinguished. BT reporting NO damage to kit on the main 'data centre' floor with the physical data centre building. Damage to the main BT NI switches within their communication area IS expected.
That suggests damage to fibre connections, though this is unclear. They hope Fire Service will grant access to technicians around 10.30. Further updates when we have them.
====
We know this is affecting:
More to follow. Please also follow @tibus on Twitter for additional information as we get it.
