We are aware of two incidents relating to the power provision to our London data-centre. The first of these was on the 10th July at around 14:30 and appeared to be a very short failure of the power to the racks which caused systems to reboot. A similar incident occurred on the 17th July at around 15:30. In both instances engineers responded to ensure that the various machines and services were rebooted correctly.
The facility provider has been very slow to provide a detailed explanation as the problem, although the second incident refers to DRUPS (Diesel Rotary Uninterruptible Power Supply) problems due to an abnormal fluctuation in the incoming supply. The load was transferred to multiple DRUPS units two of which subsequently shut themselves down due to alerts from their monitoring systems which meant remaining units struggled to handle the load and so power was switched back to the mains supply. This un-managed transition would have resulted in a momentary loss of power.
We have been actively chasing and continue to press for a full incident report with root cause analysis and remedial work detailed to prevent future similar incidents. As soon as this information is available it can be provided to affected customers.
The datacentre is a major London facility and this issue affected a large number of websites and services. Although its track record is very good and it is a modern facility, issues such as this can occur. Clearly there will be a very significant review of the situation and analysis of why test scenarios did not identify a potential issue.