Partial disruption of data center infrastructure and services due to power fluctuation on Sunday morning

On Sunday morning there were problems with the power supply. The UPS had to be switched back to battery operation manually first, the cause is not yet known. Although most of the servers in the virtualization, bw/de.NBI-Cloud, NEMO, ... ran through, there were side effects in the management cluster (Postgress, HiS) due to a defective FC switch. Likewise, some power supplies and thus services in other infrastructures are irreparably damaged.

A colleague from the company was in the IT Services building on Sunday because the UPS had switched to bypass. The emergency lighting was in operation in the buidling, which indicates problems with the external power supply. There was probably a fluctuation in the power supply that had not yet been fully analyzed, which generated some side effects even though the greater part of the infrastructure was running through. In the area of administrative IT, three terminal servers broke down and had to be reinstalled. A UPS (uninterruptible power supply) also broke in this connection, but it was to be abolished anyway.

Furthermore, the server for controlling virtualization in NEMO was affected and no longer came up cleanly; there were a number of error messages on the bwSFS storage system, and likewise some of the Isilon nodes reported the loss of a supply strip.