Partial power failures affecting racks in MS II (Site A; completely repaired)

In the course of January 3-4, 2021, there were partial failures at individual power strips or complete racks in machine room II in the basement of the RZ (server room location A), which have been completely repaired in the meantime. In connection with this, some services in the area of bioinformatics, ATLAS, NEMO and the storage system were (potentially) affected. The control and reconnection are in progress.

The cause is still unclear, but the current incident could be related to a major voltage fluctuation. Our colleagues from the Operations and Grid section were already working on Monday morning to check the first affected systems and switch power strips back on.

Further systems could be successfully supplied with power again at 15:30. Unfortunately, there are still problems with two cabinets in row 2 ... Here, we are still looking for the reason why there is no power.

In the meantime, these could also be reactivated, including the second power strip in cabinet 7 of the same row. The cause of the outages was related to the failure of some transformer stations and fluctuations in the power grid. However, the power was only gone for a very short time, so that many systems ran through. The fuses probably tripped because of the fluctuations.