All systems are operational.
All VMs have been restored and are running again.
This incident has been resolved.
We are still in the process of restoring the VM disks. This is taking a bit longer than anticipated, but we hope to finish it soon (only a few are remaining). We are still monitoring the situation.
The storage pool has now been rebuilt, we are now starting to redeploy VM images, which should result in other services coming back online.
We are closely monitoring the situation.
The hard drives have now been replaced. We are in the process to rebuilding the storage pool.
This could also take some time, please be patient.
We are currently in the process of replacing the failed drives and recreating the storage pool.
We have now successfully created backups of all virtual machines and will now start to rebuild the storage of the host machine.
We are currently making sure to secure all data from the old drives so we can completely renew the faulty storage system afterwards.
Sadly we cannot tell how long this procedure will take.
We are currently having issues with failing hard drives in one one of our VM hosts.
This results in partial downtimes of some services (web, mail) as we need to replace the drives.
