FeM e.V. Infrastructure and System Status

FeM e.V. Infrastructure and System Status

This page shows an overview of the status of infrastructure, services and systems of FeM e.V.

Please keep in mind that all helpers do this voluntarily in their free time. Therefore, there are no guarantees for any given timelines or time estimations.

If you have any questions or specific problems, feel free to send us a ticket via our helpdesk: https://helpdesk.fem.tu-ilmenau.de.

Some systems are experiencing issues.

Last updated 1 day ago
    • Internet Uplink

      Operational
      Last updated 2026-08-23 21:59:06
    • Wireless (Freescale system)

      House A

      Operational
      Last updated 2026-08-01 21:43:16
    • Wireless (Aruba system)

      House B, C, D, E, H, I, K, L, N, P, Q, CJD and FeM-Office

      Operational
      Last updated 2026-07-28 17:49:29
    • Wired

      Ethernet sockets

      Operational
      Last updated 2026-01-19 23:58:28
    • Network Management

      FeM AdminDB and network services (DHCP, DNS...)

      Operational
      Last updated 2026-04-23 21:48:54
    • Helpdesk

      Last updated 2026-01-30 15:05:44
    • Storage systems

      Storage systems for servers (e.g. Ceph, ha-storage)

      Operational
      Last updated 2026-01-17 14:09:13
    • Monitoring

      All monitoring systems, e.g. Icinga, Grafana, ...

      Last updated 2025-11-04 14:15:17
    • Databases

      Last updated 2026-08-23 19:28:39
    • VM-Hosts

      Last updated 2026-08-23 19:28:40
    • Mailserver

      Mail servers, inboxes & list systems

      Last updated 2026-01-30 15:05:22
    • Webcluster

      Last updated 2026-08-23 19:28:38
    • Mastodon

      Federated social network

      Operational
      Last updated 2026-05-10 14:39:06
    • Mattermost

      Internal Messenger

      Last updated 2026-08-23 19:28:41
    • Pretix

      A Service to Sell Tickets and Manage Events

      Last updated 2026-08-23 19:28:42
    • Dist-Mirror

      A Mirror for some selected Linux Distributions

      Operational
      Last updated 2025-10-31 12:35:48
    • XMPP

      XMPP / Jabber Instant Messaging

      Operational
      Last updated 2026-05-05 23:21:05
    • FeMCI

      FeM Content Infrastructure

      Last updated 2026-08-23 19:28:43
    • Homepage

      Our homepage

      Operational
      Last updated 2026-05-22 15:00:39
    • Matrix

      Federated Instant Messaging

      Operational
      Last updated 2026-08-23 19:28:44

Past incidents

1 day ago 2026-08-23 16:00 UTC
Watching
Affected Components: Webcluster, Mailserver, VM-Hosts, Databases, Pretix, FeMCI, Monitoring, Mattermost and Helpdesk

Watching

14 hours ago 2026-08-24 15:40 UTC

The storage pool has now been rebuilt, we are now starting to redeploy VM images, which should result in other services coming back online.

We are closely monitoring the situation.

Identified

15 hours ago 2026-08-24 15:08 UTC

The hard drives have now been replaced. We are in the process to rebuilding the storage pool.

This could also take some time, please be patient.

Identified

19 hours ago 2026-08-24 10:41 UTC

We are currently in the process of replacing the failed drives and recreating the storage pool.

Identified

1 day ago 2026-08-24 00:10 UTC

We have now successfully created backups of all virtual machines and will now start to rebuild the storage of the host machine.

Identified

1 day ago 2026-08-23 18:11 UTC

We are currently making sure to secure all data from the old drives so we can completely renew the faulty storage system afterwards.

Sadly we cannot tell how long this procedure will take.

Reported

1 day ago 2026-08-23 16:00 UTC

We are currently having issues with failing hard drives in one one of our VM hosts.

This results in partial downtimes of some services (web, mail) as we need to replace the drives.

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

1 week ago 2026-08-13 10:00 UTC
Fixed
Affected Components: VM-Hosts, Mastodon, Matrix and Storage systems

Fixed

1 week ago 2026-08-17 15:27 UTC

The storage server is back online too. This incident has been resolved.

Watching

1 week ago 2026-08-17 12:52 UTC

After replacing the HBA card in our VM host, the VMs are booting up. We are monitoring the situation.

The problem of the affected storage server is currently being investigated.

Identified

1 week ago 2026-08-14 00:00 UTC

After further investigation, we found two servers, which are malfunctioning since the UPS died (probably caused by voltage fluctuations).

The hardware RAID card in one of our VM hosts has died. This renders most vm boot images unaccessible. A replacement card was already ordered and will be installed in the next days.

In addition, one of our storage systems shows some strange behavoirs amd fails to boot. We are investigating this issue.

Investigating

1 week ago 2026-08-13 12:22 UTC

We have found that one of our VM storage pools is missing.

We are investigating the cause and if there has been any dataloss.

Reported

1 week ago 2026-08-13 10:00 UTC

We are seeing some issues with multiple servers after the power outage in house E. We are investigating.

1 week ago 2026-08-13 05:00 UTC
Fixed
Affected Components: VM-Hosts and Storage systems

Fixed

1 week ago 2026-08-13 19:24 UTC

The situation regarding the Network outage has been resolved.

Identified

1 week ago 2026-08-13 10:46 UTC

It seems like our UPS in this utility room is broken. We are investigating

Watching

1 week ago 2026-08-13 09:59 UTC

The power to our infrastructure has been restored.

We are monitoring the situation.

Investigating

1 week ago 2026-08-13 09:04 UTC

We are investigating the problem on-site and are in the process of reestablishing system power.

Reported

1 week ago 2026-08-13 05:00 UTC

We are currently investigating a complete network outage in Block E.

We are suspecting a power failure inside the utility room.

No incidents reported.