FeM e.V. Infrastructure and System Status

FeM e.V. Infrastructure and System Status

This page shows an overview of the status of infrastructure, services and systems of FeM e.V.

Please keep in mind that all helpers do this voluntarily in their free time. Therefore, there are no guarantees for any given timelines or time estimations.

If you have any questions or specific problems, feel free to send us a ticket via our helpdesk: https://helpdesk.fem.tu-ilmenau.de.

Some systems are experiencing issues.

Last updated 1 day ago
    • Internet Uplink

      Operational
      Last updated 2026-08-23 21:59:06
    • Wireless (Freescale system)

      House A

      Operational
      Last updated 2026-08-01 21:43:16
    • Wireless (Aruba system)

      House B, C, D, E, H, I, K, L, N, P, Q, CJD and FeM-Office

      Operational
      Last updated 2026-07-28 17:49:29
    • Wired

      Ethernet sockets

      Operational
      Last updated 2026-01-19 23:58:28
    • Network Management

      FeM AdminDB and network services (DHCP, DNS...)

      Operational
      Last updated 2026-04-23 21:48:54
    • Helpdesk

      Last updated 2026-01-30 15:05:44
    • Storage systems

      Storage systems for servers (e.g. Ceph, ha-storage)

      Operational
      Last updated 2026-01-17 14:09:13
    • Monitoring

      All monitoring systems, e.g. Icinga, Grafana, ...

      Last updated 2025-11-04 14:15:17
    • Databases

      Last updated 2026-08-23 19:28:39
    • VM-Hosts

      Last updated 2026-08-23 19:28:40
    • Mailserver

      Mail servers, inboxes & list systems

      Last updated 2026-01-30 15:05:22
    • Webcluster

      Last updated 2026-08-23 19:28:38
    • Mastodon

      Federated social network

      Operational
      Last updated 2026-05-10 14:39:06
    • Mattermost

      Internal Messenger

      Last updated 2026-08-23 19:28:41
    • Pretix

      A Service to Sell Tickets and Manage Events

      Last updated 2026-08-23 19:28:42
    • Dist-Mirror

      A Mirror for some selected Linux Distributions

      Operational
      Last updated 2025-10-31 12:35:48
    • XMPP

      XMPP / Jabber Instant Messaging

      Operational
      Last updated 2026-05-05 23:21:05
    • FeMCI

      FeM Content Infrastructure

      Last updated 2026-08-23 19:28:43
    • Homepage

      Our homepage

      Operational
      Last updated 2026-05-22 15:00:39
    • Matrix

      Federated Instant Messaging

      Operational
      Last updated 2026-08-23 19:28:44

Past incidents

1 day ago 2026-08-23 16:00 UTC
Watching
Affected Components: Webcluster, Mailserver, VM-Hosts, Databases, Pretix, FeMCI, Monitoring, Mattermost and Helpdesk

Watching

16 hours ago 2026-08-24 15:40 UTC

The storage pool has now been rebuilt, we are now starting to redeploy VM images, which should result in other services coming back online.

We are closely monitoring the situation.

Identified

16 hours ago 2026-08-24 15:08 UTC

The hard drives have now been replaced. We are in the process to rebuilding the storage pool.

This could also take some time, please be patient.

Identified

21 hours ago 2026-08-24 10:41 UTC

We are currently in the process of replacing the failed drives and recreating the storage pool.

Identified

1 day ago 2026-08-24 00:10 UTC

We have now successfully created backups of all virtual machines and will now start to rebuild the storage of the host machine.

Identified

1 day ago 2026-08-23 18:11 UTC

We are currently making sure to secure all data from the old drives so we can completely renew the faulty storage system afterwards.

Sadly we cannot tell how long this procedure will take.

Reported

1 day ago 2026-08-23 16:00 UTC

We are currently having issues with failing hard drives in one one of our VM hosts.

This results in partial downtimes of some services (web, mail) as we need to replace the drives.

No incidents reported.

No incidents reported.

No incidents reported.

3 weeks ago 2026-07-30 16:00 UTC
Fixed
Affected Components: Wireless (Aruba system) and Wired

Fixed

2 weeks ago 2026-08-05 14:35 UTC

We replaced the switch in CJD.

Watching

3 weeks ago 2026-08-03 18:05 UTC

The switch went down again. We're working on a replacement device.

Fixed

3 weeks ago 2026-08-03 07:37 UTC

The switch is running stable for a few days. We are still monitoring the situation and will probably still replace the device in the next time.

Investigating

3 weeks ago 2026-08-01 10:54 UTC

The currently installed switch has recovered a bit (and does not reboot twice per hour anymore). Therefore, we try to sort the switch setup over the weekend* and will hopefully replace the switch in the next week*.

* Please keep in mind, that there are no guarantees for the given timeline. Delays may occure.

Investigating

3 weeks ago 2026-07-31 20:56 UTC

We are planning to replace this switch, but are facing some crucial setup-issues (as shown in house C). We are currently working on setting up the new switch.

Reported

3 weeks ago 2026-07-30 16:00 UTC

Due to the age of the switch in CJD, we see some stability issues. We are currently working on replacing this switch. Internet connection will suffer

WiFi Expansion in House CJD

3 weeks ago 2026-07-30 17:00 UTC
Complete
Affected Components: Wireless (Freescale system)
3 weeks ago 2026-07-30 19:19 UTC

Most of the access points were now installed and are starting up. This maintenance has now been concluded

3 weeks ago 2026-07-30 09:22 UTC

We were finally able to find a potential cause of this issue. After deploying a fix for the already started access points in the ground floor, these seem to work correctly.

We will now try to enable further access points and will keep you updated.

3 weeks ago 2026-07-29 16:36 UTC

We are on track with installing the new access points, but we are still facing issues regarding onboarding the access points.

3 weeks ago 2026-07-29 08:19 UTC

We are still facing issues regarding initialization of the access points.

3 weeks ago 2026-07-28 15:28 UTC

There are some minor update issues, which delay the access point setup.

3 weeks ago 2026-07-28 13:52 UTC

We have installed access points in the ground and 1st floor. We are currently in the process of starting the access points.

No incidents reported.

3 weeks ago 2026-07-28 15:30 UTC
Fixed
Affected Components: Wireless (Aruba system) and Wired

Fixed

3 weeks ago 2026-07-30 10:23 UTC

The problem has now also been resolved for CJD

Fixed

3 weeks ago 2026-07-29 21:20 UTC

We were able to solve the stability issues for all houses except CJD.

Reported

3 weeks ago 2026-07-28 15:30 UTC

Due to unforeseen problems regarding software version differences between newly installed Access Points and the controlling infrastructure, there will be some minor network stability issues in the next hours, as parts of the systems will need to be updated to a newer software version.

As most of these systems are built redundantly, there should not be major network disruptions.

The newly installed network access points in house CJD cannot be enabled until after this update.

3 weeks ago 2026-07-28 13:30 UTC
Fixed
Affected Components: Wireless (Aruba system) and Wired

Fixed

3 weeks ago 2026-07-31 23:18 UTC

The switch ist operating normally again. This incident has been resolved.

Watching

3 weeks ago 2026-07-31 23:05 UTC

Most infrastructure in house C is back online. We are monitoring the situation.

Watching

3 weeks ago 2026-07-31 22:25 UTC

We were able to restore most of the switch system and we are currently trying to finish up.

Identified

3 weeks ago 2026-07-31 20:54 UTC

We have replaced the broken switch but suffer some severe integration problems. We are currently debugging this problem on-site.

Identified

3 weeks ago 2026-07-30 23:04 UTC

Due to unforeseen problems with configuring a replacement switch, we were unable to install the switch this evening. We hope to retry in the next day.

Identified

3 weeks ago 2026-07-30 19:38 UTC

We are currently working on preparing the switch.

Identified

3 weeks ago 2026-07-29 20:45 UTC

The switch is currently completely down. It will likely need to be replaced entirely. This will take at least one day. In addition, configuration will be necessary.

Investigating

3 weeks ago 2026-07-28 21:01 UTC

The Problem has occured again, we are investigating.

Reported

3 weeks ago 2026-07-28 13:30 UTC

We are experiencing problems with our network switch in House C.

No incidents reported.