FeM e.V. Infrastructure and System Status
This page shows an overview of the status of infrastructure, services and systems of FeM e.V.
Please keep in mind that all helpers do this voluntarily in their free time. Therefore, there are no guarantees for any given timelines or time estimations.
If you have any questions or specific problems, feel free to send us a ticket via our helpdesk: https://helpdesk.fem.tu-ilmenau.de.
All systems are operational.
-
-
Internet Uplink
OperationalLast updated 2026-08-23 21:59:06 -
Wireless (Freescale system)
House A
OperationalLast updated 2026-08-01 21:43:16 -
Wireless (Aruba system)
House B, C, D, E, H, I, K, L, N, P, Q, CJD and FeM-Office
OperationalLast updated 2026-07-28 17:49:29 -
Wired
Ethernet sockets
OperationalLast updated 2026-01-19 23:58:28 -
Network Management
FeM AdminDB and network services (DHCP, DNS...)
OperationalLast updated 2026-04-23 21:48:54
-
-
-
Helpdesk
OperationalLast updated 2026-01-30 15:05:44 -
Storage systems
Storage systems for servers (e.g. Ceph, ha-storage)
OperationalLast updated 2026-01-17 14:09:13 -
Monitoring
All monitoring systems, e.g. Icinga, Grafana, ...
OperationalLast updated 2025-11-04 14:15:17 -
Databases
OperationalLast updated 2026-08-23 19:28:39 -
VM-Hosts
OperationalLast updated 2026-08-23 19:28:40 -
Mailserver
Mail servers, inboxes & list systems
OperationalLast updated 2026-01-30 15:05:22 -
Webcluster
OperationalLast updated 2026-08-23 19:28:38
-
-
-
Mastodon
Federated social network
OperationalLast updated 2026-05-10 14:39:06 -
Mattermost
Internal Messenger
OperationalLast updated 2026-08-23 19:28:41 -
Pretix
A Service to Sell Tickets and Manage Events
OperationalLast updated 2026-08-23 19:28:42 -
Dist-Mirror
A Mirror for some selected Linux Distributions
OperationalLast updated 2025-10-31 12:35:48 -
XMPP
XMPP / Jabber Instant Messaging
OperationalLast updated 2026-05-05 23:21:05 -
FeMCI
FeM Content Infrastructure
OperationalLast updated 2026-08-23 19:28:43 -
Homepage
Our homepage
OperationalLast updated 2026-05-22 15:00:39 -
Matrix
Federated Instant Messaging
OperationalLast updated 2026-08-23 19:28:44
-
Planned maintenance
-
Sunsetting of the FreeScale WiFi (House A)
2 weeks from now 2026-09-30 22:00 UTCUpcomingWe are shutting down our old WiFi system (FreeScale-WiFi) on 2026-10-01. This WiFi system is currently only used in House A. All other houses are not affected by this change!
After this date, the usage of the FeM-Net in house A is only possible via LAN or with a private router. Please keep in mind, that the usage of registered private routers (without NAT, AP mode enabled) is only allowed in accordance with the Terms of Use in houses A, N, P and houses in the Professor-Philippow-Str.
If you have any questions, please open a ticket in our helpdesk at https://helpdesk.fem.tu-ilmenau.de/.
Affected Components: Wireless (Freescale system)
Past incidents
No incidents reported.
Fixed
The module has been replaced. We are currently working on the bootstrapping of all access points
Watching
The switch module was replaced. We are monitoring the situation and the status of all connected access points
Reported
A switch module in House H is having problems and thus some LAN ports are down.
Network expansion in House E
Caused by other issues in our AP onboarding process the installation schedule has been delayed. The expansion work will continue on 2025-10-17.
Thank you for your patience.
We are currently experiencing problems with some of the newly installed access points not booting up correctly.
Updates on our monitoring system
The update is now completed and notifications are now reenabled.
There is still a problem with our graphite/grafana subsystem so most graphs on our grafana are missing. A cause was identified and a solution is currently being worked on.
The maintenance has started. Monitoring notifications are now disabled.
Maintenance on this status page
The maintenance hase been finished.
Fixed
All affected access points are up and working again.
Watching
We downgraded the version of our DHCP servers, now the accesspoints are starting up again, we're monitoring the situation.
Reported
While trying to investigate the failure source, we restartet a network switch in house K. Unfortunately this lead to more failed access points.
Identified
We have identified a component of our network management infrastructure to be the cause of this behavoir. The access points are manually reachable by our management infrastructure but are unable to broadcast wifi networks or handle traffic.
We are currently looking into the exact cause of this issue.
Reported
We have replaced multiple affected switch modules with working ones. All wired connections are back online. The access points have problems to be readded to the cluster. We are investigating this issue
Identified
We are currently preparing a physical restart of the affected switch module. We hope to finish the on-premise effort in the next hours.
Identified
We are having issues with one of the switch modules not being recognized by the switch therefore not powering up the access points. We are currently trying to restore the module state.
Reported
The situation was degraded overnight. There are currently many offline APs in house K which were unable to start up over night. We are working on a solution for this problem.
Watching
The integration rate has stabilized by now. Almost all access points are up again. We are monitoring the situation.
Reported
We are experiencing partial problems with some of the Aruba access points caused by slow inter-cluster-migration.
Fixed
We have not found any problems with stability or the setup of access points. Therefore this incident has been resolved.
Watching
We were able to recover all crashed access points. We were able to setup all newly installed access points successfully.
We are monitoring the overall situation.
Reported
The deployed fix failed to hinder the misbehaving access point crashing half of the cluster again, which happend a few minutes ago.
We are working hard on recovering from this incident.
Watching
We have deployed the fix and are in the process of restoring access point setup operations.
Identified
We are in the process of deploying the aforementioned fix. We are monitoring the infrastructure and are ready to rollback the fix
Investigating
We are currently working on a fix to hinder newly registered access points from crashing the cluster. We are planning to deploy this fix in the early evening.
Identified
The newly enabled access point did duplicated DHCP responses which failed about half of all access points in the FeM-Net. We were able to remove the access point physically.
Most of the access points are back online by now and we are trying to reproduce this behavoir in our testing network.
To prevent the same behavoir of other newly installed access points in House E we have disabled most of them.
Reported
Migrating the APs has lead to a cluster failure. We are investigating the causal context.
Identified
We are currently doing a manual on-premise-ap migration and setup into cluster mode.
Identified
Some newly installed access points do not join the access point management cluster as configured, because of network management issues, but rather create "stand alone mesh systems" which are unable to deliver any FeM-WiFi and accept traffic.
We are currently working on a solution for migrating these APs in our cluster.
Investigating
We have identified a component of our network management infrastructure to be the cause of parts of this issue. Please see this incident regarding issues with unresponsive access points.
Reported
The access points do start the automatic configuration now, but fail midway. We are currently investigating this behavoir.
Watching
We have deployed a fix for our setup process. The newly installed access points should now undergo automatic setup. We are monitoring the situation
Identified
The access points seem to be unreachable by our DHCP server. We are currently working on a configuration update for the firewall and core router components to fix this issue.
Reported
We are experiencing issues with initial configuration of the newly installed wifi access points in house E.
Maintenance on network switches
All updates on our network switches have been installed successfully. All reboots have finished and the switch modules are operating normally.
Therefore this maintenace is finished. Thank you for your patience.
Maintenance on video storage infrastructure
Some video storage servers will be updated to a new version of their operating system in this time and a restart to apply the update is required, which will impact availability.
Migration of network edge infrastructure
As part of our regular maintenance, we will be migrating some virtual machines onto different VM hosts. Therefore these VMs need to be shut down in this timeframe and the following functions are partially or fully unavailable
- SSH proxies
No incidents reported.
No incidents reported.
No incidents reported.
VM-Host Maintenance
As part of our maintenance, we will be updating the hypervisor software on one of our VM hosts. This update requires a restart of the server, which will result in multiple services being partially or completely unavailable.
This includes:
- mailing lists
- xmpp
- web services
- linux mirrors