FeM e.V. Infrastructure and System Status

FeM e.V. Infrastructure and System Status

This page shows an overview of the status of infrastructure, services and systems of FeM e.V.

Please keep in mind that all helpers do this voluntarily in their free time. Therefore, there are no guarantees for any given timelines or time estimations.

If you have any questions or specific problems, feel free to send us a ticket via our helpdesk: https://helpdesk.fem.tu-ilmenau.de.

All systems are operational.

Last updated 2 weeks ago
    • Internet Uplink

      Operational
      Last updated 2026-08-23 21:59:06
    • Wireless (Freescale system)

      House A

      Operational
      Last updated 2026-08-01 21:43:16
    • Wireless (Aruba system)

      House B, C, D, E, H, I, K, L, N, P, Q, CJD and FeM-Office

      Operational
      Last updated 2026-07-28 17:49:29
    • Wired

      Ethernet sockets

      Operational
      Last updated 2026-01-19 23:58:28
    • Network Management

      FeM AdminDB and network services (DHCP, DNS...)

      Operational
      Last updated 2026-04-23 21:48:54
    • Helpdesk

      Operational
      Last updated 2026-01-30 15:05:44
    • Storage systems

      Storage systems for servers (e.g. Ceph, ha-storage)

      Operational
      Last updated 2026-01-17 14:09:13
    • Monitoring

      All monitoring systems, e.g. Icinga, Grafana, ...

      Operational
      Last updated 2025-11-04 14:15:17
    • Databases

      Operational
      Last updated 2026-08-23 19:28:39
    • VM-Hosts

      Operational
      Last updated 2026-08-23 19:28:40
    • Mailserver

      Mail servers, inboxes & list systems

      Operational
      Last updated 2026-01-30 15:05:22
    • Webcluster

      Operational
      Last updated 2026-08-23 19:28:38
    • Mastodon

      Federated social network

      Operational
      Last updated 2026-05-10 14:39:06
    • Mattermost

      Internal Messenger

      Operational
      Last updated 2026-08-23 19:28:41
    • Pretix

      A Service to Sell Tickets and Manage Events

      Operational
      Last updated 2026-08-23 19:28:42
    • Dist-Mirror

      A Mirror for some selected Linux Distributions

      Operational
      Last updated 2025-10-31 12:35:48
    • XMPP

      XMPP / Jabber Instant Messaging

      Operational
      Last updated 2026-05-05 23:21:05
    • FeMCI

      FeM Content Infrastructure

      Operational
      Last updated 2026-08-23 19:28:43
    • Homepage

      Our homepage

      Operational
      Last updated 2026-05-22 15:00:39
    • Matrix

      Federated Instant Messaging

      Operational
      Last updated 2026-08-23 19:28:44

Planned maintenance

  • Sunsetting of the FreeScale WiFi (House A)

    2 weeks from now 2026-09-30 22:00 UTC
    Upcoming

    We are shutting down our old WiFi system (FreeScale-WiFi) on 2026-10-01. This WiFi system is currently only used in House A. All other houses are not affected by this change!

    After this date, the usage of the FeM-Net in house A is only possible via LAN or with a private router. Please keep in mind, that the usage of registered private routers (without NAT, AP mode enabled) is only allowed in accordance with the Terms of Use in houses A, N, P and houses in the Professor-Philippow-Str.

    If you have any questions, please open a ticket in our helpdesk at https://helpdesk.fem.tu-ilmenau.de/.

    Affected Components: Wireless (Freescale system)

Past incidents

No incidents reported.

10 months ago 2025-10-17 08:00 UTC
Fixed

Fixed

10 months ago 2025-10-17 19:40 UTC

The module has been replaced. We are currently working on the bootstrapping of all access points

Watching

10 months ago 2025-10-17 18:26 UTC

The switch module was replaced. We are monitoring the situation and the status of all connected access points

Reported

10 months ago 2025-10-17 08:00 UTC

A switch module in House H is having problems and thus some LAN ports are down.

Network expansion in House E

10 months ago 2025-10-17 19:00 UTC
Complete
10 months ago 2025-10-16 19:57 UTC

Caused by other issues in our AP onboarding process the installation schedule has been delayed. The expansion work will continue on 2025-10-17.

Thank you for your patience.

10 months ago 2025-10-16 13:49 UTC

We are currently experiencing problems with some of the newly installed access points not booting up correctly.

Updates on our monitoring system

10 months ago 2025-10-17 18:00 UTC
Complete
10 months ago 2025-10-17 14:49 UTC

The update is now completed and notifications are now reenabled.

There is still a problem with our graphite/grafana subsystem so most graphs on our grafana are missing. A cause was identified and a solution is currently being worked on.

10 months ago 2025-10-16 17:24 UTC

The maintenance has started. Monitoring notifications are now disabled.

Maintenance on this status page

10 months ago 2025-10-17 08:45 UTC
Complete
10 months ago 2025-10-17 08:43 UTC

The maintenance hase been finished.

10 months ago 2025-10-16 19:30 UTC
Fixed
Affected Components: Wireless (Aruba system) and Network Management

Fixed

10 months ago 2025-10-18 20:26 UTC

All affected access points are up and working again.

Watching

10 months ago 2025-10-18 13:26 UTC

We downgraded the version of our DHCP servers, now the accesspoints are starting up again, we're monitoring the situation.

Reported

10 months ago 2025-10-18 10:21 UTC

While trying to investigate the failure source, we restartet a network switch in house K. Unfortunately this lead to more failed access points.

Identified

10 months ago 2025-10-17 21:53 UTC

We have identified a component of our network management infrastructure to be the cause of this behavoir. The access points are manually reachable by our management infrastructure but are unable to broadcast wifi networks or handle traffic.

We are currently looking into the exact cause of this issue.

Reported

10 months ago 2025-10-17 20:47 UTC

We have replaced multiple affected switch modules with working ones. All wired connections are back online. The access points have problems to be readded to the cluster. We are investigating this issue

Identified

10 months ago 2025-10-17 11:12 UTC

We are currently preparing a physical restart of the affected switch module. We hope to finish the on-premise effort in the next hours.

Identified

10 months ago 2025-10-17 10:03 UTC

We are having issues with one of the switch modules not being recognized by the switch therefore not powering up the access points. We are currently trying to restore the module state.

Reported

10 months ago 2025-10-17 08:19 UTC

The situation was degraded overnight. There are currently many offline APs in house K which were unable to start up over night. We are working on a solution for this problem.

Watching

10 months ago 2025-10-16 21:55 UTC

The integration rate has stabilized by now. Almost all access points are up again. We are monitoring the situation.

Reported

10 months ago 2025-10-16 19:30 UTC

We are experiencing partial problems with some of the Aruba access points caused by slow inter-cluster-migration.

10 months ago 2025-10-16 18:00 UTC
Fixed
Affected Components: Wireless (Aruba system) and Network Management

Fixed

10 months ago 2025-10-19 19:32 UTC

We have not found any problems with stability or the setup of access points. Therefore this incident has been resolved.

Watching

10 months ago 2025-10-19 18:44 UTC

We were able to recover all crashed access points. We were able to setup all newly installed access points successfully.

We are monitoring the overall situation.

Reported

10 months ago 2025-10-19 17:48 UTC

The deployed fix failed to hinder the misbehaving access point crashing half of the cluster again, which happend a few minutes ago.

We are working hard on recovering from this incident.

Watching

10 months ago 2025-10-19 16:29 UTC

We have deployed the fix and are in the process of restoring access point setup operations.

Identified

10 months ago 2025-10-19 16:02 UTC

We are in the process of deploying the aforementioned fix. We are monitoring the infrastructure and are ready to rollback the fix

Investigating

10 months ago 2025-10-19 13:40 UTC

We are currently working on a fix to hinder newly registered access points from crashing the cluster. We are planning to deploy this fix in the early evening.

Identified

10 months ago 2025-10-18 22:39 UTC

The newly enabled access point did duplicated DHCP responses which failed about half of all access points in the FeM-Net. We were able to remove the access point physically.

Most of the access points are back online by now and we are trying to reproduce this behavoir in our testing network.

To prevent the same behavoir of other newly installed access points in House E we have disabled most of them.

Reported

10 months ago 2025-10-18 21:37 UTC

Migrating the APs has lead to a cluster failure. We are investigating the causal context.

Identified

10 months ago 2025-10-18 21:03 UTC

We are currently doing a manual on-premise-ap migration and setup into cluster mode.

Identified

10 months ago 2025-10-18 20:31 UTC

Some newly installed access points do not join the access point management cluster as configured, because of network management issues, but rather create "stand alone mesh systems" which are unable to deliver any FeM-WiFi and accept traffic.

We are currently working on a solution for migrating these APs in our cluster.

Investigating

10 months ago 2025-10-17 21:48 UTC

We have identified a component of our network management infrastructure to be the cause of parts of this issue. Please see this incident regarding issues with unresponsive access points.

Reported

10 months ago 2025-10-17 20:44 UTC

The access points do start the automatic configuration now, but fail midway. We are currently investigating this behavoir.

Watching

10 months ago 2025-10-17 11:34 UTC

We have deployed a fix for our setup process. The newly installed access points should now undergo automatic setup. We are monitoring the situation

Identified

10 months ago 2025-10-16 21:38 UTC

The access points seem to be unreachable by our DHCP server. We are currently working on a configuration update for the firewall and core router components to fix this issue.

Reported

10 months ago 2025-10-16 18:00 UTC

We are experiencing issues with initial configuration of the newly installed wifi access points in house E.

Maintenance on network switches

10 months ago 2025-10-16 20:30 UTC
Complete
10 months ago 2025-10-16 19:50 UTC

All updates on our network switches have been installed successfully. All reboots have finished and the switch modules are operating normally.

Therefore this maintenace is finished. Thank you for your patience.

Maintenance on video storage infrastructure

10 months ago 2025-10-16 19:30 UTC
Complete

Some video storage servers will be updated to a new version of their operating system in this time and a restart to apply the update is required, which will impact availability.

Migration of network edge infrastructure

10 months ago 2025-10-16 18:30 UTC
Complete

As part of our regular maintenance, we will be migrating some virtual machines onto different VM hosts. Therefore these VMs need to be shut down in this timeframe and the following functions are partially or fully unavailable

  • SSH proxies

No incidents reported.

No incidents reported.

No incidents reported.

VM-Host Maintenance

10 months ago 2025-10-12 14:30 UTC
Complete

As part of our maintenance, we will be updating the hypervisor software on one of our VM hosts. This update requires a restart of the server, which will result in multiple services being partially or completely unavailable.

This includes:

  • mailing lists
  • xmpp
  • web services
  • linux mirrors