FeM e.V. Infrastructure and System Status

All systems are operational.

Last updated 2 weeks ago

Affected components

10 months ago 2025-10-16 18:00 UTC
Fixed
Affected Components: Wireless (Aruba system) and Network Management

Fixed

10 months ago 2025-10-19 19:32 UTC

We have not found any problems with stability or the setup of access points. Therefore this incident has been resolved.

Watching

10 months ago 2025-10-19 18:44 UTC

We were able to recover all crashed access points. We were able to setup all newly installed access points successfully.

We are monitoring the overall situation.

Reported

10 months ago 2025-10-19 17:48 UTC

The deployed fix failed to hinder the misbehaving access point crashing half of the cluster again, which happend a few minutes ago.

We are working hard on recovering from this incident.

Watching

10 months ago 2025-10-19 16:29 UTC

We have deployed the fix and are in the process of restoring access point setup operations.

Identified

10 months ago 2025-10-19 16:02 UTC

We are in the process of deploying the aforementioned fix. We are monitoring the infrastructure and are ready to rollback the fix

Investigating

10 months ago 2025-10-19 13:40 UTC

We are currently working on a fix to hinder newly registered access points from crashing the cluster. We are planning to deploy this fix in the early evening.

Identified

10 months ago 2025-10-18 22:39 UTC

The newly enabled access point did duplicated DHCP responses which failed about half of all access points in the FeM-Net. We were able to remove the access point physically.

Most of the access points are back online by now and we are trying to reproduce this behavoir in our testing network.

To prevent the same behavoir of other newly installed access points in House E we have disabled most of them.

Reported

10 months ago 2025-10-18 21:37 UTC

Migrating the APs has lead to a cluster failure. We are investigating the causal context.

Identified

10 months ago 2025-10-18 21:03 UTC

We are currently doing a manual on-premise-ap migration and setup into cluster mode.

Identified

10 months ago 2025-10-18 20:31 UTC

Some newly installed access points do not join the access point management cluster as configured, because of network management issues, but rather create "stand alone mesh systems" which are unable to deliver any FeM-WiFi and accept traffic.

We are currently working on a solution for migrating these APs in our cluster.

Investigating

10 months ago 2025-10-17 21:48 UTC

We have identified a component of our network management infrastructure to be the cause of parts of this issue. Please see this incident regarding issues with unresponsive access points.

Reported

10 months ago 2025-10-17 20:44 UTC

The access points do start the automatic configuration now, but fail midway. We are currently investigating this behavoir.

Watching

10 months ago 2025-10-17 11:34 UTC

We have deployed a fix for our setup process. The newly installed access points should now undergo automatic setup. We are monitoring the situation

Identified

10 months ago 2025-10-16 21:38 UTC

The access points seem to be unreachable by our DHCP server. We are currently working on a configuration update for the firewall and core router components to fix this issue.

Reported

10 months ago 2025-10-16 18:00 UTC

We are experiencing issues with initial configuration of the newly installed wifi access points in house E.