FeM e.V. Infrastructure and System Status

We are seeing system recovery and are monitoring the situation.

We are seeing some network outages after the switches in Houses CJD and K and C have been forcefully rebooted. We suspect a power outage. The WiFi and network connection will suffer.

We are investigeting.

All access points are back online and working correctly.

This incident has been resolved.

We were able to fix some configuration changes (made last Thursday) and to restart the faulty service and its networking interfaces.

We are now seeing recovering Freescale access points and are monitoring the situation.

We have identified a faulty service, which handles orchestrating the access points and does the traffic tunneling. This service is currently unable to connect to the AP network.

We are currently in the process of reestablishing service operations.

We are experiencing total connectivity and availability issues with our Freescale WiFi in Houses A and CJD. All access points are currently offline and do not allow network usage.

We are happy to announce that all systems (except the Freescale WiFi of houses A and CJD) are functioning normally again.

Therefor, this incident has been resolved.

We were able to pinpoint the (hopefully) final cause of this major outage.

Some technical details

The cause was the change of maximum packet size (MTU) between our core network hosts, which was deployed as part of regular maintenance on Thursday evening. This caused internal packet loss together with slow and steady builtup of query and logging backlogs, which caused a major disruption in one of the database master nodes, that handles all of the network management operations, IP assignments and DNS record storage. This database outage resulted in the DHCP issues and the partial inability to register new devices.

As we verified the of the MTU change on a testing network beforehand, we did not suspect it to be a major cause of this outage. After exhausting all other possible causes, we tested a rollback of this change today (Monday), which saw minor improvements. The full rollback was executed in early evening and caused another wave of network disruptions, as most central services needed to reboot.

But the issue still persisted, as one of the database nodes was still faulty. As a last ressort, we fully reset this database node and started a replication from other parts of the core system (which is currently in process). The query and logging backlog has been processed after the faulty database node was removed. As a result, network operations are currently returning back to normal operations (hopefully).

tl;dr;

The core database systems are currently recovering. We are monitoring the situation and hope to see a final full system recovery.

While most parts of the Aruba WiFi are already back online, the recovery of the Freescale WiFi (Houses CJD and A) ist still in progress.

The NAT systems have recovered, DNS/DHCP issues are still persisting.

As mentioned beforehand: we are seeing some issues regarding our NAT systems.

We are seeing partial recovery on DHCP and DNS systems.

We are expecting a short outage in a few minutes, as there will be a small disruption when restarting some services.

We are currently looking into the issue regarding the DNS (which seems to recover right now).

After partially applying the aforementioned changes, we are seeing some more system failures, now regarding our DNS servers.

After carefully rereading the changelogs of the maintenance of last Thrusday, we were able to pinpoint some potential causes of the widespread issues we are facing.

We are in the process of reverting one of these changes and are seeing partial recoveries in parts of our system.

We are experiencing issues with IP address assignemnt.

This results in connection problems and outages.

This incident seems to be linked to this broad network outage.

We are experiencing some performance and stability issues with our AdminDB network maintenance software.

The registering of new devices or configuring the switch ports is currently partially impossible.

This incident has been moved to a new incident

We are still monitoring the situation as parts of the system continue to behave strange.

The storage updates have been deployed and most of the systems seem to have stabilized. We are monitoring the situation bevor returning back to normal operations.

We were able to pinpoint a potential issue. After the core systems maintenance last thursday no schema version upgrade to the underlying storage pool was applied, which caused I/O problems, high load and drive fragmentation. These are potentially valid causes for the current outages.

After applying the storage upgrade on parts of the core systems, partial network recovery was observed.

We are still monitoring the situation and will keep you updated. We will continue the upgrades, if the systems stabilize further.

The root cause of this issue is still not found.

We are still experiencing regular partial network outages.

We are currently experiencing an unexpected outage with our DHCP service. As a result, the automatic assignment of IPv4 addresses is failing across the network.

This issue is connected to this incindet.

Due to currently unknown reasons, we have synchronisation problems between our core database masters, which could lead to network outages or other service disruptions.

We are currently working hard on resolving this issue.