We are happy to announce that all systems (except the Freescale WiFi of houses A and CJD) are functioning normally again.

Therefor, this incident has been resolved.

We were able to pinpoint the (hopefully) final cause of this major outage.

Some technical details

The cause was the change of maximum packet size (MTU) between our core network hosts, which was deployed as part of regular maintenance on Thursday evening. This caused internal packet loss together with slow and steady builtup of query and logging backlogs, which caused a major disruption in one of the database master nodes, that handles all of the network management operations, IP assignments and DNS record storage. This database outage resulted in the DHCP issues and the partial inability to register new devices.

As we verified the of the MTU change on a testing network beforehand, we did not suspect it to be a major cause of this outage. After exhausting all other possible causes, we tested a rollback of this change today (Monday), which saw minor improvements. The full rollback was executed in early evening and caused another wave of network disruptions, as most central services needed to reboot.

But the issue still persisted, as one of the database nodes was still faulty. As a last ressort, we fully reset this database node and started a replication from other parts of the core system (which is currently in process). The query and logging backlog has been processed after the faulty database node was removed. As a result, network operations are currently returning back to normal operations (hopefully).

tl;dr;

The core database systems are currently recovering. We are monitoring the situation and hope to see a final full system recovery.

While most parts of the Aruba WiFi are already back online, the recovery of the Freescale WiFi (Houses CJD and A) ist still in progress.

The NAT systems have recovered, DNS/DHCP issues are still persisting.

As mentioned beforehand: we are seeing some issues regarding our NAT systems.

We are seeing partial recovery on DHCP and DNS systems.

We are expecting a short outage in a few minutes, as there will be a small disruption when restarting some services.

We are currently looking into the issue regarding the DNS (which seems to recover right now).

After partially applying the aforementioned changes, we are seeing some more system failures, now regarding our DNS servers.

After carefully rereading the changelogs of the maintenance of last Thrusday, we were able to pinpoint some potential causes of the widespread issues we are facing.

We are in the process of reverting one of these changes and are seeing partial recoveries in parts of our system.

We are experiencing issues with IP address assignemnt.

This results in connection problems and outages.