Overview

While auditing your server statuses, you discover that one of your instances is inaccessible due to the shutdown of the server. When you try to turn it off, it gets caught into a shutdown cycle.

Some of these alarms might be found:

• A stop job is running for Security Auditing Service (9s / 1min 30s)

You also noticed that the startup failure event was being triggered by:

[96.036751] systemd-shutdown[123]: Failed to remount '/var' read-only: Device or resource busy
[96.046047] systemd-shutdown[1]: Failed to wait for process: Protocol error dracut
Warning: Killing all remaining processes. Powering off. [ 97.674793]
Power down.

Information

This issue might be related to the use of disk storage in such a server. You can confirm the use of Disk space on /var with the following command:

df -kh

If you find out that the use is 100%, then the cause of this behavior is that the “Auditing service” is enabled and preventing the system from start-up due to /var file system being full at 100% disk use. To solve this issue, you must disable auditing via GRUB. To do that, complete the following steps:

1. Open the /etc/default/ folder and edit the GRUB file. 
   
   
   
   A file like the following one will open:
   
   
   
   
2. Change the GRUB_CMDLINE_LINUX line to add audit=0 to the end:
   
   
   
   
3. Reboot the server.
   
   
4. Use grub2-mkconfig to regenerate the grub.cfg. grub2-mkconfig scans the hard drives of your server for installed & bootable Linux distributions and generates a GRUB 2 configuration file. This final step also adds a layer of validation between your change, and the running system.

Don't forget to free space from the location you discover it was at 100% of storage occupied.