Overview

You may face an issue where one of your RTRs is suffering from diskspace issue under /var. You notice that there are  no obvious problems and no directory is taking up a lot of space, so you suspect that there are some issues with the diskspace reporting. For instance, when you run the df -h command, you may see the /var usage reported in a manner similar to: 

/dev/xxx 79G 66G 8.9G 89% /var

Solution

1. This issue may be caused because a removed file is still in use by some other application, and so it remains available for this application. 
2. This is because the file descriptor in /proc/filesystem is held open. So if there are such open descriptors to files already removed, space occupied by them considered as used by df. However, they are not taken into account by the du command as there are no longer filenames associated with them.
3. You can find all unlinked but held open files with -
   # lsof | grep '(deleted)'
   
4. As the inconsistencies are specifically on /var, execute the following command, which will reveal all deleted files which are still claiming disk space: 
   # lsof  | grep "/var" | grep deleted

For reference, see below an example where a large file is deleted but may still be used by rsyslog: 

When the File Is In Use By rsyslog

To fix the issue where the file is still in use by rsyslog: 

• The rsyslog may be in a failed status as there are no new message files in /var/log. To confirm this, execute the following command: 

  # ls -al /var/log/
  Sample output: 
  
  
  

• Restart the syslog using the following command: 

  # service rsyslog restart
  
  
  

•  A new /var/log/messages will be created and /var disk usage will be reduced. Test this using the following command: 

  # df -kh
  
  
  

• Please keep monitoring /var/log/messages size and the weekly rotation.

In case the above steps do not solve your problem, or if an application other than rsyslog is taking up space and you need assistance in clearing it up, please contact support.

​