03 November 2010

Free Space and Inodes

Have you ever exhausted capacity on a volume when it still shows storage
space available?  The following details such a situation where the
intention is to simply copy some man pages into a directory.

HOST INFO

        host            beastie
        prompt          beastie [0]
        OS              FreeBSD 8.1
        FILESYSTEM      /mnt/somevol
        NOTE            Though the example uses FreeBSD 8.1, the same
                        scenario and commands involved translate almost
                        identically to prior versions of FreeBSD, as
                        well as Linux, and Solaris.

DETAILS

We've dropped a tarball of man pages to /tmp/manpages.tar and need to
untar it into a directory under /mnt/somevol:

        beastie [0] /bin/ls -Fa /mnt/somevol
        ./      ../     .snap/  app/    bin/    docs/   logs/   pkgs/
        beastie [0] /bin/ls -Fa /mnt/somevol/docs
        ./      ../
        beastie [0] /bin/df -h /mnt/somevol
        Filesystem     Size    Used   Avail Capacity  Mounted on
        /dev/da2s1a    496M     89M    367M    20%    /mnt/somevol
        beastie [0] /usr/bin/du -sh /tmp/manpages.tar
        40K    /tmp/manpages.tar
        beastie [0] /bin/cp /tmp/manpages.tar /mnt/somevol/docs/.
        beastie [0] cd /mnt/somevol/docs
        beastie [0] /usr/bin/tar xf manpages.tar

        /mnt/somevol: create/symlink failed, no inodes free
        manpages/: Can't create 'manpages'
        manpages/setfacl.1.gz: Failed to create dir 'manpages'Can't create 'manpages/setfacl.1.gz'
        manpages/readelf.1.gz: Failed to create dir 'manpages'Can't create 'manpages/readelf.1.gz'
        manpages/ld.so.1.gz: Failed to create dir 'manpages'Can't create 'manpages/ld.so.1.gz'
        manpages/ld.1.gz: Failed to create dir 'manpages'Can't create 'manpages/ld.1.gz'
        manpages/kenv.1.gz: Failed to create dir 'manpages'Can't create 'manpages/kenv.1.gz'
        tar: Error exit delayed from previous errors.
        beastie [1]

We started off with checking what directories were available under
/mnt/somevol, identifying 'docs' as our intended location for the
man pages.  The filesystem shows plenty of space for our 40K tarball.
We copy over the tarball from /tmp, but when we try to untar it under
/mnt/somevol/docs, we get errors about being unable to extract the
tarball.  Additionally, the first error we receive is about file / symlink
creation failed and no inodes free.  So, let's check out our inodes:

        beastie [1] /bin/df -i /mnt/somevol
        Filesystem  1K-blocks  Used  Avail Capacity iused ifree %iused  Mounted on
        /dev/da2s1a    507670 91350 375708    20%   65534     0  100%   /mnt/somevol

Alright, 0 inodes free, how do our directories under /mnt/somevol look:

        beastie [0] cd /mnt/somevol
        beastie [0] /bin/ls -lFa
        total 126
        drwxr-xr-x     8 root  wheel        512 Oct 30 23:12 ./
        drwxr-xr-x     7 root  wheel        512 Oct  7 01:54 ../
        drwxrwxr-x     2 root  operator     512 Oct 30 23:10 .snap/
        drwxr-xr-x  3249 root  wheel     113664 Oct 31 00:37 app/
        drwxr-xr-x     2 root  wheel        512 Oct 30 23:12 bin/
        drwxr-xr-x     2 root  wheel        512 Oct 31 16:34 docs/
        drwxr-xr-x     2 root  wheel        512 Oct 30 23:17 logs/
        drwxr-xr-x     2 root  wheel        512 Oct 30 23:12 pkgs/

3249 links under 'app', and given the rest of the directories have few
links, 'app' is likely our culprit.  Let's see how 'app' looks:

        beastie [0] /usr/bin/find ./app \( -type d -and \! -links 2 -exec /bin/ls -ld {} \; \)
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 00:37 ./app
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:26 ./app/dir0
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:27 ./app/dir1
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:27 ./app/dir10
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:27 ./app/dir100
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:27 ./app/dir1000
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:27 ./app/dir1001
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:28 ./app/dir1002
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:28 ./app/dir1003
        drwxr-xr-x  3249 root  wheel  113664 Oct 31 16:28 ./app/dir1004
        drwxr-xr-x  584 root  wheel  9728 Oct 31 16:28 ./app/dir1005
        drwxr-xr-x  4 root  wheel  512 Oct 31 16:32 ./app/dir384

11 directories under 'app' with more than 2 links each, most with 3249
links, like 'app'.  In totalling the link counts above, that's about
33078 links.  These directories could be normal, so let's look further:

        beastie [0] /usr/bin/find ./app \( -type d -and -links 2 \) -print | /usr/bin/wc -l
           33048

A total of 33048 directories with only two links, meaning empty
directories.  (The current directory (.) and parent directory (..) are
the only two links in these directories.)  This means more than half of
our inodes are being used by empty directories.  Let's have a further
look at the contents of 'app':

        beastie [0] /bin/ls -Fa app
        ./              dir2166/        dir414/         file1584        file2754
        ../             dir2167/        dir415/         file1585        file2755
        dir0/           dir2168/        dir416/         file1586        file2756
        dir1/           dir2169/        dir417/         file1587        file2757
        dir10/          dir217/         dir418/         file1588        file2758
        dir100/         dir2170/        dir419/         file1589        file2759
        dir1000/        dir2171/        dir42/          file159         file276
        <snip...>
        beastie [0] /bin/ls -l app/file1584
        -rw-r--r--  1 root  wheel  0 Oct 30 23:20 app/file1584

Quite a few files and directories with a rather uniform name.  A check of
'file1584', chosen at random, shows 0 bytes in size.  I wonder how many
more empty files there are:

        beastie [0] cd app
        beastie [0] /usr/bin/find . \( -type f -and -size 0 \) | /usr/bin/wc -l
           32470

Adding together the count of empty files (32470) and count of empty
directories (33048) gives us 65518 inodes in use for empty files, nearly
all available inodes for this filesystem.  (Remember, everything in UNIX
is a file (even directories) and each file requires an inode.  So why care
about inodes?  Generically speaking, they contain pointers to the blocks
of data comprising a file.  Without them, it would be fairly tedious to
locate, much less access the contents of a file.)

After reviewing the listing of empty files and directories and the
expected usage of this filesystem, we've determined that these files and
directories are unnecessary and shouldn't exist.  So let's remove them,
starting with our empty files:

        beastie [0] /usr/bin/find . \( -type f -and -size 0 \) -exec /bin/rm {} \;
        beastie [0] /bin/df -i /mnt/somevol
        Filesystem  1K-blocks  Used  Avail Capacity iused ifree %iused  Mounted on
        /dev/da2s1a    507670 91350 375708    20%   33064 32470   50%   /mnt/somevol

Half of our inodes reclaimed, let's kill out empty directories:

        beastie [0] /usr/bin/find . \( -type d -and -links 2 \) -exec /bin/rmdir {} \;
        find: ./dir0/dir0: No such file or directory
        find: ./dir0/dir1: No such file or directory
        find: ./dir0/dir10: No such file or directory
        find: ./dir0/dir100: No such file or directory
        find: ./dir0/dir1000: No such file or directory
        find: ./dir0/dir1001: No such file or directory
        find: ./dir0/dir1002: No such file or directory
        <snip...>
        beastie [0] /bin/df -i /mnt/somevol
        Filesystem  1K-blocks  Used  Avail Capacity iused ifree %iused  Mounted on
        /dev/da2s1a    507670 24244 442814     5%      10 65524    0%   /mnt/somevol
        beastie [0] /bin/ls -Fa
        ./      ../

The errors from find can be safely ignored, the directories are still
being removed.  After the 'rmdir' completed, we can see almost all
of our inodes have been reclaimed.  We also see that we've reclaimed
filesystem space.  This is because each file requires at least a block of
storage, even if it's empty.  Since we just removed all of the empties,
we also freed up storage space.  The following review of our directories
under /mnt/somevol is to illustrate an additional point.  After our
cleanup work, the 'app' directory is now empty, yet we see below that
'app' is 112 KB in size.  This is because the inode for 'app' required
more space to retain data about each of the subsequent links under the
'app' directory.  Assuming no further files are written to 'app', the
space listed should normalize over time.

        beastie [0] cd /mnt/somevol
        beastie [0] /bin/ls -lFa
        total 126
        drwxr-xr-x  8 root  wheel     512 Oct 31 16:57 ./
        drwxr-xr-x  7 root  wheel     512 Oct  7 01:54 ../
        drwxr-xr-x  2 root  wheel     512 Oct 31 16:57 .snap/
        drwxr-xr-x  2 root  wheel  113664 Oct 31 17:08 app/
        drwxr-xr-x  2 root  wheel     512 Oct 30 23:12 bin/
        drwxr-xr-x  2 root  wheel     512 Oct 31 16:34 docs/
        drwxr-xr-x  2 root  wheel     512 Oct 30 23:17 logs/
        drwxr-xr-x  2 root  wheel     512 Oct 31 16:57 pkgs/
        beastie [0] /usr/bin/du -sh ./*
        112K    ./app
        2.0K    ./bin
         42K    ./docs
         24M    ./logs
        2.0K    ./pkgs

So, now that we've cleaned up all of the bogus files and directories, we
can continue with our original plan, untar the contents of manpages.tar:

        beastie [0] cd docs
        beastie [0] ls
        manpages.tar
        beastie [0] /bin/ls -Fa
        ./              ../             manpages.tar
        beastie [0] /usr/bin/tar xf manpages.tar
        beastie [0] /bin/ls -Fa
        ./              ../             manpages/       manpages.tar
        beastie [0] /usr/bin/find . -print
        .
        ./manpages.tar
        ./manpages
        ./manpages/setfacl.1.gz
        ./manpages/readelf.1.gz
        ./manpages/ld.so.1.gz
        ./manpages/ld.1.gz
        ./manpages/kenv.1.gz

The situation above is likely old hat to an experienced UNIX sysadmin,
however, its still easy to get bit by it, especially if one is new to
UNIX administration.  Without getting too far into filesystem internals,
filesystems such as UFS, ext[2|3], and FFS are all primarily bound by
two constraints.  Those constraints are total available storage space on
the filesystem and the number of inodes allocated for use.  While with
UFS and ext[2|3] we can grow our filesystem if we exhaust all free space
(and have contiguous space available to grow into), we cannot do the same
for inodes.  The count of inodes are set during filesystem creation.  The
only way to handle exhaustion of all available inodes is either remove
some files, as seen above, or create a new filesystem configured with an
appropriate number of inodes and move our data into it.  As always, if
you have anything to add to this, such as from your experiences, or any
suggestions or questions, feel free to leave a comment.