I had carelessly allowed the root filesystem on one of my Ubuntu servers to become full. Here's how I recovered from it.

Setup: Ubuntu server 16.04, entire disk for LVM, all disk space assigned to one volume group (VG), root (4GB) and swap (1.5GB) in their own logical volumes (LV), with the remaining disk space allocated to LXD containers using LVM thin provisioning.

The root filesystem, from here on referred to as root if the context is clear, had become full because I hadn't been purging old Linux kernels. Being full, the filesystem was then unable to complete "apt-get autoremove". I needed to add some space to root, which I took from swap.

Following commands were performed online with the server in multi-user mode. First, turn off swap.

# swapoff -a

Then, reduce the size of swap LV by 1GB.

# lvresize -L-1G /dev/srv-vg/swap_1

Add the 1GB taken to root; the command grows the root LV and resizes the filesystem within the LV.

# lvextend -L+1G /dev/srv-vg/root -r -v

"df -k" shows that root now has an additional 1GB and is no longer full. Fix any broken apt dependencies and purge the old Linux kernels.

# apt-get -f install
# apt-get autoremove

"df -k" now shows that root has even more free space.

Now I set about to give the 1GB back to swap. While it is possible to grow root's LV and filesystem online, shrinking root needs to be done offline by booting from live DVD into single user mode at the server console. (The server runs virtually somewhere "in the cloud". Console access is via VNC. I use the Linux client Vinagre. Obligatory Smalltalk content: Vinagre is also the client I use to connect to RFBServer running in the Pharo image that runs this blog.)

For below commands, I'm at single user mode, booted from live DVD, with all server filesystems unmounted. First, check the root filesystem.

# e2fsck -f /dev/srv-vg/root

Remember the root filesystem is contained within the root LV. Now resize the filesystem. Here "4G" refers to the absolute size.

# resize2fs /dev/srv-vg/root 4G

Next, resize the root LV.

# lvresize -L-1G /dev/srv-vg/root

Check root filesystem again.

# e2fsck -fy /dev/srv-vg/root

Now, give 1GB back to swap.

# lvresize -L+1G /dev/srv-vg/swap_1

Reboot into multi-user mode, turn swap back on, and look around.

This incident happened to the server that is running this blog. That you are now reading this blog post means that the above procedure succeeded. :-P

Tags: Linux