Rollback an apt-get upgrade if something goes wrong on Debian / Ubuntu Linux

Rolling back to the previous version may solve the problem or free the disk space. Both CentOS/RHEL support rollback feature, but I learned hard way both Debian and Ubuntu does not support rollback feature yet.

Know the problem before choosing the solution

I remotely administer a Ubuntu 14.04.3 LTS server in one of those dirt cheap clouds, and I will periodically use apt-get command/apt command to install packages or do upgrades. I wanted to set up “Planet Venus” ( a flexible feed aggregator ) software on my system. For some weird reason, I ran:
$ sudo apt-get -y --install-suggests install planet-venus
I should have stopped the installation. But, I was too tired and overworked that day. The result was a disaster. I ran out of disk space, and it installed 1764 packages on the system. My planet-venus installation broke down too. It was one of those days where I made mistakes and forgot to take snapshot/backups. Long story short:

 

Related: This is not the first time I f*cked up. See my 10 Unix command line mistakes.

Solution

I tried Google and found this wiki page not so helpful. A couple of mailing list and forum posts did not help at all. Finally, a hint come from Twitter:

Does anyone know a simple way to rollback packages on @Ubuntu Linux 14.04 LTS server? messed up.. I need to rollback over 1k+ packages :(

How to rollback an apt-get upgrade from command line?

I quickly ran:
# grep -A 2 'Start-Date: 2016-01-17 07:56:42' /var/log/apt/history.log
Sample output (full dump here):

Start-Date: 2016-01-17  07:56:42
Commandline: apt-get --install-suggests install planet-venus
Install: libmaa3:amd64 (1.3.2-2, automatic), ant:amd64 (1.9.3-2build1, automatic), libmimic0:amd64 (1.0.4-2.1ubuntu1,
....
...
....
b5.3-util:amd64 (5.3.28-3ubuntu3, automatic), libopencore-amrnb0:amd64 (0.1.3-2ubuntu1, automatic), firebird2.5-examples:amd64 (2.5.2.26540.ds4-9ubuntu1, automatic), libboost-random1.54-dev:amd64 (1.54.0-4ubuntu3.1, automatic), libtbb2:amd64 (4.2~20130725-1.1ubuntu1, automatic), libwxgtk2.8-0:amd64 (2.8.12.1+dfsg-2ubuntu2, automatic), libc6-x32:amd64 (2.19-0ubuntu6.6, automatic), magicfilter:amd64 (1.2-64, automatic)

Rollback / undo an apt-get install command

Rest was easy.

Create the list:

grep -A 2 'Start-Date: 2016-01-17  07:56:42' /var/log/apt/history.log | tail -1 >/tmp/packages.txt

Edit the /tmp/packages.txt file and delete Install: word:

vi /tmp/packages.txt

OR

sed -i 's/Install://' /tmp/packages.txt

Finally, I need to clean up a few things:

tr ',' '\n' < /tmp/packages.txt | sed '/automatic)/d' | awk '{ print $1}' > /tmp/final.packages.txt
wc -l /tmp/final.packages.txt

Sample outputs:

1764 /tmp/final.packages.txt

Delete the packages

Now, I have an entire list of all packages installed on that unfaithful day

# less /tmp/final.packages.txt 
libmaa3:amd64
ant:amd64
libmimic0:amd64
dc:amd64
libparse-yapp-perl:amd64
gir1.2-clutter-1.0:amd64
libjna-java:amd64
python-egenix-mxbeebase:amd64
libxkbcommon-x11-0:amd64
libmpeg2-4:amd64
libopencv-core2.4:amd64
libdvdread4:amd64
libhunspell-1.3-0:amd64
fonts-lobster:amd64
libtotem-plparser18:amd64
libodbcinstq4-1:amd64
jed-common:amd64
....
..
...
xfonts-cyrillic:amd64
postgresql:amd64
db5.3-util:amd64
libopencore-amrnb0:amd64
firebird2.5-examples:amd64
libboost-random1.54-dev:amd64
libtbb2:amd64
libwxgtk2.8-0:amd64
libc6-x32:amd64
magicfilter:amd64

Just uninstall it:

# Run as root
# Store packages name in $p
p="$(</tmp/final.packages.txt)"
 
# Nuke it
apt-get --purge remove $p
 
#clears out the local repository of retrieved package files
apt-get clean
 
# Just in case ...
apt-get autoremove
 
# Verify disk space
df -H

Conclusion

To help yourself, you must be yourself. Be the best that you can be. When you make a mistake, learn from it, pick yourself up and move on. –Dave Pelzer

I learned that:

1. The best time to backup is before you do major stuff on the server.
2. Think twice. Hit enter once.
3. Never trust blindly the apt-get or any command that has -y option.
4. Always make the snapshot. Unfortunately, this box still uses ext4. There is no option to set my filesystem to BTRFS/ZFS (Linux on ZFS) with this cloud server provider. So I’m stuck with ext4 for now.

Another day, another problem sysadmin solved.