How to rollback an apt-get or apt command upgrade if something goes wrong on Debian and Ubuntu Linux
Rolling back to the previous version may solve the problem or free the disk space. Both CentOS/RHEL support rollback feature, but I learned hard way both Debian and Ubuntu does not support rollback feature yet.

Know the problem before choosing the solution

I remotely administer a Ubuntu 14.04.3 LTS server in one of those dirt cheap clouds, and I will periodically use apt-get command/apt command to install packages or do upgrades. I wanted to set up “Planet Venus” ( a flexible feed aggregator ) software on my system. For some weird reason, I ran:
$ sudo apt-get -y --install-suggests install planet-venus
I should have stopped the installation. But, I was too tired and overworked that day. The result was a disaster. I ran out of disk space, and it installed 1764 packages on the system. My planet-venus installation broke down too. It was one of those days where I made mistakes and forgot to take snapshot/backups. Long story short:

 

Related: This is not the first time I f*cked up. See my 10 Unix command line mistakes.

Solution

I tried Google and found this wiki page not so helpful. A couple of mailing list and forum posts did not help at all. Finally, a hint come from Twitter:

Does anyone know a simple way to rollback packages on @Ubuntu Linux 14.04 LTS server? messed up.. I need to rollback over 1k+ packages :(

How to rollback an apt-get upgrade from command line?

I quickly ran:
# grep -A 2 'Start-Date: 2016-01-17 07:56:42' /var/log/apt/history.log
Sample output (full dump here):

Start-Date: 2016-01-17  07:56:42
Commandline: apt-get --install-suggests install planet-venus
Install: libmaa3:amd64 (1.3.2-2, automatic), ant:amd64 (1.9.3-2build1, automatic), libmimic0:amd64 (1.0.4-2.1ubuntu1,
....
...
....
b5.3-util:amd64 (5.3.28-3ubuntu3, automatic), libopencore-amrnb0:amd64 (0.1.3-2ubuntu1, automatic), firebird2.5-examples:amd64 (2.5.2.26540.ds4-9ubuntu1, automatic), libboost-random1.54-dev:amd64 (1.54.0-4ubuntu3.1, automatic), libtbb2:amd64 (4.2~20130725-1.1ubuntu1, automatic), libwxgtk2.8-0:amd64 (2.8.12.1+dfsg-2ubuntu2, automatic), libc6-x32:amd64 (2.19-0ubuntu6.6, automatic), magicfilter:amd64 (1.2-64, automatic)

Rollback / undo an apt-get install command

Rest was easy.

Create the list:

grep -A 2 'Start-Date: 2016-01-17  07:56:42' /var/log/apt/history.log | tail -1 >/tmp/packages.txt

Edit the /tmp/packages.txt file and delete Install: word:

vi /tmp/packages.txt

OR

sed -i 's/Install://' /tmp/packages.txt

Finally, I need to clean up a few things:

tr ',' '\n' < /tmp/packages.txt | sed '/automatic)/d' | awk '{ print $1}' > /tmp/final.packages.txt
wc -l /tmp/final.packages.txt

Sample outputs:

1764 /tmp/final.packages.txt

Delete the packages

Now, I have an entire list of all packages installed on that unfaithful day

# less /tmp/final.packages.txt 
libmaa3:amd64
ant:amd64
libmimic0:amd64
dc:amd64
libparse-yapp-perl:amd64
gir1.2-clutter-1.0:amd64
libjna-java:amd64
python-egenix-mxbeebase:amd64
libxkbcommon-x11-0:amd64
libmpeg2-4:amd64
libopencv-core2.4:amd64
libdvdread4:amd64
libhunspell-1.3-0:amd64
fonts-lobster:amd64
libtotem-plparser18:amd64
libodbcinstq4-1:amd64
jed-common:amd64
....
..
...
xfonts-cyrillic:amd64
postgresql:amd64
db5.3-util:amd64
libopencore-amrnb0:amd64
firebird2.5-examples:amd64
libboost-random1.54-dev:amd64
libtbb2:amd64
libwxgtk2.8-0:amd64
libc6-x32:amd64
magicfilter:amd64

Just uninstall it:

# Run as root
# Store packages name in $p
p="$(</tmp/final.packages.txt)"
 
# Nuke it
apt-get --purge remove $p
 
#clears out the local repository of retrieved package files
apt-get clean
 
# Just in case ...
apt-get autoremove
 
# Verify disk space
df -H

Conclusion

To help yourself, you must be yourself. Be the best that you can be. When you make a mistake, learn from it, pick yourself up and move on. –Dave Pelzer

I learned that:

  1. The best time to backup is before you do major stuff on the server.
  2. Think twice. Hit enter once.
  3. Never trust blindly the apt-get or any command that has -y option.
  4. Always make the snapshot. Unfortunately, this box still uses ext4. There is no option to set my filesystem to BTRFS/ZFS (Linux on ZFS) with this cloud server provider. So I’m stuck with ext4 for now.

Another day, another problem sysadmin solved.


🐧 Get the latest tutorials on Linux, Open Source & DevOps via RSS feed or Weekly email newsletter.

🐧 13 comments so far... add one

CategoryList of Unix and Linux commands
Disk space analyzersdf ncdu pydf
File Managementcat tree
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network UtilitiesNetHogs dig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
13 comments… add one
  • Bruno Maximo e Melo Jan 19, 2016 @ 13:13

    PC-BSD boot environments saves me a lot.

    • Grant McWilliams Jan 25, 2016 @ 1:00

      LVM snapshot if you FS doesn’t support it.

  • Joseph E. Brown Jan 19, 2016 @ 16:28

    I’m pretty confident sudo apt-get remove -y planet-venus && sudo apt-get autoremove -y would’ve done the trick.

    • 🐧 nixcraft Jan 19, 2016 @ 17:09

      Of course, I tried. But, it didn’t work out for me. Otherwise, I will not write this ;)

      • Joseph E. Brown Jan 19, 2016 @ 17:28

        REALLY….!
        Now I’m intrigued, I shall investigate…

        • Ed McDougal Apr 15, 2016 @ 17:40

          It would seem like apt-get ran out of disk space, so perhaps that made it exit ugly and therefore apt didn’t know what state it was in. Just a thought.

  • Calin Jan 20, 2016 @ 13:42

    OpenSuse with snapper saved my live from a similar issue this morning. You might want to write an article about snapper also.

  • David Bern Jan 27, 2016 @ 14:00

    nice article.

  • qequinet35 Mar 18, 2016 @ 9:18

    I try to do the same thing on Ubuntu 16.04 it doesn’t work … But this is a good base. I needed to change the last command by this one :

    tr ‘,’ ‘n’ /tmp/final.packages.txt

    The results is the same as your by changing this last command.

  • SSDN Technologies May 16, 2016 @ 12:37

    i think i found best post. thanks.

  • ziggoto Jul 28, 2016 @ 14:39

    You could use Silver Searcher (“ag” command) instead of traditional gnu-grep. It would be much more faster.

  • Perry Sep 11, 2016 @ 16:35

    Nice, what I have done to avoid this situation I create a bash script that does the updates but only if disk space is lower then 90%. This way avoiding to full out my disk space. : ))

  • whoami Dec 30, 2020 @ 12:27

    You are my savior! thanks

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre> for code samples. Still have questions? Post it on our forum