≡ Menu

Rollback an apt-get upgrade if something goes wrong on Debian / Ubuntu Linux

Rolling back to the previous version may solve the problem or free the disk space. Both CentOS/RHEL support rollback feature, but I learned hard way both Debian and Ubuntu does not support rollback feature yet.

Know the problem before choosing the solution

I remotely administer a Ubuntu 14.04.3 LTS server in one of those dirt cheap clouds, and I will periodically use apt-get command to install packages or do upgrades. I wanted to set up “Planet Venus” ( a flexible feed aggregator ) software on my system. For some weird reason, I ran:
$ sudo apt-get -y --install-suggests install planet-venus
I should have stopped the installation. But, I was too tired and overworked that day. The result was a disaster. I ran out of disk space, and it installed 1764 packages on the system. My planet-venus installation broke down too. It was one of those days where I made mistakes and forgot to take snapshot/backups. Long story short:

I fucked up :(

 

Related: This is not the first time I f*cked up. See my 10 Unix command line mistakes.

Solution

I tried Google and found this wiki page not so helpful. A couple of mailing list and forum posts did not help at all. Finally, a hint come from Twitter:


How to rollback an apt-get upgrade from command line?

I quickly ran:
# grep -A 2 'Start-Date: 2016-01-17 07:56:42' /var/log/apt/history.log
Sample output (full dump here):

Fig.01: history.log to rescue

Fig.01: history.log to rescue

Rollback / undo an apt-get install command

Rest was easy.

Create the list:

grep -A 2 'Start-Date: 2016-01-17  07:56:42' /var/log/apt/history.log | tail -1 >/tmp/packages.txt

Edit the /tmp/packages.txt file and delete Install: word:

vi /tmp/packages.txt

OR

sed -i 's/Install://' /tmp/packages.txt

Finally, I need to clean up a few things:

tr ',' '\n' < /tmp/packages.txt | sed '/automatic)/d' | awk '{ print $1}' > /tmp/final.packages.txt
wc -l /tmp/final.packages.txt

Sample outputs:

1764 /tmp/final.packages.txt

Delete the packages

Now, I have an entire list of all packages installed on that unfaithful day

# less /tmp/final.packages.txt 
libmaa3:amd64
ant:amd64
libmimic0:amd64
dc:amd64
libparse-yapp-perl:amd64
gir1.2-clutter-1.0:amd64
libjna-java:amd64
python-egenix-mxbeebase:amd64
libxkbcommon-x11-0:amd64
libmpeg2-4:amd64
libopencv-core2.4:amd64
libdvdread4:amd64
libhunspell-1.3-0:amd64
fonts-lobster:amd64
libtotem-plparser18:amd64
libodbcinstq4-1:amd64
jed-common:amd64
....
..
...
xfonts-cyrillic:amd64
postgresql:amd64
db5.3-util:amd64
libopencore-amrnb0:amd64
firebird2.5-examples:amd64
libboost-random1.54-dev:amd64
libtbb2:amd64
libwxgtk2.8-0:amd64
libc6-x32:amd64
magicfilter:amd64

Just uninstall it:

# Run as root
# Store packages name in $p
p="$(</tmp/final.packages.txt)"
 
# Nuke it
apt-get --purge remove $p
 
#clears out the local repository of retrieved package files
apt-get clean
 
# Just in case ...
apt-get autoremove
 
# Verify disk space
df -H

Conclusion

To help yourself, you must be yourself. Be the best that you can be. When you make a mistake, learn from it, pick yourself up and move on. –Dave Pelzer

I learned that:

  1. The best time to backup is before you do major stuff on the server.
  2. Think twice. Hit enter once.
  3. Never trust blindly the apt-get or any command that has -y option.
  4. Always make the snapshot. Unfortunately, this box still uses ext4. There is no option to set my filesystem to BTRFS/ZFS (Linux on ZFS) with this cloud server provider. So I’m stuck with ext4 for now.
Share this on:
{ 12 comments… add one }
  • Perry September 11, 2016, 4:35 pm

    Nice, what I have done to avoid this situation I create a bash script that does the updates but only if disk space is lower then 90%. This way avoiding to full out my disk space. : ))

    Reply
  • ziggoto July 28, 2016, 2:39 pm

    You could use Silver Searcher (“ag” command) instead of traditional gnu-grep. It would be much more faster.

    Reply
  • SSDN Technologies May 16, 2016, 12:37 pm

    i think i found best post. thanks.

    Reply
  • qequinet35 March 18, 2016, 9:18 am

    I try to do the same thing on Ubuntu 16.04 it doesn’t work … But this is a good base. I needed to change the last command by this one :

    tr ‘,’ ‘n’ /tmp/final.packages.txt

    The results is the same as your by changing this last command.

    Reply
  • David Bern January 27, 2016, 2:00 pm

    nice article.

    Reply
  • Calin January 20, 2016, 1:42 pm

    OpenSuse with snapper saved my live from a similar issue this morning. You might want to write an article about snapper also.

    Reply
  • Joseph E. Brown January 19, 2016, 4:28 pm

    I’m pretty confident sudo apt-get remove -y planet-venus && sudo apt-get autoremove -y would’ve done the trick.

    Reply
    • nixcraft January 19, 2016, 5:09 pm

      Of course, I tried. But, it didn’t work out for me. Otherwise, I will not write this ;)

      Reply
      • Joseph E. Brown January 19, 2016, 5:28 pm

        REALLY….!
        Now I’m intrigued, I shall investigate…

        Reply
        • Ed McDougal April 15, 2016, 5:40 pm

          It would seem like apt-get ran out of disk space, so perhaps that made it exit ugly and therefore apt didn’t know what state it was in. Just a thought.

          Reply
  • Bruno Maximo e Melo January 19, 2016, 1:13 pm

    PC-BSD boot environments saves me a lot.

    Reply
Security: Are you a robot or human?

Leave a Comment