≡ Menu

Rollback an apt-get upgrade if something goes wrong on Debian / Ubuntu Linux

Rolling back to the previous version may solve the problem or free the disk space. Both CentOS/RHEL support rollback feature, but I learned hard way both Debian and Ubuntu does not support rollback feature yet.

Know the problem before choosing the solution

I remotely administer a Ubuntu 14.04.3 LTS server in one of those dirt cheap clouds, and I will periodically use apt-get command to install packages or do upgrades. I wanted to set up “Planet Venus” ( a flexible feed aggregator ) software on my system. For some weird reason, I ran:
$ sudo apt-get -y --install-suggests install planet-venus
I should have stopped the installation. But, I was too tired and overworked that day. The result was a disaster. I ran out of disk space, and it installed 1764 packages on the system. My planet-venus installation broke down too. It was one of those days where I made mistakes and forgot to take snapshot/backups. Long story short:

I fucked up :(

 

Related: This is not the first time I f*cked up. See my 10 Unix command line mistakes.

Solution

I tried Google and found this wiki page not so helpful. A couple of mailing list and forum posts did not help at all. Finally, a hint come from Twitter:

How to rollback an apt-get upgrade from command line?

I quickly ran:
# grep -A 2 'Start-Date: 2016-01-17 07:56:42' /var/log/apt/history.log
Sample output (full dump here):

Fig.01: history.log to rescue

Fig.01: history.log to rescue

Rollback / undo an apt-get install command

Rest was easy.

Create the list:

grep -A 2 'Start-Date: 2016-01-17  07:56:42' /var/log/apt/history.log | tail -1 >/tmp/packages.txt

Edit the /tmp/packages.txt file and delete Install: word:

vi /tmp/packages.txt

OR

sed -i 's/Install://' /tmp/packages.txt

Finally, I need to clean up a few things:

tr ',' '\n' < /tmp/packages.txt | sed '/automatic)/d' | awk '{ print $1}' > /tmp/final.packages.txt
wc -l /tmp/final.packages.txt

Sample outputs:

1764 /tmp/final.packages.txt

Delete the packages

Now, I have an entire list of all packages installed on that unfaithful day

# less /tmp/final.packages.txt 
libmaa3:amd64
ant:amd64
libmimic0:amd64
dc:amd64
libparse-yapp-perl:amd64
gir1.2-clutter-1.0:amd64
libjna-java:amd64
python-egenix-mxbeebase:amd64
libxkbcommon-x11-0:amd64
libmpeg2-4:amd64
libopencv-core2.4:amd64
libdvdread4:amd64
libhunspell-1.3-0:amd64
fonts-lobster:amd64
libtotem-plparser18:amd64
libodbcinstq4-1:amd64
jed-common:amd64
....
..
...
xfonts-cyrillic:amd64
postgresql:amd64
db5.3-util:amd64
libopencore-amrnb0:amd64
firebird2.5-examples:amd64
libboost-random1.54-dev:amd64
libtbb2:amd64
libwxgtk2.8-0:amd64
libc6-x32:amd64
magicfilter:amd64

Just uninstall it:

# Run as root
# Store packages name in $p
p="$(</tmp/final.packages.txt)"
 
# Nuke it
apt-get --purge remove $p
 
#clears out the local repository of retrieved package files
apt-get clean
 
# Just in case ...
apt-get autoremove
 
# Verify disk space
df -H

Conclusion

To help yourself, you must be yourself. Be the best that you can be. When you make a mistake, learn from it, pick yourself up and move on. –Dave Pelzer

I learned that:

  1. The best time to backup is before you do major stuff on the server.
  2. Think twice. Hit enter once.
  3. Never trust blindly the apt-get or any command that has -y option.
  4. Always make the snapshot. Unfortunately, this box still uses ext4. There is no option to set my filesystem to BTRFS/ZFS (Linux on ZFS) with this cloud server provider. So I’m stuck with ext4 for now.
Share this on:

Your support makes a big difference:
I have a small favor to ask. More people are reading the nixCraft. Many of you block advertising which is your right, and advertising revenues are not sufficient to cover my operating costs. So you can see why I need to ask for your help. The nixCraft, takes a lot of my time and hard work to produce. If you use nixCraft, who likes it, helps me with donations:
Become a Supporter →    Make a contribution via Paypal/Bitcoin →   

Don't Miss Any Linux and Unix Tips

Get nixCraft in your inbox. It's free:



{ 12 comments… add one }
  • Perry September 11, 2016, 4:35 pm

    Nice, what I have done to avoid this situation I create a bash script that does the updates but only if disk space is lower then 90%. This way avoiding to full out my disk space. : ))

    Reply
  • ziggoto July 28, 2016, 2:39 pm

    You could use Silver Searcher (“ag” command) instead of traditional gnu-grep. It would be much more faster.

    Reply
  • SSDN Technologies May 16, 2016, 12:37 pm

    i think i found best post. thanks.

    Reply
  • qequinet35 March 18, 2016, 9:18 am

    I try to do the same thing on Ubuntu 16.04 it doesn’t work … But this is a good base. I needed to change the last command by this one :

    tr ‘,’ ‘n’ /tmp/final.packages.txt

    The results is the same as your by changing this last command.

    Reply
  • David Bern January 27, 2016, 2:00 pm

    nice article.

    Reply
  • Calin January 20, 2016, 1:42 pm

    OpenSuse with snapper saved my live from a similar issue this morning. You might want to write an article about snapper also.

    Reply
  • Joseph E. Brown January 19, 2016, 4:28 pm

    I’m pretty confident sudo apt-get remove -y planet-venus && sudo apt-get autoremove -y would’ve done the trick.

    Reply
    • nixcraft January 19, 2016, 5:09 pm

      Of course, I tried. But, it didn’t work out for me. Otherwise, I will not write this ;)

      Reply
      • Joseph E. Brown January 19, 2016, 5:28 pm

        REALLY….!
        Now I’m intrigued, I shall investigate…

        Reply
        • Ed McDougal April 15, 2016, 5:40 pm

          It would seem like apt-get ran out of disk space, so perhaps that made it exit ugly and therefore apt didn’t know what state it was in. Just a thought.

          Reply
  • Bruno Maximo e Melo January 19, 2016, 1:13 pm

    PC-BSD boot environments saves me a lot.

    Reply

Leave a Comment