Rolling back to the previous version may solve the problem or free the disk space. Both CentOS/RHEL support rollback feature, but I learned hard way both Debian and Ubuntu does not support rollback feature yet.
Know the problem before choosing the solution
I remotely administer a Ubuntu 14.04.3 LTS server in one of those dirt cheap clouds, and I will periodically use apt-get command/apt command to install packages or do upgrades. I wanted to set up “Planet Venus” ( a flexible feed aggregator ) software on my system. For some weird reason, I ran:
$ sudo apt-get -y --install-suggests install planet-venus
I should have stopped the installation. But, I was too tired and overworked that day. The result was a disaster. I ran out of disk space, and it installed 1764 packages on the system. My planet-venus installation broke down too. It was one of those days where I made mistakes and forgot to take snapshot/backups. Long story short:
Related: This is not the first time I f*cked up. See my 10 Unix command line mistakes.
Solution
I tried Google and found this wiki page not so helpful. A couple of mailing list and forum posts did not help at all. Finally, a hint come from Twitter:
Does anyone know a simple way to rollback packages on @Ubuntu Linux 14.04 LTS server? messed up.. I need to rollback over 1k+ packages :(
How to rollback an apt-get upgrade from command line?
I quickly ran:
# grep -A 2 'Start-Date: 2016-01-17 07:56:42' /var/log/apt/history.log
Sample output (full dump here):
Start-Date: 2016-01-17 07:56:42 Commandline: apt-get --install-suggests install planet-venus Install: libmaa3:amd64 (1.3.2-2, automatic), ant:amd64 (1.9.3-2build1, automatic), libmimic0:amd64 (1.0.4-2.1ubuntu1, .... ... .... b5.3-util:amd64 (5.3.28-3ubuntu3, automatic), libopencore-amrnb0:amd64 (0.1.3-2ubuntu1, automatic), firebird2.5-examples:amd64 (2.5.2.26540.ds4-9ubuntu1, automatic), libboost-random1.54-dev:amd64 (1.54.0-4ubuntu3.1, automatic), libtbb2:amd64 (4.2~20130725-1.1ubuntu1, automatic), libwxgtk2.8-0:amd64 (2.8.12.1+dfsg-2ubuntu2, automatic), libc6-x32:amd64 (2.19-0ubuntu6.6, automatic), magicfilter:amd64 (1.2-64, automatic)
Rollback / undo an apt-get install command
Rest was easy.
Create the list:
grep -A 2 'Start-Date: 2016-01-17 07:56:42' /var/log/apt/history.log | tail -1 >/tmp/packages.txt
Edit the /tmp/packages.txt file and delete Install: word:
vi /tmp/packages.txt
OR
sed -i 's/Install://' /tmp/packages.txt
Finally, I need to clean up a few things:
tr ',' '\n' < /tmp/packages.txt | sed '/automatic)/d' | awk '{ print $1}' > /tmp/final.packages.txt wc -l /tmp/final.packages.txt
Sample outputs:
1764 /tmp/final.packages.txt
Delete the packages
Now, I have an entire list of all packages installed on that unfaithful day
# less /tmp/final.packages.txt libmaa3:amd64 ant:amd64 libmimic0:amd64 dc:amd64 libparse-yapp-perl:amd64 gir1.2-clutter-1.0:amd64 libjna-java:amd64 python-egenix-mxbeebase:amd64 libxkbcommon-x11-0:amd64 libmpeg2-4:amd64 libopencv-core2.4:amd64 libdvdread4:amd64 libhunspell-1.3-0:amd64 fonts-lobster:amd64 libtotem-plparser18:amd64 libodbcinstq4-1:amd64 jed-common:amd64 .... .. ... xfonts-cyrillic:amd64 postgresql:amd64 db5.3-util:amd64 libopencore-amrnb0:amd64 firebird2.5-examples:amd64 libboost-random1.54-dev:amd64 libtbb2:amd64 libwxgtk2.8-0:amd64 libc6-x32:amd64 magicfilter:amd64
Just uninstall it:
# Run as root # Store packages name in $p p="$(</tmp/final.packages.txt)" # Nuke it apt-get --purge remove $p #clears out the local repository of retrieved package files apt-get clean # Just in case ... apt-get autoremove # Verify disk space df -H
Conclusion
To help yourself, you must be yourself. Be the best that you can be. When you make a mistake, learn from it, pick yourself up and move on. –Dave Pelzer
I learned that:
- The best time to backup is before you do major stuff on the server.
- Think twice. Hit enter once.
- Never trust blindly the apt-get or any command that has -y option.
- Always make the snapshot. Unfortunately, this box still uses ext4. There is no option to set my filesystem to BTRFS/ZFS (Linux on ZFS) with this cloud server provider. So I’m stuck with ext4 for now.
Another day, another problem sysadmin solved.
🐧 13 comments so far... add one ↓
| Category | List of Unix and Linux commands |
|---|---|
| File Management | cat |
| Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
| Network Utilities | dig • host • ip • nmap |
| OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
| Package Manager | apk • apt |
| Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
| Searching | grep • whereis • which |
| User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
| WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
PC-BSD boot environments saves me a lot.
LVM snapshot if you FS doesn’t support it.
I’m pretty confident sudo apt-get remove -y planet-venus && sudo apt-get autoremove -y would’ve done the trick.
Of course, I tried. But, it didn’t work out for me. Otherwise, I will not write this ;)
REALLY….!
Now I’m intrigued, I shall investigate…
It would seem like apt-get ran out of disk space, so perhaps that made it exit ugly and therefore apt didn’t know what state it was in. Just a thought.
OpenSuse with snapper saved my live from a similar issue this morning. You might want to write an article about snapper also.
nice article.
I try to do the same thing on Ubuntu 16.04 it doesn’t work … But this is a good base. I needed to change the last command by this one :
tr ‘,’ ‘n’ /tmp/final.packages.txt
The results is the same as your by changing this last command.
i think i found best post. thanks.
You could use Silver Searcher (“ag” command) instead of traditional gnu-grep. It would be much more faster.
Nice, what I have done to avoid this situation I create a bash script that does the updates but only if disk space is lower then 90%. This way avoiding to full out my disk space. : ))
You are my savior! thanks