≡ Menu

CentOS: rpmdb: PANIC: fatal region error detected; run recovery error and solution

I am using a CentOS Linux v6.5 on my home server and I am getting the following error when I execute yum command:

rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(-30974) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 – (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

How do I fix this problem on a CentOS / RHEL based system?

This is quite the messy situation. You may fix this by cleaning out rpm database. To minimize risk, make a backup of files in /var/lib/rpm/ using cp command:

mkdir /root/backups.rpm.mm_dd_yyyy/
cp -avr /var/lib/rpm/ /root/backups.rpm.mm_dd_yyyy/

To find list of __db* file, enter:
# ls -l /var/lib/rpm/_*
Sample outputs:

-rw-r--r-- 1 root root   24576 Jan 28 04:00 /var/lib/rpm/__db.001
-rw-r--r-- 1 root root  229376 Jan 28 04:00 /var/lib/rpm/__db.002
-rw-r--r-- 1 root root 1318912 Jan 28 04:00 /var/lib/rpm/__db.003
-rw-r--r-- 1 root root  753664 Jan 28 04:00 /var/lib/rpm/__db.004

To fix this problem, try:
# rm -f /var/lib/rpm/__db*
# db_verify /var/lib/rpm/Packages
# rpm --rebuilddb
# yum clean all

Verify that error has gone with the following yum command
# yum update
Sample outputs:

Loaded plugins: downloadonly, fastestmirror, security
Determining fastest mirrors
 * base: centos.mirrors.tds.net
 * extras: centos.corenetworks.net
 * updates: mirror.fdcservers.net
base                                                     | 3.7 kB     00:00
base/primary_db                                          | 4.4 MB     00:00
extras                                                   | 3.4 kB     00:00
extras/primary_db                                        |  19 kB     00:00
updates                                                  | 3.4 kB     00:00
updates/primary_db                                       | 2.0 MB     00:00
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-431.5.1.el6 will be installed
---> Package kernel-firmware.noarch 0:2.6.32-431.3.1.el6 will be updated
---> Package kernel-firmware.noarch 0:2.6.32-431.5.1.el6 will be an update
---> Package mysql-libs.x86_64 0:5.1.71-1.el6 will be updated
---> Package mysql-libs.x86_64 0:5.1.73-3.el6_5 will be an update
---> Package nss.x86_64 0:3.15.3-3.el6_5 will be updated
---> Package nss.x86_64 0:3.15.3-6.el6_5 will be an update
---> Package nss-sysinit.x86_64 0:3.15.3-3.el6_5 will be updated
---> Package nss-sysinit.x86_64 0:3.15.3-6.el6_5 will be an update
---> Package nss-tools.x86_64 0:3.15.3-3.el6_5 will be updated
---> Package nss-tools.x86_64 0:3.15.3-6.el6_5 will be an update
---> Package openldap.x86_64 0:2.4.23-32.el6_4.1 will be updated
---> Package openldap.x86_64 0:2.4.23-34.el6_5.1 will be an update
---> Package p11-kit.x86_64 0:0.18.5-2.el6 will be updated
---> Package p11-kit.x86_64 0:0.18.5-2.el6_5.2 will be an update
---> Package p11-kit-trust.x86_64 0:0.18.5-2.el6 will be updated
---> Package p11-kit-trust.x86_64 0:0.18.5-2.el6_5.2 will be an update
---> Package tzdata.noarch 0:2013i-1.el6 will be updated
---> Package tzdata.noarch 0:2013i-2.el6 will be an update
---> Package wget.x86_64 0:1.12-1.8.el6 will be updated
---> Package wget.x86_64 0:1.12-1.11.el6_5 will be an update
---> Package yum-plugin-downloadonly.noarch 0:1.1.30-14.el6 will be updated
---> Package yum-plugin-downloadonly.noarch 0:1.1.30-17.el6_5 will be an update
---> Package yum-plugin-fastestmirror.noarch 0:1.1.30-14.el6 will be updated
---> Package yum-plugin-fastestmirror.noarch 0:1.1.30-17.el6_5 will be an update
---> Package yum-plugin-security.noarch 0:1.1.30-14.el6 will be updated
---> Package yum-plugin-security.noarch 0:1.1.30-17.el6_5 will be an update
---> Package yum-utils.noarch 0:1.1.30-14.el6 will be updated
---> Package yum-utils.noarch 0:1.1.30-17.el6_5 will be an update
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-358.18.1.el6 will be erased
--> Finished Dependency Resolution
 
Dependencies Resolved
 
================================================================================
 Package                     Arch      Version                Repository   Size
================================================================================
Installing:
 kernel                      x86_64    2.6.32-431.5.1.el6     updates      28 M
Updating:
 kernel-firmware             noarch    2.6.32-431.5.1.el6     updates      13 M
 mysql-libs                  x86_64    5.1.73-3.el6_5         updates     1.2 M
 nss                         x86_64    3.15.3-6.el6_5         updates     822 k
 nss-sysinit                 x86_64    3.15.3-6.el6_5         updates      40 k
 nss-tools                   x86_64    3.15.3-6.el6_5         updates     358 k
 openldap                    x86_64    2.4.23-34.el6_5.1      updates     265 k
 p11-kit                     x86_64    0.18.5-2.el6_5.2       updates      94 k
 p11-kit-trust               x86_64    0.18.5-2.el6_5.2       updates      71 k
 tzdata                      noarch    2013i-2.el6            updates     448 k
 wget                        x86_64    1.12-1.11.el6_5        updates     483 k
 yum-plugin-downloadonly     noarch    1.1.30-17.el6_5        updates      21 k
 yum-plugin-fastestmirror    noarch    1.1.30-17.el6_5        updates      28 k
 yum-plugin-security         noarch    1.1.30-17.el6_5        updates      38 k
 yum-utils                   noarch    1.1.30-17.el6_5        updates     102 k
Removing:
 kernel                      x86_64    2.6.32-358.18.1.el6    @updates    116 M
 
Transaction Summary
================================================================================
Install       1 Package(s)
Upgrade      14 Package(s)
Remove        1 Package(s)
 
Total download size: 45 M
Is this ok [y/N]: y
Downloading Packages:
(1/15): kernel-2.6.32-431.5.1.el6.x86_64.rpm             |  28 MB     00:01
(2/15): kernel-firmware-2.6.32-431.5.1.el6.noarch.rpm    |  13 MB     00:00
(3/15): mysql-libs-5.1.73-3.el6_5.x86_64.rpm             | 1.2 MB     00:00
(4/15): nss-3.15.3-6.el6_5.x86_64.rpm                    | 822 kB     00:00
(5/15): nss-sysinit-3.15.3-6.el6_5.x86_64.rpm            |  40 kB     00:00
(6/15): nss-tools-3.15.3-6.el6_5.x86_64.rpm              | 358 kB     00:00
(7/15): openldap-2.4.23-34.el6_5.1.x86_64.rpm            | 265 kB     00:00
(8/15): p11-kit-0.18.5-2.el6_5.2.x86_64.rpm              |  94 kB     00:00
(9/15): p11-kit-trust-0.18.5-2.el6_5.2.x86_64.rpm        |  71 kB     00:00
(10/15): tzdata-2013i-2.el6.noarch.rpm                   | 448 kB     00:00
(11/15): wget-1.12-1.11.el6_5.x86_64.rpm                 | 483 kB     00:00
(12/15): yum-plugin-downloadonly-1.1.30-17.el6_5.noarch. |  21 kB     00:00
(13/15): yum-plugin-fastestmirror-1.1.30-17.el6_5.noarch |  28 kB     00:00
(14/15): yum-plugin-security-1.1.30-17.el6_5.noarch.rpm  |  38 kB     00:00
(15/15): yum-utils-1.1.30-17.el6_5.noarch.rpm            | 102 kB     00:00
--------------------------------------------------------------------------------
Total                                            18 MB/s |  45 MB     00:02
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
2:irqbalance-1.0.4-8.el6_5.x86_64 is a duplicate with 2:irqbalance-1.0.4-6.el6.x86_64
  Updating   : nss-sysinit-3.15.3-6.el6_5.x86_64                           1/30
  Updating   : nss-3.15.3-6.el6_5.x86_64                                   2/30
  Updating   : nss-tools-3.15.3-6.el6_5.x86_64                             3/30
  Updating   : p11-kit-0.18.5-2.el6_5.2.x86_64                             4/30
  Updating   : kernel-firmware-2.6.32-431.5.1.el6.noarch                   5/30
  Installing : kernel-2.6.32-431.5.1.el6.x86_64                            6/30
  Updating   : p11-kit-trust-0.18.5-2.el6_5.2.x86_64                       7/30
  Updating   : openldap-2.4.23-34.el6_5.1.x86_64                           8/30
  Updating   : wget-1.12-1.11.el6_5.x86_64                                 9/30
  Updating   : yum-plugin-fastestmirror-1.1.30-17.el6_5.noarch            10/30
  Updating   : mysql-libs-5.1.73-3.el6_5.x86_64                           11/30
  Updating   : yum-plugin-downloadonly-1.1.30-17.el6_5.noarch             12/30
  Updating   : yum-utils-1.1.30-17.el6_5.noarch                           13/30
  Updating   : yum-plugin-security-1.1.30-17.el6_5.noarch                 14/30
  Updating   : tzdata-2013i-2.el6.noarch                                  15/30
  Cleanup    : kernel-2.6.32-358.18.1.el6.x86_64                          16/30
  Cleanup    : kernel-firmware-2.6.32-431.3.1.el6.noarch                  17/30
  Cleanup    : yum-plugin-fastestmirror-1.1.30-14.el6.noarch              18/30
  Cleanup    : yum-plugin-downloadonly-1.1.30-14.el6.noarch               19/30
  Cleanup    : yum-utils-1.1.30-14.el6.noarch                             20/30
  Cleanup    : yum-plugin-security-1.1.30-14.el6.noarch                   21/30
  Cleanup    : tzdata-2013i-1.el6.noarch                                  22/30
  Cleanup    : openldap-2.4.23-32.el6_4.1.x86_64                          23/30
  Cleanup    : nss-tools-3.15.3-3.el6_5.x86_64                            24/30
  Cleanup    : nss-sysinit-3.15.3-3.el6_5.x86_64                          25/30
  Cleanup    : nss-3.15.3-3.el6_5.x86_64                                  26/30
  Cleanup    : p11-kit-trust-0.18.5-2.el6.x86_64                          27/30
  Cleanup    : p11-kit-0.18.5-2.el6.x86_64                                28/30
  Cleanup    : wget-1.12-1.8.el6.x86_64                                   29/30
  Cleanup    : mysql-libs-5.1.71-1.el6.x86_64                             30/30
 
 
Removed:
  kernel.x86_64 0:2.6.32-358.18.1.el6
 
Installed:
  kernel.x86_64 0:2.6.32-431.5.1.el6
 
Updated:
  kernel-firmware.noarch 0:2.6.32-431.5.1.el6
  mysql-libs.x86_64 0:5.1.73-3.el6_5
  nss.x86_64 0:3.15.3-6.el6_5
  nss-sysinit.x86_64 0:3.15.3-6.el6_5
  nss-tools.x86_64 0:3.15.3-6.el6_5
  openldap.x86_64 0:2.4.23-34.el6_5.1
  p11-kit.x86_64 0:0.18.5-2.el6_5.2
  p11-kit-trust.x86_64 0:0.18.5-2.el6_5.2
  tzdata.noarch 0:2013i-2.el6
  wget.x86_64 0:1.12-1.11.el6_5
  yum-plugin-downloadonly.noarch 0:1.1.30-17.el6_5
  yum-plugin-fastestmirror.noarch 0:1.1.30-17.el6_5
  yum-plugin-security.noarch 0:1.1.30-17.el6_5
  yum-utils.noarch 0:1.1.30-17.el6_5
 
Complete!
{ 12 comments… add one }
  • danfalos May 28, 2014, 2:05 pm

    Thanks alot for this information.Thanks again.

  • Cihan June 29, 2014, 2:28 am

    That was very helpful and made me solve the problem in minutes. Thanks!

  • Dave July 28, 2014, 10:44 am

    Well written article and very helpful. Thank you – Resolved my problem.

  • Jackson August 24, 2014, 10:24 pm

    Thanks for this. I had a power outage during an update that caused some corruption. Solved my issue.

  • Raz October 10, 2014, 12:04 am

    Very helpful.
    Thanks a lot

  • Vijay January 22, 2015, 10:09 am

    Awesome solution….i was stuck in this issue for past 3 days….the above steps resolved my issue….thanks man!!!

  • Robert Morley May 19, 2015, 6:21 am

    Thank you! As someone who only has linux installed for one specific purpose and has nearly no understanding of it beyond that, this gave me exactly what I needed.

  • Aleksey Tsalolikhin July 1, 2015, 3:08 pm

    Thank you for having consistently excellent and useful content!!

  • virtual July 13, 2015, 3:32 pm

    Never written comments. However, in earnest I have learned a lot Linux and UNIX from cyberciti.biz, specifically your NixCraft site. With guys like Vivek Gite you can’t go wrong.

    Great and trusted work buddies – is all I can say.

    Wow.

  • Turrican August 21, 2015, 6:18 am

    This solved my rpm database problem after a filesystem crash on a SLES machine, thank you very much!

    Under SLES, if you have zypper installed instead of yum, you need to run “zypper clean” instead of “yum clean all”.

    Thanks again!

  • Helio Paixao January 18, 2016, 11:50 am

    Thanks for sharing.

    Helio Paixao

  • Mike Solomon March 23, 2016, 10:43 am

    Thanks for that

    Fixed my problem :)

Leave a Comment


   Tagged with: