Why my Linux server ext3 filesystem go read-only?

From my mailbag:

Advertisement

We have 5 Dell server collocated running CentOS 4.x and 5.x server operating system. Sometime my file system (ext3) goes read-only. I’d like to know what could be causing such a problem?

My guess:
a) Hardware problem / hard disk problem, check harddisk for errors.

b) High disk I/O aka busy I/O retry error can mark low level disk call as failed. This will force ext3 to go into read only mode.

c) High disk I/O on SAN

d) SAN is not configured properly for the path failover.

In all sort of problems ext3 goes read-only to protect the filesystem and further damage. If you are using VMWARE, check out official webpage to download SCSI patches or workaround for vmware problems.

So what could be causing the file system on Linux go read-only?

Apart from above generic problem, any other error can trigger filesystem on Linux go read only. I hope our reader / seasoned Linux admin can help to answer this question. Please share the experiences and advice in the comments.

🥺 Was this helpful? Please add a comment to show your appreciation or feedback.

nixCrat Tux Pixel Penguin
Hi! 🤠
I'm Vivek Gite, and I write about Linux, macOS, Unix, IT, programming, infosec, and open source. Subscribe to my RSS feed or email newsletter for updates.

45 comments… add one
  • KP Oct 15, 2011 @ 22:57

    Thank you for this article, I’ve encountered the same problem. My server was not accessible anymore, nothing to find in the logs after reboot.
    Now I was connected with ssh to update the system and it suddenly was read-only. I guess my usb drive is defect somehow.

    • Arun Oct 28, 2011 @ 10:22

      Try below am succees
      echo j > /proc/sysrq-trigger

      -j Forcibly “Just thaw it” – filesystems frozen by the FIFREEZE ioctl

  • rynn May 14, 2012 @ 2:49

    Posted it in the wrong discussion – so here we go again:
    I think my drive may have crapped. I saw a “Read only file system” trying to log some things. Logged in as root and still no luck writing to the disk. Then forced a fcsk with a reboot – still no luck. A ‘df’ does not even show that drive now. Any thoughts/ideas?

    • KP May 14, 2012 @ 9:17

      My problem was the (USB) disk. I replaced it and it works fine now for over 2 months!

      • rynn May 14, 2012 @ 16:14

        Well, mine is a standalone server – CentOS6.2, 64 cores, 512GB RAM. No external drives/usb sticks connected to it.

  • Laurent Nov 10, 2012 @ 9:14

    I met the same problem, on my company development web server.

    I lost a looot of work… backups were on the same server…

    This ubuntu server was running as a virtual machine on ESXI server.

    I set up a new VM, and a few hours later, I am having now the same problem on a totally new linux installation.

    So, I think I’ll go for a hardrive problem…

    • Benjamin May 4, 2014 @ 1:51

      I hope we have learned not to store backups on a development server.

      And this issues is almost always related to a failing hard drive.

  • Miguel online Jun 14, 2013 @ 9:20

    if the filesystem is damage, does this comand damages filesystem more:

    mount -o remount /

  • nimatramon May 31, 2014 @ 7:55

    mount -o remount / woked for me too
    Thanks a lot

  • Sanjula Jun 10, 2014 @ 6:01

    Hi,
    We are having a server installed Red Hat, kernel (2.6.18-274.3.1.el5) and Red Hat version (Red Hat Enterprise Linux Server release 5.7 (Tikanga)) with Raid 10.
    When we try to copy files to one mounted partition it will change autocratically to read-only. There were some bad sectors in one of Hard disk and it also replaced with a new HDD.
    Kindly required your advice’s for what can cause for the issue even we change the HDD.

    Thanks !

  • olumba Jul 20, 2014 @ 4:27

    mount -o remount / works for me too, thanks man, you saved my life!!!

  • Nuthan Nov 5, 2014 @ 10:43

    ERIK…You saved my Day…thx dude…

  • sachin Nov 27, 2014 @ 16:10

    if vi /etc/sysconfig/readonly-root command run and my system going on read only mode then what can i do?return by in previous mode

  • kevin Mar 21, 2015 @ 4:06

    in every O/S I have seen this occur in, the RO mode is a protective measure,Tthere is an inconsistency in the volume metadata, which may be cached. So, when this happens on Linux, I just reboot the machine. FSCK typically will be called at boot time, and this approach has worked every time.

  • rakesh Jan 1, 2016 @ 18:08

    how to delete wtmp log file last login detail

  • Husain Oct 27, 2016 @ 14:21

    Hi All,
    My server RHEL 6.4 and, my root (/) file system goes in readonly mode after 3 to 4 days.
    it is happening from last 1 month.
    Every time i reboot the server and server comes in writemode.
    Can anyone tell me how can i fix it permanently. guys please help me.

  • VINOD Jan 4, 2017 @ 7:50

    my file system having 777 permissions, but it shows that read only, what is the reason?, and solution

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre> for code samples. Your comment will appear only after approval by the site admin.