"Read-only file system" on Linux: why the disk went read-only, and how to fix it safely

1. Confirm it’s mounted read-only

findmnt /data

TARGET SOURCE    FSTYPE OPTIONS
/data  /dev/sdb1 ext4   ro,relatime

ro at the start of the options is the answer. To see every read-only mount: grep ' ro,' /proc/mounts. Note the device (/dev/sdb1) — you’ll need it.

2. Find out why: the kernel log

sudo dmesg | tail -n 4

[ 8421.553012] EXT4-fs error (device sdb1): ext4_lookup:1855: inode #131074: comm rsync: deleted inode referenced: 131090
[ 8421.553219] Aborting journal on device sdb1-8.
[ 8421.554901] EXT4-fs (sdb1): Remounting filesystem read-only

(On Ubuntu, reading dmesg needs sudo. sudo journalctl -k shows the same, from earlier boots too.)

• EXT4-fs error … Remounting filesystem read-only — ext4 found an inconsistency, and the mount option errors=remount-ro (the default for root filesystems in Ubuntu’s /etc/fstab) told it to stop writing. The filesystem needs a check (section 3).
• I/O error, blk_update_request, Buffer I/O error, critical medium error — the disk itself is failing, or a cloud volume / SAN path is gone. Copy off important data first, while you can still read it (section 4).
• Nothing like that — maybe it was mounted read-only on purpose (section 5).

3. Repair the filesystem

fsck must run on an unmounted filesystem — on a mounted one it refuses, and forcing it can destroy data:

/dev/sdb1 is mounted.
e2fsck: Cannot continue, aborting.

Stop what uses the filesystem, unmount, check, mount again:

sudo umount /data
sudo fsck -y /dev/sdb1
sudo mount /data

umount: /data: target is busy. means some process (or a shell whose current folder is there) still uses it. sudo fuser -vm /data lists them; stop those services and cd out of the folder.

fsck -y answers “yes” to every repair question. Read the output: lines like Clear? yes and FILE SYSTEM WAS MODIFIED mean it fixed something. Files it couldn’t place end up in lost+found on that filesystem. Then check that the app’s data is complete.

XFS uses sudo xfs_repair /dev/sdb1 instead of fsck (also unmounted). When XFS finds corruption it shuts the filesystem down, and you see Input/output error rather than “read-only”.

The root filesystem can’t be unmounted while the system runs. Schedule the check for the next boot: in the GRUB menu press e, add fsck.mode=force fsck.repair=yes at the end of the linux line, and boot with Ctrl+X (it applies to that boot only). On a cloud server without console access, attach the disk to another machine and check it there.

4. When the disk is failing

If the log shows I/O errors, the filesystem is only the messenger. Check the disk’s own health (package smartmontools):

sudo smartctl -H -a /dev/sdb

SMART overall-health self-assessment test result: FAILED, or growing Reallocated_Sector_Ct / Current_Pending_Sector values, mean: copy the data elsewhere now, and replace the disk. Running fsck on a dying disk can speed up the end. On SD cards (Raspberry Pi) and cheap USB sticks, read-only is often how a worn-out card announces itself. In the cloud, check the volume’s status in the provider’s console.

5. Read-only on purpose

Not every read-only mount is an accident:

• /etc/fstab has ro for that filesystem — someone set it up that way. Change the option and remount.
• ISO images, snapshots, squashfs (including snap packages under /snap) are always read-only.
• Docker: a volume mounted with :ro, or a container started with --read-only / read_only: true. Write to a separate volume or a tmpfs instead.
• Kubernetes: readOnlyRootFilesystem: true in the security context. Give the app an emptyDir volume for the folders it must write to.