Fun with debugfs and e2fsck
Hello there,
We are going to corrupt the system and then repair, restore sanity. Things we are going to do:
fun1 → Corrupt the superblock.fun2 → Multiply owned block. fun3 → Corrupted inode.fun4 → Orphaned inode.
Lets begin with perfectly working file system, which has say 10 files.
mkfs.ext3 /dev/xvdb
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Go ahead and mount it then create 10 files.
mount /dev/xvdb /mnt
for i in {1..10}; do echo "this is file$i" > /mnt/file$i.txt ; done
Verify everything looks fine on mount point.
Let the fun begin!
Fun1: Attack Super block
Our first task is to corrupt the super block. Run following command which will zero-out super block!
dd if=/dev/zero of=/dev/xvdb count=1 bs=1024 seek=1
above command,writes 0's on superblock location at offset 1024!
Now lets try to mount it :
umount /mnt && mount /dev/xvdb /mnt
mount: you must specify the filesystem type
failed right? that's what we wanted :)
Now how to fix this ? For cases like this ext3 has backup superblock (have look at above mkfs.ext3 output more closely!) All you need to do is tell mount command to use that copy of superblock.
mount command expects block address in 1KB. so we need to convert our address like 98304 * 4 = 393216 we will try again with alternate superblock option "-o sb=" now:
mount -o sb=393216 /dev/xvdb /mnt
Worked! Just go ahead and explore everything is fine on mountpoint!
Fun 2: Multiply owned blocks - No, Its MY block!
a
Lets check out our existing file contents.
cat /mnt/file1.txt /mnt/file2.txt
this is file1
this is file2
We are going to corrupt file system in such way that file2.txt will also point to file1.txt content!!!!
To modify file system structure, we will a command called debugfs - an ext2/ext3/ext4 file system debugger
First fetch the data block address of file1.txt (i.e where 'this is file1' is stored).
debugfs -s 98304 -b 4096 /dev/xvdb -w
debugfs: stat file1.txt
Inode: 12 Type: regular Mode: 0644 Flags: 0x0
Generation: 1325309953 Version: 0x00000000
User: 0 Group: 0 Size: 14
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x561ff500 -- Thu Oct 15 18:48:32 2015
atime: 0x561ff7f1 -- Thu Oct 15 19:01:05 2015
mtime: 0x561ff500 -- Thu Oct 15 18:48:32 2015
BLOCKS:
(0):2048
TOTAL: 1
Okay, debugfs stat output tells us the file1.txt data block address is 2048
Now lets point file2.txt to this block with another debugfs command named
mi which is powerful command. It gives
up ability to modify inode contents! run
debugfs: mi file2.txt
and keep pressing the enter key
Mode [0100644]
User ID [0]
Group ID [0]
Size [14]
Creation time [1444934912]
Modification time [1444934912]
Access time [1444935703]
Deletion time [0]
Link count [1]
Block count high [0]
until you reach
Direct Block #0
Here type file1.txt block address here (2048)
Direct Block #0 [2049] 2048
Now just press enter until debugfs
debugfs: quit
and quit. To summarize, we figure-out the block address (2048) of file1.txt with
debugfs-stat command, then replaced file2.txt block address 2049 with 2048.
By following unmounting and remounting - we ensure our above changes written
to disk.
umount /mnt
mount -o sb=393216 /dev/xvdb /mnt -t ext3
Now go head print file1.txt file2.txt
cat /mnt/file1.txt /mnt/file2.txt
this is file1
this is file1
I hope you can spot the difference between our previous file1.txt
and file2.txt output ans this one, right? Now lets run fsck to repair this.
Since we can't run file system checker (fsck) on mounted partitions
umount /mnt
e2fsck /dev/xvdb
e2fsck 1.41.12 (17-May-2010)
e2fsck: Superblock invalid, trying backup blocks...
/dev/xvdb was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 12: 2048
Multiply-claimed block(s) in inode 13: 2048
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 2 inodes containing multiply-claimed blocks.)
File /file1.txt (inode #12, mod time Thu Oct 15 18:48:32 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
/file2.txt (inode #13, mod time Thu Oct 15 18:48:32 2015)
Clone multiply-claimed blocks? yes
Hey! see it figures our file1.txt and file2.txt has shared
block 2038 which is wrong. It prompts how to deal with this case, we indeed cloned it.
Now it go on to update other book keeping entries: Just say yes to them.
lock bitmap differences: -2049
Fix? yes
Free blocks count wrong for group #0 (31863, counted=31853).
Fix? yes
Free blocks count wrong (1512305, counted=1512295).
Fix? yes
Free inodes count wrong for group #0 (8181, counted=8171).
Fix? yes
Free inodes count wrong (393205, counted=393195).
Fix? yes
Fun 3: Corrupt permission/type - can you fix it?
Lets use debugfs again, open the drive with debugfs -this time we are going corrupt file 'mode' with modify inode option.
debugfs: mi file3.txt
Mode [0100644] 0
//keep pressing enter key
and then quit debugfs.
debugfs: quit
unmount and mount again if needed, and do:
ls -l /mnt/file3.txt
?---------. 1 root root 14 Oct 15 18:48 /mnt/file3.txt
hihi :) can you see that, we corrupted file permission!!!
I want you to fix this 'mode' with debugfs mi. If you are wondering
what values to put, just look above..no,not the sky :p just scroll above for
original mode values
Fun 4: Sad, Orphaned inodes
You might have seen this error message sometimes, lets recreate
it.
we will create 2 files on a new directory 'dir1'
mkdir /mnt/dir1
echo "this is file-a" > /mnt/dir1/a
echo "this is file-b" > /mnt/dir1/b
with the help of debugfs,we will corrupt dir1 entries!
debugfs clri dir1
what this will do is unlink the chain between directory and its file.
Now go back and list ls -l /mnt/dir1 won't show this file.
You get error like
ls: cannot access /mnt/dir1: Input/output error
Remember file 'a' and 'b' are not deleted. Its laying on our file system
without any directory pointing to it.
If we run e2fcck now, it will find these orphaned inodes and put it under lost+found .
So ahead and run it, you get message like
Pass 4: Checking reference counts
Inode 2 ref count is 4, should be 3. Fix? yes
Unattached inode 32770
Connect to /lost+found? yes
Inode 32770 ref count is 2, should be 1. Fix? yes
Unattached inode 32771
Connect to /lost+found? yes
Inode 32771 ref count is 2, should be 1. Fix? yes
e2fsck has done its job, lets go ahead and explore the end result.
ls -l /mnt/lost+found/
total 8
-rw-r--r-- 1 root root 15 Oct 15 20:28 #32770
-rw-r--r-- 1 root root 15 Oct 15 20:28 #32771
Verify the content of files:
cat \#32770
this is file-a
cat \#32771
this is file-b
Can you guess the how these filename are created?
Think..think..
hmm..No? Have a close look at our e2fsck output. Yes, these the inode numbers
of 'a' and 'b'.
With this we end this document, If you like it let me know, I'll cover corruption cases :)but for now,
I hope hereafter you don't fear these messages and know whats going on your file system.
ps: Last year I wrote a C-Python based corruption framework for OneFS, though I'm not actively seeking employment
right now (working on my project www.webminal.org),
if you have some crazy Btrfs/ext* tasks, do ping me :)