[LU-11685] removing file names and freeing inodes are not atomic on the OST

Related — Fri, 15 Mar 2019 11:13:14 +0000

Destroying objects on OSTs from ldiskfs layout POV includes removing them from their directories and freeing their inodes. Since in ext4 they are usually performed in different jbd2 transactions, the first part in ext4 also links the inode to the orphan list. However, in Lustre 2.11 and master there are two different transactions and no orphan list linkage between them.

           <...>-112262 [000] .... 37386.078026: ldiskfs_free_inode <-ldiskfs_evict_inode
           <...>-112262 [000] .... 37386.078050: <stack trace>
 => evict
 => iput
 => osd_object_delete
 => lu_object_free.isra.30
 => lu_object_put
 => ofd_destroy_by_fid
 => ofd_destroy_hdl
 => tgt_request_handle
 => ptlrpc_server_handle_request
 => ptlrpc_main
 => kthread
 => ret_from_fork

The issue can be easily reproduced by forcing transaction commit between those two and crashing the system:

diff --git a/lustre/ofd/ofd_objects.c b/lustre/ofd/ofd_objects.c
index 7605de4..f973923 100644
--- a/lustre/ofd/ofd_objects.c
+++ b/lustre/ofd/ofd_objects.c
@@ -876,6 +876,11 @@ stop:
                rc = rc2;
 unlock:
        ofd_write_unlock(env, fo);
+
+       set_current_state(TASK_UNINTERRUPTIBLE);
+       schedule_timeout(msecs_to_jiffies(MSEC_PER_SEC * 6));
+       BUG();
+
        RETURN(rc);
 }

e2fsck finds inconsistency on the OST image:

[root@panda-vbox lustre-release]# e2fsck -f /dev/sdd
e2fsck 1.42.13.wc6 (05-Feb-2017)
lustre-OST0000: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 134 has zero dtime.  Fix<y>? yes <==============
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (80310, counted=79979).
Fix<y>? yes
Inode bitmap differences:  -134
Fix<y>? yes
Free inodes count wrong for group #0 (24835, counted=24836).
Fix<y>? yes
Free inodes count wrong (99987, counted=99668).
Fix<y>? yes
[QUOTA WARNING] Usage inconsistent for ID 0:actual (1392640, 323) != expected (1392640, 324)
Update quota info for quota type 0<y>? yes
[QUOTA WARNING] Usage inconsistent for ID 0:actual (1392640, 323) != expected (1392640, 324)
Update quota info for quota type 1<y>? yes

lustre-OST0000: ***** FILE SYSTEM WAS MODIFIED *****
lustre-OST0000: 332/100000 files (0.3% non-contiguous), 20021/100000 blocks
[root@panda-vbox lustre-release]#

Before suggesting a patch, I'd like to discuss possible solutions. Should we try extending the transaction so as to include the final iput? Should we use any other approach?

Whamcloud Community JIRA

[LU-11685] removing file names and freeing inodes are not atomic on the OST