[LU-10650] cslco1705 crash: dt_statfs()) ASSERTION( dev ) failed: LBUG, Pid: 3372, comm: lctl

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.11.0
• Labels: patch
• Severity: 3
• Assignee: Alexander Boyko
• Reporter: Alexander Boyko

Description

[1534439.553868] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,errors=panic,journal_checksum,no_mbcache,nodelalloc
[1534439.883377] LustreError: 3372:0:(dt_object.h:2509:dt_statfs()) ASSERTION( dev ) failed: 
[1534439.892393] LustreError: 3372:0:(dt_object.h:2509:dt_statfs()) LBUG
[1534439.899543] Pid: 3372, comm: lctl
[1534439.903791] 
Call Trace:
[1534439.909528]  [<ffffffffa095c7ee>] libcfs_call_trace+0x4e/0x60 [libcfs]
[1534439.917285]  [<ffffffffa095c87c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[1534439.924847]  [<ffffffffa0d3221a>] tgt_statfs_internal+0x2ea/0x350 [ptlrpc]
[1534439.932712]  [<ffffffffa133f306>] ofd_statfs+0x66/0x470 [ofd]
[1534439.939464]  [<ffffffffa0a64dd6>] lprocfs_filesfree_seq_show+0xf6/0x520 [obdclass]
[1534439.948022]  [<ffffffff811b80ea>] ? alloc_pages_vma+0x9a/0x150
[1534439.954855]  [<ffffffff811781ae>] ? lru_cache_add+0xe/0x10
[1534439.959871] Lustre: testfs-OST0000: Imperative Recovery enabled, recovery window shrunk from 900-2700 down to 450-2700
[1534439.973161]  [<ffffffff811a2121>] ? page_add_new_anon_rmap+0x91/0x130
[1534439.980700]  [<ffffffff811c2585>] ? __kmalloc+0x55/0x230
[1534439.987099]  [<ffffffff81202687>] ? seq_buf_alloc+0x17/0x40
[1534439.993785]  [<ffffffffa1353212>] ofd_filesfree_seq_show+0x12/0x20 [ofd]
[1534440.001575]  [<ffffffff81202b7a>] seq_read+0xfa/0x3a0
[1534440.007748]  [<ffffffff8124911d>] proc_reg_read+0x3d/0x80
[1534440.014285]  [<ffffffff811decfc>] vfs_read+0x9c/0x170
[1534440.020499]  [<ffffffff811df84f>] SyS_read+0x7f/0xe0
[1534440.026754]  [<ffffffff81645009>] system_call_fastpath+0x16/0x1b
[1534440.033891] 
[1534440.036630] Kernel panic - not syncing: LBUG

There is a race between lctl get_param obdfilter.*.filesfree and mount operation.
ofd_statfs->tgt_statfs_internal->dt_statfs(env, lut->lut_bottom, osfs)
lut_bottom is NULL during execution and assert, but at vmcore it is initialized already.

ofd_init0() call ofd_procfs_init() and then call tgt_init() which initialize lut_bottom. So a race window exist between procfs touch and full ofd initialization.

Issue Links

• is related to: LU-10820 Interop 2.10.3 <->2.11 sanity test_276: (dt_object.h:2509:dt_statfs()) ASSERTION( dev ) failed