Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-2251] MDS crash will running mdtest https://jira.whamcloud.com/browse/LU-2251 Lustre <p>While testing for stability with mdtest the MDS experienced a Oops and I had to restart the file system which recovered successfully. Will attach the oops.</p> DDN 9950 sotrage backend with 4 OSS and a MDS and MGS server. Client are a Cray XE cage. LU-2251

MDS crash will running mdtest

Bug Blocker Resolved Fixed Alex Zhuravlev James A Simmons Tue, 30 Oct 2012 14:16:47 +0000 Fri, 19 Apr 2013 20:43:11 +0000 Tue, 27 Nov 2012 23:37:32 +0000 Lustre 2.4.0 Lustre 2.4.0 0 5 <p>please try with <a href="http://review.whamcloud.com/#change,4390" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4390</a></p> <p>Appears to have worked. Thanks. I will report if the problem shows again.</p> <p>I was wrong. I get a very similar but slightly different oops.</p> <p>Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764785] LustreError: 9696:0:(osp_dev.c:337:osp_statfs()) ASSERTION( sfs->os_fprecreated <= 20000 ) failed<br/> : <br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764789] LustreError: 9696:0:(osp_dev.c:337:osp_statfs()) LBUG<br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764791] Pid: 9696, comm: mdt01_497<br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764793] <span class="error">[<ffffffffa04fcf17>]</span> lbug_with_loc+0x47/0xb0 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764795] <br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764796] Call Trace:<br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764811] <span class="error">[<ffffffffa095c7da>]</span> osp_statfs+0x28a/0x2a0 <span class="error">[osp]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764846] <span class="error">[<ffffffffa0777e8e>]</span> lod_statfs_and_check+0x8e/0x4a0 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764854] <span class="error">[<ffffffffa04fc905>]</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764856] <span class="error">[<ffffffffa077a430>]</span> lod_alloc_rr.clone.2+0x500/0xd90 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764868] <span class="error">[<ffffffffa04fcf17>]</span> lbug_with_loc+0x47/0xb0 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764870] <span class="error">[<ffffffffa077c04c>]</span> lod_qos_prep_create+0xe5c/0x1848 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764886] <span class="error">[<ffffffffa095c7da>]</span> osp_statfs+0x28a/0x2a0 <span class="error">[osp]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764922] <span class="error">[<ffffffffa0777e8e>]</span> lod_statfs_and_check+0x8e/0x4a0 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764924] <span class="error">[<ffffffffa0e6c692>]</span> ? osd_declare_inode_qid+0x1a2/0x270 <span class="error">[osd_ldiskfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764933] <span class="error">[<ffffffffa077a430>]</span> lod_alloc_rr.clone.2+0x500/0xd90 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764935] <span class="error">[<ffffffffa0776c7b>]</span> lod_declare_striped_object+0x14b/0x920 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764943] <span class="error">[<ffffffffa077c04c>]</span> lod_qos_prep_create+0xe5c/0x1848 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764945] <span class="error">[<ffffffffa0777758>]</span> lod_declare_object_create+0x308/0x4f0 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764974] <span class="error">[<ffffffffa0d3eccf>]</span> mdd_declare_object_create_internal+0xaf/0x1d0 <span class="error">[mdd]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764991] <span class="error">[<ffffffffa0e6c692>]</span> ? osd_declare_inode_qid+0x1a2/0x270 <span class="error">[osd_ldiskfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.764994] <span class="error">[<ffffffffa0d4fc9a>]</span> mdd_create+0x39a/0x1550 <span class="error">[mdd]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765002] <span class="error">[<ffffffffa0776c7b>]</span> lod_declare_striped_object+0x14b/0x920 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765012] <span class="error">[<ffffffffa0777758>]</span> lod_declare_object_create+0x308/0x4f0 <span class="error">[lod]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765034] <span class="error">[<ffffffffa0d3eccf>]</span> mdd_declare_object_create_internal+0xaf/0x1d0 <span class="error">[mdd]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765048] <span class="error">[<ffffffffa0d4fc9a>]</span> mdd_create+0x39a/0x1550 <span class="error">[mdd]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765056] <span class="error">[<ffffffffa0efc519>]</span> mdt_reint_open+0x1079/0x1860 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765068] <span class="error">[<ffffffffa0d55bee>]</span> ? md_ucred+0x1e/0x60 <span class="error">[mdd]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765083] <span class="error">[<ffffffffa0ec7765>]</span> ? mdt_ucred+0x15/0x20 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765085] <span class="error">[<ffffffffa0efc519>]</span> mdt_reint_open+0x1079/0x1860 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765095] <span class="error">[<ffffffffa0d55bee>]</span> ? md_ucred+0x1e/0x60 <span class="error">[mdd]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765097] <span class="error">[<ffffffffa0ee73e1>]</span> mdt_reint_rec+0x41/0xe0 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765108] <span class="error">[<ffffffffa0ec7765>]</span> ? mdt_ucred+0x15/0x20 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765111] <span class="error">[<ffffffffa0ee0b73>]</span> mdt_reint_internal+0x4f3/0x7f0 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765122] <span class="error">[<ffffffffa0ee73e1>]</span> mdt_reint_rec+0x41/0xe0 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765125] <span class="error">[<ffffffffa0ee113d>]</span> mdt_intent_reint+0x1ed/0x500 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765136] <span class="error">[<ffffffffa0ee0b73>]</span> mdt_reint_internal+0x4f3/0x7f0 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765138] <span class="error">[<ffffffffa0edcb0e>]</span> mdt_intent_policy+0x38e/0x770 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765279] <span class="error">[<ffffffffa07c3b21>]</span> ldlm_lock_enqueue+0x361/0x8f0 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765281] <span class="error">[<ffffffffa07c3b21>]</span> ldlm_lock_enqueue+0x361/0x8f0 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765330] <span class="error">[<ffffffffa07eb767>]</span> ldlm_handle_enqueue0+0x4e7/0x1000 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765343] <span class="error">[<ffffffffa0edd046>]</span> mdt_enqueue+0x46/0x130 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765356] <span class="error">[<ffffffffa0ed0322>]</span> mdt_handle_common+0x932/0x1740 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765364] <span class="error">[<ffffffffa07eb767>]</span> ldlm_handle_enqueue0+0x4e7/0x1000 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765370] <span class="error">[<ffffffffa0ed1205>]</span> mdt_regular_handle+0x15/0x20 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765378] <span class="error">[<ffffffffa0edd046>]</span> mdt_enqueue+0x46/0x130 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765392] <span class="error">[<ffffffffa0ed0322>]</span> mdt_handle_common+0x932/0x1740 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765405] <span class="error">[<ffffffffa0ed1205>]</span> mdt_regular_handle+0x15/0x20 <span class="error">[mdt]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765412] <span class="error">[<ffffffffa081b5dc>]</span> ptlrpc_server_handle_request+0x41c/0xe00 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765424] <span class="error">[<ffffffffa04fd65e>]</span> ? cfs_timer_arm+0xe/0x10 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765440] <span class="error">[<ffffffffa050f17f>]</span> ? lc_watchdog_touch+0x6f/0x180 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765454] <span class="error">[<ffffffffa081b5dc>]</span> ptlrpc_server_handle_request+0x41c/0xe00 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765466] <span class="error">[<ffffffffa04fd65e>]</span> ? cfs_timer_arm+0xe/0x10 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765479] <span class="error">[<ffffffffa08129a9>]</span> ? ptlrpc_wait_event+0xa9/0x2a0 <span class="error">[ptlrpc]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765483] <span class="error">[<ffffffffa050f17f>]</span> ? lc_watchdog_touch+0x6f/0x180 <span class="error">[libcfs]</span><br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765486] <span class="error">[<ffffffff810602b0>]</span> ? default_wake_function+0x0/0x20<br/> Oct 31 08:38:31 barry-mds1 kernel: [ 1249.765522] <span class="error">[<ffffffffa08129a9>]</span> ? ptlrpc_wait_event+0xa9/0x2a0 <span class="error">[ptlrpc]</span></p> <p>James, were there timeouts/disconnects on MDS just before the assertion ?</p> <p>hmm, clearly the assertion is invalid: we initiate pre-creation before existing pool is exhausted, so the final pool can easily be OST_MAX_PRECERATE + number of objects from the previous pre-creation.</p> <p><a href="http://review.whamcloud.com/#change,4495" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4495</a></p> <p>No timeouts or disconnects happened before the assertion. I found you can only reproduce this problem when you have a newly formatted file system and the first thing you do is run mdtest on it. The exact command used was mdtest -n 1000 -F -i 5 -d /lustre/$MOUNT/scratch/$USER/md_test over 3 nodes. If I ran IOR first then the problem didn't happen. Will apply your patch.</p> <p>We hit the same assertion on a test filesystem running <a href="https://github.com/chaos/lustre/tree/2.3.54-2chaos" class="external-link" target="_blank" rel="nofollow noopener">2.3.54-2chaos</a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 6847:0:(osp_dev.c:337:osp_statfs()) ASSERTION( sfs->os_fprecreated <= 20000 ) failed: LustreError: 6847:0:(osp_dev.c:337:osp_statfs()) LBUG </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 6847 TASK: ffff88061df67500 CPU: 9 COMMAND: "mdt03_008" #0 [ffff8805f8c7b518] machine_kexec at ffffffff8103216b #1 [ffff8805f8c7b578] crash_kexec at ffffffff810b8d12 #2 [ffff8805f8c7b648] panic at ffffffff814eea99 #3 [ffff8805f8c7b6c8] lbug_with_loc at ffffffffa05c1fcb [libcfs] #4 [ffff8805f8c7b6e8] osp_statfs at ffffffffa0eb97da [osp] #5 [ffff8805f8c7b728] lod_statfs_and_check at ffffffffa0d6de8e [lod] #6 [ffff8805f8c7b758] lod_alloc_rr.clone.2 at ffffffffa0d70430 [lod] #7 [ffff8805f8c7b808] lod_qos_prep_create at ffffffffa0d7204c [lod] #8 [ffff8805f8c7b8a8] lod_declare_striped_object at ffffffffa0d6cc7b [lod] #9 [ffff8805f8c7b908] lod_declare_object_create at ffffffffa0d6d758 [lod] #10 [ffff8805f8c7b958] mdd_declare_object_create_internal at ffffffffa0dafccf [mdd] #11 [ffff8805f8c7b9a8] mdd_create at ffffffffa0dc0c9a [mdd] #12 [ffff8805f8c7ba78] mdt_reint_open at ffffffffa0e44529 [mdt] #13 [ffff8805f8c7bb58] mdt_reint_rec at ffffffffa0e2f401 [mdt] #14 [ffff8805f8c7bb78] mdt_reint_internal at ffffffffa0e28b93 [mdt] #15 [ffff8805f8c7bbb8] mdt_intent_reint at ffffffffa0e2915d [mdt] #16 [ffff8805f8c7bc08] mdt_intent_policy at ffffffffa0e24b2e [mdt] #17 [ffff8805f8c7bc48] ldlm_lock_enqueue at ffffffffa0897b21 [ptlrpc] #18 [ffff8805f8c7bca8] ldlm_handle_enqueue0 at ffffffffa08bf967 [ptlrpc] #19 [ffff8805f8c7bd18] mdt_enqueue at ffffffffa0e25066 [mdt] #20 [ffff8805f8c7bd38] mdt_handle_common at ffffffffa0e18322 [mdt] #21 [ffff8805f8c7bd88] mdt_regular_handle at ffffffffa0e19225 [mdt] #22 [ffff8805f8c7bd98] ptlrpc_server_handle_request at ffffffffa08ef8ac [ptlrpc] #23 [ffff8805f8c7be98] ptlrpc_main at ffffffffa08f0e9c [ptlrpc] #24 [ffff8805f8c7bf48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>I will drop the assertion from our tree with change <a href="http://review.whamcloud.com/4495" class="external-link" target="_blank" rel="nofollow noopener">4495</a>.</p> <p>Chris their was discuss on how to properly fix this bug so the patch has been updated. I tested but could you test this new version at <a href="http://review.whamcloud.com/#change,4495" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4495</a>. Thanks</p> <p>I have updated to the revision 2 of <a href="http://review.whamcloud.com/4495" class="external-link" target="_blank" rel="nofollow noopener">4495</a>.</p> <p>landed on master</p> Development Rank 1|hzvb67: Rank (Obsolete) 5350 Severity