https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5904 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/a99dab2e-68f9-11e4-9444-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a99dab2e-68f9-11e4-9444-5254006e85c2</a>.</p> <p>The sub-test test_13 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:56:02:Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries == 15:55:38 (1415404538) 15:56:02:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x196 15:56:02:Lustre: *** cfs_fail_loc=196, val=0*** 15:56:02:Lustre: Skipped 63 previous similar messages 15:56:02:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 15:56:02:Lustre: DEBUG MARKER: grep -c /mnt/ost1' ' /proc/mounts 15:56:03:Lustre: DEBUG MARKER: umount -d /mnt/ost1 15:56:03:Lustre: Failing over lustre-OST0000 15:56:03:Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' 15:56:03:Lustre: DEBUG MARKER: mkdir -p /mnt/ost1 15:56:03:Lustre: DEBUG MARKER: test -b /dev/lvm-Role_OSS/P1 15:56:03:Lustre: DEBUG MARKER: mkdir -p /mnt/ost1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_OSS/P1 /mnt/ost1 15:56:03:LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 15:56:04:LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.225@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. 15:56:04:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u 15:56:04:Lustre: DEBUG MARKER: e2label /dev/lvm-Role_OSS/P1 2&gt;/dev/null 15:56:04:Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect 15:56:04:Lustre: lustre-OST0000: Denying connection for new client 02b0a6e6-f977-0e0b-772a-b2c3b66ac87b (at 10.2.4.220@tcp), waiting for all 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 15:56:04:Lustre: lustre-OST0000: Denying connection for new client 02b0a6e6-f977-0e0b-772a-b2c3b66ac87b (at 10.2.4.220@tcp), waiting for all 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:19 15:56:04:Lustre: lustre-OST0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. 15:56:04:Lustre: lustre-OST0000: deleting orphan objects from 0x0:738 to 0x0:769 15:56:04:LustreError: 11266:0:(ldlm_resource.c:1150:ldlm_resource_get()) lustre-OST0000: lvbo_init failed for resource 0x2d8:0x0: rc = -2 15:56:04:LustreError: 11266:0:(ldlm_resource.c:1150:ldlm_resource_get()) Skipped 63 previous similar messages 15:56:04:BUG: unable to handle kernel paging request at 000000005a5a5a5a 15:56:04:IP: [&lt;ffffffffa07ea251&gt;] ldlm_lock_create+0x201/0xd70 [ptlrpc] 15:56:04:PGD 0 15:56:05:Oops: 0000 [#1] SMP 15:56:05:last sysfs file: /sys/devices/system/cpu/online 15:56:05:CPU 0 15:56:05:Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_ldiskfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic sha256_generic libcfs(U) ldiskfs(U) jbd2 nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: obdecho] 15:56:05: 15:56:05:Pid: 11261, comm: ll_ost00_014 Tainted: G W --------------- 2.6.32-431.20.3.el6_lustre.x86_64 #1 Red Hat KVM 15:56:06:RIP: 0010:[&lt;ffffffffa07ea251&gt;] [&lt;ffffffffa07ea251&gt;] ldlm_lock_create+0x201/0xd70 [ptlrpc] 15:56:06:RSP: 0018:ffff88006c515c60 EFLAGS: 00010246 15:56:06:RAX: ffff88006ccc1358 RBX: ffff88006ccc1180 RCX: ffff880037a64880 15:56:06:RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88006ccc1290 15:56:06:RBP: ffff88006c515cb0 R08: 0000000000000005 R09: 0000000000000000 15:56:06:R10: ffff88006ccc1180 R11: 0000000000000200 R12: ffff88006ccc1180 15:56:06:R13: ffffffffa0916ea0 R14: ffff88007ba5e400 R15: 000000005a5a5a5a 15:56:06:FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 15:56:06:CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b 15:56:06:CR2: 000000005a5a5a5a CR3: 000000007db32000 CR4: 00000000000006f0 15:56:06:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 15:56:07:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 15:56:07:Process ll_ost00_014 (pid: 11261, threadinfo ffff88006c514000, task ffff88006f803540) 15:56:07:Stack: 15:56:07: ffff880000000010 0000000000000000 0000000b00000004 ffffffffa05d5e31 15:56:07:&lt;d&gt; ffff88007ce36738 ffff88006e0ed850 ffff88007ce36818 ffffffffa0916ea0 15:56:07:&lt;d&gt; ffff88007ba5e400 0000000000000001 ffff88006c515d20 ffffffffa0811c74 15:56:07:Call Trace: 15:56:07: [&lt;ffffffffa05d5e31&gt;] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] 15:56:07: [&lt;ffffffffa0811c74&gt;] ldlm_handle_enqueue0+0x174/0x11d0 [ptlrpc] 15:56:07: [&lt;ffffffffa08947c2&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] 15:56:08: [&lt;ffffffffa0894b6c&gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc] 15:56:08: [&lt;ffffffffa084426a&gt;] ptlrpc_main+0xd1a/0x1980 [ptlrpc] 15:56:08: [&lt;ffffffffa0843550&gt;] ? ptlrpc_main+0x0/0x1980 [ptlrpc] 15:56:08: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 15:56:08: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 15:56:09: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 15:56:09: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 15:56:09:Code: 00 00 49 8d 84 24 d8 01 00 00 49 c7 84 24 90 00 00 00 00 00 00 00 ba 01 00 00 00 49 89 84 24 d8 01 00 00 49 89 84 24 e0 01 00 00 &lt;49&gt; 8b 07 48 8b 00 48 8b b8 88 01 00 00 e8 7d ba de ff 4d 89 24 15:56:09:RIP [&lt;ffffffffa07ea251&gt;] ldlm_lock_create+0x201/0xd70 [ptlrpc] 15:56:09: RSP &lt;ffff88006c515c60&gt; 15:56:09:CR2: 000000005a5a5a5a 15:56:09:Initializing cgroup subsys cpuset 15:56:09:Initializing cgroup subsys cpu </pre> </div></div> <p>Info required for matching: sanity-scrub 13</p> server: 2.6.0<br/> client: lustre-master build #2733 LU-5904

Interop 2.6.0<->master sanity-scrub test_13: BUG: ldlm_lock_create() on OST

Bug Minor Resolved Cannot Reproduce WC Triage Maloo Tue, 11 Nov 2014 17:52:28 +0000 Mon, 17 Apr 2017 21:31:57 +0000 Mon, 17 Apr 2017 21:31:57 +0000 Lustre 2.7.0 0 5 <p>Sarah, were both the MDS and OSS running 2.6.0 in this test?</p> <p>The crash happened at the line 'ls -ail $DIR/$tdir &gt; /dev/null 2&gt;&amp;1 &amp;&amp; error "(5) ls should fail"' after client remount. Because of some failure injection, the "ls" should could NOT access some OST-objects before the OI scrub rebuilt the /O on OST. But at that time-point, the OI scrub was not started yet. So the "ls" should get error.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_13() { check_mount_and_prep $SETSTRIPE -c 1 -i 0 $DIR/$tdir local count=$(precreated_ost_obj_count 0 0) #define OBD_FAIL_OSD_COMPAT_NO_ENTRY 0x196 do_facet ost1 $LCTL set_param fail_loc=0x196 createmany -o $DIR/$tdir/f $((count + 32)) do_facet ost1 $LCTL set_param fail_loc=0 umount_client $MOUNT || error <span class="code-quote">"(1) Fail to stop client!"</span> stop ost1 || error <span class="code-quote">"(2) Fail to stop ost1"</span> start ost1 $(ostdevname 1) $MOUNT_OPTS_NOSCRUB || error <span class="code-quote">"(3) Fail to start ost1"</span> mount_client $MOUNT || error <span class="code-quote">"(4) Fail to start client!"</span> ==&gt; ls -ail $DIR/$tdir &gt; /dev/<span class="code-keyword">null</span> 2&gt;&amp;1 &amp;&amp; error <span class="code-quote">"(5) ls should fail"</span> $START_SCRUB_ON_OST -r || error <span class="code-quote">"(6) Fail to start OI scrub on OST!"</span> wait_update_facet ost1 "$LCTL get_param -n \ osd-ldiskfs.$(facet_svc ost1).oi_scrub | awk <span class="code-quote">'/^status/ { print \\\$2 }'</span><span class="code-quote">" "</span>completed" 6 || error <span class="code-quote">"(7) Expected <span class="code-quote">'$expected'</span> on ost1"</span> ls -ail $DIR/$tdir &gt; /dev/<span class="code-keyword">null</span> || error <span class="code-quote">"(8) ls should succeed"</span> } run_test 13 <span class="code-quote">"OI scrub can rebuild missed /O entries"</span> </pre> </div></div> <p>According to the test logs on OST, the OST accessed some released RAM when handle LDLM lock.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(gdb) l * ldlm_lock_create+0x201 0x7521 is in ldlm_lock_create (include/linux/list.h:28). 23 struct list_head name = LIST_HEAD_INIT(name) 24 25 <span class="code-keyword">static</span> inline void INIT_LIST_HEAD(struct list_head *list) 26 { 27 list-&gt;next = list; 28 list-&gt;prev = list; 29 } </pre> </div></div> <p>ldlm_handle_enqueue0() =&gt; ldlm_lock_create() =&gt; ldlm_lock_new() =&gt; INIT_LIST_HEAD().<br/> It looks like that the new created LDLM lock was released by some other during such lock initialisation. So it was possible that the LDLM slab on the OST was corrupted. We need more clew for debug. Anyway, it should NOT be OI scrub issue since the OI scrub was not started yet. </p> <p>Andreas, both MDS and OSS were 2.6.0</p> <p>Close old issue.</p> Development Rank 1|hzx0qf: Rank (Obsolete) 16494 Severity