Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-5345] PPC mdc_read_page(): Page-wide hash collision: 0xfeffffffffffffff https://jira.whamcloud.com/browse/LU-5345 Lustre <p>Reading a normal (non DNE) directory on PPC will display the correct entries but also print an error. The last call to readdir() which should return 0 is returning -EIO.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>## Create and ls on x86_64 node: [root@shadow-36vm8 lustre]# mkdir d0 [root@shadow-36vm8 lustre]# touch d0/f{0..7} [root@shadow-36vm8 lustre]# ls d0 f0 f1 f2 f3 f4 f5 f6 f7 ## ls on PPC node: shadow-31:lustre# ls d0 ls: reading directory d0: Input/output error f0 f1 f2 f3 f4 f5 f6 f7 </pre> </div></div> <p>Here are the traces from the two readdir() calls:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:5.0:1405368875.648730:1552:4391:0:(dir.c:318:ll_readdir()) VFS Op:inode=[0x280000409:0xa:0x0](c000000772e69cf8) pos/size0/4096 32bit_api 0 00000002:00010000:5.0:1405368875.648735:3088:4391:0:(mdc_locks.c:1155:mdc_intent_lock()) (name: ,[0x280000409:0xa:0x0]) in obj [0x280000409:0xa:0x0], intent: readdir flags 00 00010000:00010000:5.0:1405368875.648739:4464:4391:0:(ldlm_lock.c:758:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x0 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00010000:00010000:5.0:1405368875.648745:4320:4391:0:(ldlm_lock.c:1392:ldlm_lock_match()) ### matched (0 0) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00000100:00100000:5.0:1405368875.648770:4176:4391:0:(client.c:1480:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ls:4986df11-5f39-84aa-b8fd-928839f5b18a:4391:1473632680299424:10.1.5.199@tcp:37 00000100:00100000:5.0:1405368875.648785:3952:4391:0:(client.c:2146:ptlrpc_set_wait()) set c000000798c90b80 going to sleep for 6 seconds 00000100:00100000:5.0:1405368875.650204:4368:4391:0:(client.c:1863:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ls:4986df11-5f39-84aa-b8fd-928839f5b18a:4391:1473632680299424:10.1.5.199@tcp:37 00010000:00010000:5.0:1405368875.650219:3680:4391:0:(ldlm_lock.c:811:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 3/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00010000:00010000:5.0:1405368875.650226:3504:4391:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/0,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00000002:00010000:5.0:1405368875.650237:3088:4391:0:(mdc_locks.c:1155:mdc_intent_lock()) (name: ,[0x280000409:0xa:0x0]) in obj [0x280000409:0xa:0x0], intent: readdir flags 00 00010000:00010000:5.0:1405368875.650242:4464:4391:0:(ldlm_lock.c:758:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x0 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00010000:00010000:5.0:1405368875.650248:4320:4391:0:(ldlm_lock.c:1392:ldlm_lock_match()) ### matched (0 0) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00000002:00200000:5.0:1405368875.650259:2720:4391:0:(mdc_request.c:1217:mdc_page_locate()) offset 100000000000000 [0x0 0xfeffffffffffffff], hash 0xfeffffffffffffff 00000100:00100000:5.0:1405368875.650281:4176:4391:0:(client.c:1480:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ls:4986df11-5f39-84aa-b8fd-928839f5b18a:4391:1473632680299428:10.1.5.199@tcp:37 00000100:00100000:5.0:1405368875.650296:3952:4391:0:(client.c:2146:ptlrpc_set_wait()) set c000000798c90b80 going to sleep for 6 seconds 00000100:00100000:5.0:1405368875.651660:4368:4391:0:(client.c:1863:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ls:4986df11-5f39-84aa-b8fd-928839f5b18a:4391:1473632680299428:10.1.5.199@tcp:37 00000002:00000400:5.0:1405368875.651674:2720:4391:0:(mdc_request.c:1585:mdc_read_page()) Page-wide hash collision: 0xfeffffffffffffff 00010000:00010000:5.0:1405368875.651679:3680:4391:0:(ldlm_lock.c:811:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 3/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00010000:00010000:5.0:1405368875.651686:3504:4391:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/0,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00000080:00200000:5.0:1405368875.651701:1552:4391:0:(dir.c:318:ll_readdir()) VFS Op:inode=[0x280000409:0xa:0x0](c000000772e69cf8) pos/size18374686479671623679/4096 32bit_api 0 00000002:00010000:5.0:1405368875.651707:3088:4391:0:(mdc_locks.c:1155:mdc_intent_lock()) (name: ,[0x280000409:0xa:0x0]) in obj [0x280000409:0xa:0x0], intent: readdir flags 00 00010000:00010000:5.0:1405368875.651711:4464:4391:0:(ldlm_lock.c:758:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x0 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00010000:00010000:5.0:1405368875.651718:4320:4391:0:(ldlm_lock.c:1392:ldlm_lock_match()) ### matched (0 0) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00000002:00200000:5.0:1405368875.651728:2720:4391:0:(mdc_request.c:1217:mdc_page_locate()) offset 100000000000000 [0x0 0xfeffffffffffffff], hash 0xfeffffffffffffff 00000100:00100000:5.0:1405368875.651750:4176:4391:0:(client.c:1480:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ls:4986df11-5f39-84aa-b8fd-928839f5b18a:4391:1473632680299432:10.1.5.199@tcp:37 00000100:00100000:5.0:1405368875.651766:3952:4391:0:(client.c:2146:ptlrpc_set_wait()) set c000000798c90b80 going to sleep for 6 seconds 00000100:00100000:5.0:1405368875.653407:4368:4391:0:(client.c:1863:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ls:4986df11-5f39-84aa-b8fd-928839f5b18a:4391:1473632680299432:10.1.5.199@tcp:37 00000002:00000400:5.0:1405368875.653421:2720:4391:0:(mdc_request.c:1585:mdc_read_page()) Page-wide hash collision: 0xfeffffffffffffff 00010000:00010000:5.0:1405368875.653426:3680:4391:0:(ldlm_lock.c:811:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 3/1,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00010000:00010000:5.0:1405368875.653433:3504:4391:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a888a80/0xb20b39c134a927fd lrc: 2/0,0 mode: PR/PR res: [0x280000409:0xa:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x6b92c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 00000080:00200000:5.0:1405368875.653666:1888:4391:0:(file.c:365:ll_file_release()) VFS Op:inode=[0x280000409:0xa:0x0](c000000772e69cf8) 00010000:00010000:5.0:1405368875.653672:3168:4391:0:(ldlm_lock.c:1392:ldlm_lock_match()) ### matched (0 0) ns: lustre-MDT0000-mdc-c00000079aee3f00 lock: c00000079a882880/0xb20b39c134a92804 lrc: 2/0,0 mode: CR/CR res: [0x280000409:0xa:0x0].0 bits 0x5 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x7992c6b1df4440e9 expref: -99 pid: 4391 timeout: 0 lvb_type: 0 </pre> </div></div> LU-5345 PPC mdc_read_page(): Page-wide hash collision: 0xfeffffffffffffff Bug Major Resolved Cannot Reproduce WC Triage John Hammond Mon, 14 Jul 2014 20:19:36 +0000 Mon, 22 Aug 2016 14:52:18 +0000 Mon, 22 Aug 2016 14:52:18 +0000 Lustre 2.6.0 0 6 <p>This is still an ongoing problem with current master that's somewhere at 2.7.56</p> <p>On a PPC build, we are seeing ls fail in sanity-scrub test 4b with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ls: reading directory /mnt/lustre/d4b.sanity-scrub/mds1/d_f4b.sanity-scrub/: Input/output error total 0 -r--r--r-- 1 root root 0 Aug 31 01:35 f0 -r--r--r-- 1 root root 0 Aug 31 01:35 f1 sanity-scrub test_4b: @@@@@@ FAIL: (17) fail to ls </pre> </div></div> <p>Logs at<br/> 2015-08-31 01:32:31 - <a href="https://testing.hpdd.intel.com/test_sets/86630832-4f82-11e5-9885-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/86630832-4f82-11e5-9885-5254006e85c2</a><br/> 2015-08-30 22:38:53 - <a href="https://testing.hpdd.intel.com/test_sets/3996487e-4f6a-11e5-876d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3996487e-4f6a-11e5-876d-5254006e85c2</a></p> <p>This seems to have stopped <em>totally</em> failing, at least for ppc_review-ldiskfs since 2015-09-11 17:28:29:<br/> <a href="https://testing.hpdd.intel.com/test_sets/34841034-58be-11e5-9cce-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/34841034-58be-11e5-9cce-5254006e85c2</a></p> <p>There are a couple of sanity failures since then, but attributable to other causes.</p> <p>I also never see this. Shall we close this ticket?</p> Related LU-6284 Development Rank 1|hzwref: Rank (Obsolete) 14906 Severity