https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10406 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/1bf35756-9e3e-11e7-ba27-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1bf35756-9e3e-11e7-ba27-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/a426e9c6-e3da-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a426e9c6-e3da-11e7-8027-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/a30b149e-e47b-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a30b149e-e47b-11e7-9c63-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/f72002ba-e45d-11e7-9840-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f72002ba-e45d-11e7-9840-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b717e1b8-e446-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b717e1b8-e446-11e7-a066-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/75350600-e445-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/75350600-e445-11e7-8027-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/11a47514-e425-11e7-9840-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/11a47514-e425-11e7-9840-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/a58c7e12-e492-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a58c7e12-e492-11e7-8027-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e357fa00-a919-11e7-b78a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e357fa00-a919-11e7-b78a-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b319cf8e-e330-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b319cf8e-e330-11e7-a066-52540065bddc</a></p> <p>The sub-test test_31c started failing intermittently on 2017-12-17 09:06:23 with the following error on :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(4) Fail to re-generate master LMV EA: 3 </pre> </div></div> <p>It looks like this was introduced by patch <a href="https://review.hpdd.intel.com/29078" class="external-link" target="_blank" rel="nofollow noopener">https://review.hpdd.intel.com/29078</a> <tt><a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> osp: cache non-exist EA</tt>, which landed to master on 2017-12-16 23:21, but failed twice on the patch itself before landing.</p> <p>Info required for matching: sanity-lfsck 31c</p> LU-10406

sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2

Bug Critical Resolved Fixed Alex Zhuravlev Maloo Tue, 19 Dec 2017 09:50:17 +0000 Wed, 27 Feb 2019 19:45:02 +0000 Tue, 13 Nov 2018 11:04:12 +0000 Lustre 2.11.0 Lustre 2.12.0 0 10 <p>Fan Yong, can you please take a look at this, it is causing a lot of test failures. If there isn't a straight-forward fix then can you please revert the failing patch.</p> <p>This has failed 27x in the past 5 days since this patch was landed.</p> <p>It looks like there is a revert patch already: <a href="https://review.whamcloud.com/30643" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30643</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>, I think you are right. The failure is quite possible related with the patch <a href="https://review.whamcloud.com/#/c/29078/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29078/</a>. I am out for vacation these days, no mac at hand, not convenient to debug the root reason. Please revert such patch. I will make proper patch when back.<br/> Thanks!</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/174e3f3c-e78f-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/174e3f3c-e78f-11e7-8027-52540065bddc</a></p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30694" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30694</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10406" title="sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2" class="issue-link" data-issue-key="LU-10406"><del>LU-10406</del></a> tests: stop running sanity-lfsck 31c<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a113d617c0d4b78d3775468d29f51959f1fb5bc4</p> <p>We have a patch (<a href="https://review.whamcloud.com/#/c/30643/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30643/</a>) to revert the patch for <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a>, but we are having a hard time getting the patch to pass testing. Thus, I've uploaded patch <a href="https://review.whamcloud.com/#/c/30694/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30694/</a> to temporarily add sanity-lfsck test 31c to the ALWAYS_EXCEPT list.</p> <p>Looking at the revert patch, 30643 patch set #3, we see that sanity-lfsck passes as part of review-dne-part-2 and review-zfs-part-1, <a href="https://testing.hpdd.intel.com/test_sets/b4da8d66-e76a-11e7-9840-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b4da8d66-e76a-11e7-9840-52540065bddc</a> and <a href="https://testing.hpdd.intel.com/test_sets/cd79dd76-e766-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cd79dd76-e766-11e7-9c63-52540065bddc</a>, but fails, in fact hangs in test 8 each time, when run in a custom test session created from "Test-Parameters: clientcount=2 osscount=1 ostcount=7 mdscount=2 mdtcount=2 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck,sanity-lfsck,sanity-lfsck"; <a href="https://testing.hpdd.intel.com/test_sessions/d29aeac6-05b2-44c5-88d4-560b25dbd1e2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/d29aeac6-05b2-44c5-88d4-560b25dbd1e2</a>.</p> <p>James, is it worthwhile to remove the custom test session request, and just run a normal review test session? It may be that sanity-lfsck.sh itself has some problem when running with the requested test config independent of this patch (e.g. MDTCOUNT=2 instead of the standard MDTCOUNT=4), so it would be good to limit the variables and try to get the reversion landed.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30694/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30694/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10406" title="sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2" class="issue-link" data-issue-key="LU-10406"><del>LU-10406</del></a> tests: stop running sanity-lfsck 31c<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d645c722d73c8c4226ab5695d37a541d39bb0a64</p> <p>Fan Yong - <br/> We submitted a patch to revert the patch for <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> at <a href="https://review.whamcloud.com/#/c/30643/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30643/</a>. We submitted that patch with the following test parameters:<br/> Test-Parameters: clientcount=2 osscount=1 ostcount=7 mdscount=2 mdtcount=2 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck,sanity-lfsck,sanity-lfsck</p> <p>The intention of running sanity-lfsck multiple times, was to confirm that reverting the patch for <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> actually fixed the test 31c failure. Yet, the test sessions with multiple sanity-lfsck runs failed in the same way with a hang in test 8. Results are at<br/> <a href="https://testing.hpdd.intel.com/test_sessions/e958a3e3-4a3f-4f98-83b7-b19e1513ea95" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e958a3e3-4a3f-4f98-83b7-b19e1513ea95</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/d29aeac6-05b2-44c5-88d4-560b25dbd1e2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/d29aeac6-05b2-44c5-88d4-560b25dbd1e2</a></p> <p>Do you understand why just running sanity-lfsck, with the revert of <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a>, would cause test 8 to hang?</p> <p>The hung issue happened at the beginning of sanity-lfsck test_8. According to the test logic, the scripts will reformat the system via 'formatall', then mount up the system via 'setupall'. The hung just happened during 'setupall'. Please note that, the system was in clean status since it was reformatted just now, nothing related with your patch.</p> <p>The logs on OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:01000004:0.0:1513957722.318110:0:21014:0:(obd_mount_server.c:1208:server_register_target()) Registration lustre-OST0000, fs=lustre, 10.2.8.87@tcp, index=0000, flags=0x62 10000000:01000000:0.0:1513957722.318112:0:21014:0:(mgc_request.c:1267:mgc_set_info_async()) register_target lustre-OST0000 0x10000062 10000000:01000000:0.0:1513957722.318125:0:21014:0:(mgc_request.c:1206:mgc_target_register()) register lustre-OST0000 00000100:00100000:0.0:1513957722.318134:0:21014:0:(client.c:2320:ptlrpc_set_wait()) set ffff88005b08ee80 going to sleep for 0 seconds 00000100:00100000:0.0:1513957722.318138:0:21014:0:(client.c:2026:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc mount.lustre:8e527c78-387c-cf38-1d04-d2dd8fba8f79:21014:1587498878862272:10.2.8.88@tcp:253 00000020:02020000:0.0:1513957722.318145:0:21014:0:(obd_mount_server.c:1230:server_register_target()) 15f-b: lustre-OST0000: cannot register this server with the MGS: rc = -108. Is the MGS running? </pre> </div></div> <p>Means the OST fail to register with MGS because connection with MGS shutdown. But from the logs on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:0.0:1513961335.627207:0:3242:0:(service.c:2099:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_mgs_0001:8e527c78-387c-cf38-1d04-d2dd8fba8f79+16:16679:x1587498878943424:12345-10.2.8.87@tcp:400 00000100:00100000:0.0:1513961335.627235:0:3242:0:(service.c:2145:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_mgs_0001:8e527c78-387c-cf38-1d04-d2dd8fba8f79+16:16679:x1587498878943424:12345-10.2.8.87@tcp:400 Request procesed in 28us (101us total) trans 0 rc 0/0 </pre> </div></div> <p>That means the MGS received ping RPC from the OST. But I cannot find the logs for MGS_TARGET_REG RPC from OSTs. That is strange. There are no detailed logs, we do not know what happened, it seems connection trouble just during the MGS_TARGET_REG. Anyway, it is NOT related with your patch.</p> <p>I know why the OST cannot register to the MGS. The connection between the MGC and MGS has not been setup when the OST registering to the MGS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00080000:0.0:1513957706.461000:0:16677:0:(import.c:1277:ptlrpc_connect_interpret()) ffff88007b1bf800 MGS: changing import state from CONNECTING to DISCONN ... 00000020:00000080:0.0:1513957722.315616:0:21014:0:(genops.c:1404:class_connect()) connect: client lustre-OST0000-osd_UUID, cookie 0xd7729766f2408d04 00000020:01000004:0.0:1513957722.315619:0:21014:0:(obd_mount_server.c:1893:server_fill_super()) Found service lustre-OST0000 on device /dev/mapper/lvm--Role_OSS-P1 00000020:01000004:0.0:1513957722.315658:0:21014:0:(obd_mount.c:331:lustre_start_mgc()) MGC10.2.8.88@tcp: Set MGC reconnect 1 10000000:01000000:0.0:1513957722.315660:0:21014:0:(mgc_request.c:1249:mgc_set_info_async()) InitRecov MGC10.2.8.88@tcp 1/d0:i1:r0:or0:CONNECTING 00000020:01000004:0.0:1513957722.315664:0:21014:0:(obd_mount_server.c:1303:server_start_targets()) starting target lustre-OST0000 00000020:01000004:0.0:1513957722.315670:0:21014:0:(obd_mount_server.c:294:server_mgc_set_fs()) Set mgc disk for /dev/mapper/lvm--Role_OSS-P1 00000040:01000000:0.0:1513957722.318097:0:21014:0:(llog_obd.c:210:llog_setup()) obd MGC10.2.8.88@tcp ctxt 0 is initialized 00000020:01000004:0.0:1513957722.318110:0:21014:0:(obd_mount_server.c:1208:server_register_target()) Registration lustre-OST0000, fs=lustre, 10.2.8.87@tcp, index=0000, flags=0x62 10000000:01000000:0.0:1513957722.318112:0:21014:0:(mgc_request.c:1267:mgc_set_info_async()) register_target lustre-OST0000 0x10000062 10000000:01000000:0.0:1513957722.318125:0:21014:0:(mgc_request.c:1206:mgc_target_register()) register lustre-OST0000 00000100:00100000:0.0:1513957722.318134:0:21014:0:(client.c:2320:ptlrpc_set_wait()) set ffff88005b08ee80 going to sleep for 0 seconds 00000100:00100000:0.0:1513957722.318138:0:21014:0:(client.c:2026:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc mount.lustre:8e527c78-387c-cf38-1d04-d2dd8fba8f79:21014:1587498878862272:10.2.8.88@tcp:253 00000020:02020000:0.0:1513957722.318145:0:21014:0:(obd_mount_server.c:1230:server_register_target()) 15f-b: lustre-OST0000: cannot register this server with the MGS: rc = -108. Is the MGS running? 00000020:01000004:0.0:1513957722.319303:0:21014:0:(obd_mount_server.c:312:server_mgc_clear_fs()) Unassign mgc disk ... 00000100:00080000:0.0:1513957735.454711:0:16677:0:(import.c:1195:ptlrpc_connect_interpret()) ffff88007b1bf800 MGS: changing import state from CONNECTING to EVICTED ... </pre> </div></div> <p>Please note that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10000000:01000000:0.0:1513957722.315660:0:21014:0:(mgc_request.c:1249:mgc_set_info_async()) InitRecov MGC10.2.8.88@tcp 1/d0:i1:r0:or0:CONNECTING </pre> </div></div> <p>That means the import::imp_invalid was 1. That will affect the ptlrpc behavior when send RPC:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int ptlrpc_import_delay_req(struct obd_import *imp, struct ptlrpc_request *req, int *status) { int delay = 0; ... } else if (imp-&gt;imp_invalid || imp-&gt;imp_obd-&gt;obd_no_recov) { if (!imp-&gt;imp_deactive) DEBUG_REQ(D_NET, req, "IMP_INVALID"); *status = -ESHUTDOWN; /* bz 12940 */ ... } </pre> </div></div> <p>That is why the server_register_target() failed for -108.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30762" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30762</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10406" title="sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2" class="issue-link" data-issue-key="LU-10406"><del>LU-10406</del></a> osp: cache xattr properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6725a75fb7bad8ebd51a2595b4766183339e5ad</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31935" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31935</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10406" title="sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2" class="issue-link" data-issue-key="LU-10406"><del>LU-10406</del></a> tests: stop running sanity-lfsck 31c<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 78dae657afaa7ea4b78352a159756077d64793bf</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31935/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31935/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10406" title="sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2" class="issue-link" data-issue-key="LU-10406"><del>LU-10406</del></a> tests: stop running sanity-lfsck 31c<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: ebaeb217ed12181a02c833354c15c68ddee21e23</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/31194/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31194/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10406" title="sanity-lfsck test_31c: (4) Fail to re-generate master LMV EA: 2" class="issue-link" data-issue-key="LU-10406"><del>LU-10406</del></a> mdt: invalidate cache upon LDLM lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7edc970036e118dd6fc63252054c1c9f594ce1e1</p> <p>Landed for 2.12</p> Related LU-10005 LU-11218 Development Rank 1|hzzpun: Rank (Obsolete) 9223372036854775807 Severity