https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11375 Lustre <p>Mounting client fails with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t lustre 192.168.56.12@tcp:/zfs2 /mnt/fs2 mount.lustre: mount 192.168.56.12@tcp:/zfs2 at /mnt/fs2 failed: Bad address </pre> </div></div> <p>MDT0 has the following error in debug log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:02000000:1.0:1536858056.517840:0:2585:0:(import.c:1534:ptlrpc_import_recovery_state_machine()) zfs2-MDT0000: Connection restored to 880830c7-7d38-ac3b-c683-8d949f550eaa (at 192.168.56.30@tcp) 00000100:00020000:1.0:1536858056.518193:0:2585:0:(layout.c:2067:__req_capsule_get()) @@@ Wrong buffer for field `mdt_body' (1 of 1) in format `MDS_STATFS': 0 vs. 216 (client) req@ffff9a06cd249b00 x1611512228611360/t0(0) o41-&gt;1953ab2d-accd-2169-bea6-82979f16a7d4@192.168.56.30@tcp:452/0 lens 224/0 e 0 to 0 dl 1536858067 ref 1 fl Interpret:/0/ffffffff rc 0/-1 00000020:00000080:1.0:1536858056.524976:0:2585:0:(genops.c:1561:class_disconnect()) disconnect: cookie 0x3f757c9924a4d7eb </pre> </div></div> <p>Setup:<br/> mds03: MGT, MDT1<br/> mds04: MDT0<br/> oss03: OST0, OST2<br/> oss04: OST1, OST3<br/> This is all ZFS based (ZFS 0.7.9)</p> 2 MDS, 2 OSS ~2.11.54_91 <br/> Client 2.10.4 <br/> LU-11375

Client 2.10 fails to mount Server 2.11.54

Bug Blocker Resolved Fixed Alex Zhuravlev Nathaniel Clark interop Thu, 13 Sep 2018 17:25:27 +0000 Thu, 25 Oct 2018 01:19:01 +0000 Wed, 10 Oct 2018 02:21:07 +0000 Lustre 2.12.0 Lustre 2.12.0 0 8 <p>Likely from <a href="https://review.whamcloud.com/29136" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29136</a> <a href="https://jira.whamcloud.com/browse/LU-10018" title="MDT as a statfs proxy" class="issue-link" data-issue-key="LU-10018"><del>LU-10018</del></a> protocol: MDT as a statfs proxy</p> <p>We run automated testing between 2.10.4 clients and master servers and we do see these tests failing in lustre-initialization. Looking at <a href="https://testing.whamcloud.com/test_sets/73262750-b59f-11e8-9df3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/73262750-b59f-11e8-9df3-52540065bddc</a>, we see the mount fail</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-09-11T07:51:28 Starting client: trevis-19vm1.trevis.whamcloud.com: -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre 2018-09-11T07:51:28 CMD: trevis-19vm1.trevis.whamcloud.com mkdir -p /mnt/lustre 2018-09-11T07:51:28 CMD: trevis-19vm1.trevis.whamcloud.com mount -t lustre -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre 2018-09-11T07:51:28 mount.lustre: mount trevis-19vm4@tcp:/lustre at /mnt/lustre failed: Bad address </pre> </div></div> <p>On that client (vm1), we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 221.250210] Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre [ 221.731388] LustreError: 11-0: lustre-MDT0000-mdc-ffff90943bde8800: operation mds_statfs to node 10.9.4.227@tcp failed: rc = -14 [ 221.732646] LustreError: 15598:0:(lmv_obd.c:1387:lmv_statfs()) can't stat MDS #0 (lustre MDT0000-mdc-ffff90943bde8800), error -14 [ 221.738793] LustreError: 15598:0:(lov_obd.c:878:lov_cleanup()) lustre-clilov-ffff90943bde8800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 221.742206] Lustre: Unmounted lustre-client [ 221.743475] LustreError: 15598:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount (-14) </pre> </div></div> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33162" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33162</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11375" title="Client 2.10 fails to mount Server 2.11.54" class="issue-link" data-issue-key="LU-11375"><del>LU-11375</del></a> mdc: use old statfs format<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a9914b892add64cf2b584b91c6da9872c871d6ef</p> <p>the patch isn't ready... I've met interesting issue - the initial MDS_STATFS is formed before the connection (and connection flags) is set up properly. so the client and the server have different connect flags.</p> <p>Since master build #3787, it looks like Ubuntu 18.04 clients can't mount due to this issue. In the MDS console log, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 191.353309] LustreError: 15466:0:(layout.c:2067:__req_capsule_get()) @@@ Wrong buffer for field `mdt_body' (1 of 1) in format `MDS_STATFS': 0 vs. 216 (client) </pre> </div></div> <p>Logs at<br/> <a href="https://testing.whamcloud.com/test_sets/6afaa572-b86c-11e8-8c12-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6afaa572-b86c-11e8-8c12-52540065bddc</a></p> <p>please try the updated patch.</p> <p> </p> <p>@James please suggest how to test with with appropriate Commit parameter</p> <p>Added the following to the commit message to test b2_10 clients with servers from this patch:<br/> Test-Parameters: clientjob=lustre-b2_10 clientbuildno=136 testgroup=review-ldiskfs</p> <p>Looks like this patch allows a 2.10.5 client to mount a later server. Results at <a href="https://testing.whamcloud.com/test_sessions/319c062d-ad5a-4f4c-bcb1-a7c23ebff084" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/319c062d-ad5a-4f4c-bcb1-a7c23ebff084</a></p> <p>The lustre linux client (also 2.10) also now works.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33162/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33162/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11375" title="Client 2.10 fails to mount Server 2.11.54" class="issue-link" data-issue-key="LU-11375"><del>LU-11375</del></a> mdc: use old statfs format<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e70a6fd8a6400c0460a6c66668103c23b7997d30</p> <p>Landed for 2.12</p> Related LU-10018 LU-11488 Development Rank 1|i002d3: Rank (Obsolete) 9223372036854775807 Severity