https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2645 Lustre <p>While running runtests test on Lustre b1_8 clients with master servers, it failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>copying /etc/hosts to /mnt/lustre/hosts.9085 again cp: writing `/mnt/lustre/hosts.9085': Input/output error runtests : @@@@@@ FAIL: can't cp /etc/hosts to /mnt/lustre/hosts.9085 again 6 </pre> </div></div> <p>Dmesg on the client node client-12vm1 showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.9085 again LustreError: 11-0: an error occurred while communicating with 10.10.4.209@tcp. The obd_ping operation failed with -107 Lustre: lustre-OST0000-osc-ffff88007cea8800: Connection to service lustre-OST0000 via nid 10.10.4.209@tcp was lost; in progress operations using this service will wait for recovery to complete. LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: Server lustre-OST0000_UUID version (2.3.58.0) is much newer than client version (1.8.8.60) Lustre: Skipped 7 previous similar messages LustreError: 10880:0:(ldlm_resource.c:519:ldlm_namespace_cleanup()) Namespace lustre-OST0000-osc-ffff88007cea8800 resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 10880:0:(ldlm_resource.c:524:ldlm_namespace_cleanup()) Resource: ffff88007b9de380 (1/0/0/0) (rc: 1) Lustre: lustre-OST0000-osc-ffff88007cea8800: Connection restored to service lustre-OST0000 using nid 10.10.4.209@tcp. LustreError: 10869:0:(lov_request.c:211:lov_update_enqueue_set()) enqueue objid 0x2 subobj 0x1 on OST idx 0: rc -5 Lustre: DEBUG MARKER: /usr/sbin/lctl mark runtests : @@@@@@ FAIL: can\'t cp \/etc\/hosts to \/mnt\/lustre\/hosts.9085 again 6 </pre> </div></div> <p>Dmesg on the OSS node client-12vm4 showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.9085 again LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 10.10.4.206@tcp ns: filter-ffff880037b92000 lock: ffff88007b5f6000/0xf7c07c2f873c39f lrc: 3/0,0 mode: PR/PR res: 1/0 rrc: 2 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;18446744073709551615) flags: 0x20 nid: 10.10.4.206@tcp remote: 0x904b6ab232a7b36 expref: 5 pid: 4268 timeout: 4296500377 lvb_type: 1 Lustre: DEBUG MARKER: /usr/sbin/lctl mark runtests : @@@@@@ FAIL: can\'t cp \/etc\/hosts to \/mnt\/lustre\/hosts.9085 again 6 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/9ad0fc8a-6181-11e2-be04-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9ad0fc8a-6181-11e2-be04-52540035b04c</a></p> Lustre Client: b1_8<br/> Lustre Client Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/245/">http://build.whamcloud.com/job/lustre-b1_8/245/</a><br/> <br/> Lustre Server: master<br/> Lustre Server Build: <a href="http://build.whamcloud.com/job/lustre-master/1172/">http://build.whamcloud.com/job/lustre-master/1172/</a><br/> <br/> Distro/Arch: RHEL6.3/x86_64<br/> LU-2645

1.8<->2.4 interop: enqueue objid 0x2 subobj 0x1 on OST idx 0: rc -5

Bug Blocker Resolved Fixed nasf Jian Yu HB Fri, 18 Jan 2013 10:34:06 +0000 Tue, 5 Mar 2013 10:20:20 +0000 Tue, 5 Mar 2013 10:20:20 +0000 Lustre 2.4.0 Lustre 1.8.9 Lustre 2.4.0 0 10 <p>This issue is blocking Lustre b1_8&lt;-&gt;master interop testing:</p> <p>sanity-quota: <a href="https://maloo.whamcloud.com/test_sets/f7e6eb7e-6158-11e2-be04-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f7e6eb7e-6158-11e2-be04-52540035b04c</a><br/> sanity: <a href="https://maloo.whamcloud.com/test_sets/b794edde-616a-11e2-be04-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b794edde-616a-11e2-be04-52540035b04c</a></p> <p>Bob will look into this one</p> <p>Looks like the problem is on the server side. Varying the client version has no effect, failure still happens. So far I've confirmed the problem doesn't happen with pure b2_3 servers. Working to narrow in on at exactly what change the failure first happens.</p> <p>The problem first appeared sometime between v2_3_57 and v2_3_58 tags. No failure in 2.3.57</p> <p>I think I've narrowed it down to something that went in 12/17. Unfortunately that was a very active day, more than two dozen commits that one day.</p> <p>I'll keep trying to narrow it down to a specific commit, but maybe those who did commits on that day could start looking at them?</p> <p>Bob, can you please add all of the committers from 12/17 to the CC list of this bug, and paste a list of commits using <tt>git log --pretty=short</tt> for that day. Maybe someone will find this issue more quickly.</p> <p>Andreas,<br/> I could do that but I'm just on the verge of pinning down the specific commit.<br/> It sure looks like the offending commit is</p> <p><a href="https://jira.whamcloud.com/browse/LU-2378" title="Move HSM/SOM attributes to dedicated XATTR" class="issue-link" data-issue-key="LU-2378"><del>LU-2378</del></a> lma: move HSM &amp; SOM attributes to dedicated xattrs</p> <p>author Johann Lombardi &lt;johann.lombardi@intel.com&gt; <br/> Fri, 23 Nov 2012 14:23:29 +0000 (15:23 +0100)<br/> committer Oleg Drokin &lt;green@whamcloud.com&gt; <br/> Mon, 17 Dec 2012 03:13:15 +0000 (22:13 -0500)<br/> commit d0c104aa0e96b1f1d2366bc1e4715fe3830f41b4</p> <p>If I've done the right builds and checks so far a build with that commit fails while a build with all commits up until that one doesn't fail.</p> <p>Double checking now.</p> <p>Just for reference here is the short log for 12/17:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2012-12-17 Alex Zhuravlev LU-2100 ofd: small batched precreation on a small system commit | commitdiff | tree | snapshot 2012-12-17 Peng Tao LU-1994 kernel: 3.6 dentry_open uses struct path as... commit | commitdiff | tree | snapshot 2012-12-17 yang sheng LU-1994 kernel: kernel 3.6 changes i_dentry/d_alias... commit | commitdiff | tree | snapshot 2012-12-17 Peng Tao LU-1994 kernel: 3.5 kernel encode_fh passes in parent... commit | commitdiff | tree | snapshot 2012-12-17 yang sheng LU-1994 llite: kernel 3.5 renames end_writeback to... commit | commitdiff | tree | snapshot 2012-12-17 Niu Yawei LU-2329 quota: wait longer in test_7c commit | commitdiff | tree | snapshot 2012-12-17 Bobi Jam LU-1741 test: fix conf_sanity test_18 test case commit | commitdiff | tree | snapshot 2012-12-17 Daniel Kobras LU-2302 scripts: prevent lfs_migrate data disclosure commit | commitdiff | tree | snapshot 2012-12-17 Daniel Kobras LU-2302 scripts: null-terminated file lists in lfs_migrate commit | commitdiff | tree | snapshot 2012-12-17 Daniel Kobras LU-2302 scripts: fix lfs_migrate with non-English locale commit | commitdiff | tree | snapshot 2012-12-17 Johann Lombardi LU-2371 ptlrpc: get new xid for resend on EINPROGRESS commit | commitdiff | tree | snapshot 2012-12-17 Lai Siyao LU-2388 statahead: don't statahead if it's stopped commit | commitdiff | tree | snapshot 2012-12-17 Johann Lombardi LU-2361 quota: keep slave's glb idx consistent with... commit | commitdiff | tree | snapshot 2012-12-17 Niu Yawei LU-2346 quota: set default grace time commit | commitdiff | tree | snapshot 2012-12-17 John L. Hammond LU-2358 procfs: Implement /proc/fs/lustre/mgs/MGS/fstyp... commit | commitdiff | tree | snapshot 2012-12-17 Lai Siyao LU-1287 mountconf: write failover nid config correctly commit | commitdiff | tree | snapshot 2012-12-17 wangdi LU-1632 fid: remove fid_delete in delete_inode procedure commit | commitdiff | tree | snapshot 2012-12-17 John L. Hammond LU-2363 lod: Fix statfs entries in lod procfs commit | commitdiff | tree | snapshot 2012-12-17 Peng Tao LU-1756 kernel: cleanup lustre_compat25.h commit | commitdiff | tree | snapshot 2012-12-17 Peng Tao LU-1484 kernel: fix build error with 2.6.18 kernel commit | commitdiff | tree | snapshot 2012-12-17 Lai Siyao LU-1887 ptlrpc: grant shrink rpc format is special commit | commitdiff | tree | snapshot 2012-12-17 Thomas Stibor LU-1924 build: configure can not find libgssapi_krb5.so commit | commitdiff | tree | snapshot 2012-12-17 Nikitas Angelinas LU-398 libcfs: Add libcfs heap, a binary heap implement... commit | commitdiff | tree | snapshot 2012-12-17 jcl LU-2016 mdd: add layout swap between 2 objects commit | commitdiff | tree | snapshot 2012-12-17 Jinshan Xiong LU-744 obdclass: revise cl_page refcount commit | commitdiff | tree | snapshot 2012-12-17 Nathaniel Clark LU-2194 tests: Wait for reconnect in recovery-small/19 commit | commitdiff | tree | snapshot 2012-12-17 Johann Lombardi LU-2378 lma: move HSM &amp; SOM attributes to dedicated... commit | commitdiff | tree | snapshot </pre> </div></div> <p>Looks like I was hasty in pointing the finger. Definitely after the v2_3_57 tag, but I now see failures from earlier than 12/17. Almost certainly not the Johann's change for <a href="https://jira.whamcloud.com/browse/LU-2378" title="Move HSM/SOM attributes to dedicated XATTR" class="issue-link" data-issue-key="LU-2378"><del>LU-2378</del></a>. Will keep on trying to narrow it down.</p> <p>I have a winner. Pretty sure I got it right this time as I double checked the result before reporting it here. I believe the offending commit is:</p> <p><a href="https://jira.whamcloud.com/browse/LU-1710" title="variable sized LVB support" class="issue-link" data-issue-key="LU-1710"><del>LU-1710</del></a> lvb: variable sized LVB support</p> <p>author Jinshan Xiong &lt;jinshan.xiong@intel.com&gt; <br/> Thu, 6 Dec 2012 17:03:27 +0000 (09:03 -0800)<br/> committer Oleg Drokin &lt;green@whamcloud.com&gt; <br/> Wed, 12 Dec 2012 17:21:07 +0000 (12:21 -0500)<br/> commit 929ec628e6fef5609e55d519a1eb9e2cbbf1f1e8<br/> tree 84a5009309b9ed1700b94e3d285aeb0aa67a0ac1 tree | snapshot<br/> parent caf5bdffb4eb6e3fb31724a1cb037cecfeb6ae6c commit | diff</p> <p>A build of the immediate parent in the tree, commit caf5bdffb4eb6e3fb31724a1cb037cecfeb6ae6c, succeeds while a build of this commit fails.</p> <p>Jinshan</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Actually Bob pinged about this issue on skype. Fanyong is the right person to take a look at this issue. I'll ping him.</p> <p>Since it is related with variable sized LVB patch, I will take and fix it.</p> <p>I am working on it.</p> <p>Lustre Client: b1_8<br/> Lustre Client Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/256" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/256</a></p> <p>Lustre Server: master<br/> Lustre Server Build: <a href="http://build.whamcloud.com/job/lustre-master/1256" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1256</a></p> <p>Distro/Arch: RHEL6.3/x86_64</p> <p>A full test session:<br/> <a href="https://maloo.whamcloud.com/test_sessions/487f5cd4-7805-11e2-9928-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/487f5cd4-7805-11e2-9928-52540035b04c</a></p> <p>Most of the tests failed with the issue in this ticket.</p> <p>This is the patch for master:</p> <p><a href="http://review.whamcloud.com/#change,5459" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5459</a></p> <p>Yujian, would you please to verify the patch? Thanks!</p> <blockquote><p>Yujian, would you please to verify the patch? Thanks!</p></blockquote> <p>Please add the following test parameters into the commit message to verify the patch. Thanks.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: envdefinitions=SLOW=yes,ENABLE_QUOTA=yes \ clientjob=lustre-b1_8 clientbuildno=256 testlist=sanity </pre> </div></div> <p>The patch has passed runtests on Maloo under interoperability mode:</p> <p><a href="https://maloo.whamcloud.com/test_sessions/649fb4c4-7aba-11e2-b5c8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/649fb4c4-7aba-11e2-b5c8-52540035b04c</a></p> <p>Landed for 2.4</p> Development Rank 1|hzvfnb: Rank (Obsolete) 6181 Severity