https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1999 Lustre <p>This issue was created by maloo for Oleg Drokin &lt;green@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/9d79385c-038b-11e2-83cf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9d79385c-038b-11e2-83cf-52540035b04c</a>.</p> <p>The sub-test test_106 failed with the following error:</p> <blockquote> <p>CMD: client-24vm3 lctl get_param -n *.lustre-MDT0000.num_exports | cut -d' ' -f2<br/> Update not seen after 60s: wanted '3' got '4'<br/> recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds </p></blockquote> <p>Info required for matching: recovery-small 106</p> LU-1999

recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds

Bug Major Resolved Fixed Johann Lombardi Maloo Thu, 20 Sep 2012 21:44:59 +0000 Thu, 18 Apr 2013 20:45:09 +0000 Thu, 18 Oct 2012 15:55:08 +0000 Lustre 2.4.0 Lustre 2.4.0 0 3 <p>Johann, this failure comes from a test added by you yesterday</p> <p>It looks like a test issue since the client got evicted as expected:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 167-0: lustre-MDT0000-mdc-ffff880078abb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. Lustre: Evicted from MGS (at 10.10.4.118@tcp) after server handle changed from 0x1345e746bb7e9980 to 0x1345e746bb7e9edc Lustre: MGC10.10.4.118@tcp: Reactivating import Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mdd </pre> </div></div> <p>The issue is the the client reconnected before the check in the test was made.</p> <p>I have disabled the test on master until i come up with a test script fix to avoid those failures.<br/> <a href="http://review.whamcloud.com/4066" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4066</a></p> <p>Tentative patch:<br/> <a href="http://review.whamcloud.com/4069" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4069</a></p> <p>Patch landed. No new reoccurrence since then.</p> <p>It seems that I am still hitting this in my local testing (ie SLOW=yes REFORMAT=yes sh recovery-small.sh ), though I cannot speak of the frequence.<br/> This is with today's master checkout.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 106: lightweight connection support == 23:16:05 (1350098165) fail_loc=0x805 Starting client: centos6-4.localnet: -o user_xattr,flock centos6-4.localnet@tcp:/lustre /mnt/lustre2 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.10.214@tcp:/lustre 374928 52384 302004 15% /mnt/lustre Failing mds1 on node centos6-4.localnet Stopping /mnt/mds1 (opts:) on centos6-4.localnet affected facets: mds1 Failover mds1 to centos6-4.localnet 23:16:24 (1350098184) waiting for centos6-4.localnet network 900 secs ... 23:16:24 (1350098184) network interface is UP Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1 Started lustre-MDT0000 touch: setting times of `/mnt/lustre2/f.recovery-small.106': Input/output error recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds Trace dump: = ./../tests/test-framework.sh:3844:error_noexit() = ./../tests/test-framework.sh:3866:error() = recovery-small.sh:1528:test_106() = ./../tests/test-framework.sh:4108:run_one() = ./../tests/test-framework.sh:4138:run_one_logged() = ./../tests/test-framework.sh:4009:run_test() = recovery-small.sh:1537:main() Dumping lctl log to /tmp/test_logs/1350095359/recovery-small.test_106.*.1350098192.log Dumping logs only on local client. 192.168.10.214@tcp:/lustre /mnt/lustre2 lustre rw,flock,user_xattr 0 0 Stopping client centos6-4.localnet /mnt/lustre2 (opts:) FAIL 106 (32s) </pre> </div></div> <p>Just did a bit of research, it's happening, but not super freequently:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@intelbox ~]# grep 'lightweight client not evicted by mds' /var/log/remotemessages Oct 9 03:43:14 centos6-0 kernel: [ 2489.388687] Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds Oct 9 13:27:35 centos6-1 kernel: [ 2898.929903] Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds Oct 11 13:33:30 centos6-0 kernel: [ 6723.295214] Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds Oct 12 23:16:32 centos6-4 kernel: [ 3261.398403] Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds </pre> </div></div> <p>Oleg, could you please attach dmesg output when the issue happened?</p> <p>I saw this with today's master plus my <a href="https://jira.whamcloud.com/browse/LU-1717" title="mdt_recovery.c:611:mdt_steal_ack_locks()) Resent req xid XXX has mismatched opc: new 101 old 0" class="issue-link" data-issue-key="LU-1717"><del>LU-1717</del></a> patch locally on a single-node setup:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> == recovery-small test 106: lightweight connection support == 16:41:54 (1350290514) fail_loc=0x805 Starting client: h221f: -o user_xattr,flock h221f@tcp:/lustre /mnt/lustre2 Failed to load ZFS module stack. Load the module manually by running 'insmod &lt;location&gt;/zfs.ko' as root. fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on h221f@tcp:/lustre 374928 51792 303136 15% /mnt/lustre Failing mds1 on node h221f Stopping /mnt/mds1 (opts:) on h221f affected facets: mds1 Failover mds1 to h221f 16:42:05 (1350290525) waiting for h221f network 900 secs ... 16:42:05 (1350290525) network interface is UP Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1 Failed to load ZFS module stack. Load the module manually by running 'insmod &lt;location&gt;/zfs.ko' as root. Started lustre-MDT0000 touch: setting times of `/mnt/lustre2/f.recovery-small.106': Input/output error recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds Trace dump: = ./../tests/test-framework.sh:3869:error_noexit() = ./../tests/test-framework.sh:3891:error() = recovery-small.sh:1528:test_106() = ./../tests/test-framework.sh:4133:run_one() = ./../tests/test-framework.sh:4163:run_one_logged() = ./../tests/test-framework.sh:4034:run_test() = recovery-small.sh:1537:main() Dumping lctl log to /tmp/test_logs/1350288504/recovery-small.test_106.*.1350290525.log Dumping logs only on local client. 192.168.56.4@tcp:/lustre /mnt/lustre2 lustre rw,flock,user_xattr 0 0 Stopping client h221f /mnt/lustre2 (opts:) sh: lsof: command not found FAIL 106 (12s) </pre> </div></div> <p>The logs will be attached in a minute.</p> <p>hm, strange, the message is in the lustre log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:02020000:0.0:1350290525.905481:0:6240:0:(import.c:1325:ptlrpc_import_recovery_state_machine()) 167-0: lustre-MDT0000-mdc-ffff880006762400: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. </pre> </div></div> <p>but not in dmesg output ...</p> <p>I can now reproduce. The issue comes from the console rate limit code which skips the messages expected by the test. </p> <p><a href="http://review.whamcloud.com/#change,4288" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4288</a></p> <p>Patch landed. Close the bug again.</p> Development Rank 1|hzv3j3: Rank (Obsolete) 4071 Severity