https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10361 Lustre <p>client info</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@onyx-23vm1:/tmp/test_logs/2016-10-06/171222# uname -a Linux onyx-23vm1.onyx.hpdd.intel.com 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux root@onyx-23vm1:/tmp/test_logs/2016-10-06/171222# lctl get_param version version= lustre: 2.7.18.2 kernel: patchless_client build: jenkins-arch=x86_64,build_type=client,distro=ubuntu1404,ib_stack=inkernel-115--PRISTINE-3.19.0-33-generic root@onyx-23vm1:/tmp/test_logs/2016-10-06/171222# </pre> </div></div> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 205: Verify job stats ============================================ ====================== 18:26:39 (1475803599) Waiting 90 secs for update Updated after 9s: wanted 'nodelocal' got 'nodelocal' Registered as changelog user cl5 mdt.lustre-MDT0000.job_cleanup_interval=5 jobid_name=id.205.mkdir.19052 Test: mkdir /mnt/lustre/d205.sanity Using JobID environment variable nodelocal=id.205.mkdir.19052 onyx-27: error: get_param: *//job_stats: Found no match onyx-27: error: get_param: *//job_stats: Found no match sanity test_205: @@@@@@ FAIL: No jobstats for id.205.mkdir.19052 found on mds::: *..job_stats Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4805:error_noexit() </pre> </div></div> server: EE3.1 tag-2.7.18.2 build#115 RHEL7.2 ldiskfs<br/> client: EE3.1 tag-2.7.18.2 build#115 Ubuntu14.04 LU-10361

Ubuntu1404 client sanity test_205: FAIL: No jobstats for id.205.mkdir.19052 found on mds

Bug Minor Resolved Cannot Reproduce Emoly Liu Sarah Liu Fri, 7 Oct 2016 03:09:52 +0000 Tue, 17 Mar 2020 03:31:26 +0000 Tue, 17 Mar 2020 03:31:26 +0000 0 5 <p>please see attached for logs</p> <p>Emoly, can you take a look of this issue? Thanks!</p> <p>This issue was caused by a wrong "$(convert_facet2label $facet)" output, what's why we got "*..job_stats" there.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sarah" class="user-hover" rel="sarah">sarah</a>, I built a ubuntu 3.13.0-32-generic 1404 on my local VM, I can mount a client but can't run any test scripts due to some issues. So can I access your ubuntu env? Or can you reproduce this issue with "sh -x", I want to see what happened to convert_facet2label? Thanks.</p> <p>Hello Emoly,</p> <p>Sorry but I will be out of office next week, so cannot get back to you promptly. The best suggestion I could have is logging on ONYX and set up the env there. Here is an example of what I did to setup the Ubuntu client env, hope this is helpful.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1. provision Ubuntu client by loadjenkins command loadjenkinsbuild -p test -d ubuntu1404 -a x86_64 -n xx -r 2. install the matched kernel which lustre client based on apt-get install linux-image-xxx-generic # I think the kernel version should be "3.19.0-33-generic" apt-get install linux-image-extra-xxx-generic # same as above reboot with the right kernel 3. download deb files from jenkins and install all of them 4. install pdsh and the dependency apt-get -f install apt-get -f install pdsh 5. change sh to bash instead of the default dash rm /bin/sh ln -s /bin/bash /bin/sh 6. make link to lib64 instead of the default lib ln -s lib lib64 </pre> </div></div> <p>After finish the above, you should be able to run test scripts.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sarah" class="user-hover" rel="sarah">sarah</a>, thanks, I will have a try.</p> <p>With Sarah's setup steps, I can make my ubuntu client work for me. But I can't reproduce this issue with ubuntu client and centos server.<br/> BTW, the logs showed the following errors after the failure, I suspect some network connect issue happened at that time.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>onyx-23vm1: Host key verification failed. onyx-23vm1: rsync: connection unexpectedly closed (0 bytes received so far) [sender] onyx-23vm1: rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.0] onyx-23vm4: Host key verification failed. onyx-23vm4: rsync: connection unexpectedly closed (0 bytes received so far) [sender] onyx-23vm4: rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9] onyx-28: Host key verification failed. onyx-28: rsync: connection unexpectedly closed (0 bytes received so far) [sender] onyx-28: rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9] onyx-27: Host key verification failed. onyx-27: rsync: connection unexpectedly closed (0 bytes received so far) [sender] onyx-27: rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]" </pre> </div></div> <p>So I suggest to close this ticket and reopen it if we hit it again.</p> <p>This is failing when running the command <tt>"lctl get_param <b>.</b>.job_stats | grep -c 'job_id.*mkdir'"</tt>.</p> <p>At first guess, I'd think that this is caused by the remote shell expansion dropping the "*" or something, but it might relate to a problem with <tt>/proc</tt> or <tt>/sys</tt> not having the <tt>job_stats</tt> file on the Ubuntu client?</p> <p>Closing old bug not seen in a long time.</p> Development Rank 1|hzyqrb: Rank (Obsolete) 9223372036854775807 Severity