Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-2277] Text file busy https://jira.whamcloud.com/browse/LU-2277 Lustre <ol> <li>ldd a.out<br/> /usr/bin/ldd: line 118: ./a.out: Text file busy</li> </ol> <p>see attached client and server debug traces.</p> server-2.1.3-1nas (centos6.3)<br/> client 2.1.3-1nas Suse11-1 LU-2277

Text file busy

Bug Critical Resolved Fixed Nathaniel Clark Mahmoud Hanafi Mon, 5 Nov 2012 13:05:40 +0000 Mon, 17 Mar 2014 17:51:25 +0000 Mon, 11 Mar 2013 14:16:23 +0000 Lustre 2.1.3 Lustre 2.1.5 0 5 <p>This is supposed to happen when somebody holds a file opened for write.</p> <p>How was a.out produced? Was somebody holding it open still?</p> <p>a.out was compiled in placed. It seems that error only occers on some of the clients. </p> <p>Is there any NFS reexport going on by any chance?<br/> Was the file created via NFS in particular? Do the clients where you are able to execute is on NFS and ones that fail are on lustre?</p> <p>I also see that you have standard debug enabled, if you can still reproduce this and there is no NFS involved, please increase debug level on your mds as follows:<br/> cat /proc/sys/lnet/debug<br/> remember the value it gave you</p> <p>try to minimize all other activity oing on on the cluster</p> <p>echo -1 >/proc/sys/lnet debug ; echo - trace >/proc/sys/lnet/debug</p> <p>do the reproducer</p> <ol> <li>important to run the lctl below as quickly as possible after the reproducer<br/> lctl dk >/tmp/lustre/log </li> </ol> <p>echo "YOUR SAVED VALUE FROM ABOVE" >/proc/sys/lnet/debug</p> <p>Thanks</p> <p>The Lustre filesyste is not reexported via nfs. There NFS mounts on the client for home dir, etc.</p> <p>I was able to track this issue to the stripe width of the directory. If the stripe width is > 54 it will reproduce the error.</p> <p>how it was reproduced.<br/> mhanafi@pfe1:/nobackupp5/mhanafi/60> cat test.c++<br/> #include <stdio.h><br/> int<br/> main(int argc, char *argv[])<br/> {<br/> return 0;<br/> }</p> <p>mhanafi@pfe1:/nobackupp5/mhanafi/60> icpc -O test.c++<br/> mhanafi@pfe1:/nobackupp5/mhanafi/60> ldd a.out<br/> /usr/bin/ldd: line 118: ./a.out: Text file busy</p> <p>see attached debug logs.</p> <p>Have reproduction case:<br/> 1 Client - CentOS 6.3 - Lustre 2.1.3-2.6.32_279.2.1.el6.x86_64<br/> gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)<br/> 1 MDS - CentOS 6.3 - Lustre 2.1.3-2.6.32_279.2.1.el6_lustre.gc46c389.x86_64 (1 mds/mgs partition)<br/> 1 OSS - CentOS 6.3 - Lustre 2.1.3-2.6.32_279.2.1.el6_lustre.gc46c389.x86_64 (60 osts on loop devices)</p> <p>All TCP interconnect<br/> All Server partitions created & mounted with standard autotest tools (auster -c PATH/TO/custom.sh sanity.sh --only MOUNT)</p> <p>In mounted FS:<br/> cd /mnt/lustre<br/> mkdir 58<br/> lfs setstripe -c 58 58<br/> cd 58<br/> cat << EOF > test.c<br/> #include <stdio.h><br/> int<br/> main(int argc, char *argv[])<br/> {<br/> return 0;<br/> }<br/> EOF<br/> gcc test.c<br/> ./a.out</p> <p>Expected Results:<br/> Should run w/o error. For directories with a strip width less than 54, there is no error.</p> <p>Actual Results:<br/> For stripe widths of 54 and larger (at least up to 60) the following error results:<br/> ./a.out: Text file busy</p> <p>Other Test Results:</p> <ul> <li>If the filesystem is remounted then all a.out's will run correctly.</li> <li>If any a.out is copied (either w/in or between directories, regardless of stripe width) it will run fine</li> <li>If "bad" a.out is moved it will still illicit the same error</li> </ul> <p>With 2 Clients (both CentOS 6.3 - Lustre 2.1.3-2.6.32_279.2.1.el6.x86_64)<br/> 1) First clients generates "bad" a.out<br/> 2) Second client mounts FS<br/> 3) 2nd client gets error running a.out<br/> 4) First client unmounts FS<br/> 5) 2nd Client no longer gets error running a.out</p> <p>WAG of cause:<br/> Client 1 when compiling test.c somehow continues to have a writecount ref (mot_writecount > 0) after compilation finishes. This ref is cleaned up when the client fully disconnects.</p> <p>Attached is C File which can recreate this bug in directories with a default stripe width greater than 53, by just doing a modified copy of an executable.</p> <p>What causes the executable to be in a bad state is the following order of events:<br/> 1. Create file (O_RDWR or O_WRONLY)<br/> 2. write to file<br/> 3. close file (up to here, everything is fine – this is just a copy)<br/> 4. open file (O_RDWR or O_WRONLY) - O_RDONLY do not cause this issue<br/> 5. close file</p> <p>After more careful inspection of the logs, it appears that in the failing cases, the second open is being processed twice:</p> <p>This is log lines that match the following grep expression: '((mdt_close(|mdt_mfd_open).*leaving|incoming)' from logs that are of a double open copy to a directory with a given stripe size:</p> <blockquote> <p>53<br/> (events.c:284:request_in_callback()) incoming req@ffff8800189bc850 x1421171244404975 msgsize 192<br/> (events.c:284:request_in_callback()) incoming req@ffff8800189bc050 x1421171249691684 msgsize 192<br/> (events.c:284:request_in_callback()) incoming req@ffff880014590800 x1421167733816518 msgsize 544<br/> (mdt_open.c:721:mdt_mfd_open()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff880012109400 x1421167733816520 msgsize 360<br/> (mdt_open.c:1645:mdt_close()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff880015236050 x1421167733816521 msgsize 544<br/> (events.c:284:request_in_callback()) incoming req@ffff880012043000 x1421167733816522 msgsize 544<br/> (mdt_open.c:721:mdt_mfd_open()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff8800152ca050 x1421167733816523 msgsize 360<br/> (mdt_open.c:1645:mdt_close()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff8800189bd850 x1421167733816525 msgsize 192<br/> 54<br/> (events.c:284:request_in_callback()) incoming req@ffff8800189bd050 x1421167797775378 msgsize 192<br/> (events.c:284:request_in_callback()) incoming req@ffff880012109400 x1421167797775379 msgsize 192<br/> (events.c:284:request_in_callback()) incoming req@ffff8800189be850 x1421171244404976 msgsize 192<br/> (events.c:284:request_in_callback()) incoming req@ffff880014590800 x1421167733816584 msgsize 544<br/> (mdt_open.c:721:mdt_mfd_open()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff8800148c9800 x1421167733816584 msgsize 544<br/> (events.c:284:request_in_callback()) incoming req@ffff88001458dc00 x1421167733816586 msgsize 360<br/> (mdt_open.c:1645:mdt_close()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff880015238850 x1421167733816587 msgsize 544<br/> (events.c:284:request_in_callback()) incoming req@ffff880014689800 x1421167733816588 msgsize 544<br/> (mdt_open.c:721:mdt_mfd_open()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff880012109400 x1421167733816588 msgsize 544<br/> (mdt_open.c:721:mdt_mfd_open()) Process leaving (rc=0 : 0 : 0)<br/> (events.c:284:request_in_callback()) incoming req@ffff880014590800 x1421167733816589 msgsize 360<br/> (mdt_open.c:1645:mdt_close()) Process leaving (rc=0 : 0 : 0)</p></blockquote> <p>Notice the repeated x1421167733816588 in the 54 stripe case with accompanying mdt_mfd_open but missing mdt_close(). mdt_close is skipped. This means that a request is being processed twice.</p> <p>Note on what goes over the wire:<br/> Items listed are wireshark dump of traffic, all items are requests client -> mds, all items are replied to prior to the next item being sent.</p> <p>For stripe sizes 50-53: (example is stripe 50)<br/> 1) LDLM_ENQUEUE request <span class="error">[Concurrent Write]</span>[ intent : open create ] filename : a2<br/> 2) MDS_CLOSE <span class="error">[REINT_SETATTR]</span><br/> 3) LDLM_ENQUEUE request <span class="error">[Concurrent Read]</span>[ intent : lookup ] filename : 50<br/> 4) LDLM_ENQUEUE request <span class="error">[Concurrent Read]</span>[ intent : open ]<span class="error">[REINT_OPEN]</span> filename : a2<br/> 5) MDS_CLOSE <span class="error">[REINT_SETATTR]</span></p> <p>For stripe sizes 54-59: (example is stripe 54)<br/> 1) LDLM_ENQUEUE request <span class="error">[Concurrent Write]</span>[ intent : open create ] filename : a2<br/> 2) LDLM_ENQUEUE request <span class="error">[Concurrent Write]</span>[ intent : open create ] filename : a2<br/> 3) MDS_CLOSE <span class="error">[REINT_SETATTR]</span><br/> 4) LDLM_ENQUEUE request <span class="error">[Concurrent Read]</span>[ intent : lookup ] filename : 54<br/> 5) LDLM_ENQUEUE request <span class="error">[Concurrent Read]</span>[ intent : open ] filename : a2<br/> 6) LDLM_ENQUEUE request <span class="error">[Concurrent Read]</span>[ intent : open ] filename : a2<br/> 7) MDS_CLOSE <span class="error">[REINT_SETATTR]</span></p> <p>Client seems to be re-sending open create because response data is large and reply_in_callback() registers it as truncated and resends. So the solution I'm going to look to implement is to correctly handle the replay of the open/create for write/exec in mdt_mfd_open().</p> <p><a href="http://review.whamcloud.com/4848" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4848</a></p> <p>Original Commit is now just testing updates:</p> <p>Following are cherry-picked commits that are dependent on 4848, 5002 is unchanged, 5003 needed a merge conflict resolved.<br/> <a href="http://review.whamcloud.com/5002" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5002</a><br/> <a href="http://review.whamcloud.com/5003" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5003</a></p> <p>Patches picked to branch</p> Duplicate LU-1571 LU-971 Development Rank 1|hzvbpb: Rank (Obsolete) 5443 Severity