https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4705 Lustre <p>Unexpected MDC LustreError's on most clients. </p> <p>Client 10:<br/> Mar 4 03:27:11 lustre10 kernel: LustreError: 183913:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> <p>Client 11:<br/> Mar 4 00:37:25 lustre11 kernel: LustreError: 89827:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> <p>Client 12:<br/> Mar 4 00:39:36 lustre12 kernel: LustreError: 11-0: cal-MDT0000-mdc-ffff8807b75c4000: Communicating with 192.168.20.1@tcp1, operation ldlm_enqueue failed with -116.<br/> Mar 4 00:39:36 lustre12 kernel: LustreError: 70225:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -116<br/> Mar 4 00:39:36 lustre12 kernel: LustreError: 70225:0:(vvp_io.c:1227:vvp_io_init()) cal: refresh file layout <span class="error">&#91;0x200001c0b:0x176e:0x0&#93;</span> error -116.<br/> Mar 4 03:09:33 lustre12 kernel: LustreError: 70225:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> <p>Client 13:<br/> Mar 4 00:29:54 lustre13 kernel: LustreError: 167294:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> <p>Client 14:<br/> Mar 4 01:18:04 lustre14 kernel: LustreError: 11-0: cal-MDT0000-mdc-ffff880787af8400: Communicating with 192.168.20.1@tcp1, operation ldlm_enqueue failed with -116.<br/> Mar 4 01:18:04 lustre14 kernel: LustreError: 11503:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -116<br/> Mar 4 01:18:04 lustre14 kernel: LustreError: 11503:0:(vvp_io.c:1227:vvp_io_init()) cal: refresh file layout <span class="error">&#91;0x200001c12:0xbbe2:0x0&#93;</span> error -116.</p> <p>Client 16:<br/> Mar 4 01:00:46 lustre16 kernel: LustreError: 141605:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> <p>Client 17:<br/> Mar 4 00:13:39 lustre17 kernel: LustreError: 11-0: cal-MDT0000-mdc-ffff8808038aa000: Communicating with 192.168.20.1@tcp1, operation ldlm_enqueue failed with -116.<br/> Mar 4 00:13:39 lustre17 kernel: LustreError: 126770:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -116<br/> Mar 4 00:13:39 lustre17 kernel: LustreError: 126770:0:(vvp_io.c:1227:vvp_io_init()) cal: refresh file layout <span class="error">&#91;0x200001beb:0x1aedf:0x0&#93;</span> error -116.<br/> Mar 4 02:02:43 lustre17 kernel: LustreError: 126770:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> <p>Client 18:<br/> Mar 1 05:34:03 lustre18 kernel: LustreError: 146331:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2</p> Running tip of Lustre b2_5, 1 MGS, 1 MDS, 2 OSS, 12 clients. LU-4705

LustreError: 89827:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2

Bug Minor Resolved Fixed WC Triage Brett Lee Tue, 4 Mar 2014 15:28:35 +0000 Thu, 26 Oct 2017 16:49:41 +0000 Tue, 24 Oct 2017 13:01:01 +0000 Lustre 2.5.1 Lustre 2.11.0 Lustre 2.10.2 0 8 <p>I see these same errors with a Lustre 2.5.0 Client. The do not seem to impact the usability of the filesystem. But this is listed as a Error so there could be something happening. </p> <p>Is the filesystem re-exported via NFS, or possibly have concurrent threads that are accessing and unlinking files?</p> <p>These messages mean that the client was looking up some file, but it was deleted by the time it tried to access it.</p> <p>-116 = -ESTALE, -2 = -ENOENT.</p> <p>The errors are not really fatal, and could probably be quieted from the console.</p> <p>I have seen this error with IOR no NFS. I am not sure if the errors were generated during one single file or file per process. </p> <p>No, there was no re-exporting, but each Lustre client did have four (4) mounts of the file system - each mount appearing active via the stats files in /proc.</p> <p>Brett, what was the workload being run here? Something that is creating and deleting files concurrently (e.g. racer), or possibly multiple threads doing "rm -r" on the same tree? Either this is "normal" and maybe we should quiet the error messages, or it might imply some sort of bug on the MDS with inode lookup or files unexpectedly being deleted. Are there application-visible errors that are unexpected ("No such file or directory")?</p> <p>Andreas, the workload was a mix of real jobs with varying IO patterns - most prominent of which was many small reads from large files. There was no artificial creating/deleting of files. As for the application, am now noticing that a setting disabled printing of "some" error an warning messages during this run, however, each job completed successfully. No unexpected application-visible errors were seen.</p> <p>This is being seen at Gulfstream. In their environment, there doesn't appear to be any operational consequence to it. But, it scared them. It'd be nice if we could mute these errors, as discussed in <a href="https://jira.hpdd.intel.com/browse/LU-4705?focusedCommentId=79255&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-79255" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-4705?focusedCommentId=79255&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-79255</a></p> <p>I just saw an instance of this error in the Lustre file system at TJNAF. It is the only instance I can recall of it being seen here, we are running lustre 2.5.3 pristine </p> <p>To expand a bit more... I have a test environment that I'm using to benchmark oss systems. Presently I have three osts on a single server running lustre 2.5.3. I've mounted it on a single client and am running IOR tests with the following parameters:</p> <p>mpirun -np 12 -bynode -machinefile ./nodelist ./ior -F -e -m -g -i 10 -t 1024k -b 42G -o /testL/benchmark/test</p> <p>where nodelist contains a single node.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28978" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28978</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4705" title="LustreError: 89827:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2" class="issue-link" data-issue-key="LU-4705"><del>LU-4705</del></a> mdc: improve mdc_enqueue() error message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9d8f53da6ac5482262c188ba1e0ca3fb395aedfd</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28978/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28978/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4705" title="LustreError: 89827:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2" class="issue-link" data-issue-key="LU-4705"><del>LU-4705</del></a> mdc: improve mdc_enqueue() error message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 471c5303eb29d5ea1ba5a683173bda63095dae78</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29736" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29736</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4705" title="LustreError: 89827:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2" class="issue-link" data-issue-key="LU-4705"><del>LU-4705</del></a> mdc: improve mdc_enqueue() error message<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: b51accd1a652406afbe41ad764d116d0f361a0fb</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29736/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29736/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4705" title="LustreError: 89827:0:(mdc_locks.c:916:mdc_enqueue()) ldlm_cli_enqueue: -2" class="issue-link" data-issue-key="LU-4705"><del>LU-4705</del></a> mdc: improve mdc_enqueue() error message<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: c27470755cf40ee33056011883a0d0600ce00340</p> Related LU-4973 LU-4522 Development Rank 1|hzwgpb: Rank (Obsolete) 12942 Severity