https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8081 Lustre <p>On the OSS's, a few times a day, I see in the logs:</p> <p>Apr 27 16:00:57 oss1 kernel: Lustre: 41839:0:(osd_handler.c:1265:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0<br/> Apr 27 16:00:57 oss1 kernel: Lustre: 41839:0:(osd_handler.c:1272:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0<br/> Apr 27 16:00:57 oss1 kernel: Lustre: 41839:0:(osd_handler.c:1282:osd_trans_dump_creds()) write: 2/12/0, punch: 1/4/0, quota 4/4/0<br/> Apr 27 16:00:57 oss1 kernel: Lustre: 41839:0:(osd_handler.c:1289:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0<br/> Apr 27 16:00:57 oss1 kernel: Lustre: 41839:0:(osd_handler.c:1296:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0<br/> Apr 27 16:00:57 oss1 kernel: LustreError: 41839:0:(osd_internal.h:1073:osd_trans_exec_op()) blizzard-OST0000-osd: op = 7, rb = 7</p> <p>Not sure what triggers it for sure. We have a mixed workload of mpi, and serial type jobs.</p> <p>The values are always the same: write 2/10/0, punch 1/4/0 , etc. A limit I imagine. It almost seems like debug type logs that are not super important. </p> EL 6.7, x86_64, mellanox FDR, lustre 2.8.0 client and server. 1 combined mds/mdt, and 7 combined oss/ost. LU-8081

Lustre 2.8.0, osd_trans_exec_op(), osd_trans_dump_creds()

Bug Minor Resolved Fixed Bruno Faccini christopher coffey Thu, 28 Apr 2016 16:51:29 +0000 Tue, 18 Oct 2016 21:13:44 +0000 Sat, 10 Sep 2016 04:18:28 +0000 Lustre 2.8.0 Lustre 2.9.0 0 6 <p>Bruno</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Christopher,<br/> thanks for filing this report. In addition to the above osd_trans_dump_creds() message, is there a stack trace or other debugging messages in the logs at the same time?</p> <p>You are correct that this is partly a debugging message, but it indicates that the internal ldiskfs transaction accounting is incorrect, and in some rare cases this might result in a failure in lower layers of the code. In our regression testing this would trigger an immediate assertion, but for production use the Lustre-level assertion is disabled to allow the system to continue running in the vast majority of cases.</p> <p>If you are able to take a short outage at some point in the future, you could set <tt>lctl set_param osd-ldiskfs.track_declares_assert=1</tt> on the OSS nodes so that the failure triggers an assertion and dumps the stack of the server process running this transaction.</p> <p>*<b>Please Note</b>* that <tt>track_declares_assert=1</tt> will require that the OSS be rebooted after the assertion has triggered, and should only be enabled if you are interested to debug this problem further, and rebooting the OSS is acceptable for your environment. If not, then we can try and find some other way to debug this problem.</p> <p>Another question that comes to mind is what the journal size is on your OSTs? This can be checked with <tt>dumpe2fs -h /dev/ostXXX | grep "Journal size"</tt> on each of the OST devices. Typically they should be 400MB in size, allowing individual transactions up to 100MB (25000 blocks), so it is a bit surprising that this debugging message is being printed with only a few tens of blocks reserved in this transaction.</p> <p>Hmm, actually it looks like credits were consumed outside of our tracking mechanism:<br/> we reserved 12 blocks for write, but at write we observed that &lt;12 left in the transaction.<br/> at the same time none of the previous operations consumed more than declared.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19865" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19865</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8081" title="Lustre 2.8.0, osd_trans_exec_op(), osd_trans_dump_creds()" class="issue-link" data-issue-key="LU-8081"><del>LU-8081</del></a> osd-ldiskfs: improve transaction debug message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8219dd1e5da64670ddcf84e7e779e315169e3b63</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19865/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19865/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8081" title="Lustre 2.8.0, osd_trans_exec_op(), osd_trans_dump_creds()" class="issue-link" data-issue-key="LU-8081"><del>LU-8081</del></a> osd-ldiskfs: improve transaction debug message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0cf638409812ce2533043908073b68698fb46c84</p> <p>Landed for 2.9</p> Development Rank 1|hzy9on: Rank (Obsolete) 9223372036854775807 Severity