https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2936 Lustre <p>ORNL did a full scale system test today and one of their OSSes crashed with the above assertion.</p> <p>It seems we never saw it before because we don't seriously test /proc/fs/lustre/health_check functionality in our testing, but it's actually heavily used by a lot of sites.</p> <p>I was able to reproduce the issue with racer while running this line in parallel:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">while</span> :; <span class="code-keyword">do</span> cat /proc/fs/lustre/health_check ; done </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[305098.783912] LustreError: 1075:0:(ptlrpc_internal.h:165:nrs_svcpt2nrs()) ASSERTION( (!(hp) || (nrs_svcpt_has_hp(svcpt))) ) failed: [305098.784415] LustreError: 1075:0:(ptlrpc_internal.h:165:nrs_svcpt2nrs()) LBUG [305098.784682] Pid: 1075, comm: cat [305098.784881] [305098.784881] Call Trace: [305098.785248] [&lt;ffffffffa07b7915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [305098.785524] [&lt;ffffffffa07b7f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [305098.785810] [&lt;ffffffffa11b0e22&gt;] ptlrpc_nrs_req_poll_nolock+0xc2/0x1c0 [ptlrpc] [305098.786239] [&lt;ffffffffa1170696&gt;] ptlrpc_svcpt_health_check+0x56/0x180 [ptlrpc] [305098.786664] [&lt;ffffffffa1170812&gt;] ptlrpc_service_health_check+0x52/0x70 [ptlrpc] [305098.787079] [&lt;ffffffffa05d61fd&gt;] ost_health_check+0x4d/0x90 [ost] [305098.787345] [&lt;ffffffffa0e4c8e7&gt;] obd_proc_read_health+0x2a7/0x3b0 [obdclass] [305098.792327] [&lt;ffffffffa0e6f36c&gt;] lprocfs_fops_read+0xec/0x1f0 [obdclass] [305098.793699] [&lt;ffffffffa0e6f280&gt;] ? lprocfs_fops_read+0x0/0x1f0 [obdclass] [305098.793964] [&lt;ffffffff811e1cc5&gt;] proc_reg_read+0x85/0xc0 [305098.794200] [&lt;ffffffff8117b9e5&gt;] vfs_read+0xb5/0x1a0 [305098.794429] [&lt;ffffffff8117bb21&gt;] sys_read+0x51/0x90 [305098.794657] [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b [305098.794907] [305098.992803] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Crashdump and modules are in /exports/crashdumps/192.168.10.210-2013-03-09-00\:44\:57</p> <p>The problem was seemingly added along with NRS code drop.</p> LU-2936

nrs_svcpt2nrs()) ASSERTION( (!(hp) || (nrs_svcpt_has_hp(svcpt))) ) failed

Bug Blocker Resolved Fixed Oleg Drokin Oleg Drokin HB Sat, 9 Mar 2013 00:57:25 +0000 Wed, 13 Mar 2013 08:55:41 +0000 Wed, 13 Mar 2013 08:24:57 +0000 Lustre 2.4.0 Lustre 2.4.0 0 5 <p>Ok, I now checked the dump and the situation is clear.<br/> The ptlrpc_svcpt_health_check tries to call ptlrpc_nrs_req_poll_nolock twice for every service partition that has anything pending.<br/> Once time with hp set to true and once to false.<br/> Then if we happen to be called for a service that does not have hp ops registered (like ost_create), the assertion trips as the underlying nrs code seems to assuem caller must be all smart about service types and request for a hp request for a service with no possible hp requests is a no-no (which is a bit strange, considering that it's perfectly ok to check if a service has any hp requests pendign even for non-hp services.)</p> <p>As such possible fixes are:<br/> 1. remove the assertion and restrictions on caller knowledge on underlying service when trying to fetch requests.<br/> 2. Check that a hp request is actually available before trying to fetch it from ptlrpc_svcpt_health_check</p> <p>patch in <a href="http://review.whamcloud.com/5665" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5665</a></p> <p>As mentioned in Gerrit, this bug is addressed in the NRS follow-up patch as well, but '2' from the comment above that you have used is a better solution. Maybe '1' could be used to improve things on a future patch.</p> <p>Patch landed to master.</p> Related LU-398 Development Rank 1|hzvkiv: Rank (Obsolete) 7054 Severity