https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5686 Lustre <p>We hit the following LBUG twice on one of our MDT:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[78073.117731] Lustre: 31681:0:(ldlm_lib.c:952:target_handle_connect()) work2-MDT0000: connection from 38d12a48-aabd-9279-dc69-b78c4e00321c@10.100.62.72@o2ib2 t189645377601 exp ffff880b95bb1c00 cur 1410508503 last 1410508503 [78079.176124] Lustre: 31681:0:(mdt_handler.c:1005:mdt_getattr_name_lock()) Although resent, but still not get child lockparent:[0x22f2b0783:0x34b:0x0] child:[0x22d854b6e:0x85d5:0x0] [78079.192443] LustreError: 31681:0:(mdt_handler.c:3203:mdt_intent_lock_replace()) ASSERTION( lustre_msg_get_flags(req-&gt;rq_reqmsg) &amp; 0x0002 ) failed: [78079.205971] LustreError: 31681:0:(mdt_handler.c:3203:mdt_intent_lock_replace()) LBUG [78079.215326] Pid: 31681, comm: mdt_104 [78079.220352] [78079.220353] Call Trace: [78079.227394] [&lt;ffffffffa051a7f5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [78079.236100] [&lt;ffffffffa051ae07&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [78079.243815] [&lt;ffffffffa0d9671b&gt;] mdt_intent_lock_replace+0x3bb/0x440 [mdt] [78079.252140] [&lt;ffffffffa0daad26&gt;] mdt_intent_getattr+0x3a6/0x4a0 [mdt] [78079.260391] [&lt;ffffffffa0da6c09&gt;] mdt_intent_policy+0x379/0x690 [mdt] [78079.268641] [&lt;ffffffffa07423c1&gt;] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc] [78079.276846] [&lt;ffffffffa07683cd&gt;] ldlm_handle_enqueue0+0x48d/0xf50 [ptlrpc] [78079.285614] [&lt;ffffffffa0da7586&gt;] mdt_enqueue+0x46/0x130 [mdt] [78079.292950] [&lt;ffffffffa0d9c762&gt;] mdt_handle_common+0x932/0x1750 [mdt] [78079.300987] [&lt;ffffffffa0d9d655&gt;] mdt_regular_handle+0x15/0x20 [mdt] [78079.309024] [&lt;ffffffffa07974f6&gt;] ptlrpc_main+0xd16/0x1a80 [ptlrpc] [78079.316979] [&lt;ffffffff810017cc&gt;] ? __switch_to+0x1ac/0x320 [78079.324222] [&lt;ffffffffa07967e0&gt;] ? ptlrpc_main+0x0/0x1a80 [ptlrpc] [78079.331896] [&lt;ffffffff8100412a&gt;] child_rip+0xa/0x20 [78079.338522] [&lt;ffffffffa07967e0&gt;] ? ptlrpc_main+0x0/0x1a80 [ptlrpc] [78079.346599] [&lt;ffffffffa07967e0&gt;] ? ptlrpc_main+0x0/0x1a80 [ptlrpc] [78079.354520] [&lt;ffffffff81004120&gt;] ? child_rip+0x0/0x20 [78079.361136] [78079.364683] Kernel panic - not syncing: LBUG </pre> </div></div> <p>The support engineer was able to retrieve the client node from the crash dump. Both time, the client was a login node running Lustre 2.4.3.</p> <p>It looks like <a href="https://jira.whamcloud.com/browse/LU-5314" title="Lustre 2.4.2 MDS hit LBUG and crash" class="issue-link" data-issue-key="LU-5314"><del>LU-5314</del></a>. The backported patch proposal failed on maloo ( <a href="http://review.whamcloud.com/#/c/10902/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10902/</a> )</p> Clients:<br/> &nbsp;- RHEL6 w/ patched kernel 2.6.32-431.11.2.el6 <br/> &nbsp;- Lustre 2.4.3 + bullpatches<br/> Servers:<br/> &nbsp;- RHEL6 w/ patched kernel 2.6.32-220.23.1<br/> &nbsp;- Lustre 2.1.6 + bullpatches LU-5686

(mdt_handler.c:3203:mdt_intent_lock_replace()) ASSERTION( lustre_msg_get_flags(req->rq_reqmsg) & 0x0002 ) failed

Bug Major Resolved Duplicate Bruno Faccini Bruno Travouillon p4b Tue, 30 Sep 2014 15:24:20 +0000 Wed, 13 Oct 2021 03:05:54 +0000 Wed, 13 Oct 2021 03:05:54 +0000 Lustre 2.1.6 Lustre 2.4.3 0 4 <p>Bruno T</p> <p>If this is indeed a duplicate of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> then it would be safer to use the patch from <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a>. However, better still would probably be to pick up the fix for this when you upgrade to the Bull 2.5.x version</p> <p>Bruno F</p> <p>Anything to add/correct?</p> <p>Peter</p> <p>Hello Bruno,<br/> Unfortunately, my b2_4 back-port of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> patch at <a href="http://review.whamcloud.com/#/c/10902/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10902/</a> is buggy ...<br/> My colleague Oleg Drokin has been working on this and I will gather the correct set of fixes you need to apply. Will also update <a href="https://jira.whamcloud.com/browse/LU-5314" title="Lustre 2.4.2 MDS hit LBUG and crash" class="issue-link" data-issue-key="LU-5314"><del>LU-5314</del></a> with the same infos.</p> <p>Bruno, can you also check if you run with additional patch from <a href="https://jira.whamcloud.com/browse/LU-3338" title="IOC_MDC_GETFILESTRIPE can abuse vmalloc()" class="issue-link" data-issue-key="LU-3338"><del>LU-3338</del></a> ?</p> <p>Hi,</p> <p>We run without the patch from <a href="https://jira.whamcloud.com/browse/LU-3338" title="IOC_MDC_GETFILESTRIPE can abuse vmalloc()" class="issue-link" data-issue-key="LU-3338"><del>LU-3338</del></a>.</p> <p>I do agree with Peter, a fix for 2.5+ should be sufficient enough.</p> <p>Ok fine, but can you provide the list of additional patches for both Client and Server sides ? Thanks in advance.</p> <p>Client: Lustre 2.4.3 + patches</p> <ul class="alternate" type="square"> <li>nf16987_lu4279_dot_lustre_access_by_fid.patch</li> <li>nf16815_lu4298_setstripe_does_not_create_file_with_no_stripe.patch</li> <li>nf16818_lu4136_target_health_status.patch</li> <li>lu4379_dont_always_check_max_pages_per_rpc_alignement.patch</li> <li>nf15746_nf17470_increase_oss_num_threads_max_value.patch</li> <li>nf16693_lu1165_obdclass_add_LCT_SERVER_SESSION_for_server_session.patch</li> <li>nf17550_add_extents_stats_max_processes_tunable.patch</li> <li>nf17342_lu4460_multiple_nids.patch</li> <li>lu4222_mdt_Extra_checking_for_getattr_RPC.patch</li> <li>lu4008_mdt_Shrink_default_LOVEA_reply_buffer.patch</li> <li>lu4008_mdt_actual_size_for_layout_lock_intents.patch</li> <li>lu4719_mdt_Dont_try_to_print_non-existent_objects.patch</li> <li>nf17522_lu3230_ost_umount_stuck.patch</li> <li>nf17870_lu2059_mgc_use_osd_api_for_backup_logs.patch</li> <li>nf17870_lu3915_lu4878_osd-ldiskfs_dont_assert_on_possible_upgrade.patch</li> <li>nf17891_lu4403_extra_lock_during_resend_lock_lookup.patch</li> <li>nf17811_lu4611_too_many_transaction_credits.patch</li> <li>nf17808_lu4558_clio_Solve_a_race_in_cl_lock_put.patch</li> <li>nf17755_lu4790_lu4509_ptlrpc_re-enqueue_ptlrpcd_worker__ptlrpcd_stick_to_a_single_CPU.patch</li> <li>nf17607_lu4791_lod_subtract_xattr_overhead_when_calculate_max_EA_size.patch</li> <li>nf17864_lu4659_mdd_rename_forgets_updating_target_linkEA.patch</li> <li>nf17977_lu4554_lfsck_Old_single-OI_MDT_always_scrubbed.patch</li> <li>nf17971_lu4878_lu3126_osd_remove_fld_lookup_during_configuration.patch</li> <li>nf17704_lu4881_lu4381_lov_to_not_hold_sub_locks_at_initialization.patch</li> <li>nf13212_lu4650_lu4257_clio_replace_semaphore_with_mutex.patch</li> </ul> <p>Server: Lustre 2.1.6 + patches</p> <ul class="alternate" type="square"> <li>ornl22_general_ptlrpcd_threads_pool_support.patch</li> <li>316929_lu1144_NUMA_aware_ptlrpcd_bind_policy.patch</li> <li>316874_lu1110_open_by_fid_oops.patch</li> <li>319248_lu2613_to_much_unreclaimable_slab_space.patch</li> <li>319615_lu2624_ptlrpc_fix_thread_stop.patch</li> <li>317941_lu2683_client_deadlock_in_cl-lock-mutex-get.patch</li> <li>319613_too_many_ll-inode-revalidate-fini-failure_msg_in_syslog.patch</li> <li>nf15746_increase_oss_num_threads_max_value.patch</li> <li>nf13204_lu2665_keep_resend_flocks.patch</li> <li>nf15559_lu1306_protect_l-flags_with_locking_to_prevent_race.patch</li> <li>nf13194_lu2943_LBUG_in_mdt_reconstruct_open.patch</li> <li>nf13204_lu3701_only_resend_F_UNLCKs.patch</li> </ul> <p>Hope this helps <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Hello Bruno,</p> <p>In fact there are regression issues with b2_4 back-port (<a href="http://review.whamcloud.com/#/c/10902/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10902/</a>) of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> changes. And I checked that the b2_5 version (<a href="http://review.whamcloud.com/#/c/10492/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10492/</a>) is ok and that it will land soon now.</p> <p>Hi,</p> <p>We are now running Lustre 2.5.3 + b2_5 patch <a href="http://review.whamcloud.com/#/c/10492/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10492/</a>. Since the upgrade, we are hitting several issues on MDS/OSS around the ldlm. Are you aware of any complementary fix that we should apply with this one?</p> <p>In the meantime, we are still investigating those issues onsite and will report them asap in new JIRA tickets.</p> <p>Hello Bruno,<br/> I wonder if your new "ldlm-related" issues could be like those reported in <a href="https://jira.whamcloud.com/browse/LU-5530" title="MDS thread lockup witrh patched 2.5 server" class="issue-link" data-issue-key="LU-5530"><del>LU-5530</del></a> ?</p> <p>Hello Bruno,</p> <p>Yes, one of our issue is very close to <a href="https://jira.whamcloud.com/browse/LU-5530" title="MDS thread lockup witrh patched 2.5 server" class="issue-link" data-issue-key="LU-5530"><del>LU-5530</del></a>. I see that there is a couple of patches to apply on top of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a>. Thanks for the tip. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Bruno,<br/> You are right this must be made clear here also, so to be complete, the full list of post <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> related tickets/patches has been well documented and in detail by Oleg in its <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> 2x comments dated 11/Sep/14 for <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a>.</p> Related LU-4584 LU-5530 Development End date Thu, 12 Feb 2015 15:24:20 +0000 Rank 1|hzwxf3: Rank (Obsolete) 15925 Severity Start date Tue, 30 Sep 2014 15:24:20 +0000