https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14578 Lustre <p>This filesystem has 2 servers and both hit LBUG</p> <p>First one</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [4308300.501701] LustreError: 17867:0:(tgt_grant.c:463:tgt_grant_space_left()) nbp12-OST0000: cli 55a893a2-37fc-a91e-0edd-c13e39c0bcd7/ffff93b5024b4000 left 51980938248192 &lt; tot_grant 4611686279107159457 unstable 85434368 pending 90030080 dirty 390602752 [4308300.568648] LustreError: 17867:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 69 previous similar messages [4308305.988484] LustreError: 80109:0:(tgt_grant.c:463:tgt_grant_space_left()) nbp12-OST0000: cli ba1ae12e-b75a-7684-bce5-34097cf4072c/ffff93b488caa400 left 51980913885184 &lt; tot_grant 4611686277031540129 unstable 0 pending 0 dirty 344813568 [4308306.051765] LustreError: 80109:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 44 previous similar messages [4308314.057174] LustreError: 86494:0:(tgt_grant.c:463:tgt_grant_space_left()) nbp12-OST0000: cli b5875278-4309-10d6-2c2d-557864167838/ffff93a576975c00 left 51980868866048 &lt; tot_grant 4611686246882669986 unstable 27496448 pending 31715328 dirty 373313536 [4308314.124119] LustreError: 86494:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 38 previous similar messages [4308330.021959] LustreError: 17949:0:(tgt_grant.c:463:tgt_grant_space_left()) nbp12-OST0000: cli fcefe8b5-bf4c-3d12-56f9-fb1b93134dfc/ffff939c3f41a800 left 51980592369664 &lt; tot_grant 4611686224816031140 unstable 25616384 pending 25669632 dirty 388505600 [4308330.088908] LustreError: 17949:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 183 previous similar messages [4308337.687240] LustreError: 19209:0:(tgt_grant.c:151:tgt_check_export_grants()) nbp12-OST0000: cli 5fc11b40-37df-3838-564b-bfebf0abf25a/ffff93ad4aeb9000 ted_grant(4611686037896666112) + ted_pending(0) &gt; maxsize(77690215612416) [4308337.747390] LustreError: 19209:0:(tgt_grant.c:223:tgt_grant_sanity_check()) LBUG [4308337.749201] LustreError: 25433:0:(tgt_grant.c:223:tgt_grant_sanity_check()) LBUG [4308337.749203] Pid: 25433, comm: kworker/11:0 3.10.0-1127.19.1.el7_lustre2125.x86_64 #1 SMP Mon Nov 2 14:50:02 PST 2020 [4308337.749203] Call Trace: [4308337.749221]  [&lt;ffffffffc09f67cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [4308337.749225]  [&lt;ffffffffc09f687c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [4308337.749275]  [&lt;ffffffffc135aa20&gt;] tgt_grant_sanity_check+0x520/0x560 [ptlrpc] [4308337.749281]  [&lt;ffffffffc18dccd8&gt;] ofd_destroy_export+0x88/0x110 [ofd] [4308337.749304]  [&lt;ffffffffc0e2459e&gt;] class_export_destroy+0xee/0x490 [obdclass] [4308337.749316]  [&lt;ffffffffc0e24955&gt;] obd_zombie_exp_cull+0x15/0x20 [obdclass] [4308337.749319]  [&lt;ffffffffa28be6bf&gt;] process_one_work+0x17f/0x440 [4308337.749321]  [&lt;ffffffffa28bf7d6&gt;] worker_thread+0x126/0x3c0 [4308337.749323]  [&lt;ffffffffa28c6691&gt;] kthread+0xd1/0xe0 [4308337.749325]  [&lt;ffffffffa2f92d1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [4308337.749340]  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [4308337.749341] Kernel panic - not syncing: LBUG [4308337.749343] CPU: 11 PID: 25433 Comm: kworker/11:0 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1127.19.1.el7_lustre2125.x86_64 #1 [4308337.749343] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 06/15/2018 [4308337.749356] Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] [4308337.749356] Call Trace: [4308337.749360]  [&lt;ffffffffa2f7ffa5&gt;] dump_stack+0x19/0x1b [4308337.749362]  [&lt;ffffffffa2f79541&gt;] panic+0xe8/0x21f [4308337.749367]  [&lt;ffffffffc09f68cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [4308337.749399]  [&lt;ffffffffc135aa20&gt;] tgt_grant_sanity_check+0x520/0x560 [ptlrpc] [4308337.749403]  [&lt;ffffffffc18dccd8&gt;] ofd_destroy_export+0x88/0x110 [ofd] [4308337.749416]  [&lt;ffffffffc0e2459e&gt;] class_export_destroy+0xee/0x490 [obdclass] [4308337.749428]  [&lt;ffffffffc0e24955&gt;] obd_zombie_exp_cull+0x15/0x20 [obdclass] [4308337.749430]  [&lt;ffffffffa28be6bf&gt;] process_one_work+0x17f/0x440 [4308337.749431]  [&lt;ffffffffa28bf7d6&gt;] worker_thread+0x126/0x3c0 [4308337.749433]  [&lt;ffffffffa28bf6b0&gt;] ? manage_workers.isra.26+0x2a0/0x2a0 [4308337.749434]  [&lt;ffffffffa28c6691&gt;] kthread+0xd1/0xe0 [4308337.749436]  [&lt;ffffffffa28c65c0&gt;] ? insert_kthread_work+0x40/0x40 [4308337.749437]  [&lt;ffffffffa2f92d1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [4308337.749438]  [&lt;ffffffffa28c65c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>Second one </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [3879290.271375] LustreError: 16716:0:(tgt_grant.c:463:tgt_grant_space_left()) nbp12-OST000b: cli ad8e8405-8dcc-dcec-4983-af2b3509ed40/ffff9a21dd688400 left 51431169011712 &lt; tot_grant 4611686156105684284 unstable 0 pending 0 dirty 55066624 [3879290.334406] LustreError: 16716:0:(tgt_grant.c:463:tgt_grant_space_left()) Skipped 6 previous similar messages [3879348.214404] LustreError: 87087:0:(tgt_grant.c:151:tgt_check_export_grants()) nbp12-OST000b: cli 5fc11b40-37df-3838-564b-bfebf0abf25a/ffff9a32405cac00 ted_grant(4611686037969102848) + ted_pending(0) &gt; maxsize(77690215612416) [3879348.274549] LustreError: 87087:0:(tgt_grant.c:223:tgt_grant_sanity_check()) LBUG [3879348.297309] Pid: 87087, comm: kworker/17:3 3.10.0-1127.19.1.el7_lustre2125.x86_64 #1 SMP Mon Nov 2 14:50:02 PST 2020 [3879348.297314] Call Trace: [3879348.297326] [&lt;ffffffffc0b447cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [3879348.302057] [3879348.302061] [&lt;ffffffffc0b4487c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [3879348.302061] [3879348.302107] [&lt;ffffffffc15d6a20&gt;] tgt_grant_sanity_check+0x520/0x560 [ptlrpc] [3879348.302113] [&lt;ffffffffc125ecd8&gt;] ofd_destroy_export+0x88/0x110 [ofd] [3879348.302133] [&lt;ffffffffc0f8f59e&gt;] class_export_destroy+0xee/0x490 [obdclass] [3879348.302145] [&lt;ffffffffc0f8f955&gt;] obd_zombie_exp_cull+0x15/0x20 [obdclass] [3879348.302149] [&lt;ffffffffaccbe6bf&gt;] process_one_work+0x17f/0x440 [3879348.302150] [&lt;ffffffffaccbf7d6&gt;] worker_thread+0x126/0x3c0 [3879348.302152] [&lt;ffffffffaccc6691&gt;] kthread+0xd1/0xe0 [3879348.302154] [&lt;ffffffffad392d1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [3879348.302169] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [3879348.302170] Kernel panic - not syncing: LBUG [3879348.302172] CPU: 17 PID: 87087 Comm: kworker/17:3 Kdump: loaded Tainted: G OE ------------ 3.10.0-1127.19.1.el7_lustre2125.x86_64 #1 [3879348.302172] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 06/15/2018 [3879348.302185] Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] [3879348.302185] Call Trace: [3879348.302188] [&lt;ffffffffad37ffa5&gt;] dump_stack+0x19/0x1b [3879348.302190] [&lt;ffffffffad379541&gt;] panic+0xe8/0x21f [3879348.302196] [&lt;ffffffffc0b448cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [3879348.302227] [&lt;ffffffffc15d6a20&gt;] tgt_grant_sanity_check+0x520/0x560 [ptlrpc] [3879348.302231] [&lt;ffffffffc125ecd8&gt;] ofd_destroy_export+0x88/0x110 [ofd] [3879348.302243] [&lt;ffffffffc0f8f59e&gt;] class_export_destroy+0xee/0x490 [obdclass] [3879348.302255] [&lt;ffffffffc0f8f955&gt;] obd_zombie_exp_cull+0x15/0x20 [obdclass] [3879348.302257] [&lt;ffffffffaccbe6bf&gt;] process_one_work+0x17f/0x440 [3879348.302258] [&lt;ffffffffaccbf7d6&gt;] worker_thread+0x126/0x3c0 [3879348.302260] [&lt;ffffffffaccbf6b0&gt;] ? manage_workers.isra.26+0x2a0/0x2a0 [3879348.302261] [&lt;ffffffffaccc6691&gt;] kthread+0xd1/0xe0 [3879348.302263] [&lt;ffffffffaccc65c0&gt;] ? insert_kthread_work+0x40/0x40 [3879348.302264] [&lt;ffffffffad392d1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [3879348.302265] [&lt;ffffffffaccc65c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> LU-14578

LustreError: 25433:0:(tgt_grant.c:223:tgt_grant_sanity_check()) LBUG

Bug Minor Resolved Won't Fix Mikhail Pershin Mahmoud Hanafi Thu, 1 Apr 2021 18:19:47 +0000 Thu, 29 Jun 2023 02:33:55 +0000 Mon, 17 Jan 2022 11:39:13 +0000 Lustre 2.12.5 0 6 <p>Mike</p> <p>I believe that you've looked into some issues in this area recently. What do you advise here?</p> <p>Peter</p> <p>It looks like grants were overflow at some point causing <tt>tot_grant</tt> to become too large. I will check known issues first</p> <p>problem should be solved in the latest 2.12 releases</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/93a6a37b-819c-44d2-b248-8fc24bc85404" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/93a6a37b-819c-44d2-b248-8fc24bc85404</a></p> Development Rank 1|i01r67: Rank (Obsolete) 9223372036854775807 Severity