https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17185 Lustre <p>After running lctl conf_param lustrefc-OST0018.osc.active=0 on the MGS for multiple OSTs, some clients see the OSTs as inactive and work just fine. Some clients see the OSTs as active and hang. The software stack is the same on working and non-working clients.</p> <p>Here is some sample output from a working client:<br/> 4 IN osc lustrefc-OST0007-osc-ffff9d035c824000 c75633d5-3afe-6370-7e49-dcad475a6bc2 4<br/> 5 IN osc lustrefc-OST000e-osc-ffff9d035c824000 c75633d5-3afe-6370-7e49-dcad475a6bc2 4<br/> 6 IN osc lustrefc-OST000f-osc-ffff9d035c824000 c75633d5-3afe-6370-7e49-dcad475a6bc2 4</p> <ol> <li>Note disabled OSTs not listed</li> <li>lfs df -h partial ouput<br/> lustrefc-MDT0000_UUID 229.3G 61.3G 131.4G 32% /data<span class="error">&#91;MDT:0&#93;</span><br/> lustrefc-OST0012_UUID 64.9T 20.3T 44.6T 32% /data<span class="error">&#91;OST:18&#93;</span></li> </ol> <p>Output from a non-working client<br/> 4 UP osc lustrefc-OST0007-osc-ffff958c1995b800 4910e4fd-accd-1685-c6d2-3418a29afbd1 3<br/> 5 UP osc lustrefc-OST000e-osc-ffff958c1995b800 4910e4fd-accd-1685-c6d2-3418a29afbd1 3<br/> 6 UP osc lustrefc-OST000f-osc-ffff958c1995b800 4910e4fd-accd-1685-c6d2-3418a29afbd1 3</p> <ol> <li>lfs df -h partial output<br/> lustrefc-MDT0000_UUID 229.3G 61.3G 131.4G 32% /data<span class="error">&#91;MDT:0&#93;</span><br/> OST0007 : Invalid argument<br/> OST000e : Invalid argument<br/> OST000f : Invalid argument</li> </ol> <p>This has rendered our cluster unusable. </p> Client: <br/> CentOS Linux release 7.9.2009 (Core) <br/> 3.10.0-1160.49.1.el7.x86_64 <br/> lustre-client-2.12.9-1.el7.x86_64 <br/> lustre-client-dkms-2.12.9-1.el7.noarch <br/> <br/> Servers: <br/> Servers: <br/> CentOS Linux release 7.9.2009 (Core) <br/> 3.10.0-1160.49.1.el7_lustre.x86_64 <br/> Lustre 2.12.9 <br/> <br/> LU-17185

After deactivating OSTs, some clients see them as active

Bug Blocker Open Unresolved WC Triage Roger Sersted Wed, 11 Oct 2023 20:05:27 +0000 Fri, 13 Oct 2023 18:48:46 +0000 Lustre 2.12.9 0 2 <p>I should add, I tried rebooting one of the problem nodes and that did not resolve this issue.</p> <p>I can't say for sure why this setting is not being applied to some of the clients. As a workaround, you could manually deactivate these OSTs on the affected clients like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client# lctl set_param osc.*OST{0007,000e,000f}*.active=0 </pre> </div></div> <p>possibly using <tt>pdsh</tt> or other tool to execute it on multiple clients at once. It shouldn't be harmful if this is run on clients that already have the OSTs deactivated.</p> <p>It would be worthwhile to check that the conf_param command is present in the client config log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mgs# lctl --device MGS llog_print lustrefc-client | grep OST0007 </pre> </div></div> <p>There would be initial commands to add the OST and then the last one should be the one to mark it inactive. In newer releases there is an "<tt>lctl del_ost</tt>" command that will remove the OST setup commands from the configuration log completely. </p> <p>Thank you. The lctl ...active=0 command on the clients fixed the hang problem. I'll run the llog_print command later and attach the results.</p> <p>I thought it fixed the problem. On the problem clients, I'm seeing this:</p> <p> <span class="error">&#91;root@puppy83 ~&#93;</span># lfs df -h<br/> UUID bytes Used Available Use% Mounted on<br/> lustrefc-MDT0000_UUID 229.3G 61.3G 131.4G 32% /data<span class="error">&#91;MDT:0&#93;</span><br/> OST0007 : Invalid argument<br/> OST000e : Invalid argument<br/> OST000f : Invalid argument<br/> OST0010 : Invalid argument<br/> OST0011 : Invalid argument<br/> lustrefc-OST0012_UUID 64.9T 20.1T 44.8T 31% /data<span class="error">&#91;OST:18&#93;</span><br/> lustrefc-OST0013_UUID 64.9T 19.4T 45.6T 30% /data<span class="error">&#91;OST:19&#93;</span><br/> lustrefc-OST0014_UUID 64.9T 49.4T 15.5T 77% /data<span class="error">&#91;OST:20&#93;</span><br/> lustrefc-OST0015_UUID 64.9T 48.8T 16.1T 76% /data<span class="error">&#91;OST:21&#93;</span><br/> OST0016 : Invalid argument<br/> OST0017 : Invalid argument<br/> OST0018 : Invalid argument<br/> <span class="error">&#91;root@puppy83 ~&#93;</span># lfs osts<br/> OBDS:<br/> 7: lustrefc-OST0007_UUID INACTIVE<br/> 14: lustrefc-OST000e_UUID INACTIVE<br/> 15: lustrefc-OST000f_UUID INACTIVE<br/> 16: lustrefc-OST0010_UUID INACTIVE<br/> 17: lustrefc-OST0011_UUID INACTIVE<br/> 18: lustrefc-OST0012_UUID ACTIVE</p> <p><span class="error">&#91;root@puppy83 ~&#93;</span># lctl dl<br/> 0 UP mgc MGC172.17.1.112@o2ib 8214fcc9-bf1d-148d-8c9e-8040314a4b34 4<br/> 1 UP lov lustrefc-clilov-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 3<br/> 2 UP lmv lustrefc-clilmv-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 4<br/> 3 UP mdc lustrefc-MDT0000-mdc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 4<br/> 4 UP osc lustrefc-OST0007-osc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 3<br/> 5 UP osc lustrefc-OST000e-osc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 3<br/> 6 UP osc lustrefc-OST000f-osc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 3<br/> 7 UP osc lustrefc-OST0010-osc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 3<br/> 8 UP osc lustrefc-OST0011-osc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 3<br/> 9 UP osc lustrefc-OST0012-osc-ffff9cb9fd858000 9e88710e-fc2d-cda9-b054-cef11bc3aa18 4</p> Development Epic client mgs Epic/Theme client mgs Rank 1|i03y4v: Rank (Obsolete) 9223372036854775807 Severity