https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11844 Lustre <p>We have a fortran code that does small writes to a status file. This code worked well on Lustre 2.5.x. On Lustre 2.10.4 we’re experiencing a bug with the same code where one of the writes will hang for about 20 seconds. During that period all writes to the affect OST will hang. Io to other OSTs will work, and io from other nodes are unaffected.</p> <p> </p> <p>The strace of the application shows that it opens a file, truncates to 0, writes a char, tuncates to 1, and continues writing,closes then repeats. After a few cycles the write that passes 4k offset into the file will hang for about 30 seconds.</p> <p> </p> <p>The strace looks like this:</p> <p>…</p> <p>open("short_file", O_RDWR|O_CREAT, 0700) = 3</p> <p>ftruncate(3, 0) = 0</p> <p>write(3, "/", 1) = 1</p> <p>ftruncate(3, 1) = 0</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130) = 130</p> <p>write(3, "atHUpw+orbPHuU55Em+XvliyYOwQg2le"..., 130 ← Hangs Here</p> <p> </p> <p>The Lustre debug traces from the client shows the hang and indicates multiple calls to genops.c:1990:obd_stale_export_get. Attached are a reproducer, and the lctl dk output after issuing: echo "trace nettrace dlmtrace rpctrace vfstrace" &gt; /proc/sys/lnet/debug</p> <p> </p> <p> </p> LU-11844

IO pattern causing writes to hang to OST

Bug Major Open Unresolved WC Triage Alex Parga Wed, 9 Jan 2019 14:31:49 +0000 Wed, 9 Jan 2019 20:36:14 +0000 Lustre 2.10.4 0 6 <p>do you use ldiskfs or ZFS on OST? how much free space OSTs report?</p> <p> </p> <p>We use zfs 0.7.9-1. We can replicate the problem on all of our Lustre2.10.4 filesystems that are freshly formated and up to 35% utilized.</p> <p>the symptoms look like <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> </p> <p>Alex Z, why did you think <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a>? I thought that was due to some incompatibility between 2.8 clients and 2.10 servers?</p> <p>Alex P, does this issue happen immediately on a newly mounted client, or does it take some time before it hits? On a client that is having the problem, can you check "<tt>lctl get_param osc.&#42;.cur_grant_bytes</tt>" to see if the client is running out of grant?</p> <p>It does happen on a freshly mounted client. Just to confirm we are using Lustre-2.10.4 on the clients and servers. We did a test based <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> comments and set the recordsize back to 128K, and then ran the reproducer while monitoring the cur_grant_bytes. The problem still occurred however with each hang the cur_grant_bytes  would cycle in order between 262144, 524288, 0 and then repeat.</p> Related LU-11798 Development Rank 1|i0091z: Rank (Obsolete) 9223372036854775807 Severity