https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16251 Lustre <p>As per DDN-3114:</p> <p>jobid_interpret_string() is not an atomic operation. It is possible that two processes hit the same file+offset (so the same request) but with different jobid schemes. So that jobid_interpret_string() is re-entered and generate a "combined" jobid.</p> LU-16251

Fill jobid in an atomic way

Bug Minor Resolved Fixed Feng Lei Feng Lei Wed, 19 Oct 2022 03:05:28 +0000 Thu, 5 Jan 2023 00:32:44 +0000 Tue, 8 Nov 2022 13:58:23 +0000 Lustre 2.16.0 0 5 <p>"Feng Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48915" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48915</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16251" title="Fill jobid in an atomic way" class="issue-link" data-issue-key="LU-16251"><del>LU-16251</del></a> obdclass: fill jobid in an atomic way<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d4c2887f4b0c170b918e65fff72ae4e76bd82d4a</p> <p>The bug is reproduced after inserting a msleep(1) in jobid_interpret_string(). Here is the patch to reproduce the bug:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/obdclass/jobid.c b/lustre/obdclass/jobid.c index 1b4bdd3168..b9b33044f8 100644 --- a/lustre/obdclass/jobid.c +++ b/lustre/obdclass/jobid.c @@ -32,6 +32,7 @@   */  #define DEBUG_SUBSYSTEM S_RPC +#include &lt;linux/delay.h&gt;  #include &lt;linux/user_namespace.h&gt;  #include &lt;linux/uidgid.h&gt;  #include &lt;linux/utsname.h&gt; @@ -696,6 +697,7 @@ static int jobid_interpret_string(const char *jobfmt, char *jobid,                         l = snprintf(jobid, joblen, "%u", current-&gt;pid);                         break;                 case 'u': /* user ID */ +                       msleep(1);                         l = snprintf(jobid, joblen, "%u",                                      from_kuid(&amp;init_user_ns, current_fsuid()));                         break; diff --git a/lustre/tests/sanity.sh b/lustre/tests/sanity.sh index 437d86aa96..bf47268373 100755 --- a/lustre/tests/sanity.sh +++ b/lustre/tests/sanity.sh @@ -28908,6 +28908,27 @@ test_906() {  }  run_test 906 "Simple test for io_uring I/O engine via fio" + +test_1001() { +       local file=$DIR/$tdir/$tfile + +       test_mkdir $DIR/$tdir + +       for i in \{1..2}; do +               echo "start worker $i..." +               dd if=/dev/zero of=$file bs=1M count=100 &amp; +       done + +       echo "wait for all workers done..." +       wait + +       $LCTL get_param *.*.job_stats | grep job_id | grep 'dd\.$' +       if (( $? == 0)); then +               error "found corrupted jobid" +       fi +} +run_test 1001 "test corrupted jobid" +  complete $SECONDS  [ -f $EXT2_DEV ] &amp;&amp; rm $EXT2_DEV || true  check_and_cleanup_lustre </pre> </div></div> <p>The test result:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-bash"> == sanity test 1001: test corrupted jobid ================ 09:20:20 (1666833620) start worker 1... start worker 2... wait <span class="code-object">for</span> all workers <span class="code-object">done</span>... 100+0 records <span class="code-object">in</span> 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.63228 s, 64.2 MB/s 100+0 records <span class="code-object">in</span> 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.65471 s, 63.4 MB/s - job_id:          dd.  sanity test_1001: @@@@@@ FAIL: found corrupted jobid    Trace dump:   = ./../tests/test-framework.sh:6524:error()   = sanity.sh:29016:test_1001()   = ./../tests/test-framework.sh:6860:run_one()   = ./../tests/test-framework.sh:6910:run_one_logged()   = ./../tests/test-framework.sh:6732:run_test()   = sanity.sh:29019:main() Dumping lctl log to /tmp/test_logs/1666833512/sanity.test_1001.*.1666833622.log FAIL 1001 (3s) </pre> </div></div> <p>In this case we generate a corrupted jobid "dd." without the uid because there is a delay before appending uid to jobid. It should also be possible to generate other kinds of corruption easily if we insert the delay at other points and trigger it with other kinds of setting.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48915/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48915/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16251" title="Fill jobid in an atomic way" class="issue-link" data-issue-key="LU-16251"><del>LU-16251</del></a> obdclass: fill jobid in a safe way<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9a0a89520e8b57bd63a9343fe3cdc56c61c41f6d</p> <p>Landed for 2.16</p> <p>I have just had a quick look at this patch given I am talking around this in a few weeks time.</p> <p> </p> <p>Given:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>  - job_id:          kworker/u518:6eamtrynka   snapshot_time:   1668009798   read_bytes:      { samples:           1, unit: bytes, min:  663552, max:  663552, sum:          663552 }   write_bytes:     { samples:           0, unit: bytes, min:       0, max:       0, sum:               0 }   - job_id:          kworker/u130:0   snapshot_time:   1668009796   read_bytes:      { samples:       21459, unit: bytes, min:    4096, max: 2297856, sum:       160104448 }   write_bytes:     { samples:           0, unit: bytes, min:       0, max:       0, sum:               0 }   - job_id:          thread-pool-14113912   snapshot_time:   1668009775   read_bytes:      { samples:           2, unit: bytes, min:   36864, max:  983040, sum:         1019904 }   write_bytes:     { samples:           0, unit: bytes, min:       0, max:       0, sum:               0 }   [root@lus23-oss1 ~]# lctl get_param obdfilter.lus23-OST0000.job_stats | grep job_id | grep thread-pool | wc -l 126 [root@lus23-oss1 ~]# lctl get_param obdfilter.lus23-OST0000.job_stats | grep job_id | grep kworker | wc -l 155 [root@lus23-oss1 ~]# lctl get_param obdfilter.lus23-OST0000.job_stats | grep job_id | wc -l 330 </pre> </div></div> <p> </p> <p> </p> <p>And <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48915/4/lustre/obdclass/jobid.c#478" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48915/4/lustre/obdclass/jobid.c#478</a></p> <p> </p> <p>should thread-pool be on the internal list ( I don't know the answer its an honest question ).</p> <p> </p> <p>( and for user communtity, Is this a is aserver side fix or a client fix ).</p> Related LU-16228 Development Rank 1|i0338f: Rank (Obsolete) 9223372036854775807 Severity