https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16720 Lustre <p>Starting from March 30, right after landings on that day, a new assertion crash appeared in large-scale test 3a (only gets run in full testing I guess, so flew under radar)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 676976:0:(osp_precreate.c:488:osp_precreate_rollover_new_seq()) ASSERTION( fid_seq(fid) != fid_seq(last_fid) ) failed: fid [0x240000bd0:0x1:0x0], last_fid [0x240000bd0:0x3fff:0x0] LustreError: 676976:0:(osp_precreate.c:488:osp_precreate_rollover_new_seq()) LBUG Pid: 676976, comm: osp-pre-0-0 4.18.0-425.10.1.el8_lustre.x86_64 #1 SMP Thu Mar 2 00:54:22 UTC 2023 Call Trace TBD: [&lt;0&gt;] libcfs_call_trace+0x6f/0xa0 [libcfs] [&lt;0&gt;] lbug_with_loc+0x3f/0x70 [libcfs] [&lt;0&gt;] osp_precreate_thread+0x121d/0x1230 [osp] [&lt;0&gt;] kthread+0x10b/0x130 [&lt;0&gt;] ret_from_fork+0x35/0x40 </pre> </div></div> <p> </p> <p>Example crashes:</p> <p><a href="https://testing.whamcloud.com/test_sets/5173c0c5-ff80-4f5b-aec2-d6e1419cbd85" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5173c0c5-ff80-4f5b-aec2-d6e1419cbd85</a></p> <p><a href="https://testing.whamcloud.com/test_sets/68c90481-1450-4526-a659-b6d5d6b97f0a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/68c90481-1450-4526-a659-b6d5d6b97f0a</a></p> <p><a href="https://testing.whamcloud.com/test_sets/20a4a76a-e1bf-4f46-985c-b8cbed94e51b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/20a4a76a-e1bf-4f46-985c-b8cbed94e51b</a></p> <p>I suspect this is due to <a href="https://jira.whamcloud.com/browse/LU-11912" title="reduce number of OST objects created per MDS Sequence" class="issue-link" data-issue-key="LU-11912"><del>LU-11912</del></a> patch landing, the timing checks out.</p> LU-16720

large-scale test_3a osp_precreate_rollover_new_seq()) ASSERTION( fid_seq(fid) != fid_seq(last_fid) ) failed: fid [0x240000bd0:0x1:0x0], last_fid [0x240000bd0:0x3fff:0x0]

Bug Minor Open Unresolved WC Triage Oleg Drokin Fri, 7 Apr 2023 00:48:09 +0000 Wed, 12 Apr 2023 00:06:41 +0000 Lustre 2.16.0 Lustre 2.16.0 0 4 <p>Dongyang, this shouldn't be a case with replay_barrier, just creating a lot of files. It isn't exactly the same as <a href="https://jira.whamcloud.com/browse/LU-16692" title="replay-single: test_70c osp_fid_diff()) ASSERTION( fid_seq(fid1) == fid_seq(fid2) )" class="issue-link" data-issue-key="LU-16692">LU-16692</a>, since this is LASSERT that the sequences are different, while that ticket is LASSERT that they are the same. </p> <p>It seems like there is an off-by-one in the rollover? Also, it may be that we need to replace the LASSERT with error handling, since they seem too easily hit. </p> <p>This is a different issue to <a href="https://jira.whamcloud.com/browse/LU-16692" title="replay-single: test_70c osp_fid_diff()) ASSERTION( fid_seq(fid1) == fid_seq(fid2) )" class="issue-link" data-issue-key="LU-16692">LU-16692</a>. Looks like the LASSERT happened in osp_precreate_rollover_new_seq()<br/> During SEQ rollover we get a new SEQ id, and then it has to be different to the previous using SEQ saved in last_used_fid, note the object id from the last_used_fid is 0x3fff(the reduced SEQ width), which means the SEQ is used up and due to be changed.<br/> I feel like this is actually a bug found by changing the SEQ more frequently, maybe a race when changing the SEQ?</p> <p>I think I know what's going on.<br/> before large-scale, it was replay-ost-single, and it does replay_barrier on ost1, and from the logs the MDT0 osp got a new SEQ after the replay_barrier on ost1.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 9541.509199] Lustre: DEBUG MARKER: == replay-ost-single test 12b: write after OST failover to a missing object ========================================================== 03:08:10 (1680059290) [ 9545.683083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 9546.092712] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 [ 9546.795469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000401 to 0x240000bd0 </pre> </div></div> <p>the replay_barrier on ost1 dropping writes so we lost the seq range update, after that as we progress to large-scale when we need to allocate new SEQ from ofd we still got the old one because the seq range update is lost.<br/> forcing new seq on all mdts in replay-ost-single should fix this, I've updated <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50478" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50478</a></p> Related LU-11912 Development Rank 1|i03icv: Rank (Obsolete) 9223372036854775807 Severity