https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10740 Lustre <p>2.7 (IEEL) based server, 2.10.58 client</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == replay-single test 2d: setdirstripe replay ======================================================== 00:25:51 (1519604751) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 344152 2752 317200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 344152 2492 317460 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 664400 17712 597688 3% /mnt/lustre[OST:0] lustre-OST0001_UUID 664400 17712 597688 3% /mnt/lustre[OST:1] filesystem_summary: 1328800 35424 1195376 3% /mnt/lustre Failing mds1 on fre1209 Stopping /mnt/lustre-mds1 (opts:) on fre1209 pdsh@fre1211: fre1209: ssh exited with exit code 1 reboot facets: mds1 Failover mds1 to fre1209 00:26:02 (1519604762) waiting <span class="code-keyword">for</span> fre1209 network 900 secs ... 00:26:02 (1519604762) network <span class="code-keyword">interface</span> is UP mount facets: mds1 Starting mds1: -o rw,user_xattr /dev/vdb /mnt/lustre-mds1 pdsh@fre1211: fre1209: ssh exited with exit code 1 pdsh@fre1211: fre1209: ssh exited with exit code 1 Started lustre-MDT0000 fre1212: fre1212: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid fre1211: fre1211: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid fre1212: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec fre1211: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec Can't lstat /mnt/lustre/d2d.replay-single: No such file or directory replay-single test_2d: @@@@@@ FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed </pre> </div></div> <p>100% reproducible with review-dne-zfs-part-4</p> LU-10740

replay-single test_2d: FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed

Bug Major Resolved Duplicate Lai Siyao Elena Gryaznova dne zfs Wed, 28 Feb 2018 16:06:36 +0000 Tue, 26 Feb 2019 07:44:11 +0000 Tue, 26 Feb 2019 07:44:11 +0000 Lustre 2.11.0 Lustre 2.12.0 0 7 <p>also hit on master branch, zfs DNE</p> <p><a href="https://testing.hpdd.intel.com/test_sets/d05367b0-2799-11e8-9e0e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d05367b0-2799-11e8-9e0e-52540065bddc</a></p> <p>mds dmesg</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [64823.580533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 2d: setdirstripe replay ======================================================== 17:06:03 \(1520960763\) [64823.808722] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ======================================================== 17:06:03 (1520960763) [64824.017795] Lustre: DEBUG MARKER: sync; sync; sync [64825.224138] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [64825.565253] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [64826.042689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [64826.212687] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [64826.695274] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true [64827.035062] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [64827.574756] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' || true [64827.930760] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 &gt;/dev/null 2&gt;&amp;1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 [64838.339263] Lustre: DEBUG MARKER: hostname [64838.749896] Lustre: DEBUG MARKER: lsmod | grep zfs &gt;&amp;/dev/null || modprobe zfs; zpool list -H lustre-mdt1 &gt;/dev/null 2&gt;&amp;1 || zpool import -f -o cachefile=none -o failmode=panic -d /dev/lvm-Role_MDS lustre-mdt1 [64839.417169] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [64839.751101] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [64841.981159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [64842.324523] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [64842.971945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm8.onyx.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [64842.974147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm8.onyx.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [64843.192600] Lustre: DEBUG MARKER: onyx-43vm8.onyx.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [64843.192672] Lustre: DEBUG MARKER: onyx-43vm8.onyx.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [64843.395032] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [64843.733809] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2&gt;/dev/null | grep -E ':[a-zA-Z]\{3}[0-9]\{4}' [64844.079675] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2&gt;/dev/null | grep -E ':[a-zA-Z]\{3}[0-9]\{4}' [64844.425889] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2&gt;/dev/null [64845.202422] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [64845.389052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm5.onyx.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [64845.389056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm6.onyx.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [64845.655584] Lustre: DEBUG MARKER: onyx-43vm5.onyx.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [64845.687723] Lustre: DEBUG MARKER: onyx-43vm6.onyx.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [64846.599842] LustreError: 29449:0:(llog.c:1012:llog_write_rec()) lustre-MDT0000-osd: loghandle ffff88004c471e00 with no header [64846.601307] LustreError: 29449:0:(llog_cat.c:500:llog_cat_add_rec()) llog_write_rec -71: lh=ffff88004c471e00 [64846.602541] LustreError: 29449:0:(update_trans.c:994:top_trans_stop()) lustre-MDT0000-mdtlov: write updates failed: rc = -71 [64846.603937] LustreError: 29449:0:(tgt_lastrcvd.c:1226:tgt_last_rcvd_update()) lustre-MDT0000: replay transno 38654705669 failed: rc = -71 [64846.613367] Lustre: lustre-MDT0000: disconnecting 1 stale clients [64846.614170] LustreError: 29449:0:(tgt_grant.c:248:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [64847.016802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [64847.244576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [64854.015289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [64854.220999] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [64855.647689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-single test_2d: @@@@@@ FAIL: checkstat -v \/mnt\/lustre\/d2d.replay-single check failed [64855.876070] Lustre: DEBUG MARKER: replay-single test_2d: @@@@@@ FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed </pre> </div></div> <p> </p> <p>The error being returned is <tt>-71 = -EREMOTE</tt>, which suggests that there is some problem sending the RPC to the wrong MDT. However, that may also be some kind of fallout from "<tt>loghandle XXX with no header</tt>", which implies that the llogfile is zero length and wasn't written correctly? It's hard to see why the <tt>-EREMOTE</tt> problem could relate to only DNE+ZFS, while the second one seems more plausible.</p> <p>Lai, could you please take a look? James reports that this is failing 100% of the time with review-dne-zfs-part-4, which is currently an optional test run, but we want to make it a required test.</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32976" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32976</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10740" title="replay-single test_2d: FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed " class="issue-link" data-issue-key="LU-10740"><del>LU-10740</del></a> test: test replay-single 2d with full debug<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 958da17e9abf0504a802963ba8a7fa0a9afc075b</p> <p>The failed call trace is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> llog_add() -&gt; llog_cat_add_rec() -&gt; llog_cat_new_log() -&gt; llog_osd_create() -&gt; osd_create() -&gt; osd_zap_add() return -EEXIST </pre> </div></div> <p>but <tt>loghandle-&gt;lgh_hdr</tt> is NULL, which caused <tt>llog_write_rec() return -EPROTO</tt> (-71). But I don't understand why it fails on ZFS only.</p> <p>I found something strange in the log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>40000000:00000040:1.0:1535051095.957588:0:12953:0:(fid_handler.c:348:__seq_server_alloc_meta()) srv-lustre-MDT0000: Allocated meta-sequence [0x0000000200001b70-0x0000000200001b71]:0:mdt 40000000:00000040:1.0:1535051095.957598:0:12953:0:(fid_request.c:400:seq_client_alloc_fid()) cli-lustre-MDT0000: Allocated FID [0x200001b70:0x1:0x0] 00000004:00000040:1.0:1535051095.957601:0:12953:0:(lod_object.c:1984:lod_prep_md_striped_create()) Get idx 0, for stripe 0 [0x200001b70:0x1:0x0] 00000004:00000040:1.0:1535051095.957607:0:12953:0:(mdt_handler.c:5324:mdt_object_init()) object init, fid = [0x200001b70:0x1:0x0] 00000001:00000002:1.0:1535051095.958937:0:12953:0:(linkea.c:173:linkea_add_buf()) New link_ea name '[0x2000013a0:0xb:0x0]:[0x200001b70:0x1:0x0]:0' is added 00000020:00000040:1.0:1535051095.960470:0:12953:0:(update_records.c:240:update_records_update_pack()) 4th [0x200001b70:0x1:0x0] create param_count = 7 00000040:00000001:0.0:1535051123.894990:0:16387:0:(llog_osd.c:1225:llog_osd_open()) Process entered 00000040:00000001:0.0:1535051123.894991:0:16387:0:(obd_class.h:911:obd_fid_alloc()) Process entered 40000000:00000040:0.0:1535051123.941029:0:16387:0:(fid_request.c:400:seq_client_alloc_fid()) cli-lustre-MDT0000: Allocated FID [0x200001b70:0x1:0x0] </pre> </div></div> <p>The scenario is as below:<br/> 1. setdirstripe allocated <span class="error">&#91;0x200001b70:0x1:0x0&#93;</span> for its first stripe on MDT0, and it also allocated a new meta sequence.<br/> 2. replay barrier<br/> 3. replay setdirstripe, MDT0 got update logs from MDT1, and it needed to create an update log, now it allocated fid for this update log, but it allocated the same meta sequence as step 1, and allocated the same fid as the first stripe.<br/> 4. llog_osd_create() <del>&gt; osd_create() -&gt; osd_zap_add() found the fid in OI, so it skipped llog_init_handle(), which cause loghandle</del>&gt;lgh_hdr not set, and finally llog_write_rec() return -EPROTO.</p> <p>I'm not sure whether it's related with ZFS, because seq_store_update() has updated FLDB with th_sync set.</p> <p>Is it possible the MDT is running with <tt>osd_object_sync_delay_us=0</tt>? This was being proposed in a recent patch, but I didn't think it was landed yet, and was initially limited to a single test. </p> <p>It doesn't look to be related with osd_object_sync_delay_us. I added some debug log, the result shows th_sync doesn't work on ZFS, the debug log shows seq file was sync written (osd_trans_commit_cb() was called before osd_trans_stop()), but after replay barrier, server read stale seq file on start up. I also tweaked the code to do create in sync mode, the result also shows file not exist after restart.</p> <p>Is there a simple way to verify this?</p> <p>One problem we have with ZFS TXG sync is that this doesn't force a commit, only block the thread until the TXG commit happens. That should be equivalent from the point of view of the client, but it is possible the test doesn't expect this?</p> <p>Until we have ZIL support, it would be nice to have a way to force TXG commit, especially with flash devices that have no seek latency. </p> <p>FID seq file is updated in sync mode, so upon MDS fail, it won't allocate duplicate FID.</p> <p>replay-single.sh 2d failed because it happens to allocate a new super sequence before fail, If it's run separately it can pass.</p> <p>Andreas, I added a simple debugging data - print time it takes to commit TXG upon enforced commit and initiated that many times during sanity.sh:</p> <p>sync took 8276 usec<br/> sync took 26489 usec<br/> sync took 15471 usec<br/> sync took 8526 usec<br/> sync took 36187 usec<br/> sync took 16634 usec<br/> sync took 283 usec<br/> sync took 8142 usec<br/> sync took 7335 usec<br/> sync took 609 usec<br/> sync took 641 usec<br/> sync took 660 usec<br/> sync took 9278 usec<br/> sync took 396 usec<br/> sync took 17179 usec<br/> sync took 401 usec<br/> sync took 455 usec<br/> sync took 403 usec<br/> sync took 532 usec<br/> sync took 424 usec<br/> sync took 9057 usec<br/> sync took 444 usec<br/> sync took 11116 usec<br/> sync took 16007 usec<br/> sync took 16611 usec<br/> sync took 680 usec<br/> sync took 240 usec<br/> sync took 289 usec<br/> sync took 256 usec<br/> sync took 289 usec<br/> sync took 311 usec<br/> sync took 7353 usec<br/> sync took 285 usec<br/> sync took 9369 usec<br/> sync took 13867 usec<br/> sync took 275 usec<br/> sync took 6440 usec<br/> sync took 327 usec<br/> sync took 14827 usec<br/> sync took 15640 usec<br/> sync took 17490 usec<br/> sync took 113 usec<br/> sync took 524 usec<br/> sync took 268 usec<br/> sync took 375 usec<br/> sync took 15034 usec<br/> sync took 297 usec<br/> sync took 247 usec<br/> sync took 401 usec<br/> sync took 9492 usec</p> <p>Alex, what are you using to force a TXG commit in your tests, and are you sure that the commit is actually making it to disk? It seems very fast in some cases, 0.1ms for a commit...</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> osd_trans_stop(<span class="code-keyword">const</span> struct lu_env *env, struct dt_device *dt, struct thandle *th) { <span class="code-keyword">if</span> (sync) { <span class="code-keyword">if</span> (osd_txg_sync_delay_us &lt; 0) txg_wait_synced(dmu_objset_pool(osd-&gt;od_os), txg); <span class="code-keyword">else</span> udelay(osd_txg_sync_delay_us); } </pre> </div></div> <p>My understanding is that <tt>txg_wait_synced()</tt> will just wait for the TXG commit to happen, it can't necessarily force a TXG commit like <tt>jbd2_journal_stop()</tt> can. While we don't want every thread to be forcing a TXT commit, IMHO the jbd2 code has it right - wait a short time for more threads to join the TXG and commit, then force the commit. The length of time to wait should be based on the current storage commit time, as <tt>jbd2_journal_stop()</tt> does, so that it is fast on flash and slower on disks to be most efficient.</p> <p>the patch I used:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @@ -276,6 +276,17 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> zfs_osd_mntdev_seq_show(struct seq_file *m, void *data) } LPROC_SEQ_FOPS_RO(zfs_osd_mntdev); +struct osd_sync_cb_data { + ktime_t time; +}; + +<span class="code-keyword">static</span> void osd_sync_cb(void *cb_data, <span class="code-object">int</span> error) +{ + struct osd_sync_cb_data *cb = cb_data; + printk(<span class="code-quote">"sync took %llu usec\n"</span>, ktime_us_delta(ktime_get(), cb-&gt;time)); + OBD_FREE_PTR(cb); +} + <span class="code-keyword">static</span> ssize_t lprocfs_osd_force_sync_seq_write(struct file *file, <span class="code-keyword">const</span> <span class="code-object">char</span> __user *buffer, size_t count, loff_t *off) @@ -288,6 +299,19 @@ lprocfs_osd_force_sync_seq_write(struct file *file, <span class="code-keyword">const</span> <span class="code-object">char</span> __user *buffer, rc = lu_env_init(&amp;env, LCT_LOCAL); <span class="code-keyword">if</span> (rc) <span class="code-keyword">return</span> rc; + { + struct osd_device *osd = osd_dt_dev(dt); + struct osd_sync_cb_data *cb; + dmu_tx_t *tx; + + OBD_ALLOC_PTR(cb); + cb-&gt;time = ktime_get(); + tx = dmu_tx_create(osd-&gt;od_os); + dmu_tx_assign(tx, TXG_WAIT); + dmu_tx_callback_register(tx, osd_sync_cb, cb); + dmu_tx_commit(tx); + } </pre> </div></div> <p>i.e. I registered a commit callback and just calc/printk when it's called.</p> <p>txg_wait_synced() asks sync thread to initiate txg commit:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (tx-&gt;tx_sync_txg_waiting &lt; txg) tx-&gt;tx_sync_txg_waiting = txg; ... cv_broadcast(&amp;tx-&gt;tx_sync_more_cv); </pre> </div></div> <p> </p> <p>Alex, is your testing running with an HDD or SSD or ramdisk? On an HDD I can't see how a TXG commit could ever complete in less than 4x seek time (just for the überblock writes, 2x each at the start and end of the device), which is about 32-40 ms.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33106" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33106</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10740" title="replay-single test_2d: FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed " class="issue-link" data-issue-key="LU-10740"><del>LU-10740</del></a> tests: disable tests for replay-dne-zfs-part-4<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a7675ca33ec579525b4d39ad114b54ca3b462a62</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33106/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33106/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10740" title="replay-single test_2d: FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed " class="issue-link" data-issue-key="LU-10740"><del>LU-10740</del></a> tests: disable tests for replay-dne-zfs-part-4<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 16e92e2d01a71c2a97cae89c70c58abf409c12c0</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>, sorry, missed your question.. I'm using ramdisk.</p> <p>I think this is a dup of <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a></p> Duplicate LU-10143 LU-9157 Related LU-11336 LU-11366 Development Rank 1|hzzthz: Rank (Obsolete) 9223372036854775807 Severity