https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3573 Lustre <p>On ZFS an error was seen with lustre-rsync test 8 the logs are here:<br/> <a href="https://maloo.whamcloud.com/test_sets/e3492890-e901-11e2-ae91-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e3492890-e901-11e2-ae91-52540035b04c</a></p> <p>I don't really know how to read lrsync_log. </p> <p>The test error reports a very basic:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== lustre-rsync-test test 8: Replicate multiple file/directory moves == 16:00:59 (1373410859) CMD: wtm-10vm7 lctl --device lustre-MDT0000 changelog_register -n lustre-MDT0000: Registered changelog user cl13 CMD: wtm-10vm7 lctl get_param -n mdd.lustre-MDT0000.changelog_users Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl13 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 107 seconds Changelog records consumed: 1881 Only in /tmp/target/d0.lustre-rsync-test/d8/d08/d083: a3 lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4066:error_noexit() = /usr/lib64/lustre/tests/test-framework.sh:4093:error() ..... </pre> </div></div> <p>Out of the last 100 runs it reports 1 error so it could be related to the base patch or a rare error. </p> a patch pushed to autotest LU-3573

lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found.

Bug Critical Resolved Fixed Nathaniel Clark Keith Mannthey prz zfs Wed, 10 Jul 2013 22:14:14 +0000 Mon, 20 Jul 2015 21:43:55 +0000 Fri, 12 Dec 2014 15:31:56 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.7.0 Lustre 2.5.3 Lustre 2.7.0 Lustre 2.5.4 0 11 <p>Nathaniel,<br/> Could you have a look at this one?<br/> Thank you!</p> <p>It appears that this has been failing for several weeks. I searched for "lustre-rsync-test test8" in Maloo Results-&gt;Search-&gt;SubTest and found quite a few hits, dating back to 2013-03-25 after <a href="https://jira.whamcloud.com/browse/LU-2886" title="create local files using local_storage library" class="issue-link" data-issue-key="LU-2886"><del>LU-2886</del></a> was fixed (which totally broke lustre-rsync-test for interop):</p> <p><a href="https://maloo.whamcloud.com/sub_tests/1d923f9a-959d-11e2-bc9e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/1d923f9a-959d-11e2-bc9e-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/b0fef176-99d8-11e2-8bb4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/b0fef176-99d8-11e2-8bb4-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/92a06174-b1a7-11e2-802d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/92a06174-b1a7-11e2-802d-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/7f4534c4-b2b0-11e2-9b6a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/7f4534c4-b2b0-11e2-9b6a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/8a0326fc-bf47-11e2-88e0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/8a0326fc-bf47-11e2-88e0-52540035b04c</a><br/> :<br/> :</p> <p>The actual file reported changes ("a8", "a9", etc). The Maloo search also confirmed that none of the recent test_8 failures happened on ldiskfs, only ZFS.</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/395e5f0a-6ec4-11e3-a57c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/395e5f0a-6ec4-11e3-a57c-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/41/</a> (2.5.1 RC3)<br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/f858f01c-aa58-11e3-b4b1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f858f01c-aa58-11e3-b4b1-52540035b04c</a></p> <p>This bug may be fixed by <a href="https://jira.whamcloud.com/browse/LU-4838" title="Test failure sanity-lfsck test_9a: Expect &#39;completed&#39;, but got &#39;scanning-phase1&#39;" class="issue-link" data-issue-key="LU-4838"><del>LU-4838</del></a> specifically patch <a href="http://review.whamcloud.com/9838" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9838</a></p> <p>Nathaniel</p> <p>Could you please prove/disprove this theory by porting the <a href="https://jira.whamcloud.com/browse/LU-4838" title="Test failure sanity-lfsck test_9a: Expect &#39;completed&#39;, but got &#39;scanning-phase1&#39;" class="issue-link" data-issue-key="LU-4838"><del>LU-4838</del></a> patch to b2_5 and using test parameters to run the test in question?</p> <p>Thanks</p> <p>Peter </p> <p><a href="https://jira.whamcloud.com/browse/LU-4838" title="Test failure sanity-lfsck test_9a: Expect &#39;completed&#39;, but got &#39;scanning-phase1&#39;" class="issue-link" data-issue-key="LU-4838"><del>LU-4838</del></a> did not fix this issue.</p> <p>While testing patch <a href="http://review.whamcloud.com/12012" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12012</a> on Lustre b2_5 branch with FSTYPE=zfs, the same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/983a9d84-431e-11e4-bf48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/983a9d84-431e-11e4-bf48-5254006e85c2</a></p> <p>I've been running this on my local test cluster (tests 1-8 repetitively) and have not had the failure happen locally to debug the system in the failed state. This failure has only happened 4 times out of 121 test runs (zfs on master) in the last week (about 3%).</p> <p>Finally reproduced:</p> <p>File reported as missing from lustre but present in rsync'd dir is in-fact missing from lustre. Remounting client then then mdt did not cause file to re-appear, and there is no log of the file being deleted.</p> <p>This behaviour can be reproduced by just running:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">for</span> i in 1 2 3 4 5 6 7 8 9; <span class="code-keyword">do</span> mkdir $DIR/$tdir/d$i <span class="code-keyword">for</span> j in 1 2 3 4 5 6 7 8 9; <span class="code-keyword">do</span> mkdir $DIR/$tdir/d$i/d$i$j createmany -o $DIR/$tdir/d$i/d$i$j/a 10 \ &gt; /dev/<span class="code-keyword">null</span> mv $DIR/$tdir/d$i/d$i$j $DIR/$tdir/d$i/d0$i$j createmany -o $DIR/$tdir/d$i/d0$i$j/b 10 \ &gt; /dev/<span class="code-keyword">null</span> mv $DIR/$tdir/d$i/d0$i$j/a0 $DIR/$tdir/d$i/d0$i$j/c0 done mv $DIR/$tdir/d$i $DIR/$tdir/d0$i done </pre> </div></div> <p>and comparing the created files to what would be expected.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">for</span> i in 1 2 3 4 5 6 7 8 9; <span class="code-keyword">do</span> <span class="code-keyword">for</span> j in 1 2 3 4 5 6 7 8 9; <span class="code-keyword">do</span> <span class="code-keyword">for</span> k in 1 2 3 4 5 6 7 8 9; <span class="code-keyword">do</span> echo $D/d0$i/d0$i$j/a$k &gt;&gt; $F done <span class="code-keyword">for</span> k in 0 1 2 3 4 5 6 7 8 9; <span class="code-keyword">do</span> echo $D/d0$i/d0$i$j/b$k &gt;&gt; $F done echo $D/d0$i/d0$i$j/c0 &gt;&gt; $F done done find $DIR/$tdir -type f | sort | diff $F - || error <span class="code-quote">"Differences Found"</span> </pre> </div></div> <p>Nathaniel,<br/> Is the solution to fix this clear or do we need to involve anyone else?</p> <p>File actually does exist in MDT's zfs filesystem, and can be accessed directly from client, but ls does not show it in the directory. This reminds of a "recent" bug, but I'm having trouble locating it<br/> <b>Update:</b><br/> Similar bugs are <a href="https://jira.whamcloud.com/browse/LU-5475" title="readdir missing a directory" class="issue-link" data-issue-key="LU-5475"><del>LU-5475</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5254" title="readdir missing a directory" class="issue-link" data-issue-key="LU-5254"><del>LU-5254</del></a></p> <p>Status of current testing:<br/> The issue appears to be in the iteration of the directory. All the files are correctly created on the MDT and are visible (as far as I can tell via mounting MDT as ZFS), but iterating across that directory using zap_cursor_advance and zap_cursor_retrieve seem to somehow miss that one file. No errors are return in the creation of the object or adding the object to the directory, the object is just missing during iteration.</p> <p>Nathaniel, does this problem disappear if the client is unmounted/remounted, or is it a function of the directory itself? Is it possible that there are hash collisions in the name entries that are not being handled properly?</p> <p>Andreas,<br/> It is a function of the directory itself. client can be umounted then remounted w/o any change, the MDT can be mounted / exported/ imported and remounted w/o any change. It feels like it might be an issue in ZFS itself, but I don't have a smoking gun.</p> <p>Not sure if this is related but, I was trying to see if it was something in the file structures on disk vs. something under the hood for ZFS so I followed the directions for <a href="https://build.hpdd.intel.com/job/lustre-manual/565/artifact/lustre_manual.xhtml#dbdoclet.50438207_21638" class="external-link" target="_blank" rel="nofollow noopener">Backup and Restore of a File-Level backup</a>. I restored and then tried to mount the MDT resulting in this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 46380:0:(osd_oi.c:232:osd_fld_lookup()) ASSERTION( ss != ((void *)0) ) failed: LustreError: 46380:0:(osd_oi.c:232:osd_fld_lookup()) LBUG Kernel panic - not syncing: LBUG Pid: 46380, comm: mount.lustre Tainted: P --------------- 2.6.32-431.20.3.el6_lustre.g5a7c614.x86_64 #1 Call Trace: [&lt;ffffffff8152859c&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa0665eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0e9c78f&gt;] ? osd_fld_lookup+0xcf/0xd0 [osd_zfs] [&lt;ffffffffa0e9c88e&gt;] ? fid_is_on_ost+0xfe/0x310 [osd_zfs] [&lt;ffffffffa0392869&gt;] ? dbuf_rele_and_unlock+0x169/0x1e0 [zfs] [&lt;ffffffffa0e9cbac&gt;] ? osd_get_name_n_idx+0x4c/0xe80 [osd_zfs] [&lt;ffffffffa03b3f53&gt;] ? dsl_dataset_block_freeable+0x43/0x60 [zfs] [&lt;ffffffffa03a8d93&gt;] ? dmu_tx_hold_zap+0x1a3/0x200 [zfs] [&lt;ffffffffa0e9dd1a&gt;] ? osd_convert_root_to_new_seq+0x33a/0x670 [osd_zfs] [&lt;ffffffffa0e8c4b8&gt;] ? osd_mount+0xbc8/0xf60 [osd_zfs] [&lt;ffffffffa0e8ed66&gt;] ? osd_device_alloc+0x2a6/0x3b0 [osd_zfs] [&lt;ffffffffa07ebf0f&gt;] ? obd_setup+0x1bf/0x290 [obdclass] [&lt;ffffffffa07ec1e8&gt;] ? class_setup+0x208/0x870 [obdclass] [&lt;ffffffffa07f438c&gt;] ? class_process_config+0xc5c/0x1ac0 [obdclass] [&lt;ffffffffa07f94b5&gt;] ? lustre_cfg_new+0x4f5/0x6f0 [obdclass] [&lt;ffffffffa07f9808&gt;] ? do_lcfg+0x158/0x450 [obdclass] [&lt;ffffffffa07f9b94&gt;] ? lustre_start_simple+0x94/0x200 [obdclass] [&lt;ffffffffa0832df1&gt;] ? server_fill_super+0x1061/0x1720 [obdclass] [&lt;ffffffffa07ff6b8&gt;] ? lustre_fill_super+0x1d8/0x550 [obdclass] [&lt;ffffffffa07ff4e0&gt;] ? lustre_fill_super+0x0/0x550 [obdclass] [&lt;ffffffff8118c01f&gt;] ? get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa07f72b5&gt;] ? lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff8118b67b&gt;] ? vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff8118b822&gt;] ? do_kern_mount+0x52/0x130 [&lt;ffffffff8119e422&gt;] ? vfs_ioctl+0x22/0xa0 [&lt;ffffffff811ad1fb&gt;] ? do_mount+0x2fb/0x930 [&lt;ffffffff811ad8c0&gt;] ? sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I had fully and cleanly unmounted everything prior to creating the backup, I then reformatted the ZFS filesystem.</p> <p><b>update</b><br/> This happens on seq is FID_SEQ_ROOT.</p> <p>I don't think it is supported to do file-level backup and restore for ZFS. For ZFS you need to do "zfs send" and "zfs recv" or else the OI files will be broken, and there is not currently OI Scrub functionality for osd-zfs.</p> <p>If that isn't clearly documented in the Lustre User Manual's Backup and Restore section, then that is a defect in the manual that should be fixed.</p> <p>Doing update restore using <a href="https://jira.whamcloud.com/browse/LUDOC-161" title="document backup/restore process for ZFS backing filesystems" class="issue-link" data-issue-key="LUDOC-161"><del>LUDOC-161</del></a> information (zfs send/recv) instead of tar works much better. It shows no change after restore, the file is still missing form lustre client.</p> <p>One more instance on master branch: <a href="https://testing.hpdd.intel.com/test_sets/a2161baa-5bea-11e4-a35f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a2161baa-5bea-11e4-a35f-5254006e85c2</a></p> <p>The file itself isn't hidden, the name is "cloaked".<br/> In this example d074/a9 has been "lost", notice that a9 will never appear to ls:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lubuilder d074]# ls a1 a2 a3 a4 a5 a6 a7 a8 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 c0 [root@lubuilder d074]# mv a9 a9a [root@lubuilder d074]# ls a1 a2 a3 a4 a5 a6 a7 a8 a9a b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 c0 [root@lubuilder d074]# mv a9a a9 [root@lubuilder d074]# ls a1 a2 a3 a4 a5 a6 a7 a8 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 c0 [root@lubuilder d074]# mv a9 a9a [root@lubuilder d074]# ls a1 a2 a3 a4 a5 a6 a7 a8 a9a b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 c0 [root@lubuilder d074]# mv a8 a9 [root@lubuilder d074]# ls a1 a2 a3 a4 a5 a6 a7 a9a b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 c0 [root@lubuilder d074]# mv a9 a8 [root@lubuilder d074]# ls a1 a2 a3 a4 a5 a6 a7 a8 a9a b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 c0 </pre> </div></div> <p>Does ZFS have whiteout entries, or is there some problem with the hashing that prevents the directory entry from appearing? I know that "ls" will not show entries that have inode == 0, but the inode number should be independent of the filename, so that wouldn't behave in this manner. Maybe there is a ZFS "hidden" flag that is not being initialized correctly and in some cases this flag is set? Does anything appear with "strace" or with "zdb"?</p> <p>The inode is not 0, the inode is tied to the file and this issue is with the name in that directory.<br/> I can delete the file and create a new one, and it will still be hidden. It must be in how ZFS stores the name, I'm trying to debug that now.</p> <p>One more instance on Lustre b2_5 branch with FSTYPE=zfs: <a href="https://testing.hpdd.intel.com/test_sets/5310f46c-61fd-11e4-bd1f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5310f46c-61fd-11e4-bd1f-5254006e85c2</a></p> <p><a href="http://review.whamcloud.com/12582" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12582</a></p> <p>Fix landed for 2.7</p> <p>in b2_5:<br/> <a href="http://review.whamcloud.com/12649" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12649</a></p> <p>Unfortunately, version 2 of the <a href="http://review.whamcloud.com/12582" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12582</a> patch failed review-zfs conf-sanity test_32b with a real problem that is now causing serious test failures on master. The fact that it failed on the unlanded patch, when no other tests were failing before that time makes it very likely that this patch caused the problem. I've opened <a href="https://jira.whamcloud.com/browse/LU-5924" title="conf-sanity test_32b: list verification failed " class="issue-link" data-issue-key="LU-5924"><del>LU-5924</del></a> for that issue, but we might consider reverting this patch if it cannot be addressed quickly.</p> <p>Patch landed to Master. b2_5 patch ready to land and tracked outside of this ticket.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12887" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12887</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> osd-zfs: Only advance zap cursor as needed"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5f6d6b371a18a8b164f0414a5dc7356f066fd2c4</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12904" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12904</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> osd-zfs: Only advance zap cursor as needed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 90a5fa16f900ef24412bad534c848e99e091e44b</p> <p>This should be a fix for this bug without re-introducing <a href="https://jira.whamcloud.com/browse/LU-5924" title="conf-sanity test_32b: list verification failed " class="issue-link" data-issue-key="LU-5924"><del>LU-5924</del></a><br/> <a href="http://review.whamcloud.com/12904" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12904</a></p> <p>When <a href="http://review.whamcloud.com/12887" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12887</a> is landed this bug will be open</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12887/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12887/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> osd-zfs: Only advance zap cursor as needed"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 038d747f2ac5971875734701f5319047d6703844</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12904/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12904/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> osd-zfs: Only advance zap cursor as needed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0780fde4d415aa1f407ed0094f20f30e6f1b0d3a</p> <p>Patches landed to Master.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13357" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13357</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> osd-zfs: Only advance zap cursor as needed<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 78d6d3aeea75334fa7af3332c2835fcf573a8a7a</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13357/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13357/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> osd-zfs: Only advance zap cursor as needed<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: ce75d8f368a506993730da4224f4cfc18c2a0260</p> Duplicate LU-5765 Related LU-5254 LU-5924 LU-6101 LU-5475 LU-3580 Development Rank 1|hzvv07: Rank (Obsolete) 9026 Severity