https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10898 Lustre <p>Looking at logs at <a href="https://testing.hpdd.intel.com/test_sets/6f53a458-3c92-11e8-8f8a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6f53a458-3c92-11e8-8f8a-52540065bddc</a>, we see conf-sanity test_32a and test_32d fail with the following in the client test_log after trying to rmmod the ZFS module 19 times</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-49vm4: trevis-49vm4.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod zfs trevis-49vm4: rmmod: ERROR: Module zfs is in use CMD: trevis-49vm4 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh check_mem_leak trevis-49vm4: trevis-49vm4.trevis.hpdd.intel.com: executing check_mem_leak Unloading modules on trevis-49vm4: Attempt 19 CMD: trevis-49vm4 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh /usr/sbin/lustre_rmmod zfs trevis-49vm4: trevis-49vm4.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod zfs trevis-49vm4: rmmod: ERROR: Module zfs is in use CMD: trevis-49vm4 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh check_mem_leak trevis-49vm4: trevis-49vm4.trevis.hpdd.intel.com: executing check_mem_leak Unloading modules on trevis-49vm4: Given up  conf-sanity test_32a: @@@@@@ FAIL: Reloading modules   Trace dump:   = /usr/lib64/lustre/tests/test-framework.sh:5726:error_noexit()   = /usr/lib64/lustre/tests/conf-sanity.sh:2292:t32_test() </pre> </div></div> <p> </p> <p>Looking at the console log on vm4, the MDS, we see some errors prior to trying to unload the ZFS module</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 7085.963959] Lustre: DEBUG MARKER: mount -t lustre -onomgs -omgsnode=10.9.6.58@tcp t32fs-ost1/ost1 /tmp/t32/mnt/ost [ 7086.669656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [ 7087.000523] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [ 7087.335563] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.6.58@tcp [ 7087.663836] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [ 7087.993803] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.6.58@tcp [ 7088.322614] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [ 7093.175575] LustreError: 9067:0:(mgc_request.c:1576:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.6.58@tcp [ 7093.177860] Lustre: 9067:0:(mgc_request.c:1802:mgc_process_recover_nodemap_log()) MGC10.9.6.58@tcp: error processing recovery log t32fs-mdtir: rc = -2 [ 7093.181698] LustreError: 9067:0:(mgc_request.c:2132:mgc_process_log()) MGC10.9.6.58@tcp: recover log t32fs-mdtir failed, not fatal: rc= -2 [ 7093.187057] Lustre: 10864:0:(obd_mount.c:972:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [ 7093.191117] LustreError: 10864:0:(obd_config.c:1501:class_process_proc_param()) t32fs-OST0000-osc-MDT0000: unknown config parameter 'osc.max_dirty_mb=15' [ 7094.656698] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [ 7094.993137] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [ 7095.327475] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [ 7095.500065] Lustre: Failing over t32fs-MDT0000 [ 7095.809239] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [ 7102.179414] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7102.796961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-49vm4.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod zfs [ 7102.801643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-49vm4.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod zfs [ 7102.998133] Lustre: DEBUG MARKER: trevis-49vm4.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod zfs [ 7103.010598] Lustre: DEBUG MARKER: trevis-49vm4.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod zfs </pre> </div></div> <p> </p> <p>It looks like this these tests started failing during testing for <a href="https://jira.whamcloud.com/browse/LU-8066" title="Move lustre procfs handling to sysfs and debugfs." class="issue-link" data-issue-key="LU-8066">LU-8066</a> on 2018-04-10 01:04:07 UTC.</p> <p> </p> <p>Logs for these test failures are at</p> <p><a href="https://testing.hpdd.intel.com/test_sets/6f53a458-3c92-11e8-8f8a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6f53a458-3c92-11e8-8f8a-52540065bddc</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/32358bce-3cb3-11e8-960d-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/32358bce-3cb3-11e8-960d-52540065bddc</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/827118f8-3c8e-11e8-8f8a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/827118f8-3c8e-11e8-8f8a-52540065bddc</a></p> <p> </p> LU-10898

conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’

Bug Major Resolved Fixed Nathaniel Clark James Nunez zfs Tue, 10 Apr 2018 16:02:43 +0000 Sat, 15 Dec 2018 18:06:32 +0000 Thu, 17 May 2018 03:06:21 +0000 Lustre 2.12.0 Lustre 2.10.5 Lustre 2.12.0 Lustre 2.10.5 0 9 <p>Let me take a look at this. Its only ZFS? I haven't ported ZFS to sysfs/debugfs. I wonder if osd-zfs never cleaned up properly /procfs and now the move to sysfs/debugs exposed a long buried bug. procfs was more forgiving.</p> <p>Yes. So far, only ZFS.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31960" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31960</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> tests: disable failing conf-sanity test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eec072312c894657cc863b6c0973094c3ac4ac6e</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="https://review.whamcloud.com/31960/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31960/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> tests: disable failing conf-sanity test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 55bac2a58bd320ab675c724e925f6b58eafe757d</p> <p>It seems the problematic patch may cause this problem is from <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> <a href="https://review.whamcloud.com/#/c/31464/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31464/</a></p> <p>Here is the bisect try on previous patch <a href="https://jira.whamcloud.com/browse/LU-10773" title="soft lockup when remove changelog" class="issue-link" data-issue-key="LU-10773"><del>LU-10773</del></a>, conf-sanity passed</p> <p><a href="https://review.whamcloud.com/#/c/31974/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31974/</a></p> <p>Then try on <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> patch, it started failing</p> <p><a href="https://review.whamcloud.com/#/c/31968/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31968/</a></p> <p> </p> <p> </p> <p>I'm confused why this worked at all (even before my patch), since it's trying to rmmod zfs w/o exporting the pools.</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31991" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31991</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> tests: enable conf-sanity 32a/32d<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 14761b794bb3f3f6ff2485c0ffa25a22f4f288ff</p> <p>ZED is holding zfs module open, which is why it's not rmmod'ing correcly.</p> <p>Assigning this ticket to Nathaniel since he has a patch for this issue. Thanks, Nathaniel.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31991/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31991/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> tests: enable conf-sanity 32a/32d<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e8e6e3210e5eae78eefcc3f05e078a60c04dc80e</p> <p>Landed for 2.12</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32520" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32520</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> tests: enable conf-sanity 32a/32d<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 57e82001d462efe3001570a4f11210c6e44509a3</p> <p>The same failure occurred on Lustre b2_10 branch:<br/> <a href="https://testing.whamcloud.com/test_sets/2defd962-97b6-11e8-87f3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2defd962-97b6-11e8-87f3-52540065bddc</a></p> <p>The above patch needs land.</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32520/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32520/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> tests: enable conf-sanity 32a/32d<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 574e63fc86553510d87d02cd6d72785f341e48dc</p> Related LU-9551 LU-10933 Development Rank 1|hzzvl3: Rank (Obsolete) 9223372036854775807 Severity