Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-4190] LustreError: 18166:0:(genops.c:1570:obd_exports_barrier()) ASSERTION( list_empty(&obd->obd_exports) ) failed: https://jira.whamcloud.com/browse/LU-4190 Lustre <p>2 servers work normally at active-active status<br/> 1、Mount Lustre FS on the client and write and read data;<br/> 2、Umount the MDT on the Failnode; <br/> 3、Read data on the client from Lustre FS,successfully;<br/> 4、Mount the MDT on the Failnode;<br/> 5、Umount the MDT on the MGSnode;<br/> 6、Read data on the client from Lustre FS,failed;<br/> 7、Mount the MDT on the MGSnode,then the MGSnode crash,print information as follow:</p> <p>LustreError: 18166:0:(genops.c:320:class_newdev()) Device MGC192.168.22.50@tcp already exists at 2, won't add <br/> LustreError: 18166:0:(obd_config.c:374:class_attach()) Cannot create device MGC192.168.22.50@tcp of type mgc : -17 <br/> LustreError: 18166:0:(obd_mount.c:196:lustre_start_simple()) MGC192.168.22.50@tcp attach error -17 <br/> LustreError: 18166:0:(obd_mount_server.c:844:lustre_disconnect_lwp()) lustre-MDT0000-lwp-MDT0000: Can't end config log lustre-client. <br/> LustreError: 18166:0:(obd_mount_server.c:1426:server_put_super()) lustre-MDT0000: failed to disconnect lwp. (rc=-2) <br/> LustreError: 18166:0:(obd_mount_server.c:1456:server_put_super()) no obd lustre-MDT0000 <br/> LustreError: 18166:0:(obd_mount_server.c:135:server_deregister_mount()) lustre-MDT0000 not registered <br/> LustreError: 18166:0:(genops.c:1570:obd_exports_barrier()) ASSERTION( list_empty(&amp;obd-&gt;obd_exports) ) failed: <br/> LustreError: 18166:0:(genops.c:1570:obd_exports_barrier()) LBUG <br/> Pid: 18166, comm: mount.lustre </p> <p>Call Trace: <br/> <span class="error">&#91;&lt;ffffffffa070a8a5&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa070aeb7&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0813d91&gt;&#93;</span> obd_exports_barrier+0x181/0x190 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0f1e886&gt;&#93;</span> mgs_device_fini+0xf6/0x5c0 <span class="error">&#91;mgs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa083e837&gt;&#93;</span> class_cleanup+0x817/0xe00 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0817e2c&gt;&#93;</span> ? class_name2dev+0x7c/0xe0 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0842e9b&gt;&#93;</span> class_process_config+0x1b6b/0x2f60 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa070bb90&gt;&#93;</span> ? cfs_alloc+0x30/0x60 <span class="error">&#91;libcfs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0844723&gt;&#93;</span> class_manual_cleanup+0x493/0xe80 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffff8147a1fe&gt;&#93;</span> ? _read_unlock+0xe/0x10 <br/> <span class="error">&#91;&lt;ffffffffa0817e2c&gt;&#93;</span> ? class_name2dev+0x7c/0xe0 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa087fb9d&gt;&#93;</span> server_put_super+0x42d/0x2580 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0882440&gt;&#93;</span> server_fill_super+0x750/0x1580 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa084fc98&gt;&#93;</span> lustre_fill_super+0x1d8/0x530 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa084fac0&gt;&#93;</span> ? lustre_fill_super+0x0/0x530 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffff8114d21f&gt;&#93;</span> get_sb_nodev+0x5f/0xa0 <br/> <span class="error">&#91;&lt;ffffffffa08473f5&gt;&#93;</span> lustre_get_sb+0x25/0x30 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffff8114c74b&gt;&#93;</span> vfs_kern_mount+0x7b/0x1b0 <br/> <span class="error">&#91;&lt;ffffffff8114c8f2&gt;&#93;</span> do_kern_mount+0x52/0x130 <br/> <span class="error">&#91;&lt;ffffffff81168912&gt;&#93;</span> do_mount+0x2d2/0x8c0 <br/> <span class="error">&#91;&lt;ffffffff81168f90&gt;&#93;</span> sys_mount+0x90/0xe0 <br/> <span class="error">&#91;&lt;ffffffff81002f5b&gt;&#93;</span> system_call_fastpath+0x16/0x1b </p> <p>Message fromKernel panic - not syncing: LBUG <br/> Pid: 18166, comm: mount.lustre Tainted: GF --------------- 2.6.32-358.6.2.l2.08 #2 <br/> Call Trace: <br/> <span class="error">&#91;&lt;ffffffff81476fa7&gt;&#93;</span> ? panic+0xa1/0x163 <br/> <span class="error">&#91;&lt;ffffffffa070af0b&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0813d91&gt;&#93;</span> ? obd_exports_barrier+0x181/0x190 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0f1e886&gt;&#93;</span> ? mgs_device_fini+0xf6/0x5c0 <span class="error">&#91;mgs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa083e837&gt;&#93;</span> ? class_cleanup+0x817/0xe00 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0817e2c&gt;&#93;</span> ? class_name2dev+0x7c/0xe0 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0842e9b&gt;&#93;</span> ? class_process_config+0x1b6b/0x2f60 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa070bb90&gt;&#93;</span> ? cfs_alloc+0x30/0x60 <span class="error">&#91;libcfs&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0844723&gt;&#93;</span> ? class_manual_cleanup+0x493/0xe80 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffff8147a1fe&gt;&#93;</span> ? _read_unlock+0xe/0x10 <br/> <span class="error">&#91;&lt;ffffffffa0817e2c&gt;&#93;</span> ? class_name2dev+0x7c/0xe0 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa087fb9d&gt;&#93;</span> ? server_put_super+0x42d/0x2580 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa0882440&gt;&#93;</span> ? server_fill_super+0x750/0x1580 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa084fc98&gt;&#93;</span> ? lustre_fill_super+0x1d8/0x530 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffffa084fac0&gt;&#93;</span> ? lustre_fill_super+0x0/0x530 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffff8114d21f&gt;&#93;</span> ? get_sb_nodev+0x5f/0xa0 <br/> <span class="error">&#91;&lt;ffffffffa08473f5&gt;&#93;</span> ? lustre_get_sb+0x25/0x30 <span class="error">&#91;obdclass&#93;</span> <br/> <span class="error">&#91;&lt;ffffffff8114c74b&gt;&#93;</span> ? vfs_kern_mount+0x7b/0x1b0 <br/> <span class="error">&#91;&lt;ffffffff8114c8f2&gt;&#93;</span> ? do_kern_mount+0x52/0x130 <br/> <span class="error">&#91;&lt;ffffffff81168912&gt;&#93;</span> ? do_mount+0x2d2/0x8c0 <br/> <span class="error">&#91;&lt;ffffffff81168f90&gt;&#93;</span> ? sys_mount+0x90/0xe0 <br/> <span class="error">&#91;&lt;ffffffff81002f5b&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b <br/> *******<b>show para for nt_memcpy16</b>******* <br/> src: ffff8802e118fc40, dst: ffffc901125a8d70, len: 80 <br/> *******<b>show para for panic done</b>******* </p> Lustre2.4.0，with 2 servers and 1 client, kernel version:2.6.32-358.6.2.l2.08<br/> MGSnode: mgs, 1 mdt and 4osts<br/> Failnode: 1mdt and 4 osts LU-4190 LustreError: 18166:0:(genops.c:1570:obd_exports_barrier()) ASSERTION( list_empty(&obd->obd_exports) ) failed: Bug Major Closed Cannot Reproduce Mikhail Pershin yueyuling Wed, 30 Oct 2013 08:29:16 +0000 Sun, 15 Dec 2019 08:27:19 +0000 Sun, 15 Dec 2019 08:27:18 +0000 Lustre 2.4.0 0 10 <blockquote> <p>2 servers work normally at active-active status<br/> 1、Mount Lustre FS on the client and write and read data;<br/> 2、Umount the MDT on the Failnode;<br/> 3、Read data on the client from Lustre FS,successfully;<br/> 4、Mount the MDT on the Failnode;<br/> 5、Umount the MDT on the MGSnode;<br/> 6、Read data on the client from Lustre FS,failed;</p></blockquote> <p>Are you mounting the same MDT device (lustre-MDT0000) on both nodes? That is bad and will lead to filesystem corruption. You should only mount it on one MDS node at a time. I suggest you enable "MMP" on your devices with "tune2fs -O mmp /dev/&lt;mdt_or_ost_device&gt;" (this happens automatically if you format the filesystem with --failnode).</p> <p>Thank you for your response! But I didn't mount the same MDT device on both nodes. There are two MDS devices in my Lustre FS. I mount one MDT device on each node.<br/> So, I modify my descriptions as follow:<br/> 2 servers work normally at active-active status<br/> 1、Mount MGS, MDT0000 and 4 OSTs at MGSnode, mount MDT0001 and other 4 OSTs at Failnode, and Mount Lustre FS on the client and write and read data;<br/> 2、Umount the MDT0001 at the Failnode;<br/> 3、Read data on the client from Lustre FS,successfully;<br/> 4、Mount the MDT0001 on the Failnode;<br/> 5、Umount the MDT0000 on the MGSnode;<br/> 6、Read data on the client from Lustre FS,failed;<br/> 7、Mount the MDT0000 on the MGSnode, the MGSnode crash and output as follow:</p> <p>I tried this test on current master. </p> <p>MDT1</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-2 ~]# mkfs.lustre --reformat --mgs --mdt --index=0 --fsname lustre --failnode=10.10.4.3@tcp /dev/disk/by-id/scsi-1IET_00040001 </pre> </div></div> <p>MDT2</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-3 ~]# mkfs.lustre --reformat --mgsnode=10.10.4.2@tcp --mgsnode=10.10.4.3@tcp --mdt --index=1 --fsname lustre --failnode=10.10.4.2@tcp /dev/disk/by-id/scsi-1IET_00020001 </pre> </div></div> <p>But unfortunately when it failed when I tries to mount mdt2</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-3 ~]# mount -t lustre /dev/disk/by-id/scsi-1IET_00020001 /mnt/mds2/ mount.lustre: mount /dev/sdj at /mnt/mds2 failed: No such file or directory Is the MGS specification correct? Is the filesystem name correct? If upgrading, is the copied client log valid? (see upgrade docs) [root@client-3 ~]# </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... LDISKFS-fs (sdj): mounted filesystem with ordered data mode. quota=on. Opts: Lustre: srv-lustre-MDT0001: No data found on store. Initialize space Lustre: lustre-MDT0001: new disk, initializing LustreError: 11-0: lustre-MDT0000-osp-MDT0001: Communicating with 10.10.4.2@tcp, operation mds_connect failed with -11. LustreError: 13a-8: Failed to get MGS log params and no local copy. LustreError: 2354:0:(obd_mount_server.c:699:lustre_lwp_add_conn()) lustre-MDT0001: can't find lwp device. LustreError: 15c-8: MGC10.10.4.2@tcp: The configuration from log 'lustre-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 2242:0:(obd_mount_server.c:1321:server_start_targets()) lustre-MDT0001: failed to start LWP: -2 LustreError: 2242:0:(obd_mount_server.c:1776:server_fill_super()) Unable to start targets: -2 Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 2242:0:(obd_mount.c:1338:lustre_fill_super()) Unable to mount (-2) </pre> </div></div> <p>config log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-2 ~]# llog_reader /mnt/mds1/CONFIGS/lustre-client Header size : 8192 Time : Fri Apr 4 20:36:36 2014 Number of records: 30 Target uuid : config_uuid ----------------------- #01 (224)marker 4 (flags=0x01, v2.5.57.0) lustre-clilov 'lov setup' Fri Apr 4 20:36:36 2014- #02 (120)attach 0:lustre-clilov 1:lov 2:lustre-clilov_UUID #03 (168)lov_setup 0:lustre-clilov 1:(struct lov_desc) uuid=lustre-clilov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1 #04 (224)marker 4 (flags=0x02, v2.5.57.0) lustre-clilov 'lov setup' Fri Apr 4 20:36:36 2014- #05 (224)marker 5 (flags=0x01, v2.5.57.0) lustre-clilmv 'lmv setup' Fri Apr 4 20:36:36 2014- #06 (120)attach 0:lustre-clilmv 1:lmv 2:lustre-clilmv_UUID #07 (168)lov_setup 0:lustre-clilmv 1:(struct lov_desc) uuid=lustre-clilmv_UUID stripe:cnt=0 size=0 offset=0 pattern=0 #08 (224)marker 5 (flags=0x02, v2.5.57.0) lustre-clilmv 'lmv setup' Fri Apr 4 20:36:36 2014- #09 (224)marker 6 (flags=0x01, v2.5.57.0) lustre-MDT0000 'add mdc' Fri Apr 4 20:36:36 2014- #10 (080)add_uuid nid=10.10.4.2@tcp(0x200000a0a0402) 0: 1:10.10.4.2@tcp #11 (128)attach 0:lustre-MDT0000-mdc 1:mdc 2:lustre-clilmv_UUID #12 (136)setup 0:lustre-MDT0000-mdc 1:lustre-MDT0000_UUID 2:10.10.4.2@tcp #13 (080)add_uuid nid=10.10.4.3@tcp(0x200000a0a0403) 0: 1:10.10.4.3@tcp #14 (104)add_conn 0:lustre-MDT0000-mdc 1:10.10.4.3@tcp #15 (160)modify_mdc_tgts add 0:lustre-clilmv 1:lustre-MDT0000_UUID 2:0 3:1 4:lustre-MDT0000-mdc_UUID #16 (224)marker 6 (flags=0x02, v2.5.57.0) lustre-MDT0000 'add mdc' Fri Apr 4 20:36:36 2014- #17 (224)marker 7 (flags=0x01, v2.5.57.0) lustre-client 'mount opts' Fri Apr 4 20:36:36 2014- #18 (120)mount_option 0: 1:lustre-client 2:lustre-clilov 3:lustre-clilmv #19 (224)marker 7 (flags=0x02, v2.5.57.0) lustre-client 'mount opts' Fri Apr 4 20:36:36 2014- #20 (224)marker 11 (flags=0x01, v2.5.57.0) lustre-MDT0001 'add mdc' Fri Apr 4 20:50:05 2014- #21 (080)add_uuid nid=10.10.4.3@tcp(0x200000a0a0403) 0: 1:10.10.4.3@tcp #22 (128)attach 0:lustre-MDT0001-mdc 1:mdc 2:lustre-clilmv_UUID #23 (136)setup 0:lustre-MDT0001-mdc 1:lustre-MDT0001_UUID 2:10.10.4.3@tcp #24 (080)add_uuid nid=10.10.4.2@tcp(0x200000a0a0402) 0: 1:10.10.4.2@tcp #25 (104)add_conn 0:lustre-MDT0001-mdc 1:10.10.4.2@tcp #26 (160)modify_mdc_tgts add 0:lustre-clilmv 1:lustre-MDT0001_UUID 2:1 3:1 4:lustre-MDT0001-mdc_UUID #27 (224)marker 11 (flags=0x02, v2.5.57.0) lustre-MDT0001 'add mdc' Fri Apr 4 20:50:05 2014- #28 (224)marker 12 (flags=0x01, v2.5.57.0) lustre-client 'mount opts' Fri Apr 4 20:50:05 2014- #29 (120)mount_option 0: 1:lustre-client 2:lustre-clilov 3:lustre-clilmv #30 (224)marker 12 (flags=0x02, v2.5.57.0) lustre-client 'mount opts' Fri Apr 4 20:50:05 2014- </pre> </div></div> <p>It might be related with the change <a href="http://review.whamcloud.com/7666" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7666</a> Fan Yong, could you please comment here. Thanks!</p> <p>The original issue happened on Lustre-2.4, but the patch <a href="http://review.whamcloud.com/#/c/7666/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7666/</a> only has been applied to Lustre-2.6, then even though such patch has some issues, it should not affect Lustre-2.4, right?</p> <p>Oh, I am not asking the original issue shown in this ticket, but the failure I met in my test, which stops me continue the test on 2.6. Hmm, I will create a new ticket then. </p> <p>In addition, I've created the MGS, MDT0000 and MDT0001 separately with different device. So, the MGS and MDT0000 are in different devices.</p> <p>Mike, could you please try configuring a test system as described here to see if a similar problem still exists in master? This seems similar to the failure in <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a>.</p> <p>I've tried to repeat those steps after <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a> fix and everything works, please clarify how did you write/read data from client? I'd try to repeat all steps as close as possible.</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a></p> <p>Jodi, this is not duplicate of <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a>, this was just blocked by <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a>. Reported bug happened in Lustre 2.4 and <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a> doesn't even exist there. It looks like this issue doesn't exist in current master and is not blocker for 2.6, but it exists in 2.4 as reported.</p> <p>This is not a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a>. <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a> blocks the ability to reproduce this issue, but does not resolve it. As such, I am reopening and giving it a lower priority (since it cannot be reproduced thanks to <a href="https://jira.whamcloud.com/browse/LU-4916" title="mount failure when adding failover node to mkfs.lustre" class="issue-link" data-issue-key="LU-4916"><del>LU-4916</del></a>).</p> <p>The steps of write/read data :<br/> 1、Mount Lustre FS on the client and write and read data：<br/> Repeat 100 times as follow： <br/> a、Create a new directory;<br/> b、Copy data from client to Lustre FS, 5 files per directory, each file is 1.2GB;<br/> c、Use MD5 to read the files in the directory and record the md5 value;<br/> 3、Read data on the client from Lustre FS,successfully;<br/> Use MD5 to read the files which is writen at step 1.</p> <p>Outdated</p> Related LU-4916 Development Rank 1|hzw7bj: Rank (Obsolete) 11330 Severity