Hi Daire
Good to see you still in the Lustre world!
Bruno
Could you please look into this one?
Thanks
Peter
Hello Daire,
Was a crash-dump written during one of the LBUGs ?? Also, has the concerned file-system been formatted in 2.4 or with earlier release (1.8?) ?
Bruno,
I have a kdump vmcore - it's too big to attach. I've put it in my dropbox account:
https://dl.dropboxusercontent.com/u/24821368/vmcore.tgz
The filesystem was formatted using Lustre v2.3.
Peter,
Yes it's been a while! We have actually been using Lustre for a couple of years now at my present company. We are currently building a new filesystem and it seems like about the right time to try out v2. We just need to help you debug it some more like we did at Framestore with v1.8 
Thank's for the crash-dump Daire already! But in order to be able to analyze it I also need the Lustre-modules RPM and the kernel-debuginfo RPM, can you also provide them ??
Bruno,
We are using the whamcloud binary rpms so you can find them on the download site.
We've also seen this during our recent 2.4-ldiskfs testing at LLNL:
LustreError: 7128:0:(linkea.c:169:linkea_links_find()) ASSERTION( ldata->ld_leh != ((void *)0) ) failed: LustreError: 7128:0:(linkea.c:169:linkea_links_find()) LBUG
crash> bt PID: 7128 TASK: ffff8805cec96ae0 CPU: 10 COMMAND: "mdt02_034" #0 [ffff880605c6b878] machine_kexec at ffffffff81035bfb #1 [ffff880605c6b8d8] crash_kexec at ffffffff810c0932 #2 [ffff880605c6b9a8] panic at ffffffff8150d943 #3 [ffff880605c6ba28] lbug_with_loc at ffffffffa0646f4b [libcfs] #4 [ffff880605c6ba48] linkea_links_find at ffffffffa0838986 [obdclass] #5 [ffff880605c6bab8] mdd_linkea_prepare at ffffffffa0d9b645 [mdd] #6 [ffff880605c6bb08] mdd_links_rename at ffffffffa0d9bb01 [mdd] #7 [ffff880605c6bb88] mdd_unlink at ffffffffa0d9fae6 [mdd] #8 [ffff880605c6bc48] mdo_unlink at ffffffffa1009b98 [mdt] #9 [ffff880605c6bc58] mdt_reint_unlink at ffffffffa100cf40 [mdt] #10 [ffff880605c6bcd8] mdt_reint_rec at ffffffffa1009891 [mdt] #11 [ffff880605c6bcf8] mdt_reint_internal at ffffffffa0feeb03 [mdt] #12 [ffff880605c6bd38] mdt_reint at ffffffffa0feee04 [mdt] #13 [ffff880605c6bd58] mdt_handle_common at ffffffffa0ff3ab8 [mdt] #14 [ffff880605c6bda8] mds_regular_handle at ffffffffa102d155 [mdt] #15 [ffff880605c6bdb8] ptlrpc_server_handle_request at ffffffffa09b26d8 [ptlrpc] #16 [ffff880605c6beb8] ptlrpc_main at ffffffffa09b3a6e [ptlrpc] #17 [ffff880605c6bf48] kernel_thread at ffffffff8100c0ca
Hmm, apparently in mdd_links_read, it did not check the return value of linkea_init, so the following linked_links_find will deal with the wrong linkea, which cause the panic. Probably this patch should fix it
diff --git a/lustre/mdd/mdd_dir.c b/lustre/mdd/mdd_dir.c
index 9f3c333..6435757 100644
— a/lustre/mdd/mdd_dir.c
+++ b/lustre/mdd/mdd_dir.c
@@ -1075,8 +1075,7 @@ int mdd_links_read(const struct lu_env *env, struct mdd_object *mdd_obj,
if (rc < 0)
return rc;
- linkea_init(ldata);
- return 0;
+ return linkea_init(ldata);
}
/** Read the link EA into a temp buffer.
Thank's for the direction Di!
Master patch implementing your proposal is at http://review.whamcloud.com/6676.
Bruno,
I can verify the patch stops the LBUG but the syslog gets spammed instead with the likes of:
Jun 18 11:42:51 bmds1 kernel: LustreError: 24036:0:(mdd_dir.c:995:mdd_links_rename()) link_ea del 'from_repo_revision' failed -61 [0x200000c68:0x795d:0x0]
Jun 18 11:42:52 bmds1 kernel: LustreError: 24005:0:(mdd_dir.c:995:mdd_links_rename()) link_ea del 'from_repo_timestamp' failed -61 [0x200000c68:0x796c:0x0]
Jun 18 11:42:52 bmds1 kernel: LustreError: 24036:0:(mdd_dir.c:995:mdd_links_rename()) link_ea del 'checksum_type' failed -61 [0x200000c68:0x7941:0x0]
Jun 18 11:42:52 bmds1 kernel: LustreError: 24036:0:(mdd_dir.c:995:mdd_links_rename()) link_ea del 'from_repo_revision' failed -61 [0x200000c68:0x795d:0x0]
Humm, seems that you trigger cases where the link_ea_header has a NULL leh_reccount !!...
I will finally need to have a look into earlier crash-dump to see how this can happen.
You say syslog get spammed, but what does this mean in term of number of msgs per timing window ??
Also, the names in the msgs you attached are from Yum database and are known to be hard-links to the same inodes, so can you find how-many hard-links concern these FIDs if still present ?
We may only trigger cases where the reverse link entry was not added in the link_ea because exceeding its maximum ?? And thus the msgs could be avoided ...
Hmm, it seems to me the -ENODATA and ENOENT check is wrong in mdd_links_prepare, which cause mdd_linkea_prepare wrongly return -ENODATA. probably this patch should help.
diff --git a/lustre/mdd/mdd_dir.c b/lustre/mdd/mdd_dir.c
index f7947c8..4770c32 100644
--- a/lustre/mdd/mdd_dir.c
+++ b/lustre/mdd/mdd_dir.c
@@ -932,11 +932,10 @@ static int mdd_linkea_prepare(const struct lu_env *env,
if (oldpfid != NULL) {
rc = __mdd_links_del(env, mdd_obj, ldata, oldlname, oldpfid);
if (rc) {
- if ((check == 0) ||
- (rc != -ENODATA && rc != -ENOENT))
+ if ((check == 0 && (rc == -ENOENT || rc == -ENODATA)))
+ rc = 0;
+ else
RETURN(rc);
- /* No changes done. */
- rc = 0;
}
}
So just to give you an idea of what we are doing here. We are essentially using "rsync --link-dest" to do backups of servers (hence the yum DB files). If the file is unchanged just hardlink it to the previous backup copy. So to trigger these messages we are simply deleting old backups which in many cases simply removes the hard link count by one. This workload has been found to be a good test of metadata and IO. We have never seen this issue in Lustre v1.8 in the 2 years this workload has been running on it.
In terms of the messages maybe 20/s? It isn't constant so I guess not all files trigger it. And there can be half an hour between an influx of these messages.
Cray has been seeing this bug a lot. We'll try out the patch and report back. Speak up if you need any of our debug info.
re: http://review.whamcloud.com/6676
Should mdt_links_read() as well as mdd_links_read() be changed to return the rc from linkea_init()? I'm just noting the symmetry in the code.
Should mdt_links_read() as well as mdd_links_read() be changed to return the rc from linkea_init()? I'm just noting the symmetry in the code.
At first glance it looks like it should. And if it shouldn't, a comment explaining the difference is needed.
Di,
Since ENOENT/ENODATA indicate either the link reference was not found in link_ea or there is no more reference at all in link_ea that means the current looked-up entry was never added due to link_ea space exhausted. So yes, seems that the error return value test is wrong (in fact I think in the original version it should have been "(check == 1)" as the 1st condition !!) and that we need to ignore ENOENT/ENODATA and simply continue. BTW, actually the error will only generate the msgs and end-up beeing trashed in mdd_unlink(), with the only side effect of new entry not beeing added upon rename!
Daire,
Thank's for the details on how this happen at your site. The fact you never experienced this with 1.8 is only due to no link_ea there! And as per my last comment just before, even if annoying, the error msgs you now have running with the patch do not indicate something really bad occurs.
Prakash, Ann,
I don't think that mdt_links_read() requires same change like mdd_links_read() because its only caller mdt_path_current() does not use ldata->ld_leh reference, but anyway I will try to create a reproducer for the situation (no link_ea entry due to overflow/limit?) that triggers the original Assert during unlink() and see if it can also cause problem upon fid2path() lookup.
Just for my understanding and about the reproducer scenario, is it possible that the hard-links beeing removed/unlinked and causing the LBUGs/msgs may have been created when running with some early 2.x version (ie, with more limits in place during link_ea populate) ?
Patch to be out soon.
The filesystem was formatted using the latest v2.3 release so many of the hardlinks would have been created under that version.
Cray sees the bug on a file system formatted with 2.4.
And we have a 2.1 formatted FS upgraded to Lustre 2.4 RPMs.
Just pushed new version/patch-set #2 of change http://review.whamcloud.com/6676. It adds a few ENODATA error handling fixes, to avoid unnecessary msgs and also prevent early return, to original fix.
And http://review.whamcloud.com/6772 is cosmetic patch for similar linkea_init() error handling in mdt layer.
Cray testing on change #6676 ps1 and 6772 shows that the changes resolve our problems with the LBUG.
Thank's for the feed-back Cory, #6676 patch-set #2 should fix the LBUG AND the annoying (and erroneous!) msgs ...
I was mistaken, Cray has not yet tested w/6772 applied. However, 6676 ps1 did test successfully.
Hello
In our site, we had this problem with 2.4.50 version. It was produced moving a dir with lots of files to other destination.
I can confirm that Patch-set #2 has fixed the problem.
Hello Daniel,
Thank's for the feed-back too !
But patch-set #2 was not in accordance with the error reporting rules being used, so I just pushed patch-set #3 to fix that.
I finally got around to testing the two patches - the LBUG has returned. I patched v2.4.0:
- cd /usr/src/lustre-2.4.0/
- patch -p1 < mdd_dir.c.patch
- patch -p1 < /tmp/mdt_handler.c.patch
- ./configure
- make rpms
Jul 3 12:36:50 bmds1 kernel: LustreError: 13174:0:(linkea.c:169:linkea_links_find()) ASSERTION( ldata->ld_leh != ((void *)0) ) failed:
Jul 3 12:36:50 bmds1 kernel: LustreError: 13174:0:(linkea.c:169:linkea_links_find()) LBUG
Jul 3 12:36:50 bmds1 kernel: Pid: 13174, comm: mdt01_010
Jul 3 12:36:50 bmds1 kernel:
Jul 3 12:36:50 bmds1 kernel: Call Trace:
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa043c895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa043ce97>] lbug_with_loc+0x47/0xb0 [libcfs]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa05a47d6>] linkea_links_find+0x186/0x190 [obdclass]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0b65206>] ? mdo_xattr_get+0x26/0x30 [mdd]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0b68645>] mdd_linkea_prepare+0x95/0x430 [mdd]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0b68b01>] mdd_links_rename+0x121/0x520 [mdd]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0b6cac6>] mdd_unlink+0xb86/0xe30 [mdd]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0dddb88>] mdo_unlink+0x18/0x50 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0de0f30>] mdt_reint_unlink+0x820/0x1010 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0ddd881>] mdt_reint_rec+0x41/0xe0 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0dc2b03>] mdt_reint_internal+0x4c3/0x780 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0dc2e04>] mdt_reint+0x44/0xe0 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0dc7ab8>] mdt_handle_common+0x648/0x1660 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa0e01155>] mds_regular_handle+0x15/0x20 [mdt]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa071e388>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa043d5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa044ed8f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Jul 3 12:36:50 bmds1 kernel: [<ffffffffa07156e9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Jul 3 12:36:50 bmds1 kernel: [<ffffffff81055ab3>] ? __wake_up+0x53/0x70
Jul 3 12:36:51 bmds1 kernel: [<ffffffffa071f71e>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Jul 3 12:36:51 bmds1 kernel: [<ffffffffa071ec50>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Jul 3 12:36:51 bmds1 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 3 12:36:51 bmds1 kernel: [<ffffffffa071ec50>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Jul 3 12:36:51 bmds1 kernel: [<ffffffffa071ec50>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Wow I am sorry Daire, I don't know how this happen but patch-set#3 of http://review.whamcloud.com/6676 contained a regression from patch-set #1/#2 (in fact it did not contain the main part/change from patch-set #1 that must be in to prevent the LBUG!!) .... Can you give a try to patch-set #4 that should be definitive one ??
Daire, have you been able to test patch-set #4 of http://review.whamcloud.com/6676 finally ??
Bruno,
I have patched it in and haven't seen the issue again yet. However I have not had the opportunity to run the same workload (large unlinks) but that should happen between now and next week. I will update if we have any further issues. Thanks for the help.
FWIW, I've applied #6672 and #6676 and have not hit the issue with our test workload (we did hit it repeatedly without #6676).
Patch landed to Master. Closing ticket. Please let me know if more work is needed and I will reopen.
It seems that http://review.whamcloud.com/6676 was landed to b2_4 for 2.4.1, but http://review.whamcloud.com/6772 (which was not mentioned anywhere in this bug, but attributed to LU-3474) was only landed to master for 2.4.52 and not b2_4. This results in "lfs fid2path" on old IGIF FIDs with 2.4.2 servers to incorrectly return success if there is no linkEA, but only prints the root path:
lfs fid2path /myth [0x10b466:0xfce641b5:0x0] /myth//
There is never a linkEA for upgraded 1.x files until LFSCK 1.5 is run on a 2.5+ MDS.
Cherry-pick http://review.whamcloud.com/6772 to b2_4: http://review.whamcloud.com/10464
I think I had mentionned #6772 in this ticket, but anyway I better had, as Di suggested at that time, to merge both patches to avoid such oversight!