Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-2323] mds crash https://jira.whamcloud.com/browse/LU-2323 Lustre <p>We recently experienced two MDS crashes on our Lustre installation.</p> <p>I've attached the netconsole output of both crashes (that's all i got: there is nothing in the syslog and i wasn't able to create a screenshot of the console output as the crashed mds was already powercycled by its failover partner).</p> [<a href='mailto:root@n-mds1'>root@n-mds1</a> ~]# cat /proc/fs/lustre/version <br/> lustre: 2.2.0<br/> kernel: patchless_client<br/> build: 2.2.0-RC2--PRISTINE-2.6.32-220.4.2.el6_lustre.x86_64<br/> <br/> [<a href='mailto:root@n-mds1'>root@n-mds1</a> ~]# uname -r<br/> 2.6.32-220.4.2.el6_lustre.x86_64<br/> <br/> [<a href='mailto:root@n-mds1'>root@n-mds1</a> ~]# rpm -qa|grep lustre<br/> lustre-ldiskfs-3.3.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64<br/> lustre-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64<br/> kernel-firmware-2.6.32-220.4.2.el6_lustre.x86_64<br/> lustre-modules-2.2.0-2.6.32_220.4.2.el6_lustre.x86_64.x86_64<br/> kernel-headers-2.6.32-220.4.2.el6_lustre.x86_64<br/> kernel-2.6.32-220.4.2.el6_lustre.x86_64<br/> kernel-devel-2.6.32-220.4.2.el6_lustre.x86_64 LU-2323

mds crash

Bug Critical Resolved Fixed Niu Yawei ETHz Support server Wed, 14 Nov 2012 04:46:04 +0000 Thu, 6 Dec 2012 09:24:09 +0000 Thu, 6 Dec 2012 09:24:09 +0000 Lustre 2.2.0 0 10 <p>netconsole output</p> <p>In terms of activity, during the crashes, the mds server has memory available and the IB network have low traffic.</p> <p>In both cases there was free memory available: The crash yesterday didn't even have a full read-cache and the MDS was never swapping (it has a 50GB Swap partition).</p> <p>I also checked the system load + infiniband traffic: The MDS was doing almost nothing during the crash (load less than 0.5 / IB traffic ~100KB/s).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2012-11-08T20:14:29+01:00 n-mds1 [<ffffffffa0dd4dbb>] osd_trans_stop+0xeb/0x390 [osd_ldiskfs] 2012-11-08T20:14:29+01:00 n-mds1 RSP <ffff880bd116da70> 2012-11-08T20:14:29+01:00 n-mds1 ---[ end trace 69a06040c21c938c ]--- 2012-11-08T20:14:29+01:00 n-mds1 Kernel panic - not syncing: Fatal exception 2012-11-08T20:14:29+01:00 n-mds1 Pid: 3845, comm: mdt_52 Tainted: G D ---------------- 2.6.32-220.4.2.el6_lustre.x86_64 #1 2012-11-08T20:14:29+01:00 n-mds1 Call Trace: 2012-11-08T20:14:29+01:00 n-mds1 [<ffffffff814ec61a>] ? panic+0x78/0x143 2012-11-08T20:14:29+01:00 n-mds1 [<ffffffff814f07a4>] ? oops_end+0xe4/0x100 2012-11-08T20:14:29+01:00 n-mds1 [<ffffffff8100f26b>] ? die+0x5b/0x90 2012-11-08T20:14:29+01:00 n-mds1 [<ffffffff814f0312>] ? do_general_protection+0x152/0x160 2012-11-08T20:14:29+01:00 n-mds1 [<ffffffffa0ddaada>] ? osd_xattr_set+0x14a/0x1d0 [osd_ldiskfs] 2012-11-08T20:14:29+01:00 n-mds1 [<ffffffff814efae5>] ? general_protection+0x25/0x30 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0dd4dbb>] ? osd_trans_stop+0xeb/0x390 [osd_ldiskfs] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0cde12a>] ? mdd_trans_stop+0x1a/0x20 [mdd] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0cc1036>] ? mdd_attr_set+0xbf6/0x2030 [mdd] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0677820>] ? ldlm_completion_ast+0x0/0x6d0 [ptlrpc] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa069ae3c>] ? lustre_msg_get_versions+0x6c/0xb0 [ptlrpc] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0e12a1c>] ? cml_attr_set+0x6c/0x160 [cmm] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d4e578>] ? mdt_attr_set+0x268/0x4b0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d4eb0d>] ? mdt_reint_setattr+0x34d/0x1060 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d48e7b>] ? mdt_reint_rec+0x4b/0xa0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d41069>] ? mdt_reint_internal+0x479/0x7b0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d413ee>] ? mdt_reint+0x4e/0xb0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d37b9d>] ? mdt_handle_common+0x74d/0x1400 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0d38925>] ? mdt_regular_handle+0x15/0x20 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa06a6011>] ? ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa04373ee>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2012-11-08T20:14:30+01:00 n-mds1 [<ffffffffa0441e19>] ? lc_watchdog_touch+0x79/0x110 [libcfs] 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffffa06a00e2>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffff810519c3>] ? __wake_up+0x53/0x70 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffffa06a701f>] ? ptlrpc_main+0x71f/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffffa06a6900>] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffff8100c14a>] ? child_rip+0xa/0x20 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffffa06a6900>] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffffa06a6900>] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [<ffffffff8100c140>] ? child_rip+0x0/0x20 </pre> </div></div> <p>In the 2.2 code, we added following code in osd_xattr_set():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* version set is not real XATTR */</span> <span class="code-keyword">if</span> (strcmp(name, XATTR_NAME_VERSION) == 0) { /* <span class="code-keyword">for</span> version we are just using xattr API but change inode * field instead */ LASSERT(buf->lb_len == sizeof(dt_obj_version_t)); osd_object_version_set(env, dt, buf->lb_buf); <span class="code-keyword">return</span> sizeof(dt_obj_version_t); } </pre> </div></div> <p>we should probably check if the "name" is NULL first. Alex, any thought?</p> <p>hmm, I'd say we should assert on name=NULL.. and the caller must be fixed.</p> <p>This has become quite serious for us right now: Our filesystem is currently down: We entered a reboot -> panic -> reboot -> panic cycle:</p> <p>Our MDS crashes with the same message as soon as we reboot it:<br/> 2012-11-14T15:45:07+01:00 n-mds2 <span class="error">[<ffffffffa0cde12a>]</span> mdd_trans_stop+0x1a/0x20 <span class="error">[mdd]</span><br/> 2012-11-14T15:45:07+01:00 n-mds2 <span class="error">[<ffffffffa0cc1036>]</span> mdd_attr_set+0xbf6/0x2030 <span class="error">[mdd]</span><br/> 2012-11-14T15:45:07+01:00 n-mds2 <span class="error">[<ffffffffa0677820>]</span> ? ldlm_completion_ast+0x0/0x6d0 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa069ae3c>]</span> ? lustre_msg_get_versions+0x6c/0xb0 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0e12a1c>]</span> cml_attr_set+0x6c/0x160 <span class="error">[cmm]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d4e578>]</span> mdt_attr_set+0x268/0x4b0 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d4eb0d>]</span> mdt_reint_setattr+0x34d/0x1060 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d48e7b>]</span> mdt_reint_rec+0x4b/0xa0 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d41069>]</span> mdt_reint_internal+0x479/0x7b0 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d413ee>]</span> mdt_reint+0x4e/0xb0 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d37b9d>]</span> mdt_handle_common+0x74d/0x1400 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0d38925>]</span> mdt_regular_handle+0x15/0x20 <span class="error">[mdt]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa06a6011>]</span> ptlrpc_server_handle_request+0x3c1/0xcb0 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa04373ee>]</span> ? cfs_timer_arm+0xe/0x10 <span class="error">[libcfs]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa0441e19>]</span> ? lc_watchdog_touch+0x79/0x110 <span class="error">[libcfs]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa06a00e2>]</span> ? ptlrpc_wait_event+0xb2/0x2c0 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffff810519c3>]</span> ? __wake_up+0x53/0x70<br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa06a701f>]</span> ptlrpc_main+0x71f/0x1210 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:08+01:00 n-mds2 <span class="error">[<ffffffffa06a6900>]</span> ? ptlrpc_main+0x0/0x1210 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:09+01:00 n-mds2 <span class="error">[<ffffffff8100c14a>]</span> child_rip+0xa/0x20<br/> 2012-11-14T15:45:09+01:00 n-mds2 <span class="error">[<ffffffffa06a6900>]</span> ? ptlrpc_main+0x0/0x1210 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:09+01:00 n-mds2 <span class="error">[<ffffffffa06a6900>]</span> ? ptlrpc_main+0x0/0x1210 <span class="error">[ptlrpc]</span><br/> 2012-11-14T15:45:09+01:00 n-mds2 <span class="error">[<ffffffff8100c140>]</span> ? child_rip+0x0/0x20<br/> 2012-11-14T15:45:09+01:00 n-mds2 Code: </p> <p>Adrian, have you tried to remount the MDS with "-o abort_recov"?</p> <p>No, but it seems to be 'stable' again.</p> <p>After the 4th crash i started a 'fsck -n /dev/mapper/...', got impatient and aborted it after ~10 minutes.</p> <p>After this i was able to start the MDS without any new crash: Looks like the '10 minute downtime' was enough to time out the 'evil' client/operation !?</p> <p>Is there any way to see which client causes the crash?</p> <p>Adrian,<br/> I'm suggest to disable the cluster (to avoid pingpong) and mount manually the mds : mount -t lustre -L <MDT name> -o abort_recov <mount point></p> <p>Johann:</p> <p>Does the crash actually get triggered due to a client calling setaddr?</p> <p>One of our most frequent setattr callers is an 1.8.4 client (10.201.32.32) - could this client be the cause of the crash?</p> <p><span class="error">[root@n-mds1 exports]</span># grep setattr */stats | awk '</p> {print $2 " " $1} <p>'|sort -n | tail -5<br/> 1694 10.201.38.39@o2ib/stats:setattr<br/> 1841 10.201.38.21@o2ib/stats:setattr<br/> 2062 10.201.38.23@o2ib/stats:setattr <-- 2.2.93 client<br/> 8931 10.201.32.32@o2ib/stats:setattr <-- 1.8.4 client<br/> 17278 10.201.32.31@o2ib/stats:setattr <-- 2.3.0 client</p> <p>I do not think it is NULL "name" for osd_xattr_set() caused the failure. In this case, the "name" is from MDS internal, not from client. I do not find any internal callers passing NULL "name". On the other hand, the other failure instances have different call traces.</p> <p>Adrian, have you made any system upgrading recently when you hit the failure? There is no evidence to indicate that it is interoperability issue caused the failure, but we can try to locate the issue step by step. The first step, please migrate the system load from above non-2.2 clients to other standard-2.2 clients. Especially the 1.8.4 client, because it is too old. I do not think we have tested the interoperation between Lustre-2.2 server and Lustre-1.8.4 client when we released Lustre-2.2. So it is the most suspicious.</p> <p>in the both cases the very first messages were about inability to add llog record:</p> <p>LustreError: 3809:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff880482b99800</p> <p>probably it's a problem in the code handling this error. I think we should reproduce this locally.</p> <p>it makes sense to ls CONFIGS/ directory using ldiskfs or debugfs to see how much space can be free after orphan cleanup.</p> <p>Adrian, if you cannot abandon 1.8.4 clients, then please umount them temporarily, and try to reproduce the failure with other clients. If cannot reproduce, then it is quite possible related with the interoperability issues.</p> <p>Yong Fan:</p> <ul> <li>We need the 1.8.x client to copy data from our old Lustre 1.8 installation to the 2.2 installation.<br/> I will temporarily disable it on the next crash (there are still users moving data around).</li> </ul> <ul> <li>We didn't do any upgrades recently: We didn't touch the servers since months.</li> </ul> <ul> <li>We have 2.3 clients because the 2.2-client is too unstable for us and i don't have enough time to backport all crash fixes from 2.3 to our 2.2-client</li> </ul> <p>Alex: We get this llog_write_rec errors since months (Jun 2012) while our MDS only startet to crash recently (also note that there are ~5 hours between the error and the crash).</p> <p>What does the llog_write_rec error mean anyway?</p> <p>I'll post the output of 'ls CONFIGS/' as soon as i have a chance to remount the volume via ldiskfs (= on next crash or on monday evening)</p> <p>ok, good to know.. llog_write_rec() getting -28 mean MDS was not able to write (transactionally) a record to make some updates to OST (remove OST object, update OST object attributes).</p> <p>then could you attach osd_ldiskfs.ko please ?</p> <p>md5sum = 6a07cbbb49f63ea0f6a5e6bc067bc7c9</p> <p>requested kernel module</p> <p>Alex: I attached the requested kernel module (or did you mean with 'attach' that i should insmod it? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> )</p> <p>no, i've got what I need, thanks <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>a bit of analysis here:</p> <p>0000000000004d26 <osd_trans_stop+0x56> mov 0x50(%rbx),%r12<br/> 0000000000004d2a <osd_trans_stop+0x5a> test %r12,%r12<br/> 0000000000004d2d <osd_trans_stop+0x5d> je 0000000000004e82 <osd_trans_stop+0x1b2><br/> 0000000000004d33 <osd_trans_stop+0x63> movzbl 0x28(%r12),%eax<br/> 0000000000004d39 <osd_trans_stop+0x69> movzbl 0x4c(%rbx),%edx<br/> 0000000000004d3d <osd_trans_stop+0x6d> and $0xfffffffe,%eax<br/> 0000000000004d40 <osd_trans_stop+0x70> and $0x1,%edx<br/> 0000000000004d43 <osd_trans_stop+0x73> or %edx,%eax<br/> 0000000000004d45 <osd_trans_stop+0x75> mov %al,0x28(%r12)<br/> 0000000000004d4a <osd_trans_stop+0x7a> mov (%r12),%rax</p> <p>so rbx contains pointer to oh:</p> <p>(gdb) p/x sizeof(struct thandle)<br/> $2 = 0x50<br/> struct osd_thandle {<br/> struct thandle ot_super;<br/> handle_t *ot_handle;</p> <p>0000000000004db3 <osd_trans_stop+0xe3> mov (%rbx),%rax<br/> 0000000000004db6 <osd_trans_stop+0xe6> test %rax,%rax<br/> 0000000000004db9 <osd_trans_stop+0xe9> je 0000000000004dc4 <osd_trans_stop+0xf4><br/> 0000000000004dbb <osd_trans_stop+0xeb> mov 0x8(%rax),%rax<br/> 0000000000004dbf <osd_trans_stop+0xef> testb $0x1,(%rax)</p> <p>these lines implement:<br/> if (lu_device_is_md(&th->th_dev->dd_lu_dev)) {</p> <p>RAX: 0006000100000002 is supposed to be ld_type (and 0x8(%rax) is ld_type->ldt_tags)</p> <p>IOW, thandle was broken and pointing to garbage instead of a device.</p> <p>now the question what broke that..</p> <p>Adrian,<br/> if I remember well you can use debufs with the device mounted. Try:</p> <p>debugfs -c -R 'dump CONFIGS/ /tmp/config' /dev/<device><br/> llog_reader /tmp/config</p> <p>I did this using a snapshot from the MDS (taken at 5. November).<br/> The output of llog_reader is attached to the case (llog.txt)</p> <p>Output of CONFIGS/ via debugfs:</p> <p>$ debugfs mds.dump <br/> debugfs 1.41.12 (17-May-2010)<br/> debugfs: ls -l CONFIGS<br/> 467550721 40777 (2) 0 0 4096 9-Oct-2012 07:05 .<br/> 2 40755 (2) 0 0 4096 3-May-2012 14:58 ..<br/> 467550722 100644 (1) 0 0 12288 9-May-2012 09:21 mountdata<br/> 467550723 100644 (1) 0 0 0 3-May-2012 14:58 _mgs-sptlrpc<br/> 467550724 100644 (1) 0 0 89128 3-May-2012 14:58 nero-client<br/> 467550725 100644 (1) 0 0 0 3-May-2012 14:58 nero-sptlrpc<br/> 467550726 100644 (1) 0 0 89000 3-May-2012 14:58 nero-MDT0000<br/> 467550727 100644 (1) 0 0 0 3-May-2012 14:58 changelog_catalog<br/> 467550728 100644 (1) 0 0 0 3-May-2012 14:58 changelog_users<br/> 467550730 100644 (1) 0 0 9432 3-May-2012 15:05 nero-OST0000<br/> 467550729 100644 (1) 0 0 0 9-Oct-2012 07:05 sptlrpc<br/> 467550731 100644 (1) 0 0 9432 3-May-2012 15:08 nero-OST0008<br/> 467550732 100644 (1) 0 0 9432 3-May-2012 15:09 nero-OST0010<br/> 467550733 100644 (1) 0 0 9432 3-May-2012 15:09 nero-OST0018<br/> 467550734 100644 (1) 0 0 9432 3-May-2012 15:58 nero-OST0001<br/> 467550735 100644 (1) 0 0 9432 3-May-2012 15:58 nero-OST0009<br/> 467550736 100644 (1) 0 0 9432 3-May-2012 15:59 nero-OST0011<br/> 467550737 100644 (1) 0 0 9432 3-May-2012 15:59 nero-OST0019<br/> 467550738 100644 (1) 0 0 9432 3-May-2012 16:00 nero-OST0002<br/> 467550739 100644 (1) 0 0 9432 3-May-2012 16:01 nero-OST000a<br/> 467550740 100644 (1) 0 0 9432 3-May-2012 16:04 nero-OST0012<br/> 467550741 100644 (1) 0 0 9432 3-May-2012 16:05 nero-OST001a<br/> 467550742 100644 (1) 0 0 9432 3-May-2012 16:06 nero-OST0003<br/> 467550743 100644 (1) 0 0 9432 3-May-2012 16:06 nero-OST000b<br/> 467550744 100644 (1) 0 0 9432 3-May-2012 16:06 nero-OST0013<br/> 467550745 100644 (1) 0 0 9432 3-May-2012 16:07 nero-OST001b<br/> 467550746 100644 (1) 0 0 9432 3-May-2012 16:11 nero-OST0004<br/> 467550747 100644 (1) 0 0 9432 3-May-2012 16:11 nero-OST000c<br/> 467550748 100644 (1) 0 0 9432 3-May-2012 16:12 nero-OST0014<br/> 467550749 100644 (1) 0 0 9432 3-May-2012 16:12 nero-OST001c<br/> 467550750 100644 (1) 0 0 9432 3-May-2012 16:14 nero-OST0005<br/> 467550751 100644 (1) 0 0 9432 3-May-2012 16:14 nero-OST000d<br/> 467550752 100644 (1) 0 0 9432 3-May-2012 16:14 nero-OST0015<br/> 467550753 100644 (1) 0 0 9432 3-May-2012 16:15 nero-OST001d<br/> 467550754 100644 (1) 0 0 9432 3-May-2012 16:18 nero-OST0006<br/> 467550755 100644 (1) 0 0 9432 3-May-2012 16:18 nero-OST000e<br/> 467550756 100644 (1) 0 0 9432 3-May-2012 16:18 nero-OST0016<br/> 467550757 100644 (1) 0 0 9432 3-May-2012 16:18 nero-OST001e<br/> 467550758 100644 (1) 0 0 9432 3-May-2012 16:21 nero-OST0007<br/> 467550759 100644 (1) 0 0 9432 3-May-2012 16:21 nero-OST000f<br/> 467550760 100644 (1) 0 0 9432 3-May-2012 16:22 nero-OST0017<br/> 467550761 100644 (1) 0 0 9432 3-May-2012 16:22 nero-OST001f</p> <p>Adrian,</p> <p>Did you have opportunity to try excluding 1.8.x clients to check whether the MDS still crashes with only 2.x clients accessing it?</p> <p>Well, the problem is that i can not reproduce the crash and i did not see any new crashes since 14. November.</p> <p>(The crash was probably caused by an user job: There are about ~800 users on our cluster and i have no way to figure out what job crashed it).</p> <p>But in any case: Even if the crash was triggered by an 1.8.x client: It should get fixed, shouldn't it?</p> <p>And do we have any news about the llog_write_rec error? (did the debugfs output help?)</p> <p>Yes, even it's 1.8.x client problem we should fix it. The purpose of the question is trying to help to make out which area to find the root cause.</p> <p>I'm still investigating the llog part issue.</p> <p>I think the "LustreError: 31980:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff88042d450240" is a misleading message, the message only means the current log does not has enough space for the log record, it will create a new log for it later.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> llog_cat_add_rec(struct llog_handle *cathandle, struct llog_rec_hdr *rec, struct llog_cookie *reccookie, void *buf) { struct llog_handle *loghandle; <span class="code-object">int</span> rc; ENTRY; LASSERT(rec->lrh_len <= LLOG_CHUNK_SIZE); loghandle = llog_cat_current_log(cathandle, 1); <span class="code-keyword">if</span> (IS_ERR(loghandle)) RETURN(PTR_ERR(loghandle)); <span class="code-comment">/* loghandle is already locked by llog_cat_current_log() <span class="code-keyword">for</span> us */</span> rc = llog_write_rec(loghandle, rec, reccookie, 1, buf, -1); <span class="code-keyword">if</span> (rc < 0) CERROR(<span class="code-quote">"llog_write_rec %d: lh=%p\n"</span>, rc, loghandle); cfs_up_write(&loghandle->lgh_lock); <span class="code-keyword">if</span> (rc == -ENOSPC) { <span class="code-comment">/* to create a <span class="code-keyword">new</span> plain log */</span> loghandle = llog_cat_current_log(cathandle, 1); <span class="code-keyword">if</span> (IS_ERR(loghandle)) RETURN(PTR_ERR(loghandle)); rc = llog_write_rec(loghandle, rec, reccookie, 1, buf, -1); cfs_up_write(&loghandle->lgh_lock); } RETURN(rc); } </pre> </div></div> <p>After checking the 2.2 code carefully, I found a culprit which can cause such memory corruption:</p> <p>in mdd_declare_attr_set():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#ifdef CONFIG_FS_POSIX_ACL <span class="code-keyword">if</span> (ma->ma_attr.la_valid & LA_MODE) { mdd_read_lock(env, obj, MOR_TGT_CHILD); rc = mdo_xattr_get(env, obj, buf, XATTR_NAME_ACL_ACCESS, BYPASS_CAPA); mdd_read_unlock(env, obj); <span class="code-keyword">if</span> (rc == -EOPNOTSUPP || rc == -ENODATA) rc = 0; <span class="code-keyword">else</span> <span class="code-keyword">if</span> (rc < 0) <span class="code-keyword">return</span> rc; </pre> </div></div> <p>Our intention here is to retrieve the xattr length, but we passed an uninitialized buffer to mdo_xattr_get() (we should pass NULL here)...<br/> Actually this bug has been fixed for 2.3 & 2.4 (see <a href="http://review.whamcloud.com/#change,3928" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3928</a> & <a href="https://jira.whamcloud.com/browse/LU-1823" title="sanity/103: slab corruption" class="issue-link" data-issue-key="LU-1823"><del>LU-1823</del></a>), I think we need to backport it to 2.2.</p> <p>backport the memory corruption fix in mdd_declare_attr_set() to b2_2: <a href="http://review.whamcloud.com/4703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4703</a></p> <p>Thanks for fixing this issue: We will upgrade our MDS as soon as a new build becomes available – or should we just upgrade to 2.3?</p> <p>Adrian a build of the change backported to 2.2 already exists - <a href="http://build.whamcloud.com/job/lustre-reviews/10853/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/10853/</a> - but is still in the automated test queue at the moment. Lustre 2.3 is available now and has been thoroughly tested. It will of course include other content beyond just this one fix (both additional features and many other fixes)</p> <p>Adrian</p> <p>Have you decided which approach you will take - to patch 2.2 or upgrade to 2.3?</p> <p>Peter</p> <p>Hello Peter,</p> <p>We will upgrade to 2.3 as soon as the next opportunity arises, you can therefore close this issue.</p> <p>Thanks and best regards,<br/> Adrian</p> <p>ok thanks Adrian!</p> Development Epic metadata server Rank 1|hzvc7r: Rank (Obsolete) 5550 Severity