https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1596 Lustre <p>see attached console logs. MDS went through recover didn't finish it restarted recovery. At the end of send recovery cycle it crash</p> <p>crash is at line 1252<br/> BUG: scheduling while atomic: mdt_33/5450/0xffff8800^M<br/> BUG: unable to handle kernel paging request at 0000000000015fc0^M<br/> IP: <span class="error">&#91;&lt;ffffffff81523c5e&gt;&#93;</span> _spin_lock+0xe/0x30^M<br/> PGD 5cc604067 PUD 5cc608067 PMD 0 ^M<br/> Oops: 0002 <a href="#1" target="_blank" rel="noopener">1</a> SMP ^M<br/> last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:03:00.0/host1/rport-1:0-0/target1:0:0/1:0:0:1/state^M<br/> ^M<br/> Entering kdb (current=0xffffffff81a2d020, pid 0) on processor 0 Oops: (null)^M<br/> due to oops @ 0xffffffff81523c5e^M<br/> r15 = 0x0000000000000003 r14 = 0x0000000000015fc0 ^M<br/> r13 = 0xffff880040e03998 r12 = 0xffff8804432c9500 ^M<br/> bp = 0xffff880040e03930 bx = 0x0000000000015fc0 ^M<br/> r11 = 0x0000000000000028 r10 = 0x0000000000000000 ^M<br/> r9 = 0xfbad0278679d3602 r8 = 0x0000000000000000 ^M<br/> ax = 0x0000000000010000 cx = 0xffff880040e03ab0 ^M<br/> dx = 0x0000000000000082 si = 0xffff880040e03998 ^M<br/> di = 0x0000000000015fc0 orig_ax = 0xffffffffffffffff ^M<br/> ip = 0xffffffff81523c5e cs = 0x0000000000000010 ^M<br/> flags = 0x0000000000010006 sp = 0xffff880040e03920 ^M<br/> ss = 0x0000000000000018 &amp;regs = 0xffff880040e03888^M</p> 2.1.2 server<br/> 2.1.1and2.1.2 clients LU-1596

mds crash after recovery finished

Bug Minor Resolved Duplicate Lai Siyao Mahmoud Hanafi Tue, 3 Jul 2012 16:34:46 +0000 Thu, 20 Sep 2012 19:25:08 +0000 Thu, 20 Sep 2012 19:25:08 +0000 Lustre 2.1.2 0 4 <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>The log shows it BUG on pid 5450:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>BUG: scheduling while atomic: mdt_33/5450/0xffff8800 </pre> </div></div> <p>The backtrace for 5450 is as below:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> mdt_33 D 0000000000000246 0 5450 2 0x3005ff40 ffff8804432ca750 0000000000000018 ffff88061feaa378 0000000000000246 ffff8804432ca780 ffffffffa00023ac ffff880621185400 ffff88042f278690 ffff8804432ca7e0 ffff88061ffe8f20 ffff8804432ca7b0 ffffffff81254f74 Call Trace: [&lt;ffffffffa00023ac&gt;] ? dm_dispatch_request+0x3c/0x70 [dm_mod] [&lt;ffffffff81254f74&gt;] ? blk_unplug+0x34/0x70 [&lt;ffffffff811a7c90&gt;] ? sync_buffer+0x0/0x50 [&lt;ffffffff81520d83&gt;] ? printk+0x41/0x46 [&lt;ffffffff81056681&gt;] ? __schedule_bug+0x41/0x70 [&lt;ffffffff81521978&gt;] ? thread_return+0x638/0x760 [&lt;ffffffffa000420c&gt;] ? dm_table_unplug_all+0x5c/0x100 [dm_mod] [&lt;ffffffff8109ac19&gt;] ? ktime_get_ts+0xa9/0xe0 [&lt;ffffffff811a7c90&gt;] ? sync_buffer+0x0/0x50 [&lt;ffffffff81521b13&gt;] ? io_schedule+0x73/0xc0 [&lt;ffffffff811a7cd0&gt;] ? sync_buffer+0x40/0x50 [&lt;ffffffff815224cf&gt;] ? __wait_on_bit+0x5f/0x90 [&lt;ffffffff811a7c90&gt;] ? sync_buffer+0x0/0x50 [&lt;ffffffff81522578&gt;] ? out_of_line_wait_on_bit+0x78/0x90 [&lt;ffffffff81090030&gt;] ? wake_bit_function+0x0/0x50 [&lt;ffffffff811a7c86&gt;] ? __wait_on_buffer+0x26/0x30 [&lt;ffffffffa0abb63c&gt;] ? ldiskfs_mb_init_cache+0x24c/0xa30 [ldiskfs] [&lt;ffffffffa0abbf3e&gt;] ? ldiskfs_mb_init_group+0x11e/0x210 [ldiskfs] [&lt;ffffffffa0abc0fd&gt;] ? ldiskfs_mb_good_group+0xcd/0x110 [ldiskfs] [&lt;ffffffffa0abf0bb&gt;] ? ldiskfs_mb_regular_allocator+0x19b/0x410 [ldiskfs] [&lt;ffffffff8152286e&gt;] ? mutex_lock+0x1e/0x50 [&lt;ffffffffa0abf762&gt;] ? ldiskfs_mb_new_blocks+0x432/0x660 [ldiskfs] [&lt;ffffffffa0aa61fe&gt;] ? ldiskfs_ext_find_extent+0x2ce/0x330 [ldiskfs] [&lt;ffffffffa0aa93b4&gt;] ? ldiskfs_ext_get_blocks+0x1114/0x1a10 [ldiskfs] [&lt;ffffffffa0a7f541&gt;] ? __jbd2_journal_file_buffer+0xd1/0x220 [jbd2] [&lt;ffffffffa0a8075f&gt;] ? jbd2_journal_dirty_metadata+0xff/0x150 [jbd2] [&lt;ffffffffa0aa3f3b&gt;] ? __ldiskfs_handle_dirty_metadata+0x7b/0x100 [ldiskfs] [&lt;ffffffffa0ab1665&gt;] ? ldiskfs_get_blocks+0xf5/0x2a0 [ldiskfs] [&lt;ffffffffa0ad8048&gt;] ? __ldiskfs_journal_stop+0x68/0xa0 [ldiskfs] [&lt;ffffffffa0ab62b9&gt;] ? ldiskfs_getblk+0x79/0x1f0 [ldiskfs] [&lt;ffffffffa0ab6448&gt;] ? ldiskfs_bread+0x18/0x80 [ldiskfs] [&lt;ffffffffa0b5e2a7&gt;] ? fsfilt_ldiskfs_write_handle+0x147/0x340 [fsfilt_ldiskfs] [&lt;ffffffffa0b5e55c&gt;] ? fsfilt_ldiskfs_write_record+0xbc/0x1d0 [fsfilt_ldiskfs] [&lt;ffffffffa06382ec&gt;] ? llog_lvfs_write_blob+0x2bc/0x460 [obdclass] [&lt;ffffffffa06338b8&gt;] ? llog_init_handle+0xa18/0xa70 [obdclass] [&lt;ffffffffa0639b5a&gt;] ? llog_lvfs_write_rec+0x40a/0xf00 [obdclass] [&lt;ffffffffa0636a65&gt;] ? llog_cat_add_rec+0xf5/0x840 [obdclass] [&lt;ffffffffa063d306&gt;] ? llog_obd_origin_add+0x56/0x190 [obdclass] [&lt;ffffffffa063d4e1&gt;] ? llog_add+0xa1/0x3c0 [obdclass] [&lt;ffffffff8115c66a&gt;] ? kmem_getpages+0xba/0x170 [&lt;ffffffffa09610cc&gt;] ? lov_llog_origin_add+0xcc/0x5d0 [lov] [&lt;ffffffffa063d4e1&gt;] ? llog_add+0xa1/0x3c0 [obdclass] [&lt;ffffffffa0b7449e&gt;] ? mds_llog_origin_add+0xae/0x2e0 [mds] [&lt;ffffffffa0ab0651&gt;] ? __ldiskfs_get_inode_loc+0xf1/0x3b0 [ldiskfs] [&lt;ffffffffa0995d0b&gt;] ? lov_tgt_maxbytes+0x5b/0xb0 [lov] [&lt;ffffffffa063d4e1&gt;] ? llog_add+0xa1/0x3c0 [obdclass] [&lt;ffffffffa0b74b12&gt;] ? mds_llog_add_unlink+0x162/0x520 [mds] [&lt;ffffffffa0b75206&gt;] ? mds_log_op_unlink+0x196/0x9a0 [mds] [&lt;ffffffffa0b9d7de&gt;] ? mdd_unlink_log+0x4e/0x100 [mdd] [&lt;ffffffffa0b973ab&gt;] ? mdd_attr_get_internal+0x7ab/0xb10 [mdd] [&lt;ffffffffa0b91a8e&gt;] ? mdd_object_kill+0x14e/0x1b0 [mdd] [&lt;ffffffffa0bab8ce&gt;] ? mdd_finish_unlink+0x20e/0x2c0 [mdd] [&lt;ffffffffa0baa780&gt;] ? __mdd_ref_del+0x40/0xc0 [mdd] [&lt;ffffffffa0bb704c&gt;] ? mdd_rename+0x1ffc/0x2240 [mdd] [&lt;ffffffffa0b96dcb&gt;] ? mdd_attr_get_internal+0x1cb/0xb10 [mdd] [&lt;ffffffffa0589caf&gt;] ? cfs_hash_bd_from_key+0x3f/0xc0 [libcfs] [&lt;ffffffffa0c7f219&gt;] ? cmm_mode_get+0x109/0x320 [cmm] [&lt;ffffffffa0c7fd3a&gt;] ? cml_rename+0x33a/0xbb0 [cmm] [&lt;ffffffffa058a337&gt;] ? cfs_hash_bd_get+0x37/0x90 [libcfs] [&lt;ffffffffa0c7f49d&gt;] ? cmm_is_subdir+0x6d/0x2f0 [cmm] [&lt;ffffffffa067d8e6&gt;] ? lu_object_put+0x86/0x210 [obdclass] [&lt;ffffffffa0c09426&gt;] ? mdt_reint_rename+0x1fa6/0x2400 [mdt] [&lt;ffffffffa05909db&gt;] ? upcall_cache_get_entry+0x28b/0xa14 [libcfs] [&lt;ffffffffa0c0197f&gt;] ? mdt_rename_unpack+0x46f/0x6c0 [mdt] [&lt;ffffffffa0bb96c6&gt;] ? md_ucred+0x26/0x60 [mdd] [&lt;ffffffffa0c01c0f&gt;] ? mdt_reint_rec+0x3f/0x100 [mdt] [&lt;ffffffffa07726e4&gt;] ? lustre_msg_get_flags+0x34/0xa0 [ptlrpc] [&lt;ffffffffa0bfa004&gt;] ? mdt_reint_internal+0x6d4/0x9f0 [mdt] [&lt;ffffffffa0bef9f6&gt;] ? mdt_reint_opcode+0x96/0x160 [mdt] [&lt;ffffffffa0bfa36c&gt;] ? mdt_reint+0x4c/0x120 [mdt] [&lt;ffffffffa07721b8&gt;] ? lustre_msg_check_version+0xc8/0xe0 [ptlrpc] [&lt;ffffffffa0bedc65&gt;] ? mdt_handle_common+0x8d5/0x1810 [mdt] [&lt;ffffffffa076fe44&gt;] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc] [&lt;ffffffffa0beec75&gt;] ? mdt_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa0780b89&gt;] ? ptlrpc_main+0xbb9/0x1990 [ptlrpc] [&lt;ffffffffa077ffd0&gt;] ? ptlrpc_main+0x0/0x1990 [ptlrpc] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffffa077ffd0&gt;] ? ptlrpc_main+0x0/0x1990 [ptlrpc] [&lt;ffffffffa077ffd0&gt;] ? ptlrpc_main+0x0/0x1990 [ptlrpc] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>The backtrace looks normal (though the stack is a bit deep), but it reported schedule while atomic. Alex, could you give some hint?</p> <p>We hit this bug again last night. <br/> This time it was not during reboot as was described when we open this ticket.</p> <p>Here are what shown on console:</p> <p>Lustre: MGS: haven't heard from client af2ee3e4-cf7b-2e45-4650-a3f788979f14 (at 10.151.6.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811f27af000, cur 1344393811 expire 1344393661 last 1344393584^M<br/> Lustre: Skipped 277 previous similar messages^M<br/> Lustre: MGS: haven't heard from client 587217b0-8285-b0d5-1696-6665442381c8 (at 10.151.5.129@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811f57aa800, cur 1344393811 expire 1344393661 last 1344393584^M<br/> LustreError: 9602:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff881032b9d8c0^M<br/> BUG: scheduling while atomic: mdt_268/9115/0xffff8809^M<br/> BUG: unable to handle kernel paging request at 0000000181c1fa20^M<br/> IP: <span class="error">&#91;&lt;ffffffff81051ddd&gt;&#93;</span> task_rq_lock+0x4d/0xa0^M<br/> PGD 0 ^M<br/> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP ^M<br/> last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:03:00.0/0000:04:04.1/irq^M<br/> ^M<br/> Entering kdb (current=0xffff881227f13500, pid 0) on processor 10 Oops: (null)^M<br/> due to oops @ 0xffffffff81051ddd^M<br/> r15 = 0x0000000000000003 r14 = 0x0000000000015fc0 ^M<br/> r13 = 0xffff88095fc83978 r12 = 0xffff8811da902a80 ^M<br/> bp = 0xffff88095fc83940 bx = 0x0000000000015fc0 ^M<br/> r11 = 0x0000000000000028 r10 = 0x0000000000000000 ^M<br/> r9 = 0xf9ffdd6e00faa202 r8 = 0x0000000000000000 ^M<br/> ax = 0x0000000040010dc0 cx = 0xffff88095fc83a90 ^M<br/> dx = 0x0000000000000082 si = 0xffff88095fc83978 ^M<br/> di = 0xffff8811da902a80 orig_ax = 0xffffffffffffffff ^M<br/> ip = 0xffffffff81051ddd cs = 0x0000000000000010 ^M<br/> flags = 0x0000000000010082 sp = 0xffff88095fc83910 ^M<br/> ss = 0x0000000000000018 &amp;regs = 0xffff88095fc83878^M<br/> <span class="error">&#91;10&#93;</span>kdb&gt; </p> <p>The crash analysis showed the stack trace of the task that caused the crash:</p> <p>&#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff88095fc63ae8&#93;</span> oops_begin at ffffffff81524f1e<br/> #7 <span class="error">&#91;ffff88095fc63b00&#93;</span> no_context at ffffffff8104230c<br/> #8 <span class="error">&#91;ffff88095fc63b50&#93;</span> __bad_area_nosemaphore at ffffffff81042615<br/> #9 <span class="error">&#91;ffff88095fc63ba0&#93;</span> bad_area_nosemaphore at ffffffff810426e3<br/> #10 <span class="error">&#91;ffff88095fc63bb0&#93;</span> __do_page_fault at ffffffff81042d9d<br/> #11 <span class="error">&#91;ffff88095fc63cd0&#93;</span> do_page_fault at ffffffff81526e5e<br/> #12 <span class="error">&#91;ffff88095fc63d00&#93;</span> page_fault at ffffffff81524175<br/> <span class="error">&#91;exception RIP: update_curr+324&#93;</span><br/> RIP: ffffffff81051974 RSP: ffff88095fc63db8 RFLAGS: 00010086<br/> RAX: ffff8811da902a80 RBX: 0000000040010dc0 RCX: ffff881227d103c0<br/> RDX: 0000000000018b48 RSI: 0000000000000000 RDI: 0000000000000001<br/> RBP: ffff88095fc63de8 R8: 0000000000000000 R9: 00000000001208e0<br/> R10: 0000000000000010 R11: 00000000001208e0 R12: ffff88095fc76028<br/> R13: 000001131ede6d18 R14: 00014b731e02665b R15: ffff8811da902a80<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> #13 <span class="error">&#91;ffff88095fc63df0&#93;</span> task_tick_fair at ffffffff81052bab<br/> #14 <span class="error">&#91;ffff88095fc63e20&#93;</span> scheduler_tick at ffffffff810568d1<br/> #15 <span class="error">&#91;ffff88095fc63e60&#93;</span> update_process_times at ffffffff8107b5e2<br/> #16 <span class="error">&#91;ffff88095fc63e90&#93;</span> tick_sched_timer at ffffffff8109ffe6<br/> #17 <span class="error">&#91;ffff88095fc63ec0&#93;</span> __run_hrtimer at ffffffff8109476e<br/> #18 <span class="error">&#91;ffff88095fc63f10&#93;</span> hrtimer_interrupt at ffffffff81094b16<br/> #19 <span class="error">&#91;ffff88095fc63f90&#93;</span> smp_apic_timer_interrupt at ffffffff815297cb<br/> #20 <span class="error">&#91;ffff88095fc63fb0&#93;</span> apic_timer_interrupt at ffffffff8100bc13<br/> &#8212; &lt;IRQ stack&gt; &#8212;<br/> #21 <span class="error">&#91;ffff8811da8aa6a8&#93;</span> apic_timer_interrupt at ffffffff8100bc13<br/> <span class="error">&#91;exception RIP: vprintk+465&#93;</span><br/> RIP: ffffffff81069c61 RSP: ffff8811da8aa750 RFLAGS: 00000246<br/> RAX: 0000000000010e28 RBX: ffff8811da8aa7e0 RCX: 0000000000001b45<br/> RDX: ffff88095fc60000 RSI: 0000000000000046 RDI: 0000000000000246<br/> RBP: ffffffff8100bc0e R8: ffffffff81ba2580 R9: 0000000000000000<br/> R10: 0000000000000007 R11: 000000000000000f R12: 00000000000b29d3<br/> R13: ffffffff81068fa5 R14: ffff8811da8aa6e0 R15: 0000000000000046<br/> ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018<br/> #22 <span class="error">&#91;ffff8811da8aa7e8&#93;</span> printk at ffffffff81520d83<br/> #23 <span class="error">&#91;ffff8811da8aa848&#93;</span> __schedule_bug at ffffffff81056681<br/> #24 <span class="error">&#91;ffff8811da8aa868&#93;</span> thread_return at ffffffff81521978<br/> #25 <span class="error">&#91;ffff8811da8aa928&#93;</span> io_schedule at ffffffff81521b13<br/> #26 <span class="error">&#91;ffff8811da8aa948&#93;</span> sync_buffer at ffffffff811a7cd0<br/> #27 <span class="error">&#91;ffff8811da8aa958&#93;</span> __wait_on_bit at ffffffff815224cf<br/> #28 <span class="error">&#91;ffff8811da8aa9a8&#93;</span> out_of_line_wait_on_bit at ffffffff81522578<br/> #29 <span class="error">&#91;ffff8811da8aaa18&#93;</span> __wait_on_buffer at ffffffff811a7c86<br/> #30 <span class="error">&#91;ffff8811da8aaa28&#93;</span> ldiskfs_mb_init_cache at ffffffffa0b4f63c <span class="error">&#91;ldiskfs&#93;</span><br/> #31 <span class="error">&#91;ffff8811da8aab08&#93;</span> ldiskfs_mb_init_group at ffffffffa0b4ff3e <span class="error">&#91;ldiskfs&#93;</span><br/> #32 <span class="error">&#91;ffff8811da8aab58&#93;</span> ldiskfs_mb_good_group at ffffffffa0b500fd <span class="error">&#91;ldiskfs&#93;</span><br/> #33 <span class="error">&#91;ffff8811da8aab98&#93;</span> ldiskfs_mb_regular_allocator at ffffffffa0b530bb <span class="error">&#91;ldiskfs&#93;</span><br/> #34 <span class="error">&#91;ffff8811da8aac48&#93;</span> ldiskfs_mb_new_blocks at ffffffffa0b53762 <span class="error">&#91;ldiskfs&#93;</span><br/> #35 <span class="error">&#91;ffff8811da8aace8&#93;</span> ldiskfs_ext_get_blocks at ffffffffa0b3d3b4 <span class="error">&#91;ldiskfs&#93;</span><br/> #36 <span class="error">&#91;ffff8811da8aae58&#93;</span> ldiskfs_get_blocks at ffffffffa0b45665 <span class="error">&#91;ldiskfs&#93;</span><br/> #37 <span class="error">&#91;ffff8811da8aaed8&#93;</span> ldiskfs_getblk at ffffffffa0b4a2b9 <span class="error">&#91;ldiskfs&#93;</span><br/> #38 <span class="error">&#91;ffff8811da8aaf98&#93;</span> ldiskfs_bread at ffffffffa0b4a448 <span class="error">&#91;ldiskfs&#93;</span><br/> #39 <span class="error">&#91;ffff8811da8aafc8&#93;</span> fsfilt_ldiskfs_write_handle at ffffffffa0bf72a7 <span class="error">&#91;fsfilt_ldiskfs&#93;</span><br/> #40 <span class="error">&#91;ffff8811da8ab078&#93;</span> fsfilt_ldiskfs_write_record at ffffffffa0bf755c <span class="error">&#91;fsfilt_ldiskfs&#93;</span><br/> #41 <span class="error">&#91;ffff8811da8ab0f8&#93;</span> llog_lvfs_write_blob at ffffffffa06c32ec <span class="error">&#91;obdclass&#93;</span><br/> #42 <span class="error">&#91;ffff8811da8ab1a8&#93;</span> llog_lvfs_write_rec at ffffffffa06c502e <span class="error">&#91;obdclass&#93;</span><br/> #43 <span class="error">&#91;ffff8811da8ab288&#93;</span> llog_cat_add_rec at ffffffffa06c1a65 <span class="error">&#91;obdclass&#93;</span><br/> #44 <span class="error">&#91;ffff8811da8ab308&#93;</span> llog_obd_origin_add at ffffffffa06c8306 <span class="error">&#91;obdclass&#93;</span><br/> #45 <span class="error">&#91;ffff8811da8ab368&#93;</span> llog_add at ffffffffa06c84e1 <span class="error">&#91;obdclass&#93;</span><br/> #46 <span class="error">&#91;ffff8811da8ab3d8&#93;</span> lov_llog_origin_add at ffffffffa09f60cc <span class="error">&#91;lov&#93;</span><br/> #47 <span class="error">&#91;ffff8811da8ab488&#93;</span> llog_add at ffffffffa06c84e1 <span class="error">&#91;obdclass&#93;</span><br/> #48 <span class="error">&#91;ffff8811da8ab4f8&#93;</span> mds_llog_origin_add at ffffffffa0c1349e <span class="error">&#91;mds&#93;</span><br/> #49 <span class="error">&#91;ffff8811da8ab578&#93;</span> llog_add at ffffffffa06c84e1 <span class="error">&#91;obdclass&#93;</span><br/> #50 <span class="error">&#91;ffff8811da8ab5e8&#93;</span> mds_llog_add_unlink at ffffffffa0c13b12 <span class="error">&#91;mds&#93;</span><br/> #51 <span class="error">&#91;ffff8811da8ab668&#93;</span> mds_log_op_unlink at ffffffffa0c14206 <span class="error">&#91;mds&#93;</span><br/> #52 <span class="error">&#91;ffff8811da8ab6f8&#93;</span> mdd_unlink_log at ffffffffa0c3c7de <span class="error">&#91;mdd&#93;</span><br/> #53 <span class="error">&#91;ffff8811da8ab758&#93;</span> mdd_object_kill at ffffffffa0c30a8e <span class="error">&#91;mdd&#93;</span><br/> #54 <span class="error">&#91;ffff8811da8ab7b8&#93;</span> mdd_finish_unlink at ffffffffa0c4a8ce <span class="error">&#91;mdd&#93;</span><br/> #55 <span class="error">&#91;ffff8811da8ab838&#93;</span> mdd_rename at ffffffffa0c5604c <span class="error">&#91;mdd&#93;</span><br/> #56 <span class="error">&#91;ffff8811da8ab9b8&#93;</span> cml_rename at ffffffffa0d1ed3a <span class="error">&#91;cmm&#93;</span><br/> #57 <span class="error">&#91;ffff8811da8aba68&#93;</span> mdt_reint_rename at ffffffffa0ca8426 <span class="error">&#91;mdt&#93;</span><br/> #58 <span class="error">&#91;ffff8811da8abbc8&#93;</span> mdt_reint_rec at ffffffffa0ca0c0f <span class="error">&#91;mdt&#93;</span><br/> #59 <span class="error">&#91;ffff8811da8abc18&#93;</span> mdt_reint_internal at ffffffffa0c99004 <span class="error">&#91;mdt&#93;</span><br/> #60 <span class="error">&#91;ffff8811da8abca8&#93;</span> mdt_reint at ffffffffa0c9936c <span class="error">&#91;mdt&#93;</span><br/> #61 <span class="error">&#91;ffff8811da8abcf8&#93;</span> mdt_handle_common at ffffffffa0c8cc65 <span class="error">&#91;mdt&#93;</span><br/> #62 <span class="error">&#91;ffff8811da8abd78&#93;</span> mdt_regular_handle at ffffffffa0c8dc75 <span class="error">&#91;mdt&#93;</span><br/> #63 <span class="error">&#91;ffff8811da8abd88&#93;</span> ptlrpc_main at ffffffffa080bd49 <span class="error">&#91;ptlrpc&#93;</span><br/> #64 <span class="error">&#91;ffff8811da8abf48&#93;</span> kernel_thread at ffffffff8100c14a</p> <p>Our lustre git source is at <br/> <a href="https://github.com/jlan/lustre-nas/commits/nas-2.1.2" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/commits/nas-2.1.2</a></p> <p>The original report was built on tag 2.1.2-1nasS. The crash last night was built on tag 2.1.2-2nasS.</p> <p>Can we modify the summary to "BUG: scheduling while atomic"? The current summary does not describe the crash of last night.</p> <p>We hit this again. Can we increase this to critical</p> <p>There is a log message reporting -ENOSPC:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 9602:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff881032b9d8c0^M </pre> </div></div> <p>Could you verify it's true on MDS?</p> <p>I saw the -ENOSPC LustreError immediately before the crash. However, when I analyzed the vmcore with crash, "kmem -i" showed MDS had plenty of memory for the system:</p> <p> PAGES TOTAL PERCENTAGE<br/> TOTAL MEM 18494890 70.6 GB ----<br/> FREE 11790336 45 GB 63% of TOTAL MEM<br/> USED 6704554 25.6 GB 36% of TOTAL MEM<br/> SHARED 2663959 10.2 GB 14% of TOTAL MEM<br/> BUFFERS 2603900 9.9 GB 14% of TOTAL MEM<br/> CACHED 119807 468 MB 0% of TOTAL MEM<br/> SLAB 2185165 8.3 GB 11% of TOTAL MEM</p> <p>TOTAL SWAP 500013 1.9 GB ----<br/> SWAP USED 0 0 0% of TOTAL SWAP<br/> SWAP FREE 500013 1.9 GB 100% of TOTAL SWAP</p> <p>Did we run out of some preallocated memory? Hmmm...</p> <p>On the other hand, -ENOSPC (or -28) was not reported in the crash last night (the 2nd crash in two consecutive nights.) So, it should not be the reason.</p> <p>The problem at hand seems to be stack overflow, so patches from lu-969 should help</p> <p>so the three patches you need are:<br/> <a href="http://review.whamcloud.com/#change,2668" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2668</a><br/> then <a href="http://review.whamcloud.com/3034" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3034</a><br/> and then <a href="http://review.whamcloud.com/3072" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3072</a></p> <p>Following up on the action in the meeting earlier. I spoke to Oleg about whether it was possible to predict when stack overflows would occur and he confirmed that it was a combination of independent factors that resulted in a stack overflow so it was not possible to anticipate ahead of time.</p> <p>Oleg, how do I decide it was a stack overflow?</p> <p>PID: 9115 TASK: ffff8811da902a80 CPU: 9 COMMAND: "mdt_268"</p> <p>"bt" command showed the stack started at <br/> #64 <span class="error">&#91;ffff8811da8abf48&#93;</span> kernel_thread at ffffffff8100c14a</p> <p>and,</p> <p>struct task_struct {<br/> state = 2, <br/> stack = 0xffff8811da8aa000, </p> <p>The "RSP: ffff8811da8aa750" when interrupted, so I assumed it did not overflow<br/> the per cpu stack yet?</p> <p>And, when the PageFault EXCEPTION hit the IRQ stack, <br/> the "RSP: ffff88095fc63db8" looks still within the IRQ stack?</p> <p>How do I determine I have had a stack overflow? Is there a command in "crash"<br/> to display stack sizes being used? Thanks!</p> <p>Thanks!</p> <p>The stacktrace is pretty long, as you can see, and that's not all of it, there was more as we dipped into device driver, and that's what stepped on the struct thread_info enough to just overwrite the flags but not spill over too much into other spaces.<br/> And those drivers might be mighty fat in their stack usage too. (heck, even sync_buffer to schedule bug shaved 0x100 bytes off the stack and that's mere 3 functions in between).<br/> there are a bit less than 0x6a8 bytes left (the interrupt stack is in a different place so does not play a role).</p> <p>also note the crash happened in task_rq_lock in the interrupt after the warnign was already printed by the so happened interrupt. the task rq lock happened to be garbage already which further reinforces this theory.<br/> We have seen many of cashes like this before.</p> <p>There is no command in crash to catch a stack overflow that managed not to crash while in overflow and return back inside of the stack but merely corrupted thread info before retracting back, but you can dump struct_thread_info content and see a lot of it is garbage, I guess.</p> <p>Since the 3 patches Oleg suggested on Aug 9 were quite extensive, would it be OK if we just apply the patches to MDS only? Would the patches require OSS'es also run the same set of patches?</p> <p>You can apply them to MDS only if that's where you experience the problem, but they would be beneficial everywhere, we have seen stack overflows on clients too, usually when reexporting nfs.</p> <p>BTW, I now have combined patch instead of the three I referenced, if that makes your life easier: <a href="http://review.whamcloud.com/3623" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3623</a></p> <p>We have run with patch of <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> (commit 2d77b00).<br/> Peter indicated 2.1.3 release contains a complete solution. In that case,<br/> this ticket can be closed for duplicate of <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a>.</p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a></p> Development Rank 1|hzvgs7: Rank (Obsolete) 6368 Severity