https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3721 Lustre <p>System crash after reboot/remount of osts. We have crash dump and can upload if needed.</p> <p>ustre: nbp7-OST002b: deleting orphan objects from 0x0:131250 to 0x0:131297<br/> Lustre: Skipped 3 previous similar messages<br/> Lustre: nbp7-OST0037: deleting orphan objects from 0x0:131412 to 0x0:131457<br/> Lustre: nbp7-OST0027: deleting orphan objects from 0x0:131476 to 0x0:131585<br/> -----------<del>[ cut here ]</del>-----------<br/> WARNING: at mm/page_alloc.c:1361 get_page_from_freelist+0x818/0x830() (Tainted: G --------------- T)<br/> -----------<del>[ cut here ]</del>-----------<br/> WARNING: at mm/page_alloc.c:1361 get_page_from_freelist+0x818/0x830() (Tainted: G --------------- T)<br/> Hardware name: SUMMIT<br/> Modules linked in: osp(U) ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) jbd2 mdd(U) acpi_cpufreq freq_table mperf lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) dm_round_robin scsi_dh_rdac lpfc(U) scsi_transport_fc nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc bonding 8021q garp stp llc ib_ucm(U) rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_srp(U) scsi_transport_srp scsi_tgt ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_mad(U) ib_core(U) tcp_bic microcode sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support mlx4_core(U) memtrack(U) igb dca shpchp ext3 jbd isci libsas sd_mod crc_t10dif mpt2sas scsi_transport_sas raid_class ahci wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod gr3u1 o<span class="error">&#91;luat sto f un3l2 oacpdueds :i nsc ksidb_w,a witai_sticanng &#93;</span>f<br/> or the rest, timeout in 10 second(s)<br/> Pid: 24478, comm: ll_ost_io03_003 Tainted: G --------------- T 2.6.32-358.6.2.el6.20130607.x86_64.lustre240 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff8106d607&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> <span class="error">&#91;&lt;ffffffff8106d65a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> <span class="error">&#91;&lt;ffffffff81126ee8&gt;&#93;</span> ? get_page_from_freelist+0x818/0x830<br/> <span class="error">&#91;&lt;ffffffff81128303&gt;&#93;</span> ? __alloc_pages_nodemask+0x113/0x8d0<br/> <span class="error">&#91;&lt;ffffffff81128303&gt;&#93;</span> ? __alloc_pages_nodemask+0x113/0x8d0<br/> <span class="error">&#91;&lt;ffffffff81124bc6&gt;&#93;</span> ? __rmqueue+0x156/0x490<br/> <span class="error">&#91;&lt;ffffffff81163302&gt;&#93;</span> ? kmem_getpages+0x62/0x170<br/> <span class="error">&#91;&lt;ffffffff8116396f&gt;&#93;</span> ? cache_grow+0x2cf/0x320<br/> <span class="error">&#91;&lt;ffffffff81163bc2&gt;&#93;</span> ? cache_alloc_refill+0x202/0x240<br/> <span class="error">&#91;&lt;ffffffff811be01d&gt;&#93;</span> ? bio_integrity_prep+0x7d/0x320<br/> <span class="error">&#91;&lt;ffffffff81164a89&gt;&#93;</span> ? __kmalloc+0x1a9/0x220<br/> <span class="error">&#91;&lt;ffffffff811be01d&gt;&#93;</span> ? bio_integrity_prep+0x7d/0x320<br/> <span class="error">&#91;&lt;ffffffff81266ca0&gt;&#93;</span> ? generic_make_request+0x2b0/0x550<br/> <span class="error">&#91;&lt;ffffffff81118de3&gt;&#93;</span> ? mempool_alloc+0x63/0x140<br/> <span class="error">&#91;&lt;ffffffffa0015f67&gt;&#93;</span> ? dm_merge_bvec+0xc7/0x100 <span class="error">&#91;dm_mod&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81266fcd&gt;&#93;</span> ? submit_bio+0x8d/0x120<br/> <span class="error">&#91;&lt;ffffffffa05b238e&gt;&#93;</span> ? lprocfs_oh_tally+0x2e/0x50 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c37dac&gt;&#93;</span> ? osd_submit_bio+0x1c/0x60 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c38581&gt;&#93;</span> ? osd_do_bio+0x791/0x810 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03c802c&gt;&#93;</span> ? fsfilt_map_nblocks+0xcc/0xf0 <span class="error">&#91;fsfilt_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03c82d5&gt;&#93;</span> ? fsfilt_ldiskfs_map_inode_pages+0x85/0x90 <span class="error">&#91;fsfilt_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c3af68&gt;&#93;</span> ? osd_write_commit+0x328/0x610 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0cd7cc4&gt;&#93;</span> ? ofd_commitrw_write+0x684/0x11b0 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0cdaa2d&gt;&#93;</span> ? ofd_commitrw+0x5cd/0xbb0 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02817e5&gt;&#93;</span> ? lprocfs_counter_add+0x125/0x182 <span class="error">&#91;lvfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c9a1d8&gt;&#93;</span> ? obd_commitrw+0x128/0x3d0 <span class="error">&#91;ost&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ca40d1&gt;&#93;</span> ? ost_brw_write+0xea1/0x15d0 <span class="error">&#91;ost&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81128303&gt;&#93;</span> ? __alloc_pages_nodemask+0x113/0x8d0<br/> <span class="error">&#91;&lt;ffffffffa07301c0&gt;&#93;</span> ? target_bulk_timeout+0x0/0xc0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0caa32b&gt;&#93;</span> ? ost_handle+0x3ecb/0x48e0 <span class="error">&#91;ost&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0777bab&gt;&#93;</span> ? ptlrpc_update_export_timer+0x4b/0x560 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0780388&gt;&#93;</span> ? ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa048ab60&gt;&#93;</span> ? cfs_alloc+0x30/0x60 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa028154f&gt;&#93;</span> ? lprocfs_stats_alloc_one+0x36f/0x390 <span class="error">&#91;lvfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa078171e&gt;&#93;</span> ? ptlrpc_main+0xace/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0780c50&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> ? child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa0780c50&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0780c50&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> --<del>[ end trace b1c8704e155f856a ]</del>--<br/> BUG: unable to handle kernel NULL pointer dereference at 0000000000000036<br/> IP: <span class="error">&#91;&lt;ffffffffa0c3af3f&gt;&#93;</span> osd_write_commit+0x2ff/0x610 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> PGD 0 <br/> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP <br/> last sysfs file: /sys/devices/pci0000:80/0000:80:03.0/0000:86:00.1/host16/rport-16:0-0/target16:0:0/16:0:0:72/state<br/> .All cpus are now in kdb</p> <p>Entering kdb (current=0xffff881f9c5d0040, pid 24479) on processor 30 Oops: (null)<br/> due to oops @ 0xffffffffa0c3af3f<br/> r15 = 0x0000000000000100 r14 = 0x0000000000000100 <br/> r13 = 0xffff881f99ad3000 r12 = 0xffff880f1eac3a10 <br/> bp = 0xffff881f99acb940 bx = 0x0000000000400000 <br/> r11 = 0x0000000000000000 r10 = 0xffff881f99ac8f28 <br/> r9 = 0x0000000000000000 r8 = 0x0000000000000001 <br/> ax = 0xfffffffffffffffe cx = 0xffff881f99947000 <br/> dx = 0x0000000000000100 si = 0xffff881f99bdd800 <br/> di = 0xffff880f1eac3a10 orig_ax = 0xffffffffffffffff <br/> ip = 0xffffffffa0c3af3f cs = 0x0000000000000010 <br/> flags = 0x0000000000010246 sp = 0xffff881f99acb8d0 <br/> ss = 0x0000000000000018 &amp;regs = 0xffff881f99acb838</p> LU-3721

kernel NULL pointer dereference at 0000000000000036

Bug Critical Resolved Duplicate Hongchao Zhang Mahmoud Hanafi Wed, 7 Aug 2013 17:56:14 +0000 Thu, 14 Nov 2013 21:00:44 +0000 Thu, 14 Nov 2013 21:00:44 +0000 Lustre 2.4.0 0 5 <p>Our source is at <a href="https://github.com/jlan/lustre-nas" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas</a><br/> Branch nas-2.4.0 at tag 7dba42f is the source I used to build the servers.</p> <p>Hongchao</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>could you please print the line # at osd_write_commit+0x2ff/0x610. </p> <p>The WARNING is that there was a page alloc request with order &gt; 1, but GFP_NOWARN. It is possible that the page allocation failed and the error handling is not expecting this?</p> <p>It crashed at line 842 of lustre/osd-ldiskfs/osd_io.c:</p> <p>static int osd_write_commit(const struct lu_env ...<br/> ...<br/> if (OBD_FAIL_CHECK(OBD_FAIL_OST_MAPBLK_ENOSPC)) </p> { rc = -ENOSPC; } <p> else if (iobuf-&gt;dr_npages &gt; 0) </p> { rc = osd-&gt;od_fsops-&gt;fs_map_inode_pages(inode, iobuf-&gt;dr_pages, iobuf-&gt;dr_npages, iobuf-&gt;dr_blocks, 1, NULL); } <p> else {<br/> ...</p> <p>It was this line:<br/> rc = osd-&gt;od_fsops-&gt;fs_map_inode_pages(inode, iobuf-&gt;dr_pages,</p> <p>/usr/src/redhat/BUILD/lustre-2.4.0/lustre/osd-ldiskfs/osd_io.c:842<br/> 39aa: 48 8b 7d c0 mov -0x40(%rbp),%rdi<br/> 39ae: 48 8b 8e 28 0c 00 00 mov 0xc28(%rsi),%rcx<br/> 39b5: 45 31 c9 xor %r9d,%r9d<br/> 39b8: 48 8b b6 10 0c 00 00 mov 0xc10(%rsi),%rsi<br/> 39bf: 41 b8 01 00 00 00 mov $0x1,%r8d<br/> 39c5: 48 8b 87 40 01 00 00 mov 0x140(%rdi),%rax<br/> 39cc: 4c 89 e7 mov %r12,%rdi<br/> 39cf: ff 50 38 callq *0x38(%rax)</p> <p>where the content of rax was fffffffffffffffe</p> <p> RIP: ffffffffa0c3af3f RSP: ffff881f99acb8e0 RFLAGS: 00010246<br/> RAX: fffffffffffffffe RBX: 0000000000400000 RCX: ffff881f99947000<br/> RDX: 0000000000000100 RSI: ffff881f99bdd800 RDI: ffff880f1eac3a10<br/> RBP: ffff881f99acb940 R8: 0000000000000001 R9: 0000000000000000<br/> R10: ffff881f99ac8f28 R11: 0000000000000000 R12: ffff880f1eac3a10<br/> R13: ffff881f99ad3000 R14: 0000000000000100 R15: 0000000000000100<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018</p> <p>Andreas, there was plenty of memory at the system:<br/> crash&gt; kmem -i<br/> PAGES TOTAL PERCENTAGE<br/> TOTAL MEM 32962711 125.7 GB ----<br/> FREE 31057486 118.5 GB 94% of TOTAL MEM<br/> USED 1905225 7.3 GB 5% of TOTAL MEM<br/> SHARED 64520 252 MB 0% of TOTAL MEM<br/> BUFFERS 59339 231.8 MB 0% of TOTAL MEM<br/> CACHED 70083 273.8 MB 0% of TOTAL MEM<br/> SLAB 329459 1.3 GB 0% of TOTAL MEM</p> <p>TOTAL SWAP 500013 1.9 GB ----<br/> SWAP USED 0 0 0% of TOTAL SWAP<br/> SWAP FREE 500013 1.9 GB 100% of TOTAL SWAP<br/> crash&gt; </p> <p>this should be fixed in <a href="https://jira.whamcloud.com/browse/LU-3411" title="Encountered at NULL pointer exception for function osd_read_prep" class="issue-link" data-issue-key="LU-3411"><del>LU-3411</del></a> (the patch is <a href="http://review.whamcloud.com/#/c/6496/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6496/</a>, which is not landed on <a href="https://github.com/jlan/lustre-nas" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas</a>)</p> <p>Yep! Looks like it! Crash at the same place.<br/> I will pick up this patch. Thanks!</p> <p>Hi Jay,<br/> what is the result with the patch <a href="http://review.whamcloud.com/#/c/6496/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6496/</a>, thanks!</p> <p>Patch applied and we haven't see this issue. Please close case</p> <p>ok - thanks Mahmoud.</p> Development Rank 1|hzvxdb: Rank (Obsolete) 9581 Severity