[LU-6326] list_del corruption causes MDS crash

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.5.3
• Type: Bug
• Priority: Major
• Reporter: Sebastien Buisson (Inactive)
• Assignee: Bruno Faccini (Inactive)
• Resolution: Cannot Reproduce
• Severity: 3

Description

Hi,

When trying to umount an MDT, we hit the following bug leading to the crash of the server:

<4>------------[ cut here ]------------
<4>WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Not tainted)
<4>Hardware name: bullx
<4>list_del corruption. prev->next should be ffff88030b092a90, but was 303054534f2d7077
<4>Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc( U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipmi_devintf ipmi_si ipmi_msghandler acpi_cpufreq freq_table mperf rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) i pv6 ib_uverbs(U) ib_umad(U) mlx5_ib(U) mlx5_core(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin scsi_dh_emc dm_ multipath mic(U) uinput raid0 ses enclosure serio_raw compat(U) cxgb3 mdio lpfc scsi_transport_fc scsi_tgt igb i2c_algo_bit i2c_core ptp pps_core sg lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom aacraid ahci usb_storage ata_generic pata_jmicron dm_mirror dm _region_hash dm_log dm_mod megaraid_sas [last unloaded: scsi_wait_scan]
<4>Pid: 24459, comm: umount Not tainted 2.6.32-504.8.1.el6.Bull.70.x86_64 #1
<4>Call Trace:
<4> [<ffffffff81074df7>] ? warn_slowpath_common+0x87/0xc0
<4> [<ffffffff81074ee6>] ? warn_slowpath_fmt+0x46/0x50
<4> [<ffffffff8129f0de>] ? list_del+0x6e/0xa0
<4> [<ffffffff811a8372>] ? d_kill+0x22/0x60
<4> [<ffffffff811a8716>] ? __shrink_dcache_sb+0x366/0x3c0
<4> [<ffffffff811a8962>] ? shrink_dcache_sb+0x12/0x20
<4> [<ffffffffa0e53f76>] ? osd_umount+0x56/0x150 [osd_ldiskfs]
<4> [<ffffffffa0e5b2f7>] ? osd_device_fini+0x147/0x190 [osd_ldiskfs]
<4> [<ffffffffa0745a73>] ? class_cleanup+0x573/0xd30 [obdclass]
<4> [<ffffffffa071c0a6>] ? class_name2dev+0x56/0xe0 [obdclass]
<4> [<ffffffffa074779a>] ? class_process_config+0x156a/0x1ad0 [obdclass]
<4> [<ffffffffa07408f3>] ? lustre_cfg_new+0x2d3/0x6e0 [obdclass]
<4> [<ffffffffa0747e79>] ? class_manual_cleanup+0x179/0x6f0 [obdclass]
<4> [<ffffffffa071a57b>] ? class_export_put+0x10b/0x2c0 [obdclass]
<4> [<ffffffffa0e5d8d5>] ? osd_obd_disconnect+0x1c5/0x1d0 [osd_ldiskfs]
<4> [<ffffffffa074a41b>] ? lustre_put_lsi+0x1ab/0x11a0 [obdclass]
<4> [<ffffffffa07529c8>] ? lustre_common_put_super+0x5d8/0xbf0 [obdclass]
<4> [<ffffffffa07830dd>] ? server_put_super+0x1bd/0xf60 [obdclass]
<4> [<ffffffff811909bb>] ? generic_shutdown_super+0x5b/0xe0
<4> [<ffffffff81190aa6>] ? kill_anon_super+0x16/0x60
<4> [<ffffffffa0749d26>] ? lustre_kill_super+0x36/0x60 [obdclass]
<4> [<ffffffff81191247>] ? deactivate_super+0x57/0x80
<4> [<ffffffff811b0e7f>] ? mntput_no_expire+0xbf/0x110
<4> [<ffffffff811b19cb>] ? sys_umount+0x7b/0x3a0
<4> [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
<4>---[ end trace 08665e30da4e39f3 ]---
<4>------------[ cut here ]------------

Here is the backtrace of the faulty process:

crash> bt
PID: 24459 TASK: ffff880338436040 CPU: 1 COMMAND: "umount"
 #0 [ffff8800282239c0] machine_kexec at ffffffff8103b71b
 #1 [ffff880028223a20] crash_kexec at ffffffff810c9852
 #2 [ffff880028223af0] oops_end at ffffffff8152ec30
 #3 [ffff880028223b20] no_context at ffffffff8104c80b
 #4 [ffff880028223b70] __bad_area_nosemaphore at ffffffff8104ca95
 #5 [ffff880028223bc0] bad_area_nosemaphore at ffffffff8104cb63
 #6 [ffff880028223bd0] __do_page_fault at ffffffff8104d2bf
 #7 [ffff880028223cf0] do_page_fault at ffffffff81530b7e
 #8 [ffff880028223d20] page_fault at ffffffff8152df35
    [exception RIP: _spin_lock+14]
    RIP: ffffffff8152da1e RSP: ffff880028223dd0 RFLAGS: 00010002
    RAX: 0000000000010000 RBX: 0000000000006450 RCX: 000000000019ae76
    RDX: 0000000000005444 RSI: ffff88030b092740 RDI: 0000000000000040
    RBP: ffff880028223dd0 R8: ffff88002822d140 R9: 0001407b60dde621
    R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000282
    R13: ffff88030b092740 R14: ffff88063cd50480 R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff880028223dd8] kmem_cache_free at ffffffff81175b0f
#10 [ffff880028223e48] __d_free at ffffffff811a82cf
#11 [ffff880028223e68] d_callback at ffffffff811a8c12
#12 [ffff880028223e78] __rcu_process_callbacks at ffffffff810f05f5
#13 [ffff880028223ec8] rcu_process_callbacks at ffffffff810f083b
#14 [ffff880028223ed8] __do_softirq at ffffffff8107d8b1
#15 [ffff880028223f48] call_softirq at ffffffff8100c30c
#16 [ffff880028223f60] do_softirq at ffffffff8100fb55
#17 [ffff880028223f80] irq_exit at ffffffff8107d765
#18 [ffff880028223f90] smp_apic_timer_interrupt at ffffffff815347ca
#19 [ffff880028223fb0] apic_timer_interrupt at ffffffff8100bb93
--- <IRQ stack> ---
#20 [ffff8802326db798] apic_timer_interrupt at ffffffff8100bb93
    [exception RIP: vprintk+593]
    RIP: ffffffff81075d51 RSP: ffff8802326db848 RFLAGS: 00000246
    RAX: 0000000000010500 RBX: ffff8802326db8d8 RCX: 00000000000049b5
    RDX: ffff880028220000 RSI: 0000000000000046 RDI: 0000000000000246
    RBP: ffffffff8100bb8e R8: 0000000000000000 R9: ffffffff81649020
    R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff81074f95
    R13: ffff8802326db7d8 R14: 0000000000000046 R15: 0000000000000025
    ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018
#21 [ffff8802326db8e0] printk at ffffffff81529f80
#22 [ffff8802326db940] warn_slowpath_common at ffffffff81074dae
#23 [ffff8802326db980] warn_slowpath_fmt at ffffffff81074ee6
#24 [ffff8802326db9e0] list_del at ffffffff8129f0de
#25 [ffff8802326dba00] d_kill at ffffffff811a8372
#26 [ffff8802326dba20] __shrink_dcache_sb at ffffffff811a8716
#27 [ffff8802326dbac0] shrink_dcache_sb at ffffffff811a8962
#28 [ffff8802326dbad0] osd_umount at ffffffffa0e53f76 [osd_ldiskfs]
#29 [ffff8802326dbaf0] osd_device_fini at ffffffffa0e5b2f7 [osd_ldiskfs]
#30 [ffff8802326dbb10] class_cleanup at ffffffffa0745a73 [obdclass]
#31 [ffff8802326dbb90] class_process_config at ffffffffa074779a [obdclass]
#32 [ffff8802326dbc20] class_manual_cleanup at ffffffffa0747e79 [obdclass]
#33 [ffff8802326dbce0] osd_obd_disconnect at ffffffffa0e5d8d5 [osd_ldiskfs]
#34 [ffff8802326dbd20] lustre_put_lsi at ffffffffa074a41b [obdclass]
#35 [ffff8802326dbd50] lustre_common_put_super at ffffffffa07529c8 [obdclass]
#36 [ffff8802326dbdc0] server_put_super at ffffffffa07830dd [obdclass]
#37 [ffff8802326dbe30] generic_shutdown_super at ffffffff811909bb
#38 [ffff8802326dbe50] kill_anon_super at ffffffff81190aa6
#39 [ffff8802326dbe70] lustre_kill_super at ffffffffa0749d26 [obdclass]
#40 [ffff8802326dbe90] deactivate_super at ffffffff81191247
#41 [ffff8802326dbeb0] mntput_no_expire at ffffffff811b0e7f
#42 [ffff8802326dbee0] sys_umount at ffffffff811b19cb
#43 [ffff8802326dbf80] system_call_fastpath at ffffffff8100b072
    RIP: 00007f42273d0997 RSP: 00007fff3f71d3a8 RFLAGS: 00010202
    RAX: 00000000000000a6 RBX: ffffffff8100b072 RCX: 00000000fbad2498
    RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f4229e83be0
    RBP: 00007f4229e83bc0 R8: 00007f4229e83c00 R9: 0000000000000000
    R10: 00007fff3f71d140 R11: 0000000000000246 R12: 0000000000000000
    R13: 0000000000000000 R14: 0000000000000000 R15: 00007f4229e83c70
    ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b

Sebastien.

Comments

[Bruno Faccini (Inactive)]

Humm, the corrupted pointer value looks like something familiar when treated as an ascii string : "wp-OST00", which may likely be part of an OST target lwp/LightWeightProxy name...

Also, having a better look to the stack-trace of the panic'ing thread, it looks like we may have faced some unexpected reentry (due to RTC interrupt) inside dcache code that's likely to be the cause of this crash.

BTW, you did not mention if there is a crash-dump available or not?

[Sebastien Buisson (Inactive)]

Ah, nice catch Bruno, it seems you have the ASCII character set in mind

There is a crash-dump available, I will upload it ASAP.

Sebastien.

[Sebastien Buisson (Inactive)]

Bruno,

The crash dump has been upload on ftp.whamcloud.com, under /uploads/LU-6326: the file is LU-6326-crashMDS.tar.gz.

Sebastien.

[Bruno Faccini (Inactive)]

Seb, as usual I will need the kernel-debuginfo[-common] 2x RPMs (for vmlinux and Kernel modules symbols) and also the lustre-modules and/or lustre-debugingo RPM(s) (for the Lustre modules with symbols).

[Sebastien Buisson (Inactive)]

Bruno,

The following files have been pushed on the FTP server, in the same directory:
kernel_H-debuginfo-2.6.32-504.8.1.el6.Bull.70.x86_64.rpm
kernel_H-debuginfo-common-x86_64-2.6.32-504.8.1.el6.Bull.70.x86_64.rpm
lustre-modules_H-2.5.3-2.6.32_504.8.1.el6.Bull.70.x86_64_Bull.4.107.el6.20150221.x86_64.rpm
lustre-osd-ldiskfs_H-2.5.3-2.6.32_504.8.1.el6.Bull.70.x86_64_Bull.4.107.el6.20150221.x86_64.rpm
lustre_debuginfo_H-2.5.3-2.6.32_504.8.1.el6.Bull.70.x86_64_Bull.4.107.el6.20150221.x86_64.rpm

Please tell me if you need something else.

Sebastien.

[Bruno Faccini (Inactive)]

After spending some time to analyze crash-dump, I need to admit that my previous comment's presumption :
> Also, having a better look to the stack-trace of the panic'ing thread, it looks like we may have faced some unexpected reentry (due to
> RTC interrupt) inside dcache code that's likely to be the cause of this crash.
was somewhat wrong since the crash occurs because a RTC interrupt handling has caused a re-entry in d-cache code, but this has occurred at the time a dentry slab corruption has been discovered and was being reported/printed in list_del() routine !!

The corruption in a full dentry slab looks like :

ffff88030b092000:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092010:  303054534f2d7077 0a444955555f3430   wp-OST0004_UUID.
ffff88030b092020:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092030:  303054534f2d7077 0a444955555f3937   wp-OST0079_UUID.
ffff88030b092040:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092050:  303054534f2d7077 0a444955555f6230   wp-OST000b_UUID.
ffff88030b092060:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092070:  303054534f2d7077 0a444955555f3030   wp-OST0000_UUID.
ffff88030b092080:  4d2d3030332d7365 6c2d303030305444   es-300-MDT0000-l
ffff88030b092090:  303054534f2d7077 0a444955555f3635   wp-OST0056_UUID.
ffff88030b0920a0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0920b0:  303054534f2d7077 0a444955555f3139   wp-OST0091_UUID.
ffff88030b0920c0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0920d0:  303054534f2d7077 0a444955555f3934   wp-OST0049_UUID.
ffff88030b0920e0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0920f0:  303054534f2d7077 0a444955555f3135   wp-OST0051_UUID.
ffff88030b092100:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092110:  303054534f2d7077 0a444955555f3536   wp-OST0065_UUID.
ffff88030b092120:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092130:  303054534f2d7077 0a444955555f3931   wp-OST0019_UUID.
ffff88030b092140:  4d2d3030332d7365 6c2d303030305444   es-300-MDT0000-l
ffff88030b092150:  303054534f2d7077 0a444955555f6232   wp-OST002b_UUID.
ffff88030b092160:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092170:  303054534f2d7077 0a444955555f3632   wp-OST0026_UUID.
ffff88030b092180:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092190:  ffff8803286d4aa0 ffff880328776b90   .Jm(.....kw(....
ffff88030b0921a0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0921b0:  303054534f2d7077 0a444955555f3339   wp-OST0093_UUID.
ffff88030b0921c0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0921d0:  303054534f2d7077 0a444955555f3035   wp-OST0050_UUID.
ffff88030b0921e0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0921f0:  303054534f2d7077 0a444955555f6332   wp-OST002c_UUID.
ffff88030b092200:  4d2d3030332d7365 6c2d303030305444   es-300-MDT0000-l
ffff88030b092210:  303054534f2d7077 0a444955555f3234   wp-OST0042_UUID.
ffff88030b092220:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092230:  303054534f2d7077 0a444955555f3837   wp-OST0078_UUID.
ffff88030b092240:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092250:  303054534f2d7077 0a444955555f3535   wp-OST0055_UUID.
ffff88030b092260:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092270:  303054534f2d7077 0a444955555f3136   wp-OST0061_UUID.
ffff88030b092280:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b092290:  303054534f2d7077 0a444955555f3037   wp-OST0070_UUID.
ffff88030b0922a0:  4d2d3030332d7366 6c2d303030305444   fs-300-MDT0000-l
ffff88030b0922b0:  303054534f2d7077 0a444955555f3437   wp-OST0074_UUID.
ffff88030b0922c0:  0000000800000000 0000000000090009   ................
ffff88030b0922d0:  0000000000000000 0000000000000000   ................
ffff88030b0922e0:  ffffc9000097c450 ffff880338de2740   P.......@'.8....
ffff88030b0922f0:  000000158ece00d5 ffff88030b092360   ........`#......
ffff88030b092300:  ffff880228c197c0 ffff880228c19ac0   ...(.......(....
ffff88030b092310:  ffff880228c197d0 ffff880228c19ad0   ...(.......(....
ffff88030b092320:  ffff88030b092320 ffff88030b092320    #...... #......
ffff88030b092330:  ffff88030b092330 ffff88030b092330   0#......0#......
ffff88030b092340:  ffff88030b092338 0000000000000000   8#..............
ffff88030b092350:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092360:  6d6d6f635f6e7572 42676f6c5f646e61   run_command_logB
ffff88030b092370:  ffff004263505951 0000000000000000   QYPcB...........
ffff88030b092380:  0000001800000000 0000000000050005   ................
ffff88030b092390:  0000000000000000 0000000000000000   ................
ffff88030b0923a0:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b0923b0:  00000003002233be ffff88030b092420   .3"..... $......
ffff88030b0923c0:  ffff88030b0923c0 ffff88030b0923c0   .#.......#......
ffff88030b0923d0:  ffff88030b170550 ffffffff811a8c00   P...............
ffff88030b0923e0:  ffff88030b0923e0 ffff88030b0923e0   .#.......#......
ffff88030b0923f0:  ffff88030b0923f0 ffff88030b0923f0   .#.......#......
ffff88030b092400:  0000000000000000 0000000000000000   ................
ffff88030b092410:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b092420:  0030303000313164 003030007974696e   d11.000.nity.00.
ffff88030b092430:  ffff880315448000 0000000000000000   ..D.............
ffff88030b092440:  0000001800000000 0000000000050005   ................
ffff88030b092450:  0000000000000000 0000000000000000   ................
ffff88030b092460:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092470:  000000030022367e ffff88030b0924e0   ~6"......$......
ffff88030b092480:  ffff88030b092480 ffff88030b092480   .$.......$......
ffff88030b092490:  ffff88030b092b50 ffffffff811a8c00   P+..............
ffff88030b0924a0:  ffff88030b0924a0 ffff88030b0924a0   .$.......$......
ffff88030b0924b0:  ffff88030b0924b0 ffff88030b0924b0   .$.......$......
ffff88030b0924c0:  ffff88030b0924b8 0000000000000000   .$..............
ffff88030b0924d0:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b0924e0:  7461647400353164 0030300079740061   d15.tdata.ty.00.
ffff88030b0924f0:  0000000000001363 0000000000001000   c...............
ffff88030b092500:  0000001800000000 0000000000050005   ................
ffff88030b092510:  0000000000000000 0000000000000000   ................
ffff88030b092520:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092530:  000000030022330e ffff88030b0925a0   .3"......%......
ffff88030b092540:  ffff88030b092540 ffff88030b092540   @%......@%......
ffff88030b092550:  ffff88030b0923d0 ffffffff811a8c00   .#..............
ffff88030b092560:  ffff88030b092560 ffff88030b092560   `%......`%......
ffff88030b092570:  ffff88030b092570 ffff88030b092570   p%......p%......
ffff88030b092580:  0000000000000000 0000000000000000   ................
ffff88030b092590:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b0925a0:  3030303200303164 0073750079740030   d10.20000.ty.us.
ffff88030b0925b0:  ffff88030b092500 ffffea000aca6ee0   .%.......n......
ffff88030b0925c0:  0000000800000000 0000000000040004   ................
ffff88030b0925d0:  ffff8802397962d0 0000000000000000   .by9............
ffff88030b0925e0:  ffffc90001c02230 ffff8803286d4a40   0"......@Jm(....
ffff88030b0925f0:  0000000300223b4e ffff88030b092660   N;".....`&......
ffff88030b092600:  ffff8802326dba68 ffff88030b0336c0   h.m2.....6......
ffff88030b092610:  ffff88030b0920d0 ffff88030b0336d0   . .......6......
ffff88030b092620:  ffff88030b092620 ffff88030b092620    &...... &......
ffff88030b092630:  ffff880239796300 ffff880239796300   .cy9.....cy9....
ffff88030b092640:  0000000000000000 0000000000000000   ................
ffff88030b092650:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b092660:  2d30303000313264 003030303054444d   d21.000-MDT0000.
ffff88030b092670:  0000000000000000 0000000000000001   ................
ffff88030b092680:  0000000800000000 0000000000090009   ................
ffff88030b092690:  0000000000000000 0000000000000000   ................
ffff88030b0926a0:  ffffc900006b6bf8 ffff880338de2740   .kk.....@'.8....
ffff88030b0926b0:  000000159c988c21 ffff88030b092720   !....... '......
ffff88030b0926c0:  ffff88030b04b880 ffff88030b093ac0   .........:......
ffff88030b0926d0:  ffff88030b04b890 ffff88030b093ad0   .........:......
ffff88030b0926e0:  ffff88030b0926e0 ffff88030b0926e0   .&.......&......
ffff88030b0926f0:  ffff88030b0926f0 ffff88030b0926f0   .&.......&......
ffff88030b092700:  0000000000001000 0000000000000000   ................
ffff88030b092710:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092720:  6d6d6f635f6e7572 4d676f6c5f646e61   run_command_logM
ffff88030b092730:  ffff003065465932 0000000000000000   2YFe0...........
ffff88030b092740:  0000001800000000 0000000000050005   ................
ffff88030b092750:  0000000000000000 0000000000000000   ................
ffff88030b092760:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092770:  0000000200031a47 ffff88030b0927e0   G........'......
ffff88030b092780:  ffff88030b092780 ffff88030b092780   .'.......'......
ffff88030b092790:  ffff8803287c0ad0 ffffffff811a8c00   ..|(............
ffff88030b0927a0:  ffff88030b0927a0 ffff88030b0927a0   .'.......'......
ffff88030b0927b0:  ffff88030b0927b0 ffff88030b0927b0   .'.......'......
ffff88030b0927c0:  ffffea000ac606c8 0000000000000000   ................
ffff88030b0927d0:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b0927e0:  0065727473003364 003030303054444d   d3.stre.MDT0000.
ffff88030b0927f0:  0000000000000000 ffff88030b0927f8   .........'......
ffff88030b092800:  0000001800000000 0000000000050005   ................
ffff88030b092810:  0000000000000000 0000000000000000   ................
ffff88030b092820:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092830:  0000000200031c57 ffff88030b0928a0   W........(......
ffff88030b092840:  ffff88030b092840 ffff88030b092840   @(......@(......
ffff88030b092850:  ffff8803287c0950 ffffffff811a8c00   P.|(............
ffff88030b092860:  ffff88030b092860 ffff88030b092860   `(......`(......
ffff88030b092870:  ffff88030b092870 ffff88030b092870   p(......p(......
ffff88030b092880:  0000000000000001 0000000000000000   ................
ffff88030b092890:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b0928a0:  7300307830003664 003030000073666b   d6.0x0.skfs..00.
ffff88030b0928b0:  0000000000000000 ffffffff811c58e0   .........X......
ffff88030b0928c0:  0000000800000000 0000000000040004   ................
ffff88030b0928d0:  ffff88032878acb8 0000000000000000   ..x(............
ffff88030b0928e0:  ffffc900004044a0 ffff880338f80b40   .D@.....@..8....
ffff88030b0928f0:  0000000ca4946908 ffff88030b092960   .i......`)......
ffff88030b092900:  ffff8803287ccb40 ffff88032874a1c0   @.|(......t(....
ffff88030b092910:  ffff8802397e5290 ffff880338f80ba0   .R~9.......8....
ffff88030b092920:  ffff88030b092920 ffff88030b092920    )...... )......
ffff88030b092930:  ffff88032878ace8 ffff88032878ace8   ..x(......x(....
ffff88030b092940:  0000000000000000 0000000000000000   ................
ffff88030b092950:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092960:  6f6c5f687361622e 0030300074756f67   .bash_logout.00.
ffff88030b092970:  0000000000001000 ffff88031401b000   ................
ffff88030b092980:  0000000800000000 0000000000090009   ................
ffff88030b092990:  0000000000000000 0000000000000000   ................
ffff88030b0929a0:  ffffc90001041ac0 ffff880338de2740   ........@'.8....
ffff88030b0929b0:  00000015b6fb7972 ffff88030b092a20   ry...... *......
ffff88030b0929c0:  ffff88030b04b280 ffff88030b170240   ........@.......
ffff88030b0929d0:  ffff88030b04b290 ffff88030b170250   ........P.......
ffff88030b0929e0:  ffff88030b0929e0 ffff88030b0929e0   .).......)......
ffff88030b0929f0:  ffff88030b0929f0 ffff88030b0929f0   .).......)......
ffff88030b092a00:  ffff88030b092a00 0000000000000000   .*..............
ffff88030b092a10:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092a20:  6d6d6f635f6e7572 58676f6c5f646e61   run_command_logX
ffff88030b092a30:  ffff0070396c5564 0000000000001356   dUl9p...V.......
ffff88030b092a40:  0000001800000000 0000000000050005   ................
ffff88030b092a50:  0000000000000000 0000000000000000   ................
ffff88030b092a60:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092a70:  000000030022351e ffff88030b092ae0   .5"......*......
ffff88030b092a80:  ffff88030b092a80 ffff88030b092a80   .*.......*......
ffff88030b092a90:  0000000000000000 ffffffff811a8c00   ................
ffff88030b092aa0:  ffff88030b092aa0 ffff88030b092aa0   .*.......*......
ffff88030b092ab0:  ffff88030b092ab0 ffff88030b092ab0   .*.......*......
ffff88030b092ac0:  ffffffff811c58e0 0000000000000000   .X..............
ffff88030b092ad0:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b092ae0:  6900736d00333164 003030007974696e   d13.ms.inity.00.
ffff88030b092af0:  0000000000000000 ffff88030b092af0   .........*......
ffff88030b092b00:  0000001800000000 0000000000050005   ................
ffff88030b092b10:  0000000000000000 0000000000000000   ................
ffff88030b092b20:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092b30:  000000030022372e ffff88030b092ba0   .7"......+......
ffff88030b092b40:  ffff88030b092b40 ffff88030b092b40   @+......@+......
ffff88030b092b50:  ffff88030b004cd0 ffffffff811a8c00   .L..............
ffff88030b092b60:  ffff88030b092b60 ffff88030b092b60   `+......`+......
ffff88030b092b70:  ffff88030b092b70 ffff88030b092b70   p+......p+......
ffff88030b092b80:  ffff880314016000 0000000000000000   .`..............
ffff88030b092b90:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b092ba0:  2d74736500363164 003030303054444d   d16.est-MDT0000.
ffff88030b092bb0:  0000000000000000 0000000000000000   ................
ffff88030b092bc0:  0000000800000000 0000000000090009   ................
ffff88030b092bd0:  0000000000000000 0000000000000000   ................
ffff88030b092be0:  ffffc90000ad1b00 ffff880338de2740   ........@'.8....
ffff88030b092bf0:  0000001571d1586f ffff88030b092c60   oX.q....`,......
ffff88030b092c00:  ffff8803287cce40 ffff880507fb9e80   @.|(............
ffff88030b092c10:  ffff8803287cce50 ffff880507fb9e90   P.|(............
ffff88030b092c20:  ffff88030b092c20 ffff88030b092c20    ,...... ,......
ffff88030b092c30:  ffff88030b092c30 ffff88030b092c30   0,......0,......
ffff88030b092c40:  0000000000001351 0000000000000000   Q...............
ffff88030b092c50:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092c60:  6d6d6f635f6e7572 31676f6c5f646e61   run_command_log1
ffff88030b092c70:  ffff00667844704e ffff88030b092c70   NpDxf...p,......
ffff88030b092c80:  0000000800000000 0000000000090009   ................
ffff88030b092c90:  0000000000000000 0000000000000000   ................
ffff88030b092ca0:  ffffc900009ec6a8 ffff880338de2740   ........@'.8....
ffff88030b092cb0:  00000015784f21ea ffff88030b092d20   .!Ox.... -......
ffff88030b092cc0:  ffff880228c191c0 ffff8802397b3c00   ...(.....<{9....
ffff88030b092cd0:  ffff880228c191d0 ffff8802397b3c10   ...(.....<{9....
ffff88030b092ce0:  ffff88030b092ce0 ffff88030b092ce0   .,.......,......
ffff88030b092cf0:  ffff88030b092cf0 ffff88030b092cf0   .,.......,......
ffff88030b092d00:  ffff88030b092cf8 0000000000000000   .,..............
ffff88030b092d10:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092d20:  6d6d6f635f6e7572 32676f6c5f646e61   run_command_log2
ffff88030b092d30:  ffff003854484571 0000000000000000   qEHT8...........
ffff88030b092d40:  0000000800000000 0000000000090009   ................
ffff88030b092d50:  0000000000000000 0000000000000000   ................
ffff88030b092d60:  ffffc900011c1920 ffff880338de2740    .......@'.8....
ffff88030b092d70:  00000015b6be79df ffff88030b092de0   .y.......-......
ffff88030b092d80:  ffff8802397e5340 ffff8803287e1840   @S~9....@.~(....
ffff88030b092d90:  ffff8802397e5350 ffff8803287e1850   PS~9....P.~(....
ffff88030b092da0:  ffff88030b092da0 ffff88030b092da0   .-.......-......
ffff88030b092db0:  ffff88030b092db0 ffff88030b092db0   .-.......-......
ffff88030b092dc0:  0000000000000000 0000000000000000   ................
ffff88030b092dd0:  ffff8803384cfc00 0000000000000000   ..L8............
ffff88030b092de0:  6d6d6f635f6e7572 56676f6c5f646e61   run_command_logV
ffff88030b092df0:  ffff00544b6e4e79 0000000000000000   yNnKT...........
ffff88030b092e00:  0000001800000000 0000000000050005   ................
ffff88030b092e10:  0000000000000000 0000000000000000   ................
ffff88030b092e20:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092e30:  0000000200031db7 ffff88030b092ea0   ................
ffff88030b092e40:  ffff88030b092e40 ffff88030b092e40   @.......@.......
ffff88030b092e50:  ffff88030b092f10 ffffffff811a8c00   ./..............
ffff88030b092e60:  ffff88030b092e60 ffff88030b092e60   `.......`.......
ffff88030b092e70:  ffff88030b092e70 ffff88030b092e70   p.......p.......
ffff88030b092e80:  ffff88030b092e78 0000000000000000   x...............
ffff88030b092e90:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b092ea0:  0030303030003864 003030303054444d   d8.0000.MDT0000.
ffff88030b092eb0:  000000000000134b 0000000000001000   K...............
ffff88030b092ec0:  0000001800000000 0000000000050005   ................
ffff88030b092ed0:  0000000000000000 0000000000000000   ................
ffff88030b092ee0:  dead000000200200 ffff8803286d4a40   .. .....@Jm(....
ffff88030b092ef0:  0000000200031e67 ffff88030b092f60   g.......`/......
ffff88030b092f00:  ffff88030b092f00 ffff88030b092f00   ./......./......
ffff88030b092f10:  ffff88030b092550 ffffffff811a8c00   P%..............
ffff88030b092f20:  ffff88030b092f20 ffff88030b092f20    /...... /......
ffff88030b092f30:  ffff88030b092f30 ffff88030b092f30   0/......0/......
ffff88030b092f40:  0000000000000000 0000000000000000   ................
ffff88030b092f50:  ffff88032f346800 0000000000000000   .h4/............
ffff88030b092f60:  2d00307830003964 003030303054444d   d9.0x0.-MDT0000.
ffff88030b092f70:  ffff88030b092f68 ffffea000ac522a0   h/......."......
ffff88030b092f80:  0000000000001349 0000000000001000   I...............
ffff88030b092f90:  ffff880313c0c000 0000000000000000   ................
ffff88030b092fa0:  ffffffff811c58e0 0000000000000000   .X..............
ffff88030b092fb0:  ffff88030b092fb0 ffff88030b092fb0   ./......./......
ffff88030b092fc0:  0000000000000000 0000000000000000   ................
ffff88030b092fd0:  0000000000000000 0000000000000000   ................
ffff88030b092fe0:  0000000000000000 0000000000000000   ................
ffff88030b092ff0:  62732f7273752f00 00646873732f6e69   ./usr/sbin/sshd.

[Sebastien Buisson (Inactive)]

Hi Bruno,

Any news on this defect?

Cheers,
Sebastien.

[Andreas Dilger]

Close old bug that hasn't been seen in a long time.