[#LU-593] echo client for MDS

[LU-593] echo client for MDS Created: 15/Aug/11 Updated: 28/Mar/12 Resolved: 09/Jan/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	None
Fix Version/s:	Lustre 2.2.0

Type:

New Feature

Priority:

Minor

Reporter:

Di Wang

Assignee:

Di Wang

Resolution:

Fixed

Votes:

Labels:

None

Attachments:

echo-oops.log

Issue Links:

Related
is related to	~~LU-633~~	mds-survey script for MD echo client ...	Resolved

Rank (Obsolete):

4774

Description

Add Metadata support to echo client, so it can exercise MDS stack.

Comments

Comment by James A Simmons [ 05/Oct/11 ]

Patch is at http://review.whamcloud.com/#change,1429

Comment by James A Simmons [ 06/Oct/11 ]

Testing the patch I'm seeing

LustreError: 19139:0:(echo_client.c:1448:echo_md_create_internal()) Can not create child [0x200000400:0x3d095:0x0]
LustreError: 19139:0:(echo_client.c:1558:echo_create_md_object()) Can not create child 2
LustreError: 19150:0:(echo_client.c:1448:echo_md_create_internal()) Can not create child [0x200000400:0x3d0a0:0x0]
LustreError: 19150:0:(echo_client.c:1448:echo_md_create_internal()) Skipped 10 previous similar messages
LustreError: 19150:0:(echo_client.c:1558:echo_create_md_object()) Can not create child 47244639
LustreError: 19150:0:(echo_client.c:1558:echo_create_md_object()) Skipped 10 previous similar messages
LustreError: 20142:0:(echo_client.c:1448:echo_md_create_internal()) Can not create child [0x200000400:0x3d47e:0x0]
LustreError: 20142:0:(echo_client.c:1448:echo_md_create_internal()) Skipped 988 previous similar messages
LustreError: 20142:0:(echo_client.c:1558:echo_create_md_object()) Can not create child 4294969
LustreError: 20142:0:(echo_client.c:1558:echo_create_md_object()) Skipped 989 previous similar messages
LustreError: 21143:0:(echo_client.c:1448:echo_md_create_internal()) Can not create child [0x200000400:0x3d865:0x0]
LustreError: 21143:0:(echo_client.c:1448:echo_md_create_internal()) Skipped 1987 previous similar messages
LustreError: 21143:0:(echo_client.c:1558:echo_create_md_object()) Can not create child 2
LustreError: 21143:0:(echo_client.c:1558:echo_create_md_object()) Skipped 998 previous similar messages
LustreError: 22143:0:(echo_client.c:1665:echo_md_destroy_internal()) Can not unlink child tests

The test fail for me

Comment by James A Simmons [ 06/Oct/11 ]

I see pdir patch as well. Is this needed or is it optional?

Comment by Di Wang [ 06/Oct/11 ]

It is probably test command issue? Could you please post your lctl test command here? Thanks

Comment by James A Simmons [ 07/Oct/11 ]

lctl <<EOF
attach echo_client ecc-MDT0000 ecc-MDT0000_UUID
setup lustre-MDT0000 mdd
EOF

echo "Setting up test structure"
lctl <<EOF
cfg_device 5
test_mkdir /tests
test_mkdir /tests1
test_mkdir /tests2
test_mkdir /tests3
EOF

echo rpctrace > /proc/sys/lnet/debug
echo 500 > /proc/sys/lnet/debug_mb

Zero the log
lctl dk >/dev/null
lctl debug_daemon start $LOGFILE 10240
echo "Starting Test #1"
lctl <<EOF
cfg_device 5
--threads 1000 -1 5 test_create -d /tests -b 2 -c 0 -n 250
EOF
echo "Finished Test #1"
sleep 1

echo "Starting Test #2"
lctl <<EOF
cfg_device 5
--threads 1000 -1 5 test_create -d /tests1 -b 2 -n 250
EOF
echo "Finished Test #2"
sleep 1

echo "Starting Test #3"
lctl <<EOF
cfg_device 5
--threads 1000 -1 5 test_create -d /tests2 -b 2 -n 250
EOF
echo "Finished Test #3"
sleep 1

echo "Starting Test #4"
lctl <<EOF
cfg_device 5
--threads 1000 -1 5 test_create -d /tests3 -b 2 -n 250
EOF
echo "Finished Test #4"

Delete created directories
lctl <<EOF
cfg_device 5
test_rmdir /tests
test_rmdir /tests1
test_rmdir /tests2
test_rmdir /tests3
EOF
Cleanup
echo "Starting cleanup"
lctl debug_daemon stop
lctl <<EOF
device ecc-MDT0000
cleanup
device ecc-MDT0000
detach
EOF

lctl df $LOGFILE > $LOGFILE.txt
rm $LOGFILE

Comment by Di Wang [ 07/Oct/11 ]

Hi, James, Thanks for posting the command here. Is the FS refreshed before the test? Or are /tests[1-3] empty before the test? Do you have have the debuglog for the test you can share with me?

Comment by Di Wang [ 07/Oct/11 ]

Hi, James

I updated the patch to add some error msg there. http://review.whamcloud.com/#change,1287 Could you please try this patch and provide me the debug log (with -1 debug level)? And also to make sure /tests[1-3] are empty before the test. Thanks!

Comment by James A Simmons [ 10/Oct/11 ]

Done. I uploaded the results to /uploads/~~LU-593~~/echo_debug on your ftp server.

Comment by Di Wang [ 11/Oct/11 ]

Thanks. James. Did you setup OSS/OST for the test? Since the script you run here are suppose to create the object on OST as well. If you do not want create the object on OST. you should add -c 0 in your cmd line. as

--threads 1000 -1 5 test_create -d /tests3 -b 2 -n 250 -c 0

Otherwise you should mount OST as well. Thanks.

Ian also mentioned to me that you found some oops when you cleanup the system. Do you still have the trace log? Could you please post here? Thanks!

Comment by James A Simmons [ 11/Oct/11 ]

Here is the oops I seen. It happened because I attached the device and due to test failures it didn't detach the device. Once I ran the script again it attempted to attach a second time which toopled the machine.

Comment by Di Wang [ 11/Oct/11 ]

Thanks! James. I just update the patch, which should fix the oops. And also I set the default stripe to be 0 now.
http://review.whamcloud.com/#change,1287

Comment by Minh Diep [ 16/Nov/11 ]

With the latest patch 6. system panic when we run setup lustre-MDT000 (without the type)

LustreError: 8027:0:(echo_client.c:822:echo_device_alloc()) lustre-MDT0000 no type name for echo mdt setup
Unable to handle kernel NULL pointer dereference at 000000000000011c RIP:
[<ffffffff88e91c90>] :obdecho:echo_client_cleanup+0xa0/0x6a0
PGD 31d1c4067 PUD 31f6e2067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:04.0/0000:03:00.1/irq
CPU 2
Modules linked in: obdecho(U) cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs(U) mgs(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ldiskfs(U) jbd2(U) crc16(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) lockd(U) sunrpc(U) cpufreq_ondemand(U) powernow_k8(U) freq_table(U) mperf(U) be2iscsi(U) iscsi_tcp(U) bnx2i(U) cnic(U) uio(U) cxgb3i(U) iw_cxgb3(U) cxgb3(U) libiscsi_tcp(U) ib_iser(U) libiscsi2(U) scsi_transport_iscsi2(U) scsi_transport_iscsi(U) ib_srp(U) rds(U) ib_sdp(U) ib_ipoib(U) ipoib_helper(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) rdma_ucm(U) rdma_cm(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) ib_cm(U) iw_cm(U) ib_addr(U) ib_sa(U) loop(U) dm_mirror(U) dm_multipath(U) scsi_dh(U) video(U) backlight(U) sbs(U) power_meter(U) i2c_ec(U) dell_wmi(U) wmi(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) shpchp(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) sg(U) igb(U) 8021q(U) k10temp(U) hwmon(U) tpm_tis(U) tpm(U) tpm_bios(U) mlx4_core(U) amd64_edac_mod(U) dca(U) i2c_piix4(U) pcspkr(U) i2c_core(U) serio_raw(U) edac_mc(U) dm_raid45(U) dm_message(U) dm_region_hash(U) dm_log(U) dm_mod(U) dm_mem_cache(U) ahci(U) libata(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 8027, comm: lctl Tainted: G 2.6.18-238.19.1.el5_lustre #1
RIP: 0010:[<ffffffff88e91c90>] [<ffffffff88e91c90>] :obdecho:echo_client_cleanup+0xa0/0x6a0
RSP: 0018:ffff810419347ad8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff8031af28
RDX: ffffffff8031af28 RSI: 0000000000000000 RDI: ffff81010d38c0f8
RBP: ffff81010d38c0f8 R08: ffffffff8031af28 R09: 000000000000003f
R10: ffff8104193475e8 R11: 0000000000000000 R12: ffff810108331d40
R13: ffffffff88ead160 R14: ffff810109606800 R15: ffff810109606800
FS: 00002af0e5290bf0(0000) GS:ffff810223903340(0000) knlGS:00000000f7db36c0
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000000011c CR3: 000000031f1e7000 CR4: 00000000000006e0
Process lctl (pid: 8027, threadinfo ffff810419346000, task ffff810418d4d080)
Stack: 0000000000000000 0000000000000246 ffff81010d38c0f8 ffff810108331d80
ffff81021f839620 ffff810108331d40 ffffffff88ead160 ffff810108331d80
ffffffffffffffea ffffffff88ea2fcb 0000000000000000 0000000000000000
Call Trace:
[<ffffffff88ea2fcb>] :obdecho:echo_device_alloc+0x1aab/0x1c80
[<ffffffff8891fee4>] :obdclass:lu_context_init+0x1e4/0x200
[<ffffffff88903c7f>] :obdclass:class_setup+0x83f/0xd00
[<ffffffff8883caeb>] :libcfs:libcfs_debug_vmsg2+0x75b/0x9f0
[<ffffffff8000c5f5>] do_generic_mapping_read+0x347/0x359
[<ffffffff88906e38>] :obdclass:class_process_config+0x1d88/0x3200
[<ffffffff88832558>] :libcfs:cfs_alloc+0x28/0x60
[<ffffffff888d1dc7>] :obdclass:obd_ioctl_getdata+0x567/0xff0
[<ffffffff80013966>] filemap_nopage+0x193/0x360
[<ffffffff888e5254>] :obdclass:class_handle_ioctl+0x904/0x2220
[<ffffffff8000cfa2>] do_sync_read+0xc7/0x104
[<ffffffff888d157b>] :obdclass:obd_class_ioctl+0x19b/0x230
[<ffffffff800a2903>] autoremove_wake_function+0x0/0x2e
[<ffffffff800423fc>] do_ioctl+0x21/0x6b
[<ffffffff8003040b>] vfs_ioctl+0x457/0x4b9
[<ffffffff800b9656>] audit_syscall_entry+0x1a4/0x1cf
[<ffffffff8004c911>] sys_ioctl+0x59/0x78
[<ffffffff8005d28d>] tracesys+0xd5/0xe0

Comment by Di Wang [ 16/Nov/11 ]

Minh, Thanks. I just push a new patch on 1287, please check.

Comment by Minh Diep [ 17/Nov/11 ]

patch 7: this was what I did and I hit LBUG

[root@fat-amd-3 ~]# lctl dl
0 UP mgs MGS MGS 5
1 UP mgc MGC10.10.4.134@tcp 2239a36c-de1f-e9bd-0c4c-f7dee45f6230 5
2 UP lov lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4
3 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 3
4 UP mds mdd_obd-lustre-MDT0000 mdd_obd_uuid-lustre-MDT0000 3
[root@fat-amd-3 ~]# lctl
lctl >
attach echo_client lustre-MDT0000_ecc lustre-MDT0000_ecc_UUID
lctl > dl
0 UP mgs MGS MGS 5
1 UP mgc MGC10.10.4.134@tcp 2239a36c-de1f-e9bd-0c4c-f7dee45f6230 5
2 UP lov lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4
3 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 3
4 UP mds mdd_obd-lustre-MDT0000 mdd_obd_uuid-lustre-MDT0000 3
5 AT echo_client lustre-MDT0000_ecc lustre-MDT0000_ecc_UUID 1
lctl > setup lustre-MDT0000 mdd
lctl > dl
0 UP mgs MGS MGS 5
1 UP mgc MGC10.10.4.134@tcp 2239a36c-de1f-e9bd-0c4c-f7dee45f6230 5
2 UP lov lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4
3 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 3
4 UP mds mdd_obd-lustre-MDT0000 mdd_obd_uuid-lustre-MDT0000 3
5 UP echo_client lustre-MDT0000_ecc lustre-MDT0000_ecc_UUID 3
lctl > cfg_device 5
lctl > test_mkdir /tests
lctl > --threads 4 -1 5 test_create -d /tests -b 2 -n 1
Print status every 1 seconds
--threads: starting 4 threads on device 5 running test_create -d /tests -b 2 -n 1
test_create-4: start at Thu Nov 17 08:37:23 2011
test_create-1: end at Thu Nov 17 08:37:23 2011
Total: total 4 threads 4 sec 0.001066 3752.345216/second
lctl > dk /tmp/debug
Debug log: 578 lines, 578 kept, 0 dropped, 0 bad.
lctl > test_rmdir /tests
LustreError: 6157:0:(echo_client.c:1805:echo_md_destroy_internal()) Can not unlink child tests rc -39
error: test_rmdir: Directory not empty
lctl > --threads 4 -1 5 test_destroy -d /tests -b 2 -n 1
Print status every 1 seconds
--threads: starting 4 threads on device 5 running test_destroy -d /tests -b 2 -n 1
test_destroy-4: start at Thu Nov 17 08:39:01 2011
test_destroy-3: end at Thu Nov 17 08:39:01 2011
Total: total 4 threads 4 sec 0.001914 2089.864159/second
lctl > test_rmdir /tests
LustreError: 6157:0:(lu_object.h:700:lu_object_get()) ASSERTION(cfs_atomic_read(&o->lo_header->loh_ref) > 0) failed
LustreError: 6157:0:(lu_object.h:700:lu_object_get()) LBUG
Pid: 6157, comm: lctl

Call Trace:

Message from [<ffffffff888335f1>] libcfs_debug_dumpstack+0x51/0x60 [libcfs]
syslogd@ at Thu [<ffffffff88833b2a>] lbug_with_loc+0x7a/0xd0 [libcfs]
Nov 17 08:39:13 [<ffffffff8883ede0>] cfs_tracefile_init+0x0/0x10a [libcfs]
2011 ...
fat- [<ffffffff88e4f711>] osd_it_ea_init+0x1b1/0x250 [osd_ldiskfs]
amd-3 kernel: Lu [<ffffffff88da5cde>] mdd_may_delete+0xa2e/0xc30 [mdd]
streError: 6157: [<ffffffff88921ec4>] lu_context_init+0x1e4/0x200 [obdclass]
0:(lu_object.h:7 [<ffffffff88da5f74>] mdd_unlink_sanity_check+0x94/0x120 [mdd]
00:lu_object_get [<ffffffff88dabaf8>] mdd_unlink+0x388/0xbe0 [mdd]
()) ASSERTION(cf [<ffffffff88ef7da2>] echo_md_destroy_internal+0x2f2/0x490 [obdecho]
s_atomic_read(&o [<ffffffff88eff0f1>] echo_md_handler+0x261/0x19c0 [obdecho]
~~>lo_header~~>loh [<ffffffff88928f80>] cl_env_get+0x190/0x300 [obdclass]
_ref) > 0) faile [<ffffffff88eff963>] echo_md_handler+0xad3/0x19c0 [obdecho]
d

Message [<ffffffff8005c37b>] cache_alloc_refill+0x106/0x186
from syslogd@ at [<ffffffff88f018c6>] echo_client_iocontrol+0x8c6/0x2900 [obdecho]
Thu Nov 17 08:3 [<ffffffff8001aa2e>] vsnprintf+0x5df/0x627
9:13 2011 ...
[<ffffffff888331ad>] cfs_set_ptldebug_header+0x4d/0xa0 [libcfs]
fat-amd-3 kernel [<ffffffff8003aa24>] tty_ldisc_deref+0x24/0x7b
: LustreError: 6 [<ffffffff8005c37b>] cache_alloc_refill+0x106/0x186
157:0:(lu_object [<ffffffff88834558>] cfs_alloc+0x28/0x60 [libcfs]
.h:700:lu_object [<ffffffff888d3dc7>] obd_ioctl_getdata+0x567/0xff0 [obdclass]
_get()) LBUG
[<ffffffff8003aa68>] tty_ldisc_deref+0x68/0x7b
[<ffffffff888e873c>] class_handle_ioctl+0x1dec/0x2220 [obdclass]
[<ffffffff80033f06>] copy_from_read_buf+0x58/0xdc
[<ffffffff801c42d1>] uart_start+0x26/0x36
[<ffffffff801c7e86>] serial8250_start_tx+0x2b/0x90
[<ffffffff888d357b>] obd_class_ioctl+0x19b/0x230 [obdclass]
[<ffffffff800423fc>] do_ioctl+0x21/0x6b
[<ffffffff8003040b>] vfs_ioctl+0x457/0x4b9
[<ffffffff800b9656>] audit_syscall_entry+0x1a4/0x1cf
[<ffffffff8004c911>] sys_ioctl+0x59/0x78
[<ffffffff8005d28d>] tracesys+0xd5/0xe0

Kernel panic - not syncing: LBUG

Comment by Di Wang [ 17/Nov/11 ]

Minh, thanks for reporting this. I just push a new patch on 1287. Please check.

Comment by Minh Diep [ 22/Nov/11 ]

Patch #9, system panic when I hit ctrl-c during the test

Unable to handle kernel NULL pointer dereference at 0000000000000a88 RIP:
[<ffffffff8891fef1>] :obdclass:lu_env_init+0x11/0x40
PGD 9e70a067 PUD d21b6067 PMD 0
Oops: 0002 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:04.0/0000:03:00.1/irq
CPU 8
Modules linked in: obdecho(U) nfs(U) fscache(U) nfs_acl(U) cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs(U) mgs(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ldiskfs(U) jbd2(U) crc16(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) lockd(U) sunrpc(U)<3>LustreError: 13057:0:(class_obd.c:311:class_handle_ioctl()) OBD ioctl: device not setup 5
LustreError: 13048:0:(class_obd.c:311:class_handle_ioctl()) OBD ioctl: device not setup 5
cpufreq_ondemand(U) powernow_k8(U) freq_table(U) mperf(U) be2iscsi(U) iscsi_tcp(U) bnx2i(U) cnic(U) uio(U) cxgb3i(U) iw_cxgb3(U) cxgb3(U) libiscsi_tcp(U) ib_iser(U) libiscsi2(U) scsi_transport_iscsi2(U) scsi_transport_iscsi(U) ib_srp(U) rds(U) ib_sdp(U) ib_ipoib(U) ipoib_helper(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) rdma_ucm(U) rdma_cm(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) ib_cm(U) iw_cm(U) ib_addr(U) ib_sa(U) loop(U) dm_mirror(U) dm_multipath(U) scsi_dh(U) video(U) backlight(U) sbs(U) power_meter(U) i2c_ec(U) dell_wmi(U) wmi(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) shpchp(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) i2c_piix4(U) igb(U) 8021q(U) i2c_core(U) tpm_tis(U) mlx4_core(U) tpm(U) amd64_edac_mod(U) edac_mc(U) serio_raw(U) dca(U) pcspkr(U) sg(U) k10temp(U) tpm_bios(U) hwmon(U) dm_raid45(U) dm_message(U) dm_region_hash(U) dm_log(U) dm_mod(U) dm_mem_cache(U) ahci(U) libata(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 5405, comm: obd_zombid Tainted: G 2.6.18-238.19.1.el5_lustre #1
RIP: 0010:[<ffffffff8891fef1>] [<ffffffff8891fef1>] :obdclass:lu_env_init+0x11/0x40
RSP: 0018:ffff81031ee15bc0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000000a50 RCX: 000000000000000f
RDX: ffff8103dc61f880 RSI: 0000000000000002 RDI: 0000000000000a50
RBP: ffff8103c91b37c0 R08: 01eabc03765c76a9 R09: 0000000000000000
R10: ffff8103058c5280 R11: 0000000000000108 R12: 00000000000002a8
R13: 0000000000000a50 R14: ffff8103ca9c5a10 R15: ffff810218f5f200
FS: 00002b731146a6e0(0000) GS:ffff810323a2c2c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000a88 CR3: 00000000a33a0000 CR4: 00000000000006e0
Process obd_zombid (pid: 5405, threadinfo ffff81031ee14000, task ffff81021fb8b860)
Stack: ffff810319aae1c0 0000000000000000 ffff8103c91b37c0 ffffffff88e45e5f
ffff81031ee15c80 ffff81031ee15e10 003fffff00000010 0000000000000016
ffff8103c816ff70 ffff810123bb09c0 ffff8103c91b37c0 ffff81031ee15e10
Call Trace:
[<ffffffff88e45e5f>] :osd_ldiskfs:osd_object_delete+0x23f/0x350
[<ffffffff889205a9>] :obdclass:lu_object_free+0x89/0x190
[<ffffffff8892110b>] :obdclass:lu_site_purge+0x25b/0x330
[<ffffffff888ed180>] :obdclass:obd_zombie_impexp_thread+0x0/0x250
[<ffffffff88efeac9>] :obdecho:echo_device_free+0xd9/0x5d0
[<ffffffff8891fec4>] :obdclass:lu_context_init+0x1e4/0x200
[<ffffffff888ed180>] :obdclass:obd_zombie_impexp_thread+0x0/0x250
[<ffffffff88901f7a>] :obdclass:class_decref+0x34a/0x5c0
[<ffffffff888e81b2>] :obdclass:obd_zombie_impexp_cull+0x422/0x520
[<ffffffff888ed31a>] :obdclass:obd_zombie_impexp_thread+0x19a/0x250
[<ffffffff8008e438>] default_wake_function+0x0/0xe
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff888ed180>] :obdclass:obd_zombie_impexp_thread+0x0/0x250
[<ffffffff888ed180>] :obdclass:obd_zombie_impexp_thread+0x0/0x250
[<ffffffff8005dfa7>] child_rip+0x0/0x11

Code: 48 c7 47 38 00 00 00 00 e8 e2 fd ff ff 85 c0 89 c5 75 08 48
RIP [<ffffffff8891fef1>] :obdclass:lu_env_init+0x11/0x40
RSP <ffff81031ee15bc0>
CR2: 0000000000000a88
<0>Kernel panic - not syncing: Fatal exception

Comment by Di Wang [ 23/Nov/11 ]

Just push a new patch. Thanks for reporting this.

Comment by Minh Diep [ 06/Dec/11 ]

with patch 11 I still see test failed after the previous test canceled by control-c

[root@client-8 mdiep]# stripe_count=0 sh ./mdt-survey
Tue Dec 6 16:19:43 PST 2011 Mdt-survey from cliLustreError: 5326:0:(echo_client.c:1806:echo_md_destroy_internal()) Can not unlink child tests rc -39
ent-8.lab.whamclLustreError: 5326:0:(echo_client.c:1806:echo_md_destroy_internal()) Skipped 3 previous similar messages
oud.com
LustreError: 5336:0:(echo_client.c:1449:echo_md_create_internal()) Can not create child [0x200000422:0x1:0x0] rc -17
error: test_mkdir: File exists
ERROR: fail test_mkdir
created directories on localhost:lustre-MDT0000_ecc failed
program exited with error
[root@client-8 mdiep]# lctl dl
0 UP mgs MGS MGS 5
1 UP mgc MGC10.10.4.8@tcp f4aacd99-67ff-7aa3-386d-1d3432c93b3e 5
2 UP lov lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4
3 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 3
4 UP mds mdd_obd-lustre-MDT0000 mdd_obd_uuid-lustre-MDT0000 3
6 UP ost OSS OSS_uuid 3
7 UP osc lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5
8 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5

should clean/detach clear all directories

Comment by Minh Diep [ 14/Dec/11 ]

when I run mdt-survey script on a build that does not have this LU's patch, system LBUG'ed. Is this expected? We should prevent that.

[root@fat-intel-2 mdiep]# ./mdt-survey.latest

Message from syslogd@ at Wed Dec 14 14:49:27 2011 ...
fat-intel-2 kernel: LustreError: 7508:0:(mdt_handler.c:4975:mdt_obd_connect()) ASSERTION(env != NULL) failed

Message from syslogd@ at Wed Dec 14 14:49:27 2011 ...
fat-intel-2 kernel: LustreError: 7508:0:(mdt_handler.c:4975:mdt_obd_connect()) LBUG

Comment by Di Wang [ 15/Dec/11 ]

Oh, actually the echo client patch we will submit will just fix this problem, so you can ignore it now.