[LU-106] unable to handle kernel paging request in lprocfs_stats_collect() Created: 02/Mar/11  Updated: 07/Jun/12  Resolved: 12/Dec/11

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.1.0
Fix Version/s: Lustre 2.2.0, Lustre 2.1.2

Type: Bug Priority: Major
Reporter: Ned Bass Assignee: Lai Siyao
Resolution: Fixed Votes: 0
Labels: None
Environment:

RHEL6 x86_64


Severity: 3
Rank (Obsolete): 4811

 Description   

This kernel oops happened while unmounting an OST. cerebrod is a monitoring daemon that gathers Lustre data from /proc.

2011-03-01 13:06:10 Lustre: client ffff8802e6927000 umount complete
2011-03-01 13:06:13 Lustre: Failing over lustre-MDT0000
2011-03-01 13:06:13 Lustre: Failing over mdd_obd-lustre-MDT0000-0
2011-03-01 13:06:13 Lustre: mdd_obd-lustre-MDT0000-0: shutting down for failover; client state will be preserved.
2011-03-01 13:06:13 Lustre: MGS has stopped.
2011-03-01 13:06:13 Lustre: server umount lustre-MDT0000 complete
2011-03-01 13:06:15 Lustre: lustre-OST0000: shutting down for failover; client state will be preserved.
2011-03-01 13:06:15 BUG: unable to handle kernel paging request at 00000000deadcb17
2011-03-01 13:06:15 IP: [<ffffffffa0400f52>] lprocfs_stats_collect+0x102/0x160 [obdclass]
2011-03-01 13:06:15 PGD 630662067 PUD 0
2011-03-01 13:06:15 Oops: 0000 1 SMP
2011-03-01 13:06:15 last sysfs file: /sys/module/lov/initstate
2011-03-01 13:06:15 CPU 0
2011-03-01 13:06:15 Modules linked in: lustre lmv obdfilter ost cmm osd_ldiskfs mdt mdd mds fsfilt_ldiskfs exportfs mgs mgc ldiskfs mbcache jbd2 lov osc mdc fid fld ko2iblnd ptlrpc obdclass lnet lvfs libcfs ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core sg sd_mod crc_t10dif dm_mirror dm_region_hash dm_log dm_mod video output sbs sbshc power_meter hwmon acpi_pad parport serio_raw i2c_i801 i2c_core ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core mpt2sas scsi_transport_sas raid_class ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_core igb dca [last unloaded: lustre]
2011-03-01 13:06:15
2011-03-01 13:06:15 Modules linked in: lustre lmv obdfilter ost cmm osd_ldiskfs mdt mdd mds fsfilt_ldiskfs exportfs mgs mgc ldiskfs mbcache jbd2 lov osc mdc fid fld ko2iblnd ptlrpc obdclass lnet lvfs libcfs ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core sg sd_mod crc_t10dif dm_mirror dm_region_hash dm_log dm_mod video output sbs sbshc power_meter hwmon acpi_pad parport serio_raw i2c_i801 i2c_core ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core mpt2sas scsi_transport_sas raid_class ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_core igb dca [last unloaded: lustre]
2011-03-01 13:06:16 Pid: 8198, comm: cerebrod Tainted: G W ---------------- 2.6.32-14chaos #1 X8DTH-i/6/iF/6F
2011-03-01 13:06:16 RIP: 0010:[<ffffffffa0400f52>] [<ffffffffa0400f52>] lprocfs_stats_collect+0x102/0x160 [obdclass]
2011-03-01 13:06:16 RSP: 0018:ffff88061dffbd48 EFLAGS: 00010206
2011-03-01 13:06:16 RAX: 00000000deadcacf RBX: ffff880609677180 RCX: 0000000000000be0
2011-03-01 13:06:16 RDX: 0000000000000026 RSI: 0000000000000000 RDI: 0000000000000000
2011-03-01 13:06:16 RBP: ffff88061dffbd88 R08: 0000000000000018 R09: 0000000000000000
2011-03-01 13:06:16 R10: 7fffffffffffffff R11: 0000000000000bf0 R12: ffff88061dffbd98
2011-03-01 13:06:16 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
2011-03-01 13:06:16 FS: 00007ffff7ff0700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
2011-03-01 13:06:16 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2011-03-01 13:06:16 CR2: 00000000deadcb17 CR3: 000000061d597000 CR4: 00000000000006f0
2011-03-01 13:06:16 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2011-03-01 13:06:16 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2011-03-01 13:06:16 Process cerebrod (pid: 8198, threadinfo ffff88061dffa000, task ffff88063007f560)
2011-03-01 13:06:16 Stack:
2011-03-01 13:06:16 ffff880609677180 0000000000000026 0000000000000001 ffff88060970ebe0
2011-03-01 13:06:16 <0> ffff880609677180 ffff8802e67eaac0 0000000000000000 ffff88061dffbe58
2011-03-01 13:06:16 <0> ffff88061dffbe18 ffffffffa040100f 0000000000000000 0000000000000000
2011-03-01 13:06:16 Call Trace:
2011-03-01 13:06:16 [<ffffffffa040100f>] lprocfs_stats_seq_show+0x5f/0x170 [obdclass]
2011-03-01 13:06:16 [<ffffffff8118e497>] seq_read+0x267/0x3f0
2011-03-01 13:06:16 [<ffffffff811ced7e>] proc_reg_read+0x7e/0xc0
2011-03-01 13:06:16 [<ffffffff8116caf5>] vfs_read+0xb5/0x1a0
2011-03-01 13:06:16 [<ffffffff8116cc31>] sys_read+0x51/0x90
2011-03-01 13:06:16 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b
2011-03-01 13:06:16 Code: 28 4d 39 4c 24 30 7d 05 4d 89 4c 24 30 41 83 c0 01 4d 01 6c 24 38 44 39 c0 77 96 48 8b 55 c8 48 8d 04 92 48 c1 e0 04 48 03 43 10 <48> 8b 40 48 49 89 44 24 48 f6 43 04 01 74 0a 48 83 c3 08 66 ff
2011-03-01 13:06:16 RIP [<ffffffffa0400f52>] lprocfs_stats_collect+0x102/0x160 [obdclass]
2011-03-01 13:06:16 RSP <ffff88061dffbd48>
2011-03-01 13:06:16 CR2: 00000000deadcb17
2011-03-01 13:06:16 --[ end trace 210f520790780e8d ]--
2011-03-01 13:06:16 Kernel panic - not syncing: Fatal exception
2011-03-01 13:06:16 Pid: 8198, comm: cerebrod Tainted: G D W ---------------- 2.6.32-14chaos #1
2011-03-01 13:06:16 Call Trace:
2011-03-01 13:06:16 [<ffffffff814c62e3>] panic+0x78/0x137
2011-03-01 13:06:16 [<ffffffff814ca3b4>] oops_end+0xe4/0x100
2011-03-01 13:06:16 [<ffffffff8104651b>] no_context+0xfb/0x260
2011-03-01 13:06:16 [<ffffffff810467a5>] __bad_area_nosemaphore+0x125/0x1e0
2011-03-01 13:06:16 [<ffffffff810468ce>] bad_area+0x4e/0x60
2011-03-01 13:06:16 [<ffffffff814cbf00>] do_page_fault+0x390/0x3a0
2011-03-01 13:06:16 [<ffffffff814c9705>] page_fault+0x25/0x30
2011-03-01 13:06:16 [<ffffffffa0400f52>] ? lprocfs_stats_collect+0x102/0x160 [obdclass]
2011-03-01 13:06:16 [<ffffffffa0400eaa>] ? lprocfs_stats_collect+0x5a/0x160 [obdclass]
2011-03-01 13:06:16 [<ffffffffa040100f>] lprocfs_stats_seq_show+0x5f/0x170 [obdclass]
2011-03-01 13:06:16 [<ffffffff8118e497>] seq_read+0x267/0x3f0
2011-03-01 13:06:16 [<ffffffff811ced7e>] proc_reg_read+0x7e/0xc0
2011-03-01 13:06:16 [<ffffffff8116caf5>] vfs_read+0xb5/0x1a0
2011-03-01 13:06:16 [<ffffffff8116cc31>] sys_read+0x51/0x90
2011-03-01 13:06:16 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b



 Comments   
Comment by Peter Jones [ 02/Mar/11 ]

Lai

Could you please look at this one

Thanks

Peter

Comment by Lai Siyao [ 13/Mar/11 ]

There are several places where in memory struct may be released or un-initialized before proc entry creation/removal for mgs/mds/obdfilter.

I tested with the below scripts, and it could pass:

while true; do
    umount -d /mnt/ost1
    mount -t lustre <OST_DEVICE> /mnt/ost1
done
while true; do
    cat /proc/fs/lustre/obdfilter/lustre-OST0000/*
    cat /proc/fs/lustre/obdfilter/lustre-OST0000/export/*/*
done

These two scripts are running concurrently for mds/obdfilter.

Patch is at http://review.whamcloud.com/#change,325.

Comment by Build Master (Inactive) [ 13/Mar/11 ]

Integrated in reviews-centos5 #442
LU-106 unable to handle kernel pagig request in lprocfs_stats_collect()

Lai Siyao : 28030d340504ce9a9e938d7a5a7b65205a82cc7b
Files :

  • lustre/mds/handler.c
  • lustre/mds/lproc_mds.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/obdfilter/filter.c
  • lustre/mgs/lproc_mgs.c
Comment by Build Master (Inactive) [ 14/Mar/11 ]

Integrated in reviews-centos5 #444
LU-106 unable to handle kernel paging request in lprocfs_stats_collect()

Lai Siyao : 6b85a6cc4348274e29cc6d26bd958cc977c16207
Files :

  • lustre/mgs/lproc_mgs.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/mdt/mdt_lproc.c
  • lustre/mdt/mdt_handler.c
  • lustre/obdfilter/filter.c
Comment by Build Master (Inactive) [ 15/Mar/11 ]

Integrated in reviews-centos5 #458
LU-106 unable to handle kernel pagig request in lprocfs_stats_collect()

Lai Siyao : e7146c54ba1ae58e796f66c9f1619d42bc90c025
Files :

  • lustre/obdclass/lprocfs_status.c
  • lustre/mgs/lproc_mgs.c
  • lustre/obdfilter/filter.c
  • lustre/mds/handler.c
  • lustre/include/obd_class.h
  • lustre/mds/lproc_mds.c
Comment by Build Master (Inactive) [ 15/Mar/11 ]

Integrated in reviews-centos5 #459
LU-106 unable to handle kernel paging request in lprocfs_stats_collect()

Lai Siyao : 95a83d156c0b6e63d4b6ef197393839c6f9869c7
Files :

  • lustre/mdt/mdt_handler.c
  • lustre/mgs/lproc_mgs.c
  • lustre/obdfilter/filter.c
  • lustre/include/obd_class.h
  • lustre/mdt/mdt_lproc.c
  • lustre/obdclass/lprocfs_status.c
Comment by Build Master (Inactive) [ 15/Mar/11 ]

Integrated in reviews-centos5 #460
LU-106 unable to handle kernel paging request in lprocfs_stats_collect()

Lai Siyao : 95a83d156c0b6e63d4b6ef197393839c6f9869c7
Files :

  • lustre/mdt/mdt_lproc.c
  • lustre/mdt/mdt_handler.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/obdfilter/filter.c
  • lustre/include/obd_class.h
  • lustre/mgs/lproc_mgs.c
Comment by Build Master (Inactive) [ 15/Mar/11 ]

Integrated in reviews-centos5 #463
LU-106 unable to handle kernel pagig request in lprocfs_stats_collect()

Lai Siyao : e7146c54ba1ae58e796f66c9f1619d42bc90c025
Files :

  • lustre/obdfilter/filter.c
  • lustre/mgs/lproc_mgs.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/mds/lproc_mds.c
  • lustre/include/obd_class.h
  • lustre/mds/handler.c
Comment by Build Master (Inactive) [ 15/Mar/11 ]

Integrated in reviews-centos5 #468
LU-106 unable to handle kernel paging request in lprocfs_stats_collect()

Lai Siyao : 496d18c30da3998711bb3ab3379a486a4739b324
Files :

  • lustre/mdt/mdt_handler.c
  • lustre/mdt/mdt_lproc.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/mgs/lproc_mgs.c
  • lustre/obdfilter/filter.c
  • lustre/include/obd_class.h
Comment by Build Master (Inactive) [ 22/Mar/11 ]

Integrated in reviews-centos5 #533
LU-106 unable to handle kernel paging request in lprocfs_stats_collect()

Lai Siyao : 8a21280779de8135e0255e13582053e6195cc3e1
Files :

  • lustre/obdclass/lprocfs_status.c
  • lustre/mdt/mdt_lproc.c
  • lustre/mgs/lproc_mgs.c
  • lustre/include/obd_class.h
  • lustre/obdfilter/filter.c
  • lustre/mdt/mdt_handler.c
Comment by Lai Siyao [ 18/Apr/11 ]

Review is on http://review.whamcloud.com/#change,326.

Comment by Lai Siyao [ 18/Apr/11 ]

Maloo result is on https://maloo.whamcloud.com/test_sessions/a3e23b96-6588-11e0-b32b-52540025f9af.

Comment by Lai Siyao [ 27/Apr/11 ]

maloo result is on https://maloo.whamcloud.com/test_sessions/c057691e-70db-11e0-80b5-52540025f9af, and looks okay.
Oleg, do you think it's okay to land this?

Comment by Prakash Surya (Inactive) [ 02/May/11 ]

Single node test w/ patch: https://maloo.whamcloud.com/test_sessions/9a08f0b6-750e-11e0-a1b3-52540025f9af

Comment by Peter Jones [ 12/May/11 ]

Prakash\Ned

When do you think LLNL might be able to test this at scale? I have discussed this patch with Oleg and he would prefer to get some scale testing of this patch (preferably using your monitoring tool) before this is landed. It would be great if we were able to land this patch in time for 2.1 but we probably would not hold the release just for this so I am dropping the priority to reflect that

Regards

Peter

Comment by Ned Bass [ 12/May/11 ]

Hi Peter,

We are currently waiting on getting our test server cluster updated to RHEL6 and Lustre 2.x. I need to check with our system administrators on the timeframe, but my guess is that this should happen within the next two weeks or so. They've been heavily focused on other system integration projects which have priority over our test infrastructure. Once that work is done we can start at scale testing.

Thanks,
Ned

Comment by Christopher Morrone [ 12/May/11 ]

What testing have you guys done at small scale?

I honestly don't see much to gain from scale testing except that we'll have more simultaneous small scale tests...

The test for this should look like this:

  • Have a script walk and cat files in lustre /proc tree will repeatedly starting and stopping the OSTs. Run for days.

We can help with that, but I don't really think scale is an issue here.

Comment by Peter Jones [ 16/May/11 ]

Chris

I think that our concern is more around unexpected side-effects from this patch not just whether the patch corrects the reported issue.

Lai

Have you tried running tests as Chris suggests in order to validate this patch? If not, could you please look into doing so?

Thanks

Peter

Comment by Lai Siyao [ 16/May/11 ]

Peter, in the second comment I described how I tested locally: cat files under /proc/fs/lustre/[mdt|obdfilter], and it's tested for tens of minutes. To follow Chris' suggest, I need walk and cat more files under /proc/fs/lustre and /proc/sys/[lnet|lustre] and test up to 1 day. I will test it later and post result.

Comment by Lai Siyao [ 22/May/11 ]

Unable to handle kernel NULL pointer dereference at 0000000000000028 RIP:
[<ffffffff80023bb2>] filp_close+0x7/0x64
PGD 2d0057067 PUD 2d004d067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:01.0/0000:01:00.1/irq
CPU 3
Modules linked in: llite_lloop(U) lustre(U) obdfilter(U) ost(U) osd_ldiskfs(U) cmm(U) fsfilt_ldiskfs(U) exportfs(U) mdt(U) mdd(U) mds(U) mgs(U) ldiskfs(U) jbd2(U) crc16(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) nfs(U) fscache(U) nfs_acl(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) lockd(U) sunrpc(U) cpufreq_ondemand(U) acpi_cpufreq(U) freq_table(U) be2iscsi(U) iscsi_tcp(U) bnx2i(U) cnic(U) uio(U) cxgb3i(U) iw_cxgb3(U) cxgb3(U) libiscsi_tcp(U) ib_iser(U) libiscsi2(U) scsi_transport_iscsi2(U) scsi_transport_iscsi(U) ib_srp(U) rds(U) ib_sdp(U) ib_ipoib(U) ipoib_helper(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) rdma_ucm(U) rdma_cm(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) ib_cm(U) iw_cm(U) ib_addr(U) ib_sa(U) loop(U) dm_mirror(U) dm_multipath(U) scsi_dh(U) video(U) backlight(U) sbs(U) power_meter(U) hwmon(U) i2c_ec(U) dell_wmi(U) wmi(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) joydev(U) sg(U) igb(U) 8021q(U) serio_raw(U) i2c_i801(U) i2c_core(U) mlx4_core(U) shpchp(U) dca(U) pcspkr(U) dm_raid45(U) dm_message(U) dm_region_hash(U) dm_log(U) dm_mod(U) dm_mem_cache(U) ahci(U) libata(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 32369, comm: cat Tainted: G 2.6.18-194.17.1.el5_lustre.g4a053f7 #1
RIP: 0010:[<ffffffff80023bb2>] [<ffffffff80023bb2>] filp_close+0x7/0x64
RSP: 0018:ffff8102d17afc98 EFLAGS: 00010246
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8102d17afb40
RDX: 0000000000000003 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff8102d17ae000 R09: 000000000000003d
R10: ffff81010b7e4068 R11: 0000000000000080 R12: ffff8102db2d8928
R13: ffff8102db2d8670 R14: ffff8102d17afce8 R15: 0000000019844000
FS: 00002abeea526260(0000) GS:ffff81010b7636c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 00000002d165b000 CR4: 00000000000006e0
Process cat (pid: 32369, threadinfo ffff8102d17ae000, task ffff8102da4fd860)
Stack: 0000000000000001 ffff8102db2d83f8 ffff8102db2d8928 ffffffff88e43dbd
0000003000000030 ffff8102d17afdf0 ffff8102d17afcd8 0000000000000004
0000000000000000 0000000000000000 ffff81033f5cd580 ffff81030d8019c0
Call Trace:
[<ffffffff88e43dbd>] :obdfilter:filter_post+0x11d/0x240
[<ffffffff88e5629f>] :obdfilter:filter_cleanup+0x15f/0x290
[<ffffffff888bb31f>] :obdclass:class_decref+0x43f/0x5b0
[<ffffffff888a2215>] :obdclass:obd_devlist_next+0x65/0x80
[<ffffffff8888a124>] :obdclass:obd_device_list_seq_start+0x34/0x50
[<ffffffff80063b05>] mutex_lock+0xd/0x1d
[<ffffffff8003f649>] seq_read+0xe5/0x28c
[<ffffffff8000b729>] vfs_read+0xcb/0x171
[<ffffffff80011c15>] sys_read+0x45/0x6e
[<ffffffff8005d28d>] tracesys+0xd5/0xe0

Peter, during stress test I met one crash, this looks to be another issue related with procfs, should I fix it along in this ticket?

Comment by Lai Siyao [ 23/May/11 ]

crash> bt 22333
PID: 22333 TASK: ffff8102e3a46860 CPU: 3 COMMAND: "umount"
#0 [ffff8102e48e9c58] schedule at ffffffff80062f96
#1 [ffff8102e48e9d30] schedule_timeout at ffffffff800638ab
#2 [ffff8102e48e9d80] server_put_super at ffffffff88861a01
#3 [ffff8102e48e9e60] generic_shutdown_super at ffffffff800e3f3f
#4 [ffff8102e48e9e80] kill_anon_super at ffffffff800e400f
#5 [ffff8102e48e9e90] deactivate_super at ffffffff800e40c0
#6 [ffff8102e48e9eb0] sys_umount at ffffffff800edf11

server_put_super() is at server_wait_finished(), it means this obd has been cleaned up. While the crash backtrace shows the obd is been cleaned up again. After checking obd_device_list_seq related code, I found `cat /proc/fs/lustre/devices` will class_getref() on obd, and then class_putref() later, but doing this on a cleaned up obd may cause obd_cleanup() called twice, and finally cause this crash.

I will add obd->obd_stopping check in obd_devlist_first/next() to fix this.

Comment by Peter Jones [ 24/May/11 ]

> Peter, during stress test I met one crash, this looks to be another issue related with procfs, should I fix it along in this ticket?

Yes but perhaps you should track it under a new ticket if it is a distinct issue

Comment by Peter Jones [ 24/May/11 ]

Ah it seems that you have already corrected this existing patch to address the new issue. That is fine

Comment by Lai Siyao [ 24/May/11 ]

Yes, because it blocked the stress test.

I've run the stress test without failure on two setup, one for 8 hours, and another 24 hours. The test script is as below:

  • one shell running walk.sh:
    while true; do
            cat /proc/fs/lustre/*
            cat /proc/fs/lustre/*/*
            cat /proc/fs/lustre/*/*/*
            cat /proc/fs/lustre/*/*/*/*
    
            cat /proc/sys/lnet/*
            cat /proc/sys/lustre/*
    
            usleep 20
    done
    
  • the other one running failover_ost.sh:
    while true; do
            umount -d /mnt/ost1
            sleep 1
            mount -t lustre -o loop /tmp/lustre-ost1 /mnt/ost1
            sleep 9
    done
    
Comment by Andrew Perepechko (Inactive) [ 14/Nov/11 ]

We have found one more race condition in lprocfs. In osd_device_alloc(), if a /proc read happens in between dt_device_init() and osd_device initialization, a kernel crash is triggered by functions like osd_statfs().


A reproducer:

1. patch lustre with the following

diff --git a/lustre/osd-ldiskfs/osd_handler.c b/lustre/osd-ldiskfs/osd_handler.c 
index 29764c9..728c371 100644 
--- a/lustre/osd-ldiskfs/osd_handler.c 
+++ b/lustre/osd-ldiskfs/osd_handler.c 
@@ -3874,6 +3874,9 @@ static int osd_device_init(const struct lu_env *env, struct lu_device *d, 
         rc = lu_context_init(ctx, LCT_MD_THREAD|LCT_REMEMBER|LCT_NOREF); 
         if (rc == 0) { 
                 rc = osd_procfs_init(osd_dev(d), name); 
+ 
+ cfs_schedule_timeout_and_set_state(CFS_TASK_UNINT, cfs_time_seconds(10)); 
+ 
                 ctx->lc_cookie = 0x3; 
         } 
         return rc; 

2. run llmount.sh
3. wait a few seconds and run "find /proc/fs/lustre -name 'filesfree' -exec cat {} \;"


We are cooking a fix in the context of MRP-275.

Comment by Andrew Perepechko [ 15/Nov/11 ]

The following patch (applied to other pieces of osd-lprocfs as well) fixes the issue for us:

@@ -200,7 +250,17 @@ int lprocfs_osd_rd_filesfree(char *page, char **start, off_t off, int count,
                              int *eof, void *data)
 {
         struct osd_device *osd = data;
-        int rc = osd_statfs(NULL, &osd->od_dt_dev, &osd->od_kstatfs);
+        int rc;
+
+        /*
+         * We possibly haven't mounted yet. When od_mount is assigned, it
+         * should be consistent and should not deinit until lprocfs is
+         * fully unregistered.
+         */
+        if (unlikely(osd->od_mount == NULL))
+                return -EINPROGRESS;
+
+        rc = osd_statfs(NULL, &osd->od_dt_dev, &osd->od_kstatfs);
         if (!rc) {
                 *eof = 1;
                 rc = snprintf(page, count, LPU64"\n", osd->od_kstatfs.f_ffree);

The race described above will work just like that:

[root@panda ~]# find /proc/fs/lustre -name 'filesfree' -exec cat {} \;
cat: /proc/fs/lustre/osd-ldiskfs/lustre-MDT0000/filesfree: Operation now in progress
cat: /proc/fs/lustre/lov/lustre-MDT0000-mdtlov/filesfree: Input/output error
Comment by Lai Siyao [ 15/Nov/11 ]

Nice catch, I'll go through the code and see whether other places needs this check too, and then I'll include this fix. Thanks!

Comment by Lai Siyao [ 21/Nov/11 ]

Fix included in latest patch: http://review.whamcloud.com/#change,326.

Comment by Peter Jones [ 12/Dec/11 ]

Landed for 2.2

Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,client,sles11,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/obdclass/obd_config.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/include/lustre_fld.h
  • lustre/include/lprocfs_status.h
  • lustre/mdt/mdt_handler.c
  • lustre/mdt/mdt_lproc.c
  • lustre/lov/lov_obd.c
  • lustre/obdfilter/filter.c
  • lustre/cmm/cmm_device.c
  • lustre/lmv/lmv_obd.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/fld/fld_request.c
  • lustre/mgs/mgs_handler.c
  • lustre/osc/osc_request.c
  • lustre/mdc/mdc_request.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,client,el6,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/fld/fld_request.c
  • lustre/lmv/lmv_obd.c
  • lustre/obdclass/obd_config.c
  • lustre/mdt/mdt_handler.c
  • lustre/cmm/cmm_device.c
  • lustre/obdfilter/filter.c
  • lustre/mdc/mdc_request.c
  • lustre/include/lustre_fld.h
  • lustre/lov/lov_obd.c
  • lustre/osc/osc_request.c
  • lustre/include/lprocfs_status.h
  • lustre/mgs/mgs_handler.c
  • lustre/mdt/mdt_lproc.c
  • lustre/obdclass/lprocfs_status.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,server,el6,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/mdt/mdt_lproc.c
  • lustre/include/lustre_fld.h
  • lustre/mgs/mgs_handler.c
  • lustre/cmm/cmm_device.c
  • lustre/fld/fld_request.c
  • lustre/lmv/lmv_obd.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/lov/lov_obd.c
  • lustre/obdclass/obd_config.c
  • lustre/osc/osc_request.c
  • lustre/mdc/mdc_request.c
  • lustre/mdt/mdt_handler.c
  • lustre/obdfilter/filter.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/include/lprocfs_status.h
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,client,el5,ofa #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/include/lprocfs_status.h
  • lustre/cmm/cmm_device.c
  • lustre/fld/fld_request.c
  • lustre/lov/lov_obd.c
  • lustre/obdclass/obd_config.c
  • lustre/mdc/mdc_request.c
  • lustre/mdt/mdt_lproc.c
  • lustre/osc/osc_request.c
  • lustre/mgs/mgs_handler.c
  • lustre/mdt/mdt_handler.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/lmv/lmv_obd.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/obdfilter/filter.c
  • lustre/include/lustre_fld.h
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » i686,client,el6,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/mdt/mdt_handler.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/mdt/mdt_lproc.c
  • lustre/osc/osc_request.c
  • lustre/lmv/lmv_obd.c
  • lustre/mgs/mgs_handler.c
  • lustre/include/lustre_fld.h
  • lustre/mdc/mdc_request.c
  • lustre/obdclass/obd_config.c
  • lustre/cmm/cmm_device.c
  • lustre/obdfilter/filter.c
  • lustre/fld/fld_request.c
  • lustre/include/lprocfs_status.h
  • lustre/lov/lov_obd.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,client,el5,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/obdclass/obd_config.c
  • lustre/cmm/cmm_device.c
  • lustre/mdt/mdt_lproc.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/include/lustre_fld.h
  • lustre/lmv/lmv_obd.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/lov/lov_obd.c
  • lustre/include/lprocfs_status.h
  • lustre/fld/fld_request.c
  • lustre/mdc/mdc_request.c
  • lustre/osc/osc_request.c
  • lustre/mdt/mdt_handler.c
  • lustre/mgs/mgs_handler.c
  • lustre/obdfilter/filter.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/osc/osc_request.c
  • lustre/lov/lov_obd.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/mdt/mdt_handler.c
  • lustre/cmm/cmm_device.c
  • lustre/obdclass/obd_config.c
  • lustre/lmv/lmv_obd.c
  • lustre/fld/fld_request.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/mgs/mgs_handler.c
  • lustre/include/lprocfs_status.h
  • lustre/mdc/mdc_request.c
  • lustre/obdfilter/filter.c
  • lustre/mdt/mdt_lproc.c
  • lustre/include/lustre_fld.h
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,server,el5,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/mdt/mdt_handler.c
  • lustre/lmv/lmv_obd.c
  • lustre/cmm/cmm_device.c
  • lustre/mgs/mgs_handler.c
  • lustre/osc/osc_request.c
  • lustre/include/lustre_fld.h
  • lustre/obdclass/lprocfs_status.c
  • lustre/include/lprocfs_status.h
  • lustre/lov/lov_obd.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/mdt/mdt_lproc.c
  • lustre/mdc/mdc_request.c
  • lustre/obdclass/obd_config.c
  • lustre/obdfilter/filter.c
  • lustre/fld/fld_request.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » i686,client,el5,ofa #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/mdc/mdc_request.c
  • lustre/obdfilter/filter.c
  • lustre/mgs/mgs_handler.c
  • lustre/mdt/mdt_lproc.c
  • lustre/include/lprocfs_status.h
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/cmm/cmm_device.c
  • lustre/mdt/mdt_handler.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/fld/fld_request.c
  • lustre/obdclass/obd_config.c
  • lustre/lmv/lmv_obd.c
  • lustre/osc/osc_request.c
  • lustre/include/lustre_fld.h
  • lustre/lov/lov_obd.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » i686,client,el5,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/obdclass/lprocfs_status.c
  • lustre/obdfilter/filter.c
  • lustre/include/lprocfs_status.h
  • lustre/fld/fld_request.c
  • lustre/mdc/mdc_request.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/osc/osc_request.c
  • lustre/lmv/lmv_obd.c
  • lustre/mdt/mdt_lproc.c
  • lustre/obdclass/obd_config.c
  • lustre/mdt/mdt_handler.c
  • lustre/include/lustre_fld.h
  • lustre/mgs/mgs_handler.c
  • lustre/cmm/cmm_device.c
  • lustre/lov/lov_obd.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » x86_64,server,el5,ofa #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = FAILURE
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/osc/osc_request.c
  • lustre/include/lprocfs_status.h
  • lustre/lmv/lmv_obd.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/cmm/cmm_device.c
  • lustre/obdfilter/filter.c
  • lustre/obdclass/obd_config.c
  • lustre/fld/fld_request.c
  • lustre/lov/lov_obd.c
  • lustre/mdc/mdc_request.c
  • lustre/mdt/mdt_lproc.c
  • lustre/mdt/mdt_handler.c
  • lustre/mgs/mgs_handler.c
  • lustre/include/lustre_fld.h
  • lustre/osd-ldiskfs/osd_lproc.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » i686,server,el5,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/mgs/mgs_handler.c
  • lustre/osc/osc_request.c
  • lustre/include/lustre_fld.h
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/lmv/lmv_obd.c
  • lustre/mdt/mdt_handler.c
  • lustre/obdfilter/filter.c
  • lustre/mdc/mdc_request.c
  • lustre/cmm/cmm_device.c
  • lustre/include/lprocfs_status.h
  • lustre/lov/lov_obd.c
  • lustre/obdclass/obd_config.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/fld/fld_request.c
  • lustre/mdt/mdt_lproc.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » i686,server,el6,inkernel #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/obdclass/obd_config.c
  • lustre/mdt/mdt_handler.c
  • lustre/mdt/mdt_lproc.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/fld/fld_request.c
  • lustre/lov/lov_obd.c
  • lustre/include/lprocfs_status.h
  • lustre/lmv/lmv_obd.c
  • lustre/obdfilter/filter.c
  • lustre/include/lustre_fld.h
  • lustre/mgs/mgs_handler.c
  • lustre/osc/osc_request.c
  • lustre/mdc/mdc_request.c
  • lustre/cmm/cmm_device.c
Comment by Build Master (Inactive) [ 12/Dec/11 ]

Integrated in lustre-master » i686,server,el5,ofa #378
LU-106 procfs: many proc entries are not accessed safely (Revision ad538dba7f5259df31fc45e90835edc12756db40)

Result = SUCCESS
Oleg Drokin : ad538dba7f5259df31fc45e90835edc12756db40
Files :

  • lustre/cmm/cmm_device.c
  • lustre/mgs/mgs_handler.c
  • lustre/obdclass/obd_config.c
  • lustre/include/lustre_fld.h
  • lustre/lov/lov_obd.c
  • lustre/mdt/mdt_lproc.c
  • lustre/obdclass/lprocfs_status.c
  • lustre/lmv/lmv_obd.c
  • lustre/obdfilter/filter.c
  • lustre/include/lprocfs_status.h
  • lustre/osc/osc_request.c
  • lustre/fld/fld_request.c
  • lustre/osd-ldiskfs/osd_lproc.c
  • lustre/mdc/mdc_request.c
  • lustre/mdt/mdt_handler.c
Generated at Sat Feb 10 05:10:33 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.