[LU-10804] obdfilter-survey test_3a: lprocfs_alloc_obd_stats()) ASSERTION( obd->obd_proc_entry != ((void *)0) ) failed; LBUG

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0
• Affects Version/s: Lustre 2.11.0
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Maloo

Description

This issue was created by maloo for elena <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/5d126b54-2233-11e8-9ec4-52540065bddc

[ 3072.347530] LustreError: 30713:0:(lprocfs_status_server.c:647:lprocfs_alloc_obd_stats()) ASSERTION( obd->obd_proc_entry != ((void *)0) ) failed: 
[ 3072.351782] LustreError: 30713:0:(lprocfs_status_server.c:647:lprocfs_alloc_obd_stats()) LBUG
[ 3072.354022] Pid: 30713, comm: lctl
[ 3072.355821] 
[ 3072.355821] Call Trace:
[ 3072.359137]  [<ffffffffc06917ae>] libcfs_call_trace+0x4e/0x60 [libcfs]
[ 3072.361177]  [<ffffffffc069183c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[ 3072.363336]  [<ffffffffc0cbe06f>] lprocfs_alloc_obd_stats+0x18f/0x200 [obdclass]
[ 3072.365454]  [<ffffffffc0c81730>] ? lprocfs_obd_setup+0x130/0x320 [obdclass]
[ 3072.367602]  [<ffffffffc12b13ad>] echo_srv_device_alloc+0x48d/0x540 [obdecho]
[ 3072.369738]  [<ffffffffc0c91cb4>] obd_setup+0x114/0x2a0 [obdclass]
[ 3072.371892]  [<ffffffffc0c920e8>] class_setup+0x2a8/0x840 [obdclass]
[ 3072.373960]  [<ffffffffc0c9651f>] class_process_config+0x1b3f/0x27e0 [obdclass]
[ 3072.376145]  [<ffffffffc0c86ff9>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[ 3072.378240]  [<ffffffffc0c7053a>] class_handle_ioctl+0x56a/0x1df0 [obdclass]
[ 3072.380342]  [<ffffffff8121f7b8>] ? destroy_inode+0x38/0x60
[ 3072.382250]  [<ffffffff812b3ea8>] ? security_capable+0x18/0x20
[ 3072.384178]  [<ffffffffc0c567f2>] obd_class_ioctl+0xd2/0x170 [obdclass]
[ 3072.386089]  [<ffffffff8121730d>] do_vfs_ioctl+0x33d/0x540
[ 3072.387929]  [<ffffffff81224a04>] ? mntput+0x24/0x40
[ 3072.389676]  [<ffffffff81205013>] ? __fput+0x183/0x260
[ 3072.391419]  [<ffffffff812175b1>] SyS_ioctl+0xa1/0xc0
[ 3072.393128]  [<ffffffff816b8930>] ? system_call_after_swapgs+0x15d/0x214
[ 3072.394862]  [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[ 3072.396554]  [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
[ 3072.398245] 
[ 3072.399550] Kernel panic - not syncing: LBUG

Issue Links

• is related to: LU-8066 Move lustre procfs handling to sysfs and debugfs.

Comments

[Andreas Dilger]

Looking at the assertion, this appears like it is related to changes in /proc, possibly related to /sysfs changes? Possibly James already has a patch to fix this part of the code?

[Peter Jones]

Mike

Can you please investigate?

Thanks

Peter

[Elena Gryaznova]

This was not observed in the latest master test results because of test_3a was not functional, see LU-7420 :

> this test passes at Maloo, seems it does just nothing.

LU-7420 patch https://review.whamcloud.com/#/c/18443/ fixes the test to be executed. The reported issue hit with this patch.

[Oleg Drokin]

hm, so that's all after Mar 6th round of landings, I guess.

[James Nunez (Inactive)]

I've reviewed all obdfilter-survey test 3a hangs over all branches starting from January 1, 2018 to today and I've found the following test sessions that have the same assertion as in this ticket:

2018-03-09 22:59:29 UTC full-patchless - https://testing.hpdd.intel.com/test_sets/67d21086-240d-11e8-b74b-52540065bddc

2018-03-09 23:53:56 UTC full-patchless - https://testing.hpdd.intel.com/test_sets/6d7aa550-240f-11e8-b3c6-52540065bddc

2018-03-10 02:13:04 UTC full-patchless - https://testing.hpdd.intel.com/test_sets/8d7b15dc-242e-11e8-b74b-52540065bddc

2018-03-09 00:54:41 UTC full-patchless - https://testing.hpdd.intel.com/test_sets/7c8f22cc-2348-11e8-9852-52540065bddc

2018-03-08 18:53:42 UTC full-patchless - https://testing.hpdd.intel.com/test_sets/965a62b4-231d-11e8-9852-52540065bddc

2018-03-08 21:41:26 UTC full-patchless - https://testing.hpdd.intel.com/test_sets/15d7653a-233e-11e8-8d2f-52540065bddc

2018-03-08 22:23:08 UTC full-patchless – https://testing.hpdd.intel.com/test_sets/0f8acdfa-2340-11e8-88de-52540065bddc

All are from full test sessions for our patchless Lustre builds/testing.

[Oleg Drokin]

The report referenced is for a gerrit patch that is based on another patch that seems to change relevant functionality. We looked and this sort of failure was not observed in the latest master test results, so the primary suspicion is it's the patch in question that broke things, esp. since if you look, the baseline patch is "move procfs stuff to debugfs" and the assertion you hit is in procfs-related code.

[Cory Spitz]

Obdfilter-survey ought to work very reliably.  We should consider this a major issue (or greater).