[#LU-464] Test failure on test suite obdfilter-survey

[LU-464] Test failure on test suite obdfilter-survey Created: 27/Jun/11 Updated: 20/Jan/18 Resolved: 15/Jul/11
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	None
Fix Version/s:	Lustre 2.1.0

Type:

Bug

Priority:

Blocker

Reporter:

Maloo

Assignee:

Jinshan Xiong (Inactive)

Resolution:

Fixed

Votes:

Labels:

None

Severity:

Rank (Obsolete):

4953

Description

This issue was created by maloo for sarah <sarah@whamcloud.com>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/261d7db2-a032-11e0-aee5-52540025f9af.

Comments

Comment by Sarah Liu [ 27/Jun/11 ]

18:56:19:Lustre: DEBUG MARKER: == obdfilter-survey test 2a: Stripe F/S over the Network ============================================= 11:56:18 (1309114578)
18:56:20:Lustre: 6472:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000_osc->host_0_UUID netid 20000: select flavor null
18:56:20:Lustre: 6472:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
19:27:35:LustreError: 7354:0:(ldlm_lib.c:499:client_disconnect_export()) disconnecting disconnected device (lustre-OST0000_osc)
19:27:35:LustreError: 7354:0:(genops.c:1565:obd_zombie_import_add()) ASSERTION(imp->imp_sec == NULL) failed
19:27:35:LustreError: 7354:0:(genops.c:1565:obd_zombie_import_add()) LBUG
19:27:35:Pid: 7354, comm: lctl
19:27:35:
19:27:35:Call Trace:
19:27:35: [<00000000f8e095b0>] libcfs_debug_dumpstack+0x50/0x70 [libcfs]
19:27:35: [<00000000f8e09d4d>] lbug_with_loc+0x6d/0xd0 [libcfs]
19:27:35: [<00000000f8e199b6>] libcfs_assertion_failed+0x66/0x70 [libcfs]
19:27:36: [<00000000f9afcf61>] class_import_put+0x541/0x550 [obdclass]
19:27:36: [<00000000f9b1c39a>] class_handle_unhash+0x2a/0x40 [obdclass]
19:27:36: [<00000000f9b00c93>] class_destroy_import+0x23/0x80 [obdclass]
19:27:36: [<00000000f955e992>] osc_precleanup+0x402/0x560 [osc]
19:27:36: [<00000000f9b5d7b9>] lu_context_fini+0x19/0x40 [obdclass]
19:27:36: [<00000000f9b26d2c>] class_cleanup+0x2cc/0x1c70 [obdclass]
19:27:36: [<00000000c045e126>] get_page_from_freelist+0x96/0x370
19:27:36: [<00000000c045e34f>] get_page_from_freelist+0x2bf/0x370
19:27:36: [<00000000f9b2df49>] class_process_config+0x25e9/0x5ed0 [obdclass]
19:27:36: [<00000000c045fa6c>] blockable_page_cache_readahead+0x46/0x99
19:27:36: [<00000000c0429f21>] current_fs_time+0x4a/0x55
19:27:36: [<00000000c04f2d45>] copy_from_user+0x31/0x5d
19:27:36: [<00000000f9acf2ec>] obd_ioctl_getdata+0x2dc/0x22c0 [obdclass]
19:27:36: [<00000000c045a6dc>] do_generic_mapping_read+0x378/0x380
19:27:36: [<00000000f9af8ece>] class_handle_ioctl+0x43ce/0x4770 [obdclass]
19:27:36: [<00000000c045e126>] get_page_from_freelist+0x96/0x370
19:27:36: [<00000000c045e34f>] get_page_from_freelist+0x2bf/0x370
19:27:37: [<00000000f9ace70e>] obd_class_ioctl+0xae/0x470 [obdclass]
19:27:37: [<00000000c046604f>] __handle_mm_fault+0x690/0xaac
19:27:37: [<00000000f9ace660>] obd_class_ioctl+0x0/0x470 [obdclass]
19:27:37: [<00000000c0487625>] do_ioctl+0x1c/0x5d
19:27:37: [<00000000c0487bb9>] vfs_ioctl+0x47b/0x4d3
19:27:37: [<00000000c044cb5b>] audit_syscall_entry+0x18f/0x1b9
19:27:37: [<00000000c0487c59>] sys_ioctl+0x48/0x5f
19:27:37: [<00000000c0404f4b>] syscall_call+0x7/0xb
19:27:37: <IRQ>
19:27:37:Kernel panic - not syncing: LBUG

Comment by Peter Jones [ 08/Jul/11 ]

Lai will look at this one

Comment by Lai Siyao [ 11/Jul/11 ]

19:27:35:LustreError: 7354:0:(ldlm_lib.c:499:client_disconnect_export()) disconnecting disconnected device (lustre-OST0000_osc)
This message looks a bit suspicious, and I guess there may be a race in echo client setup/cleanup, will do some test to find the place tomorrow.

Comment by Lai Siyao [ 12/Jul/11 ]

The race looks like this:

1. `lctl --device $echo_client_dev detach`
     -> class_detach() puts last refcount of export
       -> obd_zombie_add_export(), and wake up zombie thread to cull this export.
2. zombie thread culls this export:
   class_destroy_export() put last refcount of echo_client_obd
     -> obd_cleanup()
       -> echo_device_free()
         -> echo_client_cleanup()
           -> osc_disconnect()
             -> client_disconnect_export() will clear cli->cl_import

3. `lctl --device $osc_dev cleanup`
     -> osc_precleanup() found cl_import is not NULL (not NULL means this import is not connected), it will call
      -> class_destroy_import()
        -> class_import_put()
          -> LASSERT(imp->imp_sec == NULL)

It's expected that step 2 is done before step 3, but if not (this can't be guaranteed currently), the ASSERT will be triggered because cli->cl_import is not cleared (see description in step 3).

I think this bug is introduced in CLIO for obdecho.

Comment by Lai Siyao [ 13/Jul/11 ]

Review is on http://review.whamcloud.com/#change,1093