[#LU-1391] lov device is released while some osc devices are still remaining on a Client node

The problem happened with Fujitsu customized Lustre, named FEFS. But after reading the source code of Lustre-1.8.7-wc1, I found that the same problem can happen in Lustre-1.8.7-wc1 environment. Then, I reported it in wc-discuss and I was recommended by Oreg to report it. So I created the ticket here... anyway, below is the problem.

After mount.lustre command is executed on a client node, ptlrpcd-recov thread activate the import objects of OSC. Then, in the middle of this way, osc_import_event() is called and it calls obd_notify_observer() like below.

static int osc_import_event(struct obd_device *obd,
                            struct obd_import *imp,
                            enum obd_import_event event)
{
        ...
        case IMP_EVENT_ACTIVE: {
                /* Only do this on the MDS OSC's */
                if (imp->imp_server_timeout) {
                        struct osc_creator *oscc = &obd->u.cli.cl_oscc;

                        spin_lock(&oscc->oscc_lock);
                        oscc->oscc_flags &= ~OSCC_FLAG_NOSPC;
                        spin_unlock(&oscc->oscc_lock);
                }
                CDEBUG(D_INFO, "notify server \n");
                rc = obd_notify_observer(obd, obd, OBD_NOTIFY_ACTIVE,NULL);
                break;
        }
        ...

}

static inline int obd_notify_observer(struct obd_device *observer,
                                      struct obd_device *observed,
                                      enum obd_notify_event ev, void *data)
{
        int rc1;
        int rc2;

        struct obd_notify_upcall *onu;

        if (observer->obd_observer)
                rc1 = obd_notify(observer->obd_observer, observed, ev,data);
        else
                rc1 = 0;
        /*
         * Also, call non-obd listener, if any
         */
        onu = &observer->obd_upcall;
        if (onu->onu_upcall != NULL)
                rc2 = onu->onu_upcall(observer, observed, ev, onu->onu_owner);

        else
                rc2 = 0;
        return rc1 ?: rc2;
 }

obd_notify_observer() calls obd_notify() with "observer->obd_observer", this must be a LOV object when the argument of "observer" is a OSC object. And then, lov_notify() works with the "observer->obd_observer". But, as you can see, obd_putref(), class_incref() and something like these have never been called to pin the "observer->obd_observer", lov object. and it's the same even when obd_observer is registered.

As a result, the "observer->obd_observer" can be released in the middile of its execution, for example, when umount is executed in just the execution. Then, it will cause OS panic. Actually, I've already succeeded to cause the same problem in purpose with the example, althought it's so rare case in real user operations.

PID: 17084  TASK: ffff81063fb770c0  CPU: 10  COMMAND: "umount"
 #0 [ffff810594801640] crash_kexec at ffffffff800aeb6b
 #1 [ffff810594801700] __die at ffffffff80066157
 #2 [ffff810594801740] die at ffffffff8006cce5
 #3 [ffff810594801770] do_general_protection at ffffffff8006659f
 #4 [ffff8105948017b0] error_exit at ffffffff8005ede9
    [exception RIP: osc_import_event+635]
    RIP: ffffffff8889a37b  RSP: ffff810594801868  RFLAGS: 00010202
    RAX: 0000000000000001  RBX: ffff8105974b23b8  RCX: 0000000000000000
    RDX: 5a5a5a5a5a5a5a5a  RSI: ffff81061222a800  RDI: ffff810593930338
    RBP: ffff810593930338   R8: ffff8105f4ab5000   R9: 00000000000000d8
    R10: 00000000ffffffff  R11: 0000000000000000  R12: ffff81061222a800
    R13: 0000000000000000  R14: ffff810c3e8f3200  R15: ffff8105939303b8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #5 [ffff8105948018c0] ptlrpc_deactivate_and_unlock_import at ffffffff887bdd06
 #6 [ffff810594801900] ptlrpc_invalidate_import at ffffffff887be31a
 #7 [ffff8105948019b0] client_disconnect_export at ffffffff8875b811
 #8 [ffff810594801a20] osc_disconnect at ffffffff88892588
 #9 [ffff810594801a70] class_disconnect_export_list at ffffffff886a1c51
#10 [ffff810594801ad0] class_disconnect_exports at ffffffff886a47bf
#11 [ffff810594801b30] class_cleanup at ffffffff886ba1fc
#12 [ffff810594801c20] class_process_config at ffffffff886be77c
#13 [ffff810594801ca0] class_manual_cleanup at ffffffff886c0544
#14 [ffff810594801d90] ll_put_super at ffffffff88a6271c
#15 [ffff810594801e60] generic_shutdown_super at ffffffff800e4bc0
#16 [ffff810594801e80] kill_anon_super at ffffffff800e4c90
#17 [ffff810594801e90] deactivate_super at ffffffff800e4d41
#18 [ffff810594801eb0] sys_umount at ffffffff800ee830
#19 [ffff810594801f80] system_call at ffffffff8005e116
    RIP: 000000394bad3dd7  RSP: 00007fffef2dfc28  RFLAGS: 00010246
    RAX: 00000000000000a6  RBX: ffffffff8005e116  RCX: 00000000004068df
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000016df0060
    RBP: 0000000016df0030   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fffef2e0138
    R13: 0000000016df0060  R14: 0000000016df00c0  R15: 0000000016def530
    ORIG_RAX: 00000000000000a6  CS: 0033  SS: 002b

PID: 16786  TASK: ffff810c34d207a0  CPU: 4   COMMAND: "ptlrpcd-recov"
 #0 [ffff81065522ef20] crash_nmi_callback at ffffffff8007bf44
 #1 [ffff81065522ef40] do_nmi at ffffffff8006688a
 #2 [ffff81065522ef50] nmi at ffffffff80065eef
    [exception RIP: lov_notify+1102]
    RIP: ffffffff889d08ee  RSP: ffff81059beddbb0  RFLAGS: 00000216
    RAX: 00000000095202b9  RBX: 0017c04c001be959  RCX: 00000005490682ff
    RDX: 0017c04e00000000  RSI: 0000000000000206  RDI: ffff810592268200
    RBP: ffff810598eb22b8   R8: 0000000000000032   R9: 0000000000000020
    R10: 000000000000001a  R11: 0000000000000000  R12: ffff8105974b2820
    R13: ffff8105974b23b8  R14: ffff810c3fecd800  R15: ffff810598eb2738
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #3 [ffff81059beddbb0] lov_notify at ffffffff889d08ee
 #4 [ffff81059beddc48] osc_import_event at ffffffff8889a83c
 #5 [ffff81059beddca8] ptlrpc_activate_import at ffffffff887bcb25
 #6 [ffff81059beddce8] ptlrpc_connect_interpret at ffffffff887bf30f
 #7 [ffff81059bedddb8] ptlrpc_check_set at ffffffff88786ab6
 #8 [ffff81059bedde68] ptlrpcd_check at ffffffff887c1add
 #9 [ffff81059beddeb8] ptlrpcd at ffffffff887c2421
#10 [ffff81059beddf48] kernel_thread at ffffffff8005efb1

As you can see. The RDX's of the umounts shown above refers to bad memory, 0x5a5a5a5a5a5a5a5a. And looking to "obd_devs" shown below indicates that MDS and LOV have been already wiped out at that time. These remaining are MGC and three OSCs, by the way.

obd_devs = $2 =
 {0xffff810c348d40b8, 0x0, 0x0, 0xffff810593930338, 0xffff810598eb22b8, 0xffff8105df0b43f8, 0x0, 0x0, 0x0, ... }

I've tried to fix the problem with some ideas like below. But all of them could n't work.

1) Add obd_gutref() in obd_register_obserever(), and create a new func, obd_unregister_observer() which calls obd_putref() and substitutes NULL to obd_observer.- This is invalid to the problem of releasing LOV obd_device prior to OSCs. Because obd_refcount reaches zero before lov_refcount reaches zero

2) Add class_incref()/class_decref() just before/after obd_getref()/obd_putref() in obd_register_observer()

3) managing MDS's and LOV's obd_refcounts in the kernel side of mount/umount causes a releasing MGC prior to MDS and LOV. then, I gave up today.

I'm really seeking the idea to fix the problem. So could someone help me with it ??
thank you.

[LU-1391] lov device is released while some osc devices are still remaining on a Client node Created: 09/May/12 Updated: 23/Jan/13 Resolved: 23/Jan/13
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 1.8.8
Fix Version/s:	None

[LU-1391] lov device is released while some osc devices are still remaining on a Client node Created: 09/May/12 Updated: 23/Jan/13 Resolved: 23/Jan/13