Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20149

ost-pools: crash due to NULL pointer dereference in qsd_op_end()

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      This issue was created by maloo for Marc Vef <mvef@whamcloud.com>

      This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/6d18dbfa-d1d4-4ee0-ac64-960c843fc693

      Test session details:
      clients: https://build.whamcloud.com/job/lustre-reviews/124049 - 5.14.0-503.40.1.el9_5.x86_64
      servers: https://build.whamcloud.com/job/lustre-reviews/124049 - 5.14.0-503.40.1_lustre.el9.x86_64

      NULL pointer dereference in qsd_op_end() due to qsd->qsd_dev being NULL

      10952.725580] Lustre: Failing over lustre-MDT0000
      [10952.734070] LustreError: 688160:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
      [10952.735420] Lustre: 686802:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery
      [10952.735472] Lustre: 686802:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages
      [10952.735525] LustreError: 686802:0:(ldlm_lib.c:1917:abort_lock_replay_queue()) @@@ aborted:  req@ffff8d13ee097400 x1862618159607680/t0(0) o101->a175cb78-4a6a-44e0-86fb-bc5363cfa578@10.240.22.147@tcp:221/0 lens 328/0 e 0 to 0 dl 1776341571 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295
      [10952.735681] Lustre: lustre-MDT0000-osd: cancel update llog [0x20003f3d0:0x1:0x0]
      [10952.738067] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400153f4:0x1:0x0]
      [10952.738535] LustreError: 686802:0:(client.c:1379:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff8d13cb2e30c0 x1862618000301696/t0(0) o700->lustre-MDT0001-osp-MDT0000@10.240.22.151@tcp:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'tgt_recover_0.0' uid:0 gid:0 projid:4294967295
      [10952.738682] LustreError: 686802:0:(fid_request.c:212:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5
      [10952.738789] LustreError: 686802:0:(fid_request.c:315:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5
      [10952.738879] Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x280005ddd:0x1:0x0]
      [10952.739251] Lustre: lustre-MDT0003-osp-MDT0000: cancel update llog [0x2c00059f4:0x1:0x0]
      [10952.739696] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 0 recovered and 5 were evicted.
      [10952.753961] Lustre: lustre-MDT0000: Not available for connect from 10.240.22.147@tcp (stopping)
      [10952.754012] Lustre: Skipped 59 previous similar messages
      [10952.897126] BUG: kernel NULL pointer dereference, address: 0000000000000060
      [10952.897134] #PF: supervisor read access in kernel mode
      [10952.897136] #PF: error_code(0x0000) - not-present page
      [10952.897138] PGD 0 P4D 0 
      [10952.897142] Oops: 0000 [#1] PREEMPT SMP NOPTI
      [10952.897145] CPU: 0 PID: 686758 Comm: ll_mgs_0002 Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.40.1_lustre.el9.x86_64 #1
      [10952.897148] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
      [10952.897150] RIP: 0010:qsd_op_end+0x11e/0x330 [lquota]
      [10952.897178] Code: c7 05 aa 73 04 00 01 00 00 00 48 c7 05 a7 73 04 00 00 00 00 00 e9 22 c5 45 ff 48 85 f6 0f 84 fc 00 00 00 48 8b 86 80 00 00 00 <f6> 40 60 02 0f 85 eb 00 00 00 4c 8d ab 08 01 00 00 4c 89 ef e8 19
      [10952.897180] RSP: 0018:ffffb2824198bb48 EFLAGS: 00010286
      [10952.897183] RAX: 0000000000000000 RBX: ffff8d13e5e4a600 RCX: 0000001e76d2e000
      [10952.897185] RDX: ffff8d13efb99b70 RSI: ffff8d13e5e4a600 RDI: ffff8d14010536c0
      [10952.897187] RBP: ffff8d13efb99b70 R08: ffffb2824198bbb0 R09: ffff8d13ec2c10a8
      [10952.897188] R10: 000000000000022e R11: 00000000000546d2 R12: ffff8d14010536c0
      [10952.897190] R13: 0000000000000000 R14: ffff8d13ec1b0000 R15: ffff8d13ee9db850
      [10952.897194] FS:  0000000000000000(0000) GS:ffff8d143bc00000(0000) knlGS:0000000000000000
      [10952.897196] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      [10952.897198] CR2: 0000000000000060 CR3: 0000000121a9e005 CR4: 0000000000770ef0
      [10952.897201] PKRU: 55555554
      [10952.897203] Call Trace:
      [10952.897205]  <TASK>
      [10952.897206]  ? srso_alias_return_thunk+0x5/0xfbef5
      [10952.897212]  ? show_trace_log_lvl+0x26e/0x2df
      [10952.897218]  ? show_trace_log_lvl+0x26e/0x2df
      [10952.897224]  ? osd_trans_stop+0x35e/0x850 [osd_ldiskfs]
      [10952.897255]  ? __die_body.cold+0x8/0xd
      [10952.897259]  ? page_fault_oops+0x134/0x170
      [10952.897264]  ? exc_page_fault+0x62/0x150
      [10952.897269]  ? asm_exc_page_fault+0x22/0x30
      [10952.897275]  ? qsd_op_end+0x11e/0x330 [lquota]
      [10952.897314]  osd_trans_stop+0x35e/0x850 [osd_ldiskfs]
      [10952.897345]  ? srso_alias_return_thunk+0x5/0xfbef5
      [10952.897348]  ? osd_write+0x10d/0x4a0 [osd_ldiskfs]
      [10952.897378]  ? srso_alias_return_thunk+0x5/0xfbef5
      [10952.897382]  nidtbl_update_version+0x21f/0x540 [mgs]
      [10952.897406]  mgs_nidtbl_write+0x1aa/0x420 [mgs]
      [10952.897427]  mgs_ir_update+0x7b/0x2a0 [mgs]
      [10952.897448]  mgs_target_reg+0x92f/0x1b60 [mgs]
      [10952.897468]  ? srso_alias_return_thunk+0x5/0xfbef5
      [10952.897471]  ? req_capsule_server_pack+0x1f1/0x2c0 [ptlrpc]
      [10952.897615]  ? srso_alias_return_thunk+0x5/0xfbef5
      [10952.897620]  tgt_handle_request0+0x14a/0x770 [ptlrpc]
      [10952.897774]  tgt_request_handle+0x3fd/0xd00 [ptlrpc]
      [10952.897921]  ptlrpc_server_handle_request.isra.0+0x2e8/0xd80 [ptlrpc]
      [10952.897990]  ? srso_alias_return_thunk+0x5/0xfbef5
      [10952.897993]  ptlrpc_main+0x9bf/0xea0 [ptlrpc]
      [10952.898057]  ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
      [10952.898119]  kthread+0xe0/0x100
      [10952.898123]  ? __pfx_kthread+0x10/0x10
      [10952.898126]  ret_from_fork+0x2c/0x50
      [10952.898132]  </TASK>
      [10952.898133] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp mlx5_ib kvm ib_uverbs macsec ib_core iTCO_wdt iTCO_vendor_support pcspkr virtio_balloon lpc_ich i2c_i801 i2c_smbus joydev fuse drm ext4 mbcache jbd2 mlx5_core crct10dif_pclmul crc32_pclmul mlxfw crc32c_intel ahci libahci libata ghash_clmulni_intel tls virtio_net virtio_blk psample net_failover pci_hyperv_intf failover serio_raw
      [10952.898182] CR2: 0000000000000060
      

            wc-triage WC Triage
            maloo Maloo
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: