Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11933

sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )

Details

    • Bug
    • Resolution: Fixed
    • Major
    • Lustre 2.13.0, Lustre 2.12.4
    • Lustre 2.13.0, Lustre 2.12.3
    • None
    • 3
    • 9223372036854775807

    Description

      This issue was created by maloo for Alexander Boyko <c17825@cray.com>

      This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/3f081d66-2971-11e9-a318-52540065bddc

      [ 6056.684483] Lustre: DEBUG MARKER: == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 14:13:40 (1549376020)
      [ 6060.313122] LustreError: 4422:0:(osd_index.c:1117:osd_declare_dir_delete()) ASSERTION( dt_object_exists(dt) ) failed: 
      [ 6060.315139] LustreError: 4422:0:(osd_index.c:1117:osd_declare_dir_delete()) LBUG
      [ 6060.316456] Pid: 4422, comm: mdt00_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Jan 11 05:58:47 UTC 2019
      [ 6060.318137] Call Trace:
      [ 6060.318641]  [<ffffffffc0a127cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]
      [ 6060.319891]  [<ffffffffc0a1287c>] lbug_with_loc+0x4c/0xa0 [libcfs]
      [ 6060.321802]  [<ffffffffc1012acd>] osd_declare_dir_delete+0x27d/0x2b0 [osd_zfs]
      [ 6060.323583]  [<ffffffffc129870e>] lod_sub_declare_delete+0xbe/0x1f0 [lod]
      [ 6060.324964]  [<ffffffffc1273be4>] lod_declare_delete+0x24/0x30 [lod]
      [ 6060.326378]  [<ffffffffc130964f>] mdd_orphan_declare_insert+0x33f/0x690 [mdd]
      [ 6060.327797]  [<ffffffffc12f0fe8>] mdd_declare_create+0xa48/0xcb0 [mdd]
      [ 6060.329315]  [<ffffffffc1310d8d>] mdd_declare_migrate_create+0x159/0x3ad [mdd]
      [ 6060.330679]  [<ffffffffc12f61bb>] mdd_migrate+0xe9b/0x1810 [mdd]
      [ 6060.332092]  [<ffffffffc1187786>] mdt_reint_migrate_internal+0xf56/0x14e0 [mdt]
      [ 6060.333573]  [<ffffffffc1187f75>] mdt_reint_rename_or_migrate.isra.51+0x265/0x860 [mdt]
      [ 6060.335338]  [<ffffffffc1188580>] mdt_reint_migrate+0x10/0x20 [mdt]
      [ 6060.336542]  [<ffffffffc1188633>] mdt_reint_rec+0x83/0x210 [mdt]
      [ 6060.337928]  [<ffffffffc1165183>] mdt_reint_internal+0x6e3/0xaf0 [mdt]
      [ 6060.339487]  [<ffffffffc11704e7>] mdt_reint+0x67/0x140 [mdt]
      [ 6060.340551]  [<ffffffffc0e1740a>] tgt_request_handle+0xafa/0x1590 [ptlrpc]
      [ 6060.342292]  [<ffffffffc0dba99e>] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc]
      [ 6060.343705]  [<ffffffffc0dbe47c>] ptlrpc_main+0xbbc/0x2090 [ptlrpc]
      [ 6060.344944]  [<ffffffffa52c1c31>] kthread+0xd1/0xe0
      [ 6060.345964]  [<ffffffffa5974c37>] ret_from_fork_nospec_end+0x0/0x39
      [ 6060.347201]  [<ffffffffffffffff>] 0xffffffffffffffff
      

      VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
      sanity test_230b - crashed during sanity test_230b
      sanity-hsm test_406 - crashed during sanity-hsm test_406

      Attachments

        Issue Links

          Activity

            [LU-11933] sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )
            pjones Peter Jones added a comment -

            Landed for 2.13

            pjones Peter Jones added a comment - Landed for 2.13

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36154/
            Subject: LU-11933 mdt: clear sp_cr_flags in migrate unpack
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: d4da3b55a8303d937828e74341b3ab5c4dfd52b2

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36154/ Subject: LU-11933 mdt: clear sp_cr_flags in migrate unpack Project: fs/lustre-release Branch: master Current Patch Set: Commit: d4da3b55a8303d937828e74341b3ab5c4dfd52b2

            Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36154
            Subject: LU-11933 mdt: clear sp_cr_flags in migrate unpack
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 9f8a7b36fd9aeca8e8a752d92091da4de988db0f

            gerrit Gerrit Updater added a comment - Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36154 Subject: LU-11933 mdt: clear sp_cr_flags in migrate unpack Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 9f8a7b36fd9aeca8e8a752d92091da4de988db0f

            It looks like this is only crashing during directory migration, mostly in sanity test_230b and much less often in sanity-hsm test_406:
            https://testing.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&warn%5Bnotice%5D=&test_set_script_id=&status%5B%5D=CRASH&query_bugs=LU-11933&horizon=2332800&commit=Update+results&buggable_class=SubTest&num_results=250

            It's hit about once a day - not terrible, but probably deserves some attention.

            adilger Andreas Dilger added a comment - It looks like this is only crashing during directory migration, mostly in sanity test_230b and much less often in sanity-hsm test_406: https://testing.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&warn%5Bnotice%5D=&test_set_script_id=&status%5B%5D=CRASH&query_bugs=LU-11933&horizon=2332800&commit=Update+results&buggable_class=SubTest&num_results=250 It's hit about once a day - not terrible, but probably deserves some attention.
            adilger Andreas Dilger added a comment - +1 on master https://testing.whamcloud.com/test_sets/3c2314e6-ca9a-11e9-90ad-52540065bddc sanity test_230b
            mdiep Minh Diep added a comment - +1 on b2_12 https://testing.whamcloud.com/test_sessions/26c1e599-2cb2-4191-9492-37ca65e906d9
            adilger Andreas Dilger added a comment - +1 on master https://testing.whamcloud.com/test_sets/149b75f8-c5a4-11e9-a2b6-52540065bddc
            laisiyao Lai Siyao added a comment -

            No, it's related with volatile file migration, I'll check details later.

            laisiyao Lai Siyao added a comment - No, it's related with volatile file migration, I'll check details later.

            laisiyao,

            Do you think this crash is related to https://review.whamcloud.com/#/c/35360/ / LU-12485?  It's ... similar looking, at least.

            pfarrell Patrick Farrell (Inactive) added a comment - laisiyao , Do you think this crash is related to  https://review.whamcloud.com/#/c/35360/  / LU-12485 ?  It's ... similar looking, at least.
            wshilong Wang Shilong (Inactive) added a comment - https://testing.whamcloud.com/test_sets/81a7f204-b9a9-11e9-a2b6-52540065bddc
            yujian Jian Yu added a comment - +1 on master branch: https://testing.whamcloud.com/test_sets/e02cf1b6-b603-11e9-b753-52540065bddc

            People

              laisiyao Lai Siyao
              aboyko Alexander Boyko
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: