Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-2640

deactivate OSD_EXEC_OP() operation accounting if operation is being undone

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • Lustre 2.4.0
    • Lustre 2.4.0
    • 4
    • 6174

    Description

      In http://review.whamcloud.com/5046 there was a change to OSD_EXEC_OP() to address problems in MDD operations that fail part-way through, where the undo of earlier operations causes internal osd-ldiskfs declare/execute accounting to fail. For example, in mdd_create()::cleanup, the failed create calls __mdd_index_delete(), mdo_ref_del(), mdo_destroy() in order to clean up the newly created object.

      One proposal is to have some kind of OSD API call/flag/method to mark the transaction handle as being used for rollback, and to disable the ot_declare_op LASSERT() checking in OSD_EXEC_OP() for this case.

      Attachments

        Issue Links

          Activity

            [LU-2640] deactivate OSD_EXEC_OP() operation accounting if operation is being undone

            With patch set 4 test looks fine.

            aboyko Alexander Boyko added a comment - With patch set 4 test looks fine.

            Alexander, would you please to try this one? Thanks.

            http://review.whamcloud.com/#change,5138,set4

            yong.fan nasf (Inactive) added a comment - Alexander, would you please to try this one? Thanks. http://review.whamcloud.com/#change,5138,set4

            of course, it's better. though I was actually hoping for a breakthrough solution to improve checks and credits calculation.

            bzzz Alex Zhuravlev added a comment - of course, it's better. though I was actually hoping for a breakthrough solution to improve checks and credits calculation.

            Yes, it's hack. Then what do you prefer to resolve it?

            We can hack in OSD, and let osd_thandle::ot_declare_ops_rb[OSD_OT_REF_ADD] increase when osd_trans_exec_op(OSD_OT_CREATE) called, can you accept it?

            yong.fan nasf (Inactive) added a comment - Yes, it's hack. Then what do you prefer to resolve it? We can hack in OSD, and let osd_thandle::ot_declare_ops_rb [OSD_OT_REF_ADD] increase when osd_trans_exec_op(OSD_OT_CREATE) called, can you accept it?

            well, this is not quite correct. osd_object_destroy() does its own hack because of limits on nlink in ldiskfs. we should not bring this assumption into the API.

            bzzz Alex Zhuravlev added a comment - well, this is not quite correct. osd_object_destroy() does its own hack because of limits on nlink in ldiskfs. we should not bring this assumption into the API.

            Seems we do not need two ref_del() when rollback for mkdir, the osd_object_destroy() can handle the last nlink for dir object.

            Alex, how do you think?

            yong.fan nasf (Inactive) added a comment - Seems we do not need two ref_del() when rollback for mkdir, the osd_object_destroy() can handle the last nlink for dir object. Alex, how do you think?

            > mdd_declare_create() has one declare_ref_add, but mdd_create() cleanup has two ref_del().

            right, because mdd_create() does not declare initial nlink set to 1 by object creation
            (we discussed this problem few times in the past..)
            adding one more declare_ref_add() in mdd_create() w/o actual ref_add() would be confusing, imho.

            bzzz Alex Zhuravlev added a comment - > mdd_declare_create() has one declare_ref_add, but mdd_create() cleanup has two ref_del(). right, because mdd_create() does not declare initial nlink set to 1 by object creation (we discussed this problem few times in the past..) adding one more declare_ref_add() in mdd_create() w/o actual ref_add() would be confusing, imho.

            I try patchset 3

            00000004:00000001:1.0:1359025663.382680:0:14733:0:(mdd_dir.c:523:__mdd_index_insert_only()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4)
            00000004:00000001:1.0:1359025663.382681:0:14733:0:(mdd_dir.c:540:__mdd_index_insert()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4)
            00000004:00000001:1.0:1359025663.382682:0:14733:0:(mdd_dir.c:1794:mdd_create()) Process leaving via cleanup (rc=18446744073709551588 : -28 : 0xffffffffffffffe4)  
            00000004:00040000:1.0:1359025663.382684:0:14733:0:(osd_internal.h:412:osd_trans_exec_op()) ASSERTION( oh->ot_declare_ops_rb[rb] > 0 ) failed: 5
            00000004:00040000:1.0:1359025663.382799:0:14733:0:(osd_internal.h:412:osd_trans_exec_op()) LBUG
            
            Pid: 14733, comm: mdt01_001
            
            Call Trace:
             [<ffffffffa073e905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
             [<ffffffffa073ef07>] lbug_with_loc+0x47/0xb0 [libcfs]
             [<ffffffffa0e7d6e0>] osd_it_ea_get+0x0/0x140 [osd_ldiskfs]
             [<ffffffffa0e86c75>] osd_object_ref_del+0x135/0x210 [osd_ldiskfs]
             [<ffffffffa0bb132b>] lod_ref_del+0x3b/0xd0 [lod]
             [<ffffffffa0dbf5fd>] mdo_ref_del+0xad/0xb0 [mdd]
             [<ffffffffa0dc5ce2>] mdd_create+0xd02/0x1500 [mdd]
             [<ffffffffa074f371>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
             [<ffffffffa0f27321>] mdt_reint_open+0x1191/0x1900 [mdt]
             [<ffffffffa074f371>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
             [<ffffffffa0f121a1>] mdt_reint_rec+0x41/0xe0 [mdt]
             [<ffffffffa0f0b843>] mdt_reint_internal+0x4e3/0x7d0 [mdt]
             [<ffffffffa0f0bdfd>] mdt_intent_reint+0x1ed/0x4f0 [mdt]
             [<ffffffffa0f074fe>] mdt_intent_policy+0x3ae/0x750 [mdt]
            

            mdd_declare_create() has one declare_ref_add, but mdd_create() cleanup has two ref_del().

            aboyko Alexander Boyko added a comment - I try patchset 3 00000004:00000001:1.0:1359025663.382680:0:14733:0:(mdd_dir.c:523:__mdd_index_insert_only()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00000004:00000001:1.0:1359025663.382681:0:14733:0:(mdd_dir.c:540:__mdd_index_insert()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00000004:00000001:1.0:1359025663.382682:0:14733:0:(mdd_dir.c:1794:mdd_create()) Process leaving via cleanup (rc=18446744073709551588 : -28 : 0xffffffffffffffe4) 00000004:00040000:1.0:1359025663.382684:0:14733:0:(osd_internal.h:412:osd_trans_exec_op()) ASSERTION( oh->ot_declare_ops_rb[rb] > 0 ) failed: 5 00000004:00040000:1.0:1359025663.382799:0:14733:0:(osd_internal.h:412:osd_trans_exec_op()) LBUG Pid: 14733, comm: mdt01_001 Call Trace: [<ffffffffa073e905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [<ffffffffa073ef07>] lbug_with_loc+0x47/0xb0 [libcfs] [<ffffffffa0e7d6e0>] osd_it_ea_get+0x0/0x140 [osd_ldiskfs] [<ffffffffa0e86c75>] osd_object_ref_del+0x135/0x210 [osd_ldiskfs] [<ffffffffa0bb132b>] lod_ref_del+0x3b/0xd0 [lod] [<ffffffffa0dbf5fd>] mdo_ref_del+0xad/0xb0 [mdd] [<ffffffffa0dc5ce2>] mdd_create+0xd02/0x1500 [mdd] [<ffffffffa074f371>] ? libcfs_debug_msg+0x41/0x50 [libcfs] [<ffffffffa0f27321>] mdt_reint_open+0x1191/0x1900 [mdt] [<ffffffffa074f371>] ? libcfs_debug_msg+0x41/0x50 [libcfs] [<ffffffffa0f121a1>] mdt_reint_rec+0x41/0xe0 [mdt] [<ffffffffa0f0b843>] mdt_reint_internal+0x4e3/0x7d0 [mdt] [<ffffffffa0f0bdfd>] mdt_intent_reint+0x1ed/0x4f0 [mdt] [<ffffffffa0f074fe>] mdt_intent_policy+0x3ae/0x750 [mdt] mdd_declare_create() has one declare_ref_add, but mdd_create() cleanup has two ref_del().

            Andreas,

            Alexander worked on backport a directory size limits patch from upstream to lustre to avoid problems from LU-2479.
            quickly - we have created large directory, remove some entries, set limit, but not able to create something because that directory over limits. patch mainline, just forbid to extent a directory instead of creating anything.

            shadow Alexey Lyashkov added a comment - Andreas, Alexander worked on backport a directory size limits patch from upstream to lustre to avoid problems from LU-2479 . quickly - we have created large directory, remove some entries, set limit, but not able to create something because that directory over limits. patch mainline, just forbid to extent a directory instead of creating anything.

            Alexander, what are you using for testing this problem? It looks like the MDT is running out of space and fails in the error cleanup path.

            adilger Andreas Dilger added a comment - Alexander, what are you using for testing this problem? It looks like the MDT is running out of space and fails in the error cleanup path.

            Miss to call "OSD_EXEC_OP(th, insert)" in "osd_index_ea_insert()". It exists in master for a long time. Thanks for trigger it!

            I will update my patches to fix it.

            yong.fan nasf (Inactive) added a comment - Miss to call "OSD_EXEC_OP(th, insert)" in "osd_index_ea_insert()". It exists in master for a long time. Thanks for trigger it! I will update my patches to fix it.

            People

              yong.fan nasf (Inactive)
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: