Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11827

Race between llog_cat_declare_add_rec and llog_cat_current_log

Details

    • 3
    • 9223372036854775807

    Description

      llog_cat_declare_add_rec() operates on &cathandle->u.chd.chd_next_log without having it protected:

      int llog_cat_declare_add_rec(const struct lu_env *env,
      ...
              rc = llog_cat_prep_log(env, cathandle,
                                     &cathandle->u.chd.chd_current_log, th);
      ...
      	rc = llog_cat_prep_log(env, cathandle, &cathandle->u.chd.chd_next_log,
                                     th);
      

      That races with llog_cat_current_log() when it switches to next log and updates cathandle->u.chd.chd_next_log:

      static struct llog_handle *llog_cat_current_log(struct llog_handle *cathandle,
      ...
              down_write_nested(&cathandle->lgh_lock, LLOGH_CAT);
      ...
              CDEBUG(D_INODE, "use next log\n");
       
              loghandle = cathandle->u.chd.chd_next_log;
              cathandle->u.chd.chd_current_log = loghandle;
              cathandle->u.chd.chd_next_log = NULL;
              down_write_nested(&loghandle->lgh_lock, LLOGH_LOG);
      ...
      

      The following trace has been observed:
      Process 177713 enters llog_cat_declare_add_rec():

      00000040:00000001:19.0:1545138333.143874:0:177713:0:(llog_cat.c:605:llog_cat_declare_add_rec()) Process entered
      00000040:00000001:19.0:1545138333.143875:0:177713:0:(llog.c:940:llog_exist()) Process leaving (rc=1 : 1 : 1)
      00000040:00000001:19.0:1545138333.143876:0:177713:0:(llog.c:940:llog_exist()) Process leaving (rc=0 : 0 : 0)
      

      Process 99986 jumps in and switches pointer to next log in cathalog handle to NULL:

      00000040:00000002:21.0:1545138333.143876:0:99986:0:(llog_cat.c:521:llog_cat_current_log()) use next log
      

      Process 177713 continues: llog_cat_prep_log->llog_declare_create->llog_handle2ops, find NULL in and fails in llog_handle2ops() with -22 as long as *ploghandle is NULL:

      00000040:00000001:19.0:1545138333.143877:0:177713:0:(llog.c:954:llog_declare_create()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea)
      

      Attachments

        Issue Links

          Activity

            [LU-11827] Race between llog_cat_declare_add_rec and llog_cat_current_log

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34455/
            Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set:
            Commit: 834d54138b250ddedc06f478e9cdfdbf0c352bda

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34455/ Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 834d54138b250ddedc06f478e9cdfdbf0c352bda

            Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34455
            Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set: 1
            Commit: 8fe07060d1dce0813ca92178d2e2541e43e99f20

            gerrit Gerrit Updater added a comment - Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34455 Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 8fe07060d1dce0813ca92178d2e2541e43e99f20

            Looks like this was hit on 2.12.0 by a customer:
            https://jira.whamcloud.com/browse/LU-11984

            pfarrell Patrick Farrell (Inactive) added a comment - Looks like this was hit on 2.12.0 by a customer: https://jira.whamcloud.com/browse/LU-11984
            pjones Peter Jones added a comment -

            Landed for 2.13

            pjones Peter Jones added a comment - Landed for 2.13

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33914/
            Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 59a62ada2e18174e5611730e8bcf5ba3165ca2b9

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33914/ Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec Project: fs/lustre-release Branch: master Current Patch Set: Commit: 59a62ada2e18174e5611730e8bcf5ba3165ca2b9
            green Oleg Drokin added a comment -

            Please see a somewhat related failure scenario in LU-12008, do you think this will help there too?

            green Oleg Drokin added a comment - Please see a somewhat related failure scenario in LU-12008 , do you think this will help there too?

            Vladimir Saveliev (c17830@cray.com) uploaded a new patch: https://review.whamcloud.com/33914
            Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 052bfbd11bad6a04821a99ba60094b55cab38ab6

            gerrit Gerrit Updater added a comment - Vladimir Saveliev (c17830@cray.com) uploaded a new patch: https://review.whamcloud.com/33914 Subject: LU-11827 llog: protect cathandle in llog_cat_declare_add_rec Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 052bfbd11bad6a04821a99ba60094b55cab38ab6

            People

              laisiyao Lai Siyao
              vsaveliev Vladimir Saveliev
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: