Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-13638

lnet: discard the callback

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      We need discard callback in some case.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-13534 Landing an LU-12678 high likely introduce a random memory corruption bug

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-14499 o2iblnd: LU-13368 changes cause shutdown procedure to not complete

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-13368 lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-13638] lnet: discard the callback
            ys Yang Sheng added a comment -

            Hi, Shuichi,

            Do you have chance to verify the patch fixes the rmmod issue?

            Thanks,
            YangSheng

            ys Yang Sheng added a comment - Hi, Shuichi, Do you have chance to verify the patch fixes the rmmod issue? Thanks, YangSheng
            ys Yang Sheng made changes -
            Link New: This issue is related to DDN-966 [ DDN-966 ]
            ys Yang Sheng added a comment -

            Hi, Shuichi,

            Could you test with patch: https://review.whamcloud.com/#/c/41970/ please? It should stack on top of https://review.whamcloud.com/#/c/40937/. TIA.

            Thanks,
            YangSheng

            ys Yang Sheng added a comment - Hi, Shuichi, Could you test with patch: https://review.whamcloud.com/#/c/41970/ please? It should stack on top of https://review.whamcloud.com/#/c/40937/ . TIA. Thanks, YangSheng
            gerrit Gerrit Updater added a comment -

            Yang Sheng (ys@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/41970
            Subject: LU-13638 o2ib: test patch
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ca80d77843ebdab6963745fc474be2e3b8985aab

            gerrit Gerrit Updater added a comment - Yang Sheng (ys@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/41970 Subject: LU-13638 o2ib: test patch Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ca80d77843ebdab6963745fc474be2e3b8985aab
            sihara Shuichi Ihara added a comment -

            attached are debug log with enabling "net" flag. there were 4 x servers (10.0.11.22[4-7]@o2ib ) for test. I've captured debug of two servers which never completed shutdown.

            sihara Shuichi Ihara added a comment - attached are debug log with enabling "net" flag. there were 4 x servers (10.0.11.22 [4-7] @o2ib ) for test. I've captured debug of two servers which never completed shutdown.
            sihara Shuichi Ihara made changes -
            Attachment New: lctl-debug-lnet-es400nvx1-vm4.txt.gz [ 37754 ]
            Attachment New: lctl-debug-lnet-es400nvx1-vm3.txt.gz [ 37755 ]
            sihara Shuichi Ihara added a comment - - edited

            patch https://review.whamcloud.com/#/c/38845/ introcued a issue that ko2iblnd_shutdown never completed.
            A reproducible test case is below.

            1. Start Lustre with LNET-MR on the Infiniband network
            2. Turn off two IB ports on one of OSSs
            3. Umount OSTs on that particular OSS (assumed OSS failover)
            4. Two IB ports are back .
            5. OSTs are remounted on that OSS (assumed OSS failback)
            6. Stop all Lustre service and cleanup (lustre_rmmod) all lustre modules

            When lustre modules were unloaded on all OSSs, some of OSS's (or all of OSS) shutdown never completed due to hanging at ko2iblnd_shutdown. I also tried second patch https://review.whamcloud.com/40937, but the problem was still exist.

            btw, if server applied patch LU-14499 (reverted LU-13638 patch), this shutdown prolbem was gone.

            sihara Shuichi Ihara added a comment - - edited patch https://review.whamcloud.com/#/c/38845/ introcued a issue that ko2iblnd_shutdown never completed. A reproducible test case is below. Start Lustre with LNET-MR on the Infiniband network Turn off two IB ports on one of OSSs Umount OSTs on that particular OSS (assumed OSS failover) Two IB ports are back . OSTs are remounted on that OSS (assumed OSS failback) Stop all Lustre service and cleanup (lustre_rmmod) all lustre modules When lustre modules were unloaded on all OSSs, some of OSS's (or all of OSS) shutdown never completed due to hanging at ko2iblnd_shutdown. I also tried second patch https://review.whamcloud.com/40937 , but the problem was still exist. btw, if server applied patch LU-14499 (reverted LU-13638 patch), this shutdown prolbem was gone.
            adilger Andreas Dilger made changes -
            Link New: This issue is related to EX-2590 [ EX-2590 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-13368 [ LU-13368 ]
            simmonsja James A Simmons made changes -
            Link New: This issue is related to LU-14499 [ LU-14499 ]

            People

              ys Yang Sheng
              ys Yang Sheng
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: