Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-3451

Editor closes file very slow because of fsync()

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • Lustre 1.8.8
    • Kernel: 2.6.32-220.el6.x86_64
      Lustre: 1.8.8
    • 3
    • 8631

    Description

      When edit a file by emacs or vim and save it, it takes a couple of seconds to close it. This problen is not limited to a specific file, client or OST, and can be reproduced easily at any file or any client. It occurs with root user. There is no error recorded on MDS/OSS/client syslog for it.

      We traced the process and got Lustre log. According to the strace, it takes seconds while fsync(). And according to Lustre log of client, mdc_sync() waits for seconds until it recieves the reply. But the log on Lustre MDS does not contain any message about mds_sync().

      We do not find any problem of other metadata operations like file creation or direcotry creation. The simple test 'fsync.c' opens, writes to and then fsync a file but it can not reproduce the problem.

      Attachments

        Activity

          [LU-3451] Editor closes file very slow because of fsync()
          pjones Peter Jones made changes -
          Resolution New: Not a Bug [ 6 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          pjones Peter Jones added a comment -

          ok - thanks Ihara

          pjones Peter Jones added a comment - ok - thanks Ihara

          After we collected metadata stats, this performance problem was caused by too many metadata operation with some applications. We found that applications that was doing such metadata operation and stop the job, the system was going to be back normal. So, please close ticket now.

          ihara Shuichi Ihara (Inactive) added a comment - After we collected metadata stats, this performance problem was caused by too many metadata operation with some applications. We found that applications that was doing such metadata operation and stop the job, the system was going to be back normal. So, please close ticket now.

          Hi Andreas,

          It is slow accasionally, but very frequently. And we found that there are a lot of open/close operations on the system. Is it the cause of the problem?

          Do you know which direction I should do more research? Or is it too complex to fix it promptly?

          Thanks!

          lixi Li Xi (Inactive) added a comment - Hi Andreas, It is slow accasionally, but very frequently. And we found that there are a lot of open/close operations on the system. Is it the cause of the problem? Do you know which direction I should do more research? Or is it too complex to fix it promptly? Thanks!

          Is it slow on every close, or just occasionally?

          I think occasional slow closes are caused by close on the MDT changing the atime, but getting stuck in the journal transaction commit. There is an old Bugzilla bug for this problem also, but I'm not able to find it (possibly because it is not public?).

          adilger Andreas Dilger added a comment - Is it slow on every close, or just occasionally? I think occasional slow closes are caused by close on the MDT changing the atime, but getting stuck in the journal transaction commit. There is an old Bugzilla bug for this problem also, but I'm not able to find it (possibly because it is not public?).

          Thank you Peter. I would be grateful for any help.

          lixi Li Xi (Inactive) added a comment - Thank you Peter. I would be grateful for any help.
          pjones Peter Jones added a comment -

          Thanks for your submission Li Xi.

          pjones Peter Jones added a comment - Thanks for your submission Li Xi.
          pjones Peter Jones made changes -
          Assignee Original: WC Triage [ wc-triage ] New: Peter Jones [ pjones ]
          lixi Li Xi (Inactive) added a comment - Here are the logs: ftp://ftp.whamcloud.com/uploads/LU-3451/MMBK_log_20130605.tar.gz ftp://ftp.whamcloud.com/uploads/LU-3451/lustre_debug_mds_cl.tar1.gz
          lixi Li Xi (Inactive) made changes -
          Attachment Original: MMBK_lustre_debug_20130610.tar.gz [ 13034 ]

          People

            pjones Peter Jones
            lixi Li Xi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: