Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-1532

only allow FID_SEQ_NORMAL from clients

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • None
    • Lustre 2.4.0, Lustre 2.1.4
    • # uname -r
      2.6.32-220.13.1.el6.l22.x86_64
      # cat /proc/fs/lustre/version
      lustre: 2.2.55
      kernel: patchless_client
      build: 2.2.55--CHANGED-2.6.32-220.13.1.el6.l22.x86_64
    • 2
    • 3
    • 3979

    Description

      [sanity@r62-lustre lustre]$ echo 'Achievement unlocked.' > .lustre/fid/[0xc:0x0:0x0]/SANITY
      [sanity@r62-lustre lustre]$ stat .lustre/fid/[0xc:0x0:0x0]/SANITY
      File: `.lustre/fid/[0xc:0x0:0x0]/SANITY'
      Size: 22 Blocks: 1 IO Block: 2097152 regular file
      Device: 2c54f966h/743766374d Inode: 144115205255725057 Links: 1
      Access: (0664/rw-rw-r-) Uid: ( 500/ sanity) Gid: ( 500/ sanity)
      Access: 2012-06-15 15:06:07.000000000 -0500
      Modify: 2012-06-15 15:06:05.000000000 -0500
      Change: 2012-06-15 15:06:05.000000000 -0500
      [sanity@r62-lustre lustre]$ cat .lustre/fid/[0xc:0x0:0x0]/SANITY
      Achievement unlocked.

      [root@r62-lustre lustre-release]# debugfs /dev/loop0
      debugfs 1.41.90.wc4 (01-Sep-2011)
      debugfs: ls CONFIGS
      12 (12) . 2 (12) .. 13 (20) mountdata 15 (20) _mgs-sptlrpc
      16 (24) lustre-client 17 (24) lustre-sptlrpc 18 (24) lustre-MDT0000
      19 (24) lustre-params 21 (28) changelog_catalog
      22 (44) changelog_users 93 (16) sptlrpc 95 (24) lustre-OST0000
      96 (24) lustre-OST0001 100 (3800) SANITY

      [sanity@r62-lustre lustre]$ rm .lustre/fid/[0xc:0x0:0x0]/mountdata

      Message from syslogd@r62-lustre at Jun 15 15:09:26 ...
      kernel:LustreError: 2858:0:(osd_internal.h:665:osd_fid2oi()) ASSERTION( !fid_is_igif(fid) ) failed:

      Message from syslogd@r62-lustre at Jun 15 15:09:26 ...
      kernel:LustreError: 2858:0:(osd_internal.h:665:osd_fid2oi()) LBUG

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. LU-1549 open-unlinked files are accessible through the PENDING directory

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-1518 Missing/bad operations in mdd_{obf,dot_lustre}_obj_op causing LBUGs

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Activity

            [LU-1532] only allow FID_SEQ_NORMAL from clients
            tappro Mikhail Pershin added a comment - - edited

            Fixed by patch for LU-1531:
            http://review.whamcloud.com/4255

            tappro Mikhail Pershin added a comment - - edited Fixed by patch for LU-1531 : http://review.whamcloud.com/4255
            adilger Andreas Dilger added a comment -

            This patch should fix this problem for the MDS: http://review.whamcloud.com/4255

            adilger Andreas Dilger added a comment - This patch should fix this problem for the MDS: http://review.whamcloud.com/4255
            adilger Andreas Dilger added a comment -

            The patch for LU-1518 (http://review.whamcloud.com/3726) blocks client access to FID_SEQ_DOT_LUSTRE for specific operations in order to prevent specific files in the .lustre tree from being renamed or otherwise modified.

            However, there are still a number of problems that may arise from clients accessing files in other FID_SEQ ranges. A safer approach would be to allow client->MDT access to only specific FID_SEQ ranges, and only open up those ranges if/when there is a specific need to do so. That will make the MDT and OST "safe by default" instead of blocking problematic ranges of objects after the fact.

            A preliminary list of acceptable SEQ/FIDs for the MDT:

            • "ROOT" (whatever specific FID it might be, as determined by MDT_GETSTATUS RPC and lookup-by-name)
            • FID_SEQ_ECHO (for mds-survey echo_client testing?)
            • FID_SEQ_IGIF..FID_SEQ_IGIF_MAX (only for filesystems that were upgraded from 1.8)
            • FID_SEQ_DOT_LUSTRE (blocked for specific operations at a lower level)
            • FID_SEQ_NORMAL

            The "ROOT" FID may have a variety of values (IGIF for 1.8 filesystems, one FID_SEQ_SPECIAL FID for new filesystems, FID_SEQ_NORMAL for filesystems upgraded by the Xyratex 2.x upgrade tool), so it should just be allowd regardless of what FID is actually in use. This can be instantiated by the reply from the MDT_GETSTATUS RPC, which is doing a lookup-by-name and should be able to bypass the by-FID sanity checks. Similarly, the FID_SEQ_IGIF range should only be allowed for upgraded 1.8 filesystems, otherwise it would allow access to any objects in the filesystem.

            I haven't thought of a good way to retroactively determine if the filesystem was newly formatted with 2.x vs. upgraded from 1.8 yet. One possibility is to set an OBD_COMPAT_IGIF flag in the last_rcvd file if the filesystem is upgraded from 1.8 (i.e. missing OBD_COMPAT_20) or if some lookup-by-name/OI Scrub finds an inode with an IGIF FID. Filesystems without OBD_COMPAT_IGIF set would refuse lookups by IGIF FIDs.

            There would have to be some delay in enforcing this restriction, since existing upgraded filesystems would not have it set and it might accidentally block IGIF lookups for previously-upgraded filesystems. Maybe the setting of OBD_COMPAT_IGIF could be added for 2.1.4/2.3/master, and enforcement only in master?

            Acceptable SEQ/FIDs for the OST:

            • FID_SEQ_OST_MDT0 (for legacy OST objects)
            • FID_SEQ_LLOG (do we still allow/need this for 2.4 OSTs?)
            • FID_SEQ_ECHO (obdfilter-survey testing)
            • FID_SEQ_IDIF..FID_SEQ_IDIF_MAX (for legacy OST objects)
            • FID_SEQ_NORMAL (for FID-on-OST)

            Unlike the MDT, there is no need to block legacy FID_SEQ_OST_MDT0/FID_SEQ_IDIF objects from clients, since they cannot be used to access objects via a different FID namespace.

            adilger Andreas Dilger added a comment - The patch for LU-1518 ( http://review.whamcloud.com/3726 ) blocks client access to FID_SEQ_DOT_LUSTRE for specific operations in order to prevent specific files in the .lustre tree from being renamed or otherwise modified. However, there are still a number of problems that may arise from clients accessing files in other FID_SEQ ranges. A safer approach would be to allow client->MDT access to only specific FID_SEQ ranges, and only open up those ranges if/when there is a specific need to do so. That will make the MDT and OST "safe by default" instead of blocking problematic ranges of objects after the fact. A preliminary list of acceptable SEQ/FIDs for the MDT: "ROOT" (whatever specific FID it might be, as determined by MDT_GETSTATUS RPC and lookup-by-name) FID_SEQ_ECHO (for mds-survey echo_client testing?) FID_SEQ_IGIF..FID_SEQ_IGIF_MAX (only for filesystems that were upgraded from 1.8) FID_SEQ_DOT_LUSTRE (blocked for specific operations at a lower level) FID_SEQ_NORMAL The "ROOT" FID may have a variety of values (IGIF for 1.8 filesystems, one FID_SEQ_SPECIAL FID for new filesystems, FID_SEQ_NORMAL for filesystems upgraded by the Xyratex 2.x upgrade tool), so it should just be allowd regardless of what FID is actually in use. This can be instantiated by the reply from the MDT_GETSTATUS RPC, which is doing a lookup-by-name and should be able to bypass the by-FID sanity checks. Similarly, the FID_SEQ_IGIF range should only be allowed for upgraded 1.8 filesystems, otherwise it would allow access to any objects in the filesystem. I haven't thought of a good way to retroactively determine if the filesystem was newly formatted with 2.x vs. upgraded from 1.8 yet. One possibility is to set an OBD_COMPAT_IGIF flag in the last_rcvd file if the filesystem is upgraded from 1.8 (i.e. missing OBD_COMPAT_20) or if some lookup-by-name/OI Scrub finds an inode with an IGIF FID. Filesystems without OBD_COMPAT_IGIF set would refuse lookups by IGIF FIDs. There would have to be some delay in enforcing this restriction, since existing upgraded filesystems would not have it set and it might accidentally block IGIF lookups for previously-upgraded filesystems. Maybe the setting of OBD_COMPAT_IGIF could be added for 2.1.4/2.3/master, and enforcement only in master? Acceptable SEQ/FIDs for the OST: FID_SEQ_OST_MDT0 (for legacy OST objects) FID_SEQ_LLOG (do we still allow/need this for 2.4 OSTs?) FID_SEQ_ECHO (obdfilter-survey testing) FID_SEQ_IDIF..FID_SEQ_IDIF_MAX (for legacy OST objects) FID_SEQ_NORMAL (for FID-on-OST) Unlike the MDT, there is no need to block legacy FID_SEQ_OST_MDT0/FID_SEQ_IDIF objects from clients, since they cannot be used to access objects via a different FID namespace.
            jhammond John Hammond added a comment -

            Hi Peter, Mike,

            Sorry I haven't had time to upload a patch on this one. For newly created filesystems it may be as simple as changing the mode to 0755 when CONFIGS is created, but I haven't tried to see if this would have any unwanted side-effects. For existing filesystems, a manual chmod may be the best solution.

            Note also that this issue applies to the OBJECTS and NIDTBL_VERSIONS directories.

            John

            jhammond John Hammond added a comment - Hi Peter, Mike, Sorry I haven't had time to upload a patch on this one. For newly created filesystems it may be as simple as changing the mode to 0755 when CONFIGS is created, but I haven't tried to see if this would have any unwanted side-effects. For existing filesystems, a manual chmod may be the best solution. Note also that this issue applies to the OBJECTS and NIDTBL_VERSIONS directories. John
            pjones Peter Jones added a comment -

            Mike

            Could you please look at this patch?

            Thanks

            Peter

            pjones Peter Jones added a comment - Mike Could you please look at this patch? Thanks Peter
            liwei Li Wei (Inactive) added a comment -

            The osd_fid2oi() side effect is being fixed under LU-1534.

            liwei Li Wei (Inactive) added a comment - The osd_fid2oi() side effect is being fixed under LU-1534 .

            People

              tappro Mikhail Pershin
              jhammond John Hammond
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: