[LU-3791] sanity-hsm test_8: 'request on 0x200002341:0x2:0x0 is not SUCCEED'

• Type: Technical task
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.5.0
• Affects Version/s: Lustre 2.5.0
• Labels: HSM revzfs
• Assignee: John Hammond
• Reporter: Maloo

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite runs:
http://maloo.whamcloud.com/test_sets/21cdd37a-094d-11e3-b004-52540035b04c
http://maloo.whamcloud.com/test_sets/2ee8358a-0931-11e3-b004-52540035b04c
http://maloo.whamcloud.com/test_sets/25a97004-094c-11e3-a9b0-52540035b04c

The sub-test test_8 failed with the following error:

request on 0x200008101:0x2:0x0 is not SUCCEED

Info required for matching: sanity-hsm 8

Comments

[John Hammond]

Bobbie, "won't fix" was probably not the best resolution here, since this issue was fixed by the configuration changes on rosso. The test session you linked above was run before that fix was applied, however.

[Bobbie Lind (Inactive)]

I realize this ticket is in a "wont fix" resolved status but I'm still hitting it even with the temporary work around in place from TEI-534 https://maloo.whamcloud.com/test_sets/db1ad8ea-170e-11e3-9d30-52540035b04c

[Jinshan Xiong (Inactive)]

Hi Aurelien,

To support multiple agents, we're using NFS share so that agents can access the same archive. To archive a lustre file, lhsmtool_posix has to copy file data, xattr and attr to the corresponding file in archive. The problem is to copy attr we have to change the file owner in NFS share.

On rosso, which is a cluster to run autotest, has root_squash enabled, so every attempt to change file owner to root will fail. This then causes archive failure this is why we have seen so many failures on sanity-hsm recently.

TEI-534 is an internal task to request administrator to disable root_squash on the NFS server.

[Aurelien Degremont (Inactive)]

It is possible to know what is in TEI-534?

[Jinshan Xiong (Inactive)]

I believe this issue will no longer exist after TEI-534 is fixed.

[John Hammond]

Please see http://review.whamcloud.com/7581. This change passes the --no-attr and --no-xattr to the copytool in sanity-hsm.sh. Let's see how it does.

[John Hammond]

But why are there no error messages in the copytool logs?

[Jinshan Xiong (Inactive)]

I think we can fix this issue by setting correct permission on nfs share.

[Jinshan Xiong (Inactive)]

I just did an experiment on rosso and it verified my guess, as follows:

[root@wtm-12vm2 0000]# ls -l
total 7172
-rw------- 1 nfsnobody nfsnobody 7340032 Sep  3 13:21 0x200002341:0x211:0x0_tmp
-rw------- 1 nfsnobody nfsnobody      56 Sep  3 13:21 0x200002341:0x211:0x0_tmp.lov
[root@wtm-12vm2 0000]# cp /etc/passwd .
[root@wtm-12vm2 0000]# chown root.root passwd
chown: changing ownership of `passwd': Operation not permitted
[root@wtm-12vm2 0000]# pwd
/home/cgearing/.autotest/shared_dir/2013-09-03/040347-70339907358780/arc1/0211/0000/2341/0000/0002/0000

I did the same thing on toro and it worked so this is why the tests only failed sometimes. If the test was running on rosso, it failed. pretty nasty, huh?

[Jinshan Xiong (Inactive)]

From what I have seen, the copy process was successful, copying xattrs should be okay as well. So from the snippet of ct_archive(), it should fail around:

        CT_TRACE("'%s' data archived to '%s' done\n", src, dst);

        /* attrs will remain on the MDS; no need to copy them, except possibly
          for disaster recovery */
        if (opt.o_copy_attrs) {
                rc = ct_copy_attr(src, dst, src_fd, dst_fd);
                if (rc < 0) {
                        CT_ERROR("'%s' attr copy failed to '%s' (%s)\n",
                                 src, dst, strerror(-rc));
                        rcf = rc;
                }
                CT_TRACE("'%s' attr file copied to archive '%s'\n",
                         src, dst);
        }

        /* xattrs will remain on the MDS; no need to copy them, except possibly
         for disaster recovery */
        if (opt.o_copy_xattrs) {
                rc = ct_copy_xattr(src, dst, src_fd, dst_fd, false);
                if (rc < 0) {
                        CT_ERROR("'%s' xattr copy failed to '%s' (%s)\n",
                                 src, dst, strerror(-rc));
                        rcf = rcf ? rcf : rc;
                }
                CT_ERROR("'%s' xattr file copied to archive '%s'\n",
                         src, dst);
        }

        if (rename_needed == true) {
                char     tmp_src[PATH_MAX];
                char     tmp_dst[PATH_MAX];

                /* atomically replace old archived file */
                ct_path_archive(src, sizeof(src), opt.o_hsm_root,
                                &hai->hai_fid);
                rc = rename(dst, src);
                if (rc < 0) {
                        CT_ERROR("'%s' renamed to '%s' failed (%s)\n", dst, src,
                                 strerror(errno));
                        rc = -errno;
                        goto fini_major;
                }
                /* rename lov file */
                snprintf(tmp_src, sizeof(tmp_src), "%s.lov", src);
                snprintf(tmp_dst, sizeof(tmp_dst), "%s.lov", dst);
                rc = rename(tmp_dst, tmp_src);
                if (rc < 0)
                        CT_ERROR("'%s' renamed to '%s' failed (%s)\n",
                                 tmp_dst, tmp_src, strerror(errno));
        }

        if (opt.o_shadow_tree) {
                /* Create a namespace of softlinks that shadows the original
                 * Lustre namespace.  This will only be current at
                 * time-of-archive (won't follow renames).
                 * WARNING: release won't kill these links; a manual
                 * cleanup of dead links would be required.
                 */

I suspect that it failed at copying attr if the archive existed on NFS share as John stated.

[John Hammond]

Test change at http://review.whamcloud.com/7576.