[LU-9367] parallel-scale test_cascading_rw: cascading_rw failed! 1

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.10.0
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Maloo

Description

This issue was created by maloo for sarah_lw <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/45d48942-2507-11e7-9de9-5254006e85c2.

The sub-test test_cascading_rw failed with the following error:

cascading_rw failed! 1

server/client: lustre-master #3558 ldiskfs el7

test log

+ su mpiuser sh -c "/usr/lib64/compat-openmpi16/bin/mpirun --mca btl tcp,self --mca btl_tcp_if_include eth0 -mca boot ssh -machinefile /tmp/parallel-scale.machines -np 4 /usr/lib64/lustre/tests/cascading_rw -g -d /mnt/lustre/d0.cascading_rw -n 300 "
--------------------------------------------------------------------------
A deprecated MCA parameter value was specified in an MCA parameter
file.  Deprecated MCA parameters should be avoided; they may disappear
in future releases.

  Deprecated parameter: plm_rsh_agent
--------------------------------------------------------------------------
--------------------------------------------------------------------------
A deprecated MCA parameter value was specified in an MCA parameter
file.  Deprecated MCA parameters should be avoided; they may disappear
in future releases.

  Deprecated parameter: plm_rsh_agent
--------------------------------------------------------------------------
--------------------------------------------------------------------------
A deprecated MCA parameter value was specified in an MCA parameter
file.  Deprecated MCA parameters should be avoided; they may disappear
in future releases.

  Deprecated parameter: plm_rsh_agent
--------------------------------------------------------------------------
--------------------------------------------------------------------------
A deprecated MCA parameter value was specified in an MCA parameter
file.  Deprecated MCA parameters should be avoided; they may disappear
in future releases.

  Deprecated parameter: plm_rsh_agent
--------------------------------------------------------------------------
--------------------------------------------------------------------------
A deprecated MCA parameter value was specified in an MCA parameter
file.  Deprecated MCA parameters should be avoided; they may disappear
in future releases.

  Deprecated parameter: plm_rsh_agent
--------------------------------------------------------------------------
--------------------------------------------------------------------------
A deprecated MCA parameter value was specified in an MCA parameter
file.  Deprecated MCA parameters should be avoided; they may disappear
in future releases.

  Deprecated parameter: plm_rsh_agent
--------------------------------------------------------------------------
/usr/lib64/lustre/tests/cascading_rw is running with 4 process(es) in DEBUG mode
23:47:45: Running test #/usr/lib64/lustre/tests/cascading_rw(iter 0)
[trevis-55vm1:21694] *** Process received signal ***
[trevis-55vm1:21694] Signal: Floating point exception (8)
[trevis-55vm1:21694] Signal code: Integer divide-by-zero (1)
[trevis-55vm1:21694] Failing at address: 0x4024c8
[trevis-55vm1:21694] [ 0] /lib64/libpthread.so.0(+0xf370) [0x7fdf9fad6370]
[trevis-55vm1:21694] [ 1] /usr/lib64/lustre/tests/cascading_rw() [0x4024c8]
[trevis-55vm1:21694] [ 2] /usr/lib64/lustre/tests/cascading_rw() [0x402be0]
[trevis-55vm1:21694] [ 3] /usr/lib64/lustre/tests/cascading_rw() [0x40158e]
[trevis-55vm1:21694] [ 4] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fdf9f727b35]
[trevis-55vm1:21694] [ 5] /usr/lib64/lustre/tests/cascading_rw() [0x40169d]
[trevis-55vm1:21694] *** End of error message ***
[trevis-55vm1.trevis.hpdd.intel.com][[36239,1],2][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
[trevis-55vm2.trevis.hpdd.intel.com][[36239,1],1][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 21694 on node trevis-55vm1.trevis.hpdd.intel.com exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------
 parallel-scale test_cascading_rw: @@@@@@ FAIL: cascading_rw failed! 1 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4905:error()
  = /usr/lib64/lustre/tests/functions.sh:734:run_cascading_rw()
  = /usr/lib64/lustre/tests/parallel-scale.sh:130:test_cascading_rw()
  = /usr/lib64/lustre/tests/test-framework.sh:5181:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5220:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:5067:run_test()
  = /usr/lib64/lustre/tests/parallel-scale.sh:132:main()

Issue Links

• is related to: LU-9429 parallel-scale test_parallel_grouplock: test failed to respond and timed out
• is related to: LU-9349 PFL known issues tracking ticket

Comments

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26915/
Subject: LU-9367 llite: restore ll_file_getstripe in ll_lov_setstripe
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 364ec95f3688ac5cc3195f7f46d0d860844796f9

[James Casper (Inactive)]

This might be related to LU-9429. The parallel_grouplock subtest started failing tag testing after 2017-04-05. In each case, parallel_grouplock times out after cascading_rw fails.

tag 55 test (b3550): 2017-04-05 (parallel_grouplock 100% passing)
several PFL landings: 2017-04-08
tag 56 test (b3565): 2017-04-23 (parallel_grouplock 100% failing)

[Zhenyu Xu]

And

#define LL_IOC_LOV_SETSTRIPE _IOW ('f', 154, long)
#define LL_IOC_LOV_GETSTRIPE _IOW ('f', 155, long)

shows that LL_IOC_LOV_SETSTRIPE should be WRITE ONLY ioctl(). And it seems that there is an error for LL_IOC_LOV_GETSTRIPE interface definition.

[Zhenyu Xu]

I am fine to restore the call to ll_file_getstripe(). And I always think the get/set methods to be with one direction data flow, and with that in mind, the ll_lov_setstripe() passes lum back to the caller is a side effect of that function to me.

[Andreas Dilger]

I'm not sure why you consider this to be a side effect? There has been code to explicitly call getstripe and return this to userspace forever. Why not just restore the call to ll_file_getstripe()?

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26915
Subject: LU-9367 mpi: get rid of SETSTRIPE side effect
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7a493a89ea6bc5ea9f7429d9507891741905d4d8

[Zhenyu Xu]

commit fafe6b4d4a6fa63cedff3bd44e6578009578b3d7 changes ll_lov_setstripe()

static int ll_lov_setstripe(struct inode *inode, struct file *file, unsigned long arg)

@@ -1694,14 +1703,6 @@ static int ll_lov_setstripe(struct inode *inode, struct file *file,
 
        lum_size = rc;
        rc = ll_lov_setstripe_ea_info(inode, file, flags, klum, lum_size);
-       if (rc == 0) {
-               __u32 gen;
-
-               put_user(0, &lum->lmm_stripe_count);
-
-               ll_layout_refresh(inode, &gen);
-               rc = ll_file_getstripe(inode, (struct lov_user_md __user *)arg);
-       }
 
        OBD_FREE(klum, lum_size);
        RETURN(rc);

The thinking behind it is that ll_lov_setstripe() only uses the lum to set the file's stripe, while it shouldn't have the side effect to retrieve the instantiated stripe info back to the lum.

And lustre/tests/mpi/cascading_rw.c just exploits the side effect of stripe retrieving during the stripe setting call.

void rw_file(char *name, long stride, unsigned int seed)

        if (rank == 0) {
                remove_file_or_dir(filename);

                lum.lmm_magic = LOV_USER_MAGIC;
                lum.lmm_stripe_size = 0;
                lum.lmm_stripe_count = 0;
                lum.lmm_stripe_offset = -1;

                fd = open(filename, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE,
                          FILEMODE);
                if (fd == -1) {
                        sprintf(errmsg, "open of file %s", filename);
                        FAIL(errmsg);
                }

                rc = ioctl(fd, LL_IOC_LOV_SETSTRIPE, &lum);
                if (rc == -1) {
                        sprintf(errmsg, "ioctl SETSTRIPE of file %s", filename);
                        FAIL(errmsg);
                }

                if (close(fd) == -1) {
                        sprintf(errmsg, "close of file %s", filename);
                        FAIL(errmsg);
                }
        }

        MPI_Barrier(MPI_COMM_WORLD);

        if (stride < 0) {
                if (rank == 0) {
                        srandom(seed);
                        while (stride < page_size/2) {
                                stride = random();
                                stride -= stride % 16;
                                if (stride < 0)
                                        stride = -stride;
                                stride %= 2 * lum.lmm_stripe_size;    // *DEVIDE BY ZERO EXCEPTION HERE*
                        }
                }

                MPI_Barrier(MPI_COMM_WORLD);

We can fix the cascading_rw exploit by calling ioctl(fd, LL_IOC_LOV_GETSTRIPE, &lum) after SETSTRIPE call so the lum.lmm_stripe_size is the real stripe_size instead of 0.

[Andreas Dilger]

This looks like right after the PFL feature landed (April 8), so it probably makes sense for Bobijam to take a look at it.