== parallel-scale-nfsv3 test racer_on_nfs: racer on NFS client ======================================= 21:38:30 (1515533910)
[357743.218692] Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test racer_on_nfs: racer on NFS client ======================================= 21:38:30 (1515533910)
Running /usr/lib64/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit
./dir_remote.sh: line 11: RANDOM % MDTCOUNT: division by 0 (error token is "MDTCOUNT")
./dir_remote.sh: line 11: RANDOM % MDTCOUNT: division by 0 (error token is "MDTCOUNT")
./dir_migrate.sh: line 11: RANDOM % MDTCOUNT: division by 0 (error token is "MDTCOUNT")
./dir_migrate.sh: line 11: RANDOM % MDTCOUNT: division by 0 (error token is "MDTCOUNT")
./dir_migrate.sh: line 11: RANDOM % MDTCOUNT: division by 0 (error token is "MDTCOUNT")
./dir_remote.sh: line 11: RANDOM % MDTCOUNT: division by 0 (error token is "MDTCOUNT")
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Trace dump:
= ./file_create.sh:10:main()
parallel-scale-nfsv3: FAIL: test-framework exiting on error
Trace dump:
= ./file_create.sh:10:main()
parallel-scale-nfsv3: FAIL: test-framework exiting on error
Trace dump:
= ./file_create.sh:10:main()
parallel-scale-nfsv3: FAIL: test-framework exiting on error
[357960.620111] INFO: task chown:10956 blocked for more than 120 seconds.
[357960.626483] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[357960.632950] chown D ffff88000e112750 0 10956 21869 0x00000080
[357960.635468] ffff880018727df0 0000000000000086 ffff88000f5d0fd0 ffff880018727fd8
[357960.637915] ffff880018727fd8 ffff880018727fd8 ffff88000f5d0fd0 ffff88000e112748
[357960.640417] ffff88000e11274c ffff88000f5d0fd0 00000000ffffffff ffff88000e112750
[357960.642902] Call Trace:
[357960.644892] [<ffffffff816aa4b9>] schedule_preempt_disabled+0x29/0x70
[357960.647213] [<ffffffff816a83e7>] __mutex_lock_slowpath+0xc7/0x1d0
[357960.649494] [<ffffffff81211aad>] ? putname+0x3d/0x60
[357960.651733] [<ffffffff816a77ff>] mutex_lock+0x1f/0x2f
[357960.653938] [<ffffffff811fdffb>] chown_common+0xeb/0x1d0
[357960.656153] [<ffffffff816b0091>] ? __do_page_fault+0x171/0x450
[357960.658395] [<ffffffff811ff5ef>] SyS_fchownat+0xcf/0x120
[357960.660597] [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[357960.662816] INFO: task setfattr:10965 blocked for more than 120 seconds.
[357960.665076] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[357960.667408] setfattr D ffff88000e115f20 0 10965 21855 0x00000080
[357960.669690] ffff88003637fd50 0000000000000086 ffff88006ff15ee0 ffff88003637ffd8
[357960.672058] ffff88003637ffd8 ffff88003637ffd8 ffff88006ff15ee0 ffff88000e115f18
[357960.674383] ffff88000e115f1c ffff88006ff15ee0 00000000ffffffff ffff88000e115f20
[357960.676681] Call Trace:
[357960.678530] [<ffffffff816aa4b9>] schedule_preempt_disabled+0x29/0x70
[357960.680677] [<ffffffff816a83e7>] __mutex_lock_slowpath+0xc7/0x1d0
[357960.682778] [<ffffffff816a77ff>] mutex_lock+0x1f/0x2f
[357960.684763] [<ffffffff81227a0f>] vfs_removexattr+0x5f/0x130
[357960.686749] [<ffffffff81227b35>] removexattr+0x55/0x80
[357960.688670] [<ffffffff81211aad>] ? putname+0x3d/0x60
[357960.690559] [<ffffffff81212cc2>] ? user_path_at_empty+0x72/0xc0
[357960.692497] [<ffffffff81203a58>] ? __sb_start_write+0x58/0x110
[357960.694401] [<ffffffff816b0091>] ? __do_page_fault+0x171/0x450
[357960.696284] [<ffffffff8122292c>] ? mnt_want_write+0x2c/0x50
[357960.698151] [<ffffffff81228924>] SyS_removexattr+0x94/0xd0
[357960.699982] [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[357960.701822] INFO: task mv:10981 blocked for more than 120 seconds.
[357960.703658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[357960.705615] mv D ffff88000e112750 0 10981 21845 0x00000080
[357960.707528] ffff880039c03c20 0000000000000082 ffff88001456af70 ffff880039c03fd8
[357960.709465] ffff880039c03fd8 ffff880039c03fd8 ffff88001456af70 ffff88000e112748
[357960.711361] ffff88000e11274c ffff88001456af70 00000000ffffffff ffff88000e112750
[357960.713237] Call Trace:
[357960.714640] [<ffffffff816aa4b9>] schedule_preempt_disabled+0x29/0x70
[357960.716340] [<ffffffff816a83e7>] __mutex_lock_slowpath+0xc7/0x1d0
[357960.717978] [<ffffffff816a77ff>] mutex_lock+0x1f/0x2f
[357960.719522] [<ffffffff816a13e3>] lookup_slow+0x33/0xa7
[357960.721017] [<ffffffff8120f0db>] path_lookupat+0x77b/0x7b0
[357960.722522] [<ffffffff811df60d>] ? kmem_cache_alloc+0x16d/0x1e0
[357960.724031] [<ffffffff81211b1f>] ? getname_flags+0x4f/0x1a0
[357960.725493] [<ffffffff8120f13b>] filename_lookup+0x2b/0xc0
[357960.726928] [<ffffffff81212cb7>] user_path_at_empty+0x67/0xc0
[357960.728375] [<ffffffff81114b22>] ? from_kgid_munged+0x12/0x20
[357960.729801] [<ffffffff8120676f>] ? cp_new_stat+0x14f/0x180
[357960.731242] [<ffffffff81212d21>] user_path_at+0x11/0x20
[357960.732635] [<ffffffff81206263>] vfs_fstatat+0x63/0xc0
[357960.734032] [<ffffffff81206831>] SYSC_newlstat+0x31/0x60
[357960.735403] [<ffffffff8111f5d6>] ? __audit_syscall_exit+0x1e6/0x280
[357960.736846] [<ffffffff81206abe>] SyS_newlstat+0xe/0x10
[357960.738208] [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[357960.739627] INFO: task ln:10982 blocked for more than 120 seconds.
[357960.741074] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[357960.742661] ln D ffff88005dd73398 0 10982 21846 0x00000080
[357960.744356] ffff880078c7bd70 0000000000000082 ffff880014568fd0 ffff880078c7bfd8
[357960.746111] ffff880078c7bfd8 ffff880078c7bfd8 ffff880014568fd0 ffff88005dd73390
[357960.747734] ffff88005dd73394 ffff880014568fd0 00000000ffffffff ffff88005dd73398
[357960.749349] Call Trace:
[357960.750511] [<ffffffff816aa4b9>] schedule_preempt_disabled+0x29/0x70
[357960.751983] [<ffffffff816a83e7>] __mutex_lock_slowpath+0xc7/0x1d0
[357960.753668] [<ffffffff816a77ff>] mutex_lock+0x1f/0x2f
[357960.755112] [<ffffffff8120f255>] filename_create+0x85/0x180
[357960.756515] [<ffffffff811df60d>] ? kmem_cache_alloc+0x16d/0x1e0
[357960.757937] [<ffffffff81211b1f>] ? getname_flags+0x4f/0x1a0
[357960.759339] [<ffffffff81211b94>] ? getname_flags+0xc4/0x1a0
[357960.760717] [<ffffffff81211e41>] user_path_create+0x41/0x60
[357960.762100] [<ffffffff8121350e>] SyS_linkat+0xae/0x320
[357960.763455] [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[357960.764866] INFO: task mkdir:10986 blocked for more than 120 seconds.
[357960.766323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[357960.767896] mkdir D ffff88005dd73398 0 10986 21843 0x00000080
[357960.769474] ffff880013bfbd90 0000000000000082 ffff880006408000 ffff880013bfbfd8
[357960.771124] ffff880013bfbfd8 ffff880013bfbfd8 ffff880006408000 ffff88005dd73390
[357960.772745] ffff88005dd73394 ffff880006408000 00000000ffffffff ffff88005dd73398
[357960.774391] Call Trace:
[357960.775583] [<ffffffff816aa4b9>] schedule_preempt_disabled+0x29/0x70
[357960.777084] [<ffffffff816a83e7>] __mutex_lock_slowpath+0xc7/0x1d0
[357960.778551] [<ffffffff816a77ff>] mutex_lock+0x1f/0x2f
[357960.779933] [<ffffffff8120f255>] filename_create+0x85/0x180
[357960.781387] [<ffffffff81211b1f>] ? getname_flags+0x4f/0x1a0
[357960.782982] [<ffffffff81211b94>] ? getname_flags+0xc4/0x1a0
[357960.784401] [<ffffffff81211e41>] user_path_create+0x41/0x60
[357960.785803] [<ffffffff812131d8>] SyS_mkdirat+0x48/0x100
[357960.787165] [<ffffffff812132a9>] SyS_mkdir+0x19/0x20
[357960.788503] [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[357960.789920] INFO: task mkdir:10988 blocked for more than 120 seconds.
[357960.791370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[357960.792928] mkdir D ffff88005dd73398 0 10988 21861 0x00000080
[357960.794496] ffff8800094bbd90 0000000000000082 ffff88006ff16eb0 ffff8800094bbfd8
[357960.796144] ffff8800094bbfd8 ffff8800094bbfd8 ffff88006ff16eb0 ffff88005dd73390
[357960.797790] ffff88005dd73394 ffff88006ff16eb0 00000000ffffffff ffff88005dd73398
[357960.799440] Call Trace:
[357960.800624] [<ffffffff816aa4b9>] schedule_preempt_disabled+0x29/0x70
[357960.802134] [<ffffffff816a83e7>] __mutex_lock_slowpath+0xc7/0x1d0
[357960.803629] [<ffffffff816a77ff>] mutex_lock+0x1f/0x2f
[357960.805043] [<ffffffff8120f255>] filename_create+0x85/0x180
[357960.806486] [<ffffffff81211b1f>] ? getname_flags+0x4f/0x1a0
[357960.807928] [<ffffffff81211b94>] ? getname_flags+0xc4/0x1a0
[357960.809375] [<ffffffff81211e41>] user_path_create+0x41/0x60
[357960.810860] [<ffffffff812131d8>] SyS_mkdirat+0x48/0x100
[357960.812242] [<ffffffff812132a9>] SyS_mkdir+0x19/0x20
[357960.813581] [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[357985.696288] nfs: server onyx-73 not responding, still trying
[357985.700798] nfs: server onyx-73 not responding, still trying
[357997.728270] nfs: server onyx-73 not responding, still trying
[357997.732791] nfs: server onyx-73 not responding, still trying
[358021.792299] nfs: server onyx-73 not responding, still trying
[358021.796371] nfs: server onyx-73 not responding, still trying
racer cleanup
./file_exec.sh: line 1: kill: (21927) - No such process
./file_exec.sh: line 1: kill: (21934) - No such process
./file_exec.sh: line 1: kill: (21914) - No such process
sleeping 5 sec ...
file_create.sh: no process found
dir_create.sh: no process found
file_rm.sh: no process found
file_rename.sh: no process found
file_link.sh: no process found
file_symlink.sh: no process found
file_list.sh: no process found
file_concat.sh: no process found
file_exec.sh: no process found
file_chown.sh: no process found
file_chmod.sh: no process found
file_mknod.sh: no process found
file_truncate.sh: no process found
file_delxattr.sh: no process found
file_getxattr.sh: no process found
file_setxattr.sh: no process found
dir_remote.sh: no process found
dir_migrate.sh: no process found
there should be NO racer processes:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
[358069.984316] nfs: server onyx-73 not responding, still trying
[358250.464082] nfs: server onyx-73 not responding, still trying
[358250.469403] nfs: server onyx-73 not responding, still trying