1109.tar.gz
sarah-nfs-nfs-client.sh
[LU-1109] NFS server not responding when running parallel-scale test_iorsff Created: 15/Feb/12 Updated: 07/Jun/12 Resolved: 29/Feb/12 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.2.0 |
| Fix Version/s: | Lustre 2.2.0, Lustre 2.1.2 |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Sarah Liu | Assignee: | Jinshan Xiong (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
server/client: 2.1.55 RHEL6-x86_64 |
||
| Attachments: |
1109.tar.gz
sarah-nfs-nfs-client.sh
|
| Severity: | 3 |
| Rank (Obsolete): | 4710 |
| Description |
|
Hit the following error when running iorssf over NFS v3 Lustre: DEBUG MARKER: == parallel-scale test iorssf: iorssf == 13:45:23 (1329342323) |
| Comments |
| Comment by Sarah Liu [ 15/Feb/12 ] |
|
dmesg and trace from the NFS server(lustre client) |
| Comment by Sarah Liu [ 15/Feb/12 ] |
|
The same error seen when running iorfpp. There is similar bug for this issue Lustre: DEBUG MARKER: == parallel-scale test iorfpp: iorfpp == 16:01:10 (1329350470) |
| Comment by Sarah Liu [ 16/Feb/12 ] |
|
NFS v4 got the same issues too. both iorsff and iorfpp failed. |
| Comment by Peter Jones [ 16/Feb/12 ] |
|
Fanyong Could you please look into this one? Thanks Peter |
| Comment by nasf (Inactive) [ 17/Feb/12 ] |
|
Hi Sarah, can you show me your test parameters? I cannot reproduce the failure, My test parameters are: build#: lustre-master 479 commands on client-16vm1: PDSH="pdsh -t 300 -S -w" DSH="rsh" NAME=ncli RCLIENTS="client-16vm2" mds_HOST=client-16vm3 MDSDEV1=/dev/vda5 MDSSIZE=2000000 ost_HOST=client-16vm4 OSTCOUNT=3 OSTSIZE=4000000 OSTDEV1=/dev/vda5 OSTDEV2=/dev/vda6 OSTDEV3=/dev/vda7 MDS_MOUNT_OPTS="-o user_xattr,acl" OST_MOUNT_OPTS=" " DEBUG_SIZE=48 SHARED_DIRECTORY="/home/nasf/test_logs" cbench_DIR=/usr/bin cnt_DIR=/opt/connectathon DBENCH_LIB=/usr/share/dbench MIRUN_OPTIONS="-mca orte_rsh_agent rsh:ssh" ENABLE_QUOTA=yes bash auster -d /home/nasf/test_logs -r -s -v -k parallel-scale-nfsv3 --only iorssf iorfpp Above test cases succeed. |
| Comment by Sarah Liu [ 17/Feb/12 ] |
|
hmm, actually I setup the environment myself and ran parallel-scale.sh instead of parallel-scale-nfsv3/4.sh. Today I tried parallel-scale-nfsv3.sh, same error. The configuration I used today is, and please find the conf file in the attached. [root@fat-amd-3 ~]# mount sh auster -r -f sarah-nfs-nfs-client parallel-scale-nfsv3 --only "iorssf iorfpp" Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test iorssf: iorssf == 12:55:01 (1329512101) |
| Comment by Sarah Liu [ 17/Feb/12 ] |
|
conf file I used for nfs testing |
| Comment by nasf (Inactive) [ 22/Feb/12 ] |
|
It may be related with flock process. Sarah, would you please to mount Lustre client with the option "-o flock", and re-test? |
| Comment by Sarah Liu [ 22/Feb/12 ] |
|
Fan Yong, I reran the tests with -o flock on NFSv3, iorssf pass while iorfpp still failed Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test iorssf: iorssf == 20:38:08 (1329971888) [root@fat-amd-3 ~]# mount |
| Comment by nasf (Inactive) [ 23/Feb/12 ] |
|
===================== This is nfsd stack trace when server hung. All the nfsd threads were self-locked unexpected. In fact, when nfsd tries to call nfsd_vfs_read() to read something from Lustre client (backend of nfs server), it just reads pages from given file and fill them into a pipe, and then reads the context from pipe by itself. Such pipe is maintained by the thread itself. Means the nfsd thread is both the pipe writer and the pipe reader. Normally, it should not block when nfsd thread fills/writes to the pipe, but according to the kernel implementation, it is possible: ssize_t splice_to_pipe(struct pipe_inode_info *pipe,
struct splice_pipe_desc *spd)
{
...
pipe_lock(pipe);
for (;;) {
if (!pipe->readers) {
send_sig(SIGPIPE, current, 0);
if (!ret)
ret = -EPIPE;
break;
}
if (pipe->nrbufs < PIPE_BUFFERS) {
int newbuf = (pipe->curbuf + pipe->nrbufs) & (PIPE_BUFFERS - 1);
struct pipe_buffer *buf = pipe->bufs + newbuf;
buf->page = spd->pages[page_nr];
buf->offset = spd->partial[page_nr].offset;
buf->len = spd->partial[page_nr].len;
buf->private = spd->partial[page_nr].private;
buf->ops = spd->ops;
if (spd->flags & SPLICE_F_GIFT)
buf->flags |= PIPE_BUF_FLAG_GIFT;
pipe->nrbufs++;
page_nr++;
ret += buf->len;
if (pipe->inode)
do_wakeup = 1;
if (!--spd->nr_pages)
break;
if (pipe->nrbufs < PIPE_BUFFERS)
continue;
break;
}
...
pipe->waiting_writers++;
====> pipe_wait(pipe);
pipe->waiting_writers--;
}
pipe_unlock(pipe);
...
}
So if the pipe::nrbufs or spd::nr_pages are processed improperly, the nfsd thread will wait at "pipe_wait(pipe)", and nobody can wakeup it any more. Jay, do you have any idea about where Lustre IO stack maybe affect above two variables? |
| Comment by Bruno Faccini (Inactive) [ 23/Feb/12 ] |
|
May be this can help, but we got this same problem with some specific Clients running particular OS/distribs versions and were able to work-around this by reducing the rsize/wsize used by such Client below the Server pipe size. |
| Comment by Bruno Faccini (Inactive) [ 23/Feb/12 ] |
|
Just some more details about which platform/reproducer triggered the problem on our site and later helped to qualify and work-aound it !!... We have experienced the NFS-Server threads problem/hang with 2 different Clients/platforms running SLS-10/SP3 and CentOS 5.2 with their original Kernels and always with the Connectathon specific/special "write_read_mmap" test. |
| Comment by Bruno Faccini (Inactive) [ 23/Feb/12 ] |
|
Oops, forgot to detail that to match Lustre default stripe-size they were originally using rsize/wsize=1MB and also that to finally workaround the issue triggered by our 2.6.32 Kernel's NFSd-Server layer (using the "splice()" feature as a pipe to xfer datas between Lustre and NFS worlds), we switched to a rsize/wsize=64KB (according to pipe-size beeing computed as PIPE_BUFFERS*PAGE_SIZE=16*4KB=64KB). |
| Comment by Jinshan Xiong (Inactive) [ 23/Feb/12 ] |
|
This problem exists if nfsd is reading across stripes and first stripe buffer happens to be 64KB(PIPE_BUFFERS*PAGE_SIZE). I'll cook a patch soon. Bruno's way is a good workaround. |
| Comment by Jinshan Xiong (Inactive) [ 23/Feb/12 ] |
|
Please try patch: http://review.whamcloud.com/2182 |
| Comment by Sarah Liu [ 25/Feb/12 ] |
|
Hi Xiong, I ran your patch on Juelich cluster and it seems there still some problem with the IOR test. NFS client hang there and below is the dmesg from the NFS server(lustre client) |
| Comment by Jinshan Xiong (Inactive) [ 25/Feb/12 ] |
|
Hi Sara, It smells not to be the same problem. Did you have recovery test During the time IOR is running? The suspicious log is this one:
But I'm not sure before I get the full log. Can you please rerun the test with the following debug setttings on the nfs server(lustre client): After you notice nfsd is hung, please do the following besides collecting lustre logs: Thanks. |
| Comment by Sarah Liu [ 27/Feb/12 ] |
|
I didn't run recovery test at that time and will keep you updated if I have any more information. |
| Comment by Sarah Liu [ 28/Feb/12 ] |
|
Hi, I reran the test on Toro instead of Juelich, both NFSv3 and v4 were pass: https://maloo.whamcloud.com/test_sets/395bfe5a-61d7-11e1-b462-5254004bbbd3 |
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Peter Jones [ 29/Feb/12 ] |
|
Landed for 2.2 |
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 29/Feb/12 ] |
|
Integrated in Result = SUCCESS
|
