[LU-1109] NFS server not responding when running parallel-scale test_iorsff

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.2
• Affects Version/s: Lustre 2.2.0
• Environment: server/client: 2.1.55 RHEL6-x86_64
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Sarah Liu

Description

Hit the following error when running iorssf over NFS v3

Lustre: DEBUG MARKER: == parallel-scale test iorssf: iorssf == 13:45:23 (1329342323)
nfs: server 10.10.4.15 not responding, still trying
nfs: server 10.10.4.15 not responding, still trying
nfs: server 10.10.4.15 not responding, still trying
nfs: server 10.10.4.15 not responding, still trying
nfs: server 10.10.4.15 not responding, still trying

Issue Links

• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,ofa #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,ofa #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,inkernel #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Peter Jones]

Landed for 2.2

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,inkernel #493
LU-1109 llite: do splice read stripe by stripe (Revision 211b00d651bbc57d9ab9d24d6d7e94b013957cf1)

Result = SUCCESS
Oleg Drokin : 211b00d651bbc57d9ab9d24d6d7e94b013957cf1
Files :

• lustre/llite/vvp_io.c

[Sarah Liu]

Hi,

I reran the test on Toro instead of Juelich, both NFSv3 and v4 were pass:

https://maloo.whamcloud.com/test_sets/395bfe5a-61d7-11e1-b462-5254004bbbd3
https://maloo.whamcloud.com/test_sets/00c195ae-61d8-11e1-b462-5254004bbbd3

[Sarah Liu]

I didn't run recovery test at that time and will keep you updated if I have any more information.

[Jinshan Xiong (Inactive)]

Hi Sara,

It smells not to be the same problem. Did you have recovery test During the time IOR is running? The suspicious log is this one:

LustreError: 31719:0:(file.c:2221:ll_inode_revalidate_fini()) failure -95 inode 1045761 LustreError: 31768:0:(mdc_locks.c:719:mdc_enqueue()) ldlm_cli_enqueue: -95
LustreError: 31768:0:(file.c:2221:ll_inode_revalidate_fini()) failure -95 inode 1045761

But I'm not sure before I get the full log.

Can you please rerun the test with the following debug setttings on the nfs server(lustre client):
1. lctl set_param debug=-1
2. lctl set_param debug=-trace
3. lctl set_param debug_mb=200
4. lctl mark "XXXX IOR test starting..."

After you notice nfsd is hung, please do the following besides collecting lustre logs:
5. echo t > /proc/sysrq-trigger
6. dmesg > dmesg.txt and upload dmesg.txt file.

Thanks.