[LU-849] NFS server not responding when running parallel-scale test_iorfpp

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Sarah Liu
• Assignee: Lai Siyao
• Resolution: Duplicate

Environment

server and client: RHEL6-x86_64 build https://newbuild.whamcloud.com/job/lustre-master/353/

• Severity: 3

Description

When running parallel-scale test_iorfpp over NFS v3 got "nfs server not responding" error

Lustre: DEBUG MARKER: ----============= acceptance-small: parallel-scale ============---- Tue Nov 15 11:54:31 PST 2011
Lustre: DEBUG MARKER: excepting tests: parallel_grouplock
Lustre: DEBUG MARKER: == parallel-scale test compilebench: compilebench == 11:54:33 (1321386873)
Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej
Lustre: DEBUG MARKER: == parallel-scale test metabench: metabench == 12:32:28 (1321389148)
Lustre: DEBUG MARKER: == parallel-scale test simul: simul == 12:33:57 (1321389237)
Lustre: DEBUG MARKER: SKIP: parallel-scale test_simul skipped for NFSCLIENT mode
Lustre: DEBUG MARKER: == parallel-scale test mdtestssf: mdtestssf == 12:33:58 (1321389238)
Lustre: DEBUG MARKER: SKIP: parallel-scale test_mdtestssf skipped for NFSCLIENT mode
Lustre: DEBUG MARKER: == parallel-scale test mdtestfpp: mdtestfpp == 12:33:58 (1321389238)
Lustre: DEBUG MARKER: SKIP: parallel-scale test_mdtestfpp skipped for NFSCLIENT mode
Lustre: DEBUG MARKER: == parallel-scale test connectathon: connectathon == 12:33:59 (1321389239)
Lustre: DEBUG MARKER: ./runtests -N 2 -b -f /mnt/lustre/d0.connectathon
Lustre: DEBUG MARKER: ./runtests -N 2 -g -f /mnt/lustre/d0.connectathon
Lustre: DEBUG MARKER: ./runtests -N 2 -s -f /mnt/lustre/d0.connectathon
Lustre: DEBUG MARKER: ./runtests -N 2 -l -f /mnt/lustre/d0.connectathon
Lustre: DEBUG MARKER: == parallel-scale test iorssf: iorssf == 12:35:05 (1321389305)
Lustre: DEBUG MARKER: == parallel-scale test iorfpp: iorfpp == 12:37:56 (1321389476)
nfs: server 10.10.4.15 not responding, still trying
nfs: server 10.10.4.15 not responding, still trying

Comments

[Sarah Liu]

NFSv4 has this issue too.

[Sarah Liu]

the attached are the logs from lustre client(NFS server)

[Peter Jones]

Lai

Could you please comment on this one?

Thanks

Peter

[Johann Lombardi (Inactive)]

hm, several nfsd threads seems to be stuck in splice_read (somewhere in cl_page_list_disown() although it is not clear if the stack is reliable).
Sarah, could you please rerun the test and collect lustre debug logs?

[Sarah Liu]

sure, will keep you updated.

[Sarah Liu]

debug log of lustre client(NFS server)

[Johann Lombardi (Inactive)]

ah, this is with the default debug mask. Could you please collect one with the debug mask set to -1? Thanks in advance.

[Sarah Liu]

debug -1 log

[Oleg Drokin]

Lai, Jinshan, we need to look into this as this potentially underscores how recent clio changes introduced some deadlocks in sendfile path.

[Lai Siyao]

nfsd uses splice read/write interface on new kernels, I tested some splice test (from LTP), and it could pass. I'll do more test later.

[Lai Siyao]

In my local test, I found it's stack overflow too. I'll try to decrease stack size a bit and verify it.

[Lai Siyao]

http://jira.whamcloud.com/browse/LU-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=26225#comment-26225

According to Jinshan's comment on LU-861, two fixes may help, this work will be continued after that.

[Peter Jones]

let's track this under lu-861 and then open a new ticket if the problems still exist with those fixes landed