[LU-394] LND failure casued by discontiguous KIOV pages - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: Lustre 2.1.0
Affects Version/s: Lustre 2.1.0
Labels:
None

Severity:
2
Rank (Obsolete):
4943

Description

Cray's gnilnd is running into a hole in kiov list in Lustre 2.1:

LustreError: 17837:0:(gnilnd_cb.c:594:kgnilnd_setup_phys_buffer()) Can't make payload
contiguous in I/O VM:page 17, offset 0, nob 6350, kiov_offset 0 kiov_len 2254
LustreError: 17837:0:(gnilnd_cb.c:1751:kgnilnd_send()) unable to setup buffer: -22

It used to be that only the first and last page in an IOV were allowed
to be of a offset + length < PAGE_SIZE.

It doesn't have this problem with 1.8 client and 2.1 server.

This problem can be reproduced by "fsx-linux -WR -dn -N 10000 junkfile".

The osc_brw() is never called and the unfragmented pages logic is not exercised in 2.1

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

c38.dk.lu394
6.02 MB
07/Jun/11 12:28 AM
nid0037.dk
722 kB
18/Jul/11 5:36 PM

Activity

[LU-394] LND failure casued by discontiguous KIOV pages

Build Master (Inactive) added a comment - 26/Jul/11 12:39 PM

Integrated in lustre-master » x86_64,server,el5,inkernel #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/obdclass/cl_lock.c
lustre/include/cl_object.h
lustre/tests/sanity.sh
lustre/osc/osc_io.c
lustre/obdclass/cl_page.c
lustre/osc/osc_lock.c
lustre/osc/osc_page.c
lustre/osc/osc_request.c
lustre/include/obd_support.h
lustre/osc/osc_object.c
lustre/llite/vvp_io.c

Build Master (Inactive) added a comment - 26/Jul/11 12:39 PM Integrated in lustre-master » x86_64,server,el5,inkernel #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/obdclass/cl_lock.c lustre/include/cl_object.h lustre/tests/sanity.sh lustre/osc/osc_io.c lustre/obdclass/cl_page.c lustre/osc/osc_lock.c lustre/osc/osc_page.c lustre/osc/osc_request.c lustre/include/obd_support.h lustre/osc/osc_object.c lustre/llite/vvp_io.c

Build Master (Inactive) added a comment - 26/Jul/11 12:33 PM

Integrated in lustre-master » i686,client,el6,inkernel #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/llite/vvp_io.c
lustre/tests/sanity.sh
lustre/obdclass/cl_page.c
lustre/osc/osc_page.c
lustre/include/cl_object.h
lustre/osc/osc_object.c
lustre/obdclass/cl_lock.c
lustre/include/obd_support.h
lustre/osc/osc_request.c
lustre/osc/osc_lock.c
lustre/osc/osc_io.c

Build Master (Inactive) added a comment - 26/Jul/11 12:33 PM Integrated in lustre-master » i686,client,el6,inkernel #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/llite/vvp_io.c lustre/tests/sanity.sh lustre/obdclass/cl_page.c lustre/osc/osc_page.c lustre/include/cl_object.h lustre/osc/osc_object.c lustre/obdclass/cl_lock.c lustre/include/obd_support.h lustre/osc/osc_request.c lustre/osc/osc_lock.c lustre/osc/osc_io.c

Build Master (Inactive) added a comment - 26/Jul/11 12:24 PM

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/osc/osc_request.c
lustre/osc/osc_lock.c
lustre/osc/osc_page.c
lustre/llite/vvp_io.c
lustre/tests/sanity.sh
lustre/osc/osc_io.c
lustre/obdclass/cl_page.c
lustre/osc/osc_object.c
lustre/obdclass/cl_lock.c
lustre/include/cl_object.h
lustre/include/obd_support.h

Build Master (Inactive) added a comment - 26/Jul/11 12:24 PM Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/osc/osc_request.c lustre/osc/osc_lock.c lustre/osc/osc_page.c lustre/llite/vvp_io.c lustre/tests/sanity.sh lustre/osc/osc_io.c lustre/obdclass/cl_page.c lustre/osc/osc_object.c lustre/obdclass/cl_lock.c lustre/include/cl_object.h lustre/include/obd_support.h

Build Master (Inactive) added a comment - 26/Jul/11 12:19 PM

Integrated in lustre-master » x86_64,client,el6,inkernel #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/llite/vvp_io.c
lustre/osc/osc_lock.c
lustre/tests/sanity.sh
lustre/osc/osc_io.c
lustre/obdclass/cl_page.c
lustre/obdclass/cl_lock.c
lustre/osc/osc_page.c
lustre/include/obd_support.h
lustre/osc/osc_object.c
lustre/osc/osc_request.c
lustre/include/cl_object.h

Build Master (Inactive) added a comment - 26/Jul/11 12:19 PM Integrated in lustre-master » x86_64,client,el6,inkernel #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/llite/vvp_io.c lustre/osc/osc_lock.c lustre/tests/sanity.sh lustre/osc/osc_io.c lustre/obdclass/cl_page.c lustre/obdclass/cl_lock.c lustre/osc/osc_page.c lustre/include/obd_support.h lustre/osc/osc_object.c lustre/osc/osc_request.c lustre/include/cl_object.h

Build Master (Inactive) added a comment - 26/Jul/11 12:17 PM

Integrated in lustre-master » x86_64,client,sles11,inkernel #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/osc/osc_lock.c
lustre/include/obd_support.h
lustre/obdclass/cl_page.c
lustre/osc/osc_page.c
lustre/osc/osc_request.c
lustre/llite/vvp_io.c
lustre/osc/osc_io.c
lustre/obdclass/cl_lock.c
lustre/include/cl_object.h
lustre/osc/osc_object.c
lustre/tests/sanity.sh

Build Master (Inactive) added a comment - 26/Jul/11 12:17 PM Integrated in lustre-master » x86_64,client,sles11,inkernel #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/osc/osc_lock.c lustre/include/obd_support.h lustre/obdclass/cl_page.c lustre/osc/osc_page.c lustre/osc/osc_request.c lustre/llite/vvp_io.c lustre/osc/osc_io.c lustre/obdclass/cl_lock.c lustre/include/cl_object.h lustre/osc/osc_object.c lustre/tests/sanity.sh

Build Master (Inactive) added a comment - 26/Jul/11 12:12 PM

Integrated in lustre-master » x86_64,client,el5,inkernel #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/obdclass/cl_lock.c
lustre/tests/sanity.sh
lustre/osc/osc_request.c
lustre/osc/osc_page.c
lustre/llite/vvp_io.c
lustre/include/cl_object.h
lustre/osc/osc_io.c
lustre/obdclass/cl_page.c
lustre/osc/osc_object.c
lustre/osc/osc_lock.c
lustre/include/obd_support.h

Build Master (Inactive) added a comment - 26/Jul/11 12:12 PM Integrated in lustre-master » x86_64,client,el5,inkernel #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/obdclass/cl_lock.c lustre/tests/sanity.sh lustre/osc/osc_request.c lustre/osc/osc_page.c lustre/llite/vvp_io.c lustre/include/cl_object.h lustre/osc/osc_io.c lustre/obdclass/cl_page.c lustre/osc/osc_object.c lustre/osc/osc_lock.c lustre/include/obd_support.h

Peter Jones added a comment - 26/Jul/11 12:11 PM

Landed for 2.1

Peter Jones added a comment - 26/Jul/11 12:11 PM Landed for 2.1

Build Master (Inactive) added a comment - 26/Jul/11 12:06 PM

Integrated in lustre-master » x86_64,client,el5,ofa #228
~~LU-394~~: LND failure casued by discontiguous KIOV

Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916
Files :

lustre/osc/osc_object.c
lustre/include/obd_support.h
lustre/osc/osc_lock.c
lustre/obdclass/cl_lock.c
lustre/osc/osc_request.c
lustre/tests/sanity.sh
lustre/llite/vvp_io.c
lustre/osc/osc_io.c
lustre/obdclass/cl_page.c
lustre/include/cl_object.h
lustre/osc/osc_page.c

Build Master (Inactive) added a comment - 26/Jul/11 12:06 PM Integrated in lustre-master » x86_64,client,el5,ofa #228 LU-394 : LND failure casued by discontiguous KIOV Oleg Drokin : 419016ac3e53e798453106ec04412a4843620916 Files : lustre/osc/osc_object.c lustre/include/obd_support.h lustre/osc/osc_lock.c lustre/obdclass/cl_lock.c lustre/osc/osc_request.c lustre/tests/sanity.sh lustre/llite/vvp_io.c lustre/osc/osc_io.c lustre/obdclass/cl_page.c lustre/include/cl_object.h lustre/osc/osc_page.c

Wally Wang (Inactive) added a comment - 19/Jul/11 5:07 PM

Sorry, it actually works.

I must have mis-labeled the image. After a total rebuild/install, it all works fine.

Wally Wang (Inactive) added a comment - 19/Jul/11 5:07 PM Sorry, it actually works. I must have mis-labeled the image. After a total rebuild/install, it all works fine.

Jinshan Xiong (Inactive) added a comment - 18/Jul/11 8:25 PM

From the log, it looks like the last page(index 49) was still added to the request. Are you sure you're using the correct patch? Can you please run the test case sanity:219 to check if it works?

I'm going to provide you a new patch with more debug info.

Jinshan Xiong (Inactive) added a comment - 18/Jul/11 8:25 PM From the log, it looks like the last page(index 49) was still added to the request. Are you sure you're using the correct patch? Can you please run the test case sanity:219 to check if it works? I'm going to provide you a new patch with more debug info.

Wally Wang (Inactive) added a comment - 18/Jul/11 5:36 PM

debug log is attached, the ost console shows:

2011-07-18T15:15:28.004617-05:00 c0-0c1s5n2 LNetError: 15810:0:(gnilnd_cb.c:594:kgnilnd_setup_phys_buffer()) Can't make payload contiguous in I/O VM:page 17, offset 0, nob 6350, kiov_offset 0 kiov_len 2254
2011-07-18T15:15:28.004650-05:00 c0-0c1s5n2 LNetError: 15810:0:(gnilnd_cb.c:1751:kgnilnd_send()) unable to setup buffer: -22

Wally Wang (Inactive) added a comment - 18/Jul/11 5:36 PM debug log is attached, the ost console shows: 2011-07-18T15:15:28.004617-05:00 c0-0c1s5n2 LNetError: 15810:0:(gnilnd_cb.c:594:kgnilnd_setup_phys_buffer()) Can't make payload contiguous in I/O VM:page 17, offset 0, nob 6350, kiov_offset 0 kiov_len 2254 2011-07-18T15:15:28.004650-05:00 c0-0c1s5n2 LNetError: 15810:0:(gnilnd_cb.c:1751:kgnilnd_send()) unable to setup buffer: -22

People

Assignee:: Jinshan Xiong (Inactive)

Reporter:: Wally Wang (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 06/Jun/11 1:50 PM

Updated:: 26/Jul/11 9:04 PM

Resolved:: 26/Jul/11 12:11 PM