[LU-4768] ost-survey hangs on client 2.4

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.2
• Fix Version/s: Lustre 2.7.0
• Type: Bug
• Priority: Minor
• Reporter: francesco de giorgi
• Assignee: James Nunez (Inactive)
• Resolution: Fixed

Environment

mds, oss: lustre 2.4.2 (CentOS 6.5 with lustre patched kernel RPM)
client: lustre 2.4.2 (CentOS 6.5 patchless client, lustre iokit 1.4.0.1)

• Epic/Theme: test
• Severity: 3

Description

ost-survey on this environment

mds, oss: lustre 2.4.2 (CentOS 6.5 with lustre patched kernel RPM)
client: lustre 2.4.2 (CentOS 6.5 patchless client, lustre iokit 1.4.0.1)

command line: ost-survey -s 100 /lustre

will hang, chewing all the CPU in the attempt to
echo 0 in /proc/fs/lustre/llite/*/max_cached_mb . This is due to the
subroutine cache_off in the ost-survey script.

In a lustre client 2.1.6 the max_cached_mb appears as
a single number

1. cat /proc/fs/lustre/llite/*/max_cached_mb
   18114

but on a 2.4.2 client is different

1. cat /proc/fs/lustre/llite/*/max_cached_mb
   users: 2
   max_cached_mb: 96766
   used_mb: 21806
   unused_mb: 74960
   reclaim_count: 0

So probably the script has not been fixed to work with the new output.

Issue Links

• is related to: LU-5030 "lctl {get,set}_param" should also ch...
• is related to: LU-5773 obdfilter-survey test 1c: oom occurre...

Comments

[Andreas Dilger]

Two things are wrong here:

• it doesn't make sense to allow the client to have no pages cached at all. ll_max_cached_mb_seq_write() should always allow at least PTLRPC_MAX_BRW_PAGES of cached data, so tha the client can make well-formed RPCs. It probably should just silently limit the input value to max(pages_number, PTLRPC_MAX_BRW_PAGES) if the requested value is smaller.
• the ost-survey script should be changed to ask for a minimum of 256 pages

[Scott Nolin]

I'm pretty sure this affects 2.5 client also, but don't have one up right now to examine.

Anyway I'd just add while it's a minor problem for things like initial set up testing, it can become a big problem if trying to debug possible bad filesystems. If you forget to remove ost-survey or modify it, and start testing from a client, that client hangs. If you're not someone aware of the problem, it can cause a bit of a sysadmin freak-out.

ost-survey has other problems too, for example:

It looks to me like it assumes you only have one filesystem mounted so when it does things like count ost's, it looks at:
/proc/fs/lustre/lov/*-clilov-*/numobd -where it seems like it should look at /proc/fs/lustre/lov/$MNT-clilov-*/numobd.

It uses the long deprecated positional parameters for setstripe, so spews a lot of errors

I think this tool really does needs to be fixed or just removed. If it's too low a priority to get a proper fix and review, then just not distributing it would be preferable.

Scott

[James Nunez (Inactive)]

Proposed master patch at: http://review.whamcloud.com/11971

This patch only allows ost-survey to run and does not modify ll_max_cached_mb_seq_write().

The comment was made to, in ost-survey, set max_cached_mb to 256 pages. The patch sets max_cached_mb to 256 pages (in MB), 256 * pagesize / 1024 * 1024, but this allows the read values to be, possibly, generous due to reading from cache. For example, on the system I was testing this patch on, the page size is 2621440. So, max_cached_mb is set to 640. Here are the results for max_cached_mb = 2 and max_cached_mb = 640:

With max_cached_mb = 2:
Ost#  Read(MB/s)  Write(MB/s)  Read-time  Write-time
----------------------------------------------------
0     258.701       125.005        3.865      8.000
1     341.453       107.399        2.929      9.311
2     340.931       117.782        2.933      8.490
3     277.857       105.958        3.599      9.438

With max_cached_mb = 640:
Ost#  Read(MB/s)  Write(MB/s)  Read-time  Write-time
----------------------------------------------------
0     1227.975       128.612        0.814      7.775
1     1109.066       108.828        0.902      9.189
2     1069.722       127.521        0.935      7.842
3     1050.668       107.104        0.952      9.337

Write performance is about the same in each case, but read performance is much larger with max_cached_mb = 640. Do we really want max_cached_mb to be 256 pages?

[James Nunez (Inactive)]

The patch for LU-5030 made significant changes to ost-survey; http://review.whamcloud.com/#/c/10534/8 .

[James A Simmons]

Yes the patch for LU-5030 removed the direct access to the proc filesystem. Instead it uses lctl [s|g]et_param as it should. Let see if the patch does the wrong thing

[James Nunez (Inactive)]

I've updated the patch to correctly get the page size and, as Andreas asked for, modified ll_max_cached_mb_seq_write() to set the requested pages to be the max of the requested pages and PTLRPC_MAX_BRW_PAGES.

[Peter Jones]

Landed for 2.7