[LU-582] 1.8<->2.1 interop: sanity test_132: FAIL: some glimpse RPC is expected

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.4.0, Lustre 2.1.5
• Affects Version/s: Lustre 2.0.0, Lustre 2.1.2, Lustre 2.1.3, Lustre 1.8.8, Lustre 1.8.6

Environment

Old Lustre Version: 1.8.6-wc1
Lustre Build: http://newbuild.whamcloud.com/job/lustre-b1_8/100/

New Lustre Version: 2.0.66.0
Lustre Build: http://newbuild.whamcloud.com/job/lustre-master/228/

Clean upgrading (Lustre servers and clients were upgraded all at once) from Lustre 1.8.6-wc1 to Lustre 2.0.66.0 under the following configuration:

OSS1: RHEL5/x86_64
OSS2: RHEL5/x86_64
MDS: RHEL5/x86_64
Client1: RHEL6/x86_64
Client2: RHEL5/x86_64

• Severity: 3
• Bugzilla ID: 23,339
• Assignee: Hongchao Zhang
• Reporter: Jian Yu

Description

After the upgrading, sanity test 132 failed on Lustre 2.0.66.0 as follows:

== sanity test 132: som avoids glimpse rpc == 03:17:56 (1312885076)
====> SOM is disabled, 0 glimpse RPC occured
 sanity test_132: @@@@@@ FAIL: some glimpse RPC is expected 

Please refer to the Maloo report for more logs: https://maloo.whamcloud.com/test_sets/a570d34e-c278-11e0-8bdf-52540025f9af

This is an known issue: bug 23339.

Issue Links

• Trackbacks: Lustre 2.1.2 release testing tracker Lustre 2.1.2 RC2 Tag: v212RC2 Build:
• Trackbacks: Lustre 2.1.3 release testing tracker Lustre 2.1.3 RC1 Tag: v213RC1 Build:

Comments

[Stephen Champion]

I've been using http://review.whamcloud.com/#change,3693 for
several months, and recommend it for b2_1.

[Jay Lan (Inactive)]

I saw this problem between 2.1.3 server and 2.3.0 client. Let me know if you want the test_logs.

[Hongchao Zhang]

the patch has been updated

[Hongchao Zhang]

there is a bug in get_ost_param

get_ost_param() {
local token=$1
local gl_sum=0
for node in $(osts_nodes); do
gl=$(do_node $node "$LCTL get_param -n ost.OSS.ost.stats" | awk '/'$token'/

{print $2}

' | head -n 1)
[ x$gl = x"" ] && gl=0
gl_sum=$((gl_sum + gl))
done
echo $gl <--- here should be "echo $gl_sum"!
}

previous, there is only one OSS in autotest, then the problem is hidden, but in upgrade test, there are 2 OSS,
then the issue shows up. the updated patch will be attached soon!

[Jian Yu]

the debug patch is tracked at http://review.whamcloud.com/#change,3692

The real patch is in http://review.whamcloud.com/#change,3693. Testing is ongoing.

[Hongchao Zhang]

Yujian help to reproduce the bug after upgrading Lustre, and the ldlm_glimpse_enqueue is indeed increased.

the debug patch is tracked at http://review.whamcloud.com/#change,3692

Hi Yujian, could you please help to test with the debug patch? thanks!!