[LU-3449] Interop failure on many testsuites: error: set_param: track_declares_assert: Found no match

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.4.1
• Fix Version/s: Lustre 2.4.1, Lustre 2.5.0
• Type: Bug
• Priority: Blocker
• Reporter: James Nunez (Inactive)
• Assignee: Bruno Faccini (Inactive)
• Resolution: Fixed
• Labels: HB

Environment

servers: b2_1 and b2_3
clients: b2_4 and master

• Severity: 3

Description

Several test suites fail during Interop testing due to a call to lctl set_param to set osd-ldiskfs.track_declares_assert=1. From test-framework.sh

if [ -n "$OSD_TRACK_DECLARES_LBUG" ] ; then
  do_nodes $(comma_list $(mdts_nodes) $(osts_nodes)) \
  "$LCTL set_param osd-*.track_declares_assert=1" \
    > /dev/null
fi

Setting track_declares_assert was added in 2.3.65.

Two Interop test runs have failed with these errors:
June 8, b2_3 servers, master clients: https://maloo.whamcloud.com/test_sessions/aaa56180-d08c-11e2-a7e3-52540035b04c

June 7, b2_3 servers, b2_4 clients: replay-vbr maybe more: https://maloo.whamcloud.com/test_sessions/f0211dda-cf83-11e2-a604-52540035b04c

I have also run into this error running test locally with b2_1 servers and master clients. So, this is a general problem for interop testing.

From the suite log for the failed test, you see

CMD: client-32vm3 /usr/sbin/lctl get_param -n version
CMD: client-32vm3,client-32vm4,client-32vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"0x33f0404\" \" 0xffb7e3ff\" 32 
CMD: client-32vm3,client-32vm4 /usr/sbin/lctl set_param 				 osd-ldiskfs.track_declares_assert=1
client-32vm3: error: set_param: /proc/{fs,sys}/{lnet,lustre}/osd-ldiskfs/track_declares_assert: Found no match
client-32vm4: error: set_param: /proc/{fs,sys}/{lnet,lustre}/osd-ldiskfs/track_declares_assert: Found no match

Issue Links

• is related to: LU-3110 Disable osd declaration tracking for ...

Comments

[Keith Mannthey (Inactive)]

So this call should be wrapped with a version check then?

[Bruno Faccini (Inactive)]

That's right Keith, I forgot interop cases when I introduced this as part of LU-3110 fixes ... Will fix that!

[Andreas Dilger]

Instead of a version check, it could just check whether the proc file exists or not.

[Bruno Faccini (Inactive)]

Sure, less elegant but more robust check !!
Master patch is at http://review.whamcloud.com/6639

[Jodi Levi (Inactive)]

Patch landed to Master so I am closing this ticket. Please let me know if something more is needed and I will reopen.

[Jian Yu]

Master patch is at http://review.whamcloud.com/6639

Hi Oleg, could you please cherry-pick the above patch to Lustre b2_4 branch? The interop testing between Lustre b2_4 clients and old servers is blocked by this issue.

[Bruno Faccini (Inactive)]

All, may be I forgot to tell you but you can disable tracking of declares by setting OSD_TRACK_DECLARES_LBUG env. var to "no". May be you can try to use it before patch lands.

[Jian Yu]

Thanks Bruno for the information. For the test sessions performed by autotest, the variable needs to be set by autotest. However, I'm concerned that this will affect all of the test sessions. So, landing the patch asap on Lustre b2_4 branch is a better way.

[Jian Yu]

Have to reopen this ticket for it's blocking the interop testing between Lustre b2_4 clients and old servers.

[Jian Yu]

Patch was cherry-picked to Lustre b2_4 branch.