[LU-3449] Interop failure on many testsuites: error: set_param: track_declares_assert: Found no match

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.4.1, Lustre 2.5.0
• Affects Version/s: Lustre 2.4.1
• Labels: HB
• Environment: servers: b2_1 and b2_3; clients: b2_4 and master
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: James Nunez (Inactive)

Description

Several test suites fail during Interop testing due to a call to lctl set_param to set osd-ldiskfs.track_declares_assert=1. From test-framework.sh

if [ -n "$OSD_TRACK_DECLARES_LBUG" ] ; then
  do_nodes $(comma_list $(mdts_nodes) $(osts_nodes)) \
  "$LCTL set_param osd-*.track_declares_assert=1" \
    > /dev/null
fi

Setting track_declares_assert was added in 2.3.65.

Two Interop test runs have failed with these errors:
June 8, b2_3 servers, master clients: https://maloo.whamcloud.com/test_sessions/aaa56180-d08c-11e2-a7e3-52540035b04c

June 7, b2_3 servers, b2_4 clients: replay-vbr maybe more: https://maloo.whamcloud.com/test_sessions/f0211dda-cf83-11e2-a604-52540035b04c

I have also run into this error running test locally with b2_1 servers and master clients. So, this is a general problem for interop testing.

From the suite log for the failed test, you see

CMD: client-32vm3 /usr/sbin/lctl get_param -n version
CMD: client-32vm3,client-32vm4,client-32vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"0x33f0404\" \" 0xffb7e3ff\" 32 
CMD: client-32vm3,client-32vm4 /usr/sbin/lctl set_param 				 osd-ldiskfs.track_declares_assert=1
client-32vm3: error: set_param: /proc/{fs,sys}/{lnet,lustre}/osd-ldiskfs/track_declares_assert: Found no match
client-32vm4: error: set_param: /proc/{fs,sys}/{lnet,lustre}/osd-ldiskfs/track_declares_assert: Found no match

Issue Links

• is related to: LU-3110 Disable osd declaration tracking for 2.4 release

Comments

[Jian Yu]

Thanks Bruno for the information. For the test sessions performed by autotest, the variable needs to be set by autotest. However, I'm concerned that this will affect all of the test sessions. So, landing the patch asap on Lustre b2_4 branch is a better way.

[Bruno Faccini (Inactive)]

All, may be I forgot to tell you but you can disable tracking of declares by setting OSD_TRACK_DECLARES_LBUG env. var to "no". May be you can try to use it before patch lands.

[Jian Yu]

Master patch is at http://review.whamcloud.com/6639

Hi Oleg, could you please cherry-pick the above patch to Lustre b2_4 branch? The interop testing between Lustre b2_4 clients and old servers is blocked by this issue.

[Jodi Levi (Inactive)]

Patch landed to Master so I am closing this ticket. Please let me know if something more is needed and I will reopen.

[Bruno Faccini (Inactive)]

Sure, less elegant but more robust check !!
Master patch is at http://review.whamcloud.com/6639

[Andreas Dilger]

Instead of a version check, it could just check whether the proc file exists or not.

[Bruno Faccini (Inactive)]

That's right Keith, I forgot interop cases when I introduced this as part of LU-3110 fixes ... Will fix that!

[Keith Mannthey (Inactive)]

So this call should be wrapped with a version check then?