[LU-10366] sanity test 410 fails with 'no inode match'

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.10.1, Lustre 2.11.0, Lustre 2.10.2, Lustre 2.12.0, Lustre 2.10.4
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: James A Simmons
• Resolution: Unresolved
• Labels: always_except, ubuntu

Environment

Ubuntu clients

• Severity: 3

Description

sanity test 410 is failing for Ubuntu clients. From the client test log, we see

== sanity test 410: Test inode number returned from kernel thread ==================================== 23:57:31 (1512691051)
 sanity test_410: @@@@@@ FAIL: no inode match 

dmesg and console logs don’t have any new information for this test failure.

This failure is seen in master and b2_10 branches.

Logs for recent failures are at
https://testing.hpdd.intel.com/test_sets/d212dae6-dc3e-11e7-a066-52540065bddc
https://testing.hpdd.intel.com/test_sets/a67135e2-dae3-11e7-9840-52540065bddc

Issue Links

• is related to: LU-10997 Ubuntu 18 support
• is related to: LU-9966 sanity test_411: fail to trigger a me...
• is related to: LU-15781 Ubuntu 22.04 LTS release support
• is related to: LU-13740 Ubuntu 20.04 LTS release
• is related to: LU-14776 Ubuntu 20.04 HWE support

Comments

[James A Simmons]

Which kernel version is this? Two LU-9558 patches landed to resolve 410 test issues. Hmm they seem to have landed to 2.10 branch.

[James Nunez (Inactive)]

This is for Ubuntu 16.04 with kernel version 4.4.0-98-generic.

[Andreas Dilger]

Is the kinode.ko module being built on Ubuntu? It is running the test.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31921
Subject: LU-10366 test: test Ubuntu xattr handling
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0f895e816c9bbbd033aa9bf437202864c736da25

[James A Simmons]

Yes kinode is built. I believe the problem is that xattr handling was broken on nwer kernels. I pushed a patch to verify this is the case.

[James A Simmons]

With the recent Ubuntu updates we can see what is going on with the test. What I discovered is that the pr_err() messages in the kinode.ko module are not being printed out to dmesg which makes the test fail. I attempted to change the printk levels but that didn't help. 

Found potential solution:

https://stackoverflow.com/questions/47589197/why-my-kernel-log-is-not-showing-the-latest-output

[James A Simmons]

Hmm. I wonder if we should scan /proc/kmsg for the results?

[Andreas Dilger]

Is this a problem with pr_err() that would be fixed by calling printk() directly? Alternately, what about using CDEBUG() and then dumping the debug logs with "lctl dk" for analysis? That would make the "kinode" module dependent on iibcfs.ko and other Lustre code, but it isn't like we need it to be a totally independent module.