[LU-17777] runtests test_1: FAIL: old and new files are different: rc=22

• Type: Bug
• Resolution: Unresolved
• Priority: Blocker
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run:
https://testing.whamcloud.com/test_sets/72dfd1b1-7efa-4dcc-a366-043549333756
https://testing.whamcloud.com/test_sets/1dde1cd4-24fd-4ae3-ab20-d7261788833f

test_1 failed with the following error:

CMD: onyx-106vm11 /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush
comparing 520 previously copied files
Files /etc/yum.repos.d/redhat.repo and /mnt/lustre/d1.runtests//etc/yum.repos.d/redhat.repo differ
Files /etc/pki/entitlement/2519028287967039457.pem and /mnt/lustre/d1.runtests//etc/pki/entitlement/2519028287967039457.pem differ
 runtests test_1: @@@@@@ FAIL: old and new files are different: rc=22 

Test session details:
clients: https://build.whamcloud.com/job/lustre-master-next/784 - 5.14.0-362.18.1.el9_3.x86_64
servers: https://build.whamcloud.com/job/lustre-master-next/784 - 5.14.0-362.18.1_lustre.el9.x86_64

This failed for the first time with this error on 2024-04-24 for on two separate test runs, one an unlanded patch, and one a "full" test run on master. Strangely, both failures were reported on the same two files. There don't appear to be any Lustre console errors immediately before this failure (a few back when the filesystem is remounted in the test).

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
runtests test_1 - old and new files are different: rc=22

Comments

[Gerrit Updater]

"Arshad Hussain <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58129
Subject: LU-17777 tests: Exclude /etc/yum*/* and /etc/pki/*
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d230e830e6eb168d59ce0fb4fe3fb626ee4f843b

[Andreas Dilger]

ys it should be enough to exclude the /etc/yum_repos.d and /etc/pki subdirectories during the test. While copying the whole /etc tree to another path would work, this slows down the test and is probably not needed.

[Yang Sheng]

Can we copy the /etc to somewhere in local FS, then copy it to lustre to compare?

[Emoly Liu]

+1 on master: https://testing.whamcloud.com/test_sets/435ee982-749c-4a7e-871b-c187cc35013b

[Nikitas Angelinas]

+1 on master: https://testing.whamcloud.com/test_sets/d0abfffa-0bcc-4624-b540-5f733c47b357

[Minh Diep]

yes I think we should skip some directories that change

[Andreas Dilger]

Something is updating /etc/yum.repos.d/redhat.repo and /etc/pki/entitlement/xxxx.pem during the test run. Not very often, but enough to cause some failures. If we don't know what that is, I guess we need a patch to make the runtests script more robust.
 

[Minh Diep]

I don't think RHEL license check nor change anything in /etc

[Charlie Olmstead]

mdiep or mkvardakov would be better suited to answer this question

[Andreas Dilger]

+1 on b2_15:
https://testing.whamcloud.com/test_sets/6ffef452-6051-4199-90f0-cec559c8aaf6

colmstea, is there some RHEL license manager that is running which is randomly updating these files in /etc while the test is running? We can make the test more robust by having it re-check and exclude files in /etc that have been modified since the test was started, but that is not retroactive for all of the branches and interop tests, so it would be nice if whatever is triggering this was turned off, or done during provisioning instead while the test was running...