[LU-15978] directory deletion fails on multiple MDS for 64K PAGE_SIZE - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: Lustre 2.16.0
Affects Version/s: None
Labels:
- DNE
- arm
- ppc
Environment:

Hide
aarch64 server centos 8
[centos@liuxl-centos-aio lustre-release]$ lsb_release -a
LSB Version: :core-4.1-aarch64:core-4.1-noarch:cxx-4.1-aarch64:cxx-4.1-noarch:desktop-4.1-aarch64:desktop-4.1-noarch:languages-4.1-aarch64:languages-4.1-noarch:printing-4.1-aarch64:printing-4.1-noarch
Distributor ID: CentOSStream
Description: CentOS Stream release 8
Release: 8
Codename: n/a
[centos@liuxl-centos-aio lustre-release]$ uname -a
Linux liuxl-centos-aio.novalocal 4.18.0-348.2.1.el8_lustre.aarch64 #1 SMP Fri Apr 15 07:36:36 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Show
aarch64 server centos 8 [ centos@liuxl-centos-aio lustre-release]$ lsb_release -a LSB Version: :core-4.1-aarch64:core-4.1-noarch:cxx-4.1-aarch64:cxx-4.1-noarch:desktop-4.1-aarch64:desktop-4.1-noarch:languages-4.1-aarch64:languages-4.1-noarch:printing-4.1-aarch64:printing-4.1-noarch Distributor ID: CentOSStream Description: CentOS Stream release 8 Release: 8 Codename: n/a [ centos@liuxl-centos-aio lustre-release]$ uname -a Linux liuxl-centos-aio.novalocal 4.18.0-348.2.1.el8_lustre.aarch64 #1 SMP Fri Apr 15 07:36:36 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Severity:
2
Rank (Obsolete):
9223372036854775807

Description

Directory deletion fails on multiple MDS for aarch64 64K PAGE_SIZE.

It fails on many tests which creating directories on multi MDS. E.g. sanity test_1

sudo MDSCOUNT=2 PTLDEBUG=-1 RUNAS_ID=1000 ~/work/lustre-release/lustre/tests/auster -D ~/log-28 -rv sanity --only 1
...
== sanity test 1: mkdir; remkdir; rmdir ================== 01:44:19 (1656294259)
striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d1.sanity
striped dir -i1 -c2 -H all_char /mnt/lustre/d1.sanity/d2
mkdir: cannot create directory '/mnt/lustre/d1.sanity/d2': File exists
/mnt/lustre/d1.sanity/d2 has type dir OK
rmdir: failed to remove '/mnt/lustre/d1.sanity/d2': Invalid argument
rmdir: failed to remove '/mnt/lustre/d1.sanity': Directory not empty
/mnt/lustre/d1.sanity exists
sanity test_1: @@@@@@ FAIL: d1.sanity was not removed
Trace dump:
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error()
= /home/centos/work/lustre-release/lustre/tests/sanity.sh:280:test_1()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6743:run_one()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6790:run_one_logged()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6616:run_test()
= /home/centos/work/lustre-release/lustre/tests/sanity.sh:282:main()
Dumping lctl log to /home/centos/log-26/sanity.test_1.*.1656294260.log
Dumping logs only on local client.
FAIL 1 (2s)
resend_count is set to 4 4
resend_count is set to 4 4
resend_count is set to 4 4
resend_count is set to 4 4
resend_count is set to 4 4
== sanity test complete, duration 7 sec ================== 01:44:23 (1656294263)
sanity: FAIL: test_1 d1.sanity was not removed
rm: cannot remove '/mnt/lustre/d1.sanity': Directory not empty
sanity test_904: @@@@@@ FAIL: remove sub-test dirs failed
Trace dump:
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:5892:check_and_cleanup_lustre()
= /home/centos/work/lustre-release/lustre/tests/sanity.sh:28307:main()
Dumping lctl log to /home/centos/log-26/sanity.test_904.*.1656294263.log
Dumping logs only on local client.
sanity returned 1
...

Attachments

Activity

[LU-15978] directory deletion fails on multiple MDS for 64K PAGE_SIZE

Peter Jones made changes - 19/Aug/23 12:08 PM

Fix Version/s		New: Lustre 2.16.0 [ 15190 ]
Resolution		New: Fixed [ 1 ]
Status	Original: In Progress [ 3 ]	New: Resolved [ 5 ]

Xinliang Liu made changes - 28/Jun/22 10:40 AM

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Xinliang Liu made changes - 28/Jun/22 9:17 AM

Labels

Original: arm ppc

New: DNE arm ppc

Xinliang Liu made changes - 28/Jun/22 9:15 AM

Description

Original: Directory deletion fails on multiple MDS for aarch64 64K PAGE_SIZE.

It fails on many tests which creating directories on multi MDS. E.g. sanity test_1

{code:bash}
sudo MDSCOUNT=2 PTLDEBUG=-1 RUNAS_ID=1000 ~/work/lustre-release/lustre/tests/auster -D ~/log-28 -rv sanity --only 1
== sanity test 1: mkdir; remkdir; rmdir ================== 01:44:19 (1656294259)striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d1.sanitystriped dir -i1 -c2 -H all_char /mnt/lustre/d1.sanity/d2mkdir: cannot create directory '/mnt/lustre/d1.sanity/d2': File exists/mnt/lustre/d1.sanity/d2 has type dir OKrmdir: failed to remove '/mnt/lustre/d1.sanity/d2': Invalid argumentrmdir: failed to remove '/mnt/lustre/d1.sanity': Directory not empty/mnt/lustre/d1.sanity exists sanity test_1: @@@@@@ FAIL: d1.sanity was not removed Trace dump: = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:280:test_1() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6743:run_one() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6790:run_one_logged() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6616:run_test() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:282:main()Dumping lctl log to /home/centos/log-26/sanity.test_1.*.1656294260.logDumping logs only on local client.FAIL 1 (2s)resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4== sanity test complete, duration 7 sec ================== 01:44:23 (1656294263)sanity: FAIL: test_1 d1.sanity was not removedrm: cannot remove '/mnt/lustre/d1.sanity': Directory not empty sanity test_904: @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:5892:check_and_cleanup_lustre() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:28307:main()Dumping lctl log to /home/centos/log-26/sanity.test_904.*.1656294263.logDumping logs only on local client.sanity returned 1{code}

New: Directory deletion fails on multiple MDS for aarch64 64K PAGE_SIZE.

It fails on many tests which creating directories on multi MDS. E.g. sanity test_1

sudo MDSCOUNT=2 PTLDEBUG=-1 RUNAS_ID=1000 ~/work/lustre-release/lustre/tests/auster -D ~/log-28 -rv sanity --only 1
...
== sanity test 1: mkdir; remkdir; rmdir ================== 01:44:19 (1656294259)
striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d1.sanity
striped dir -i1 -c2 -H all_char /mnt/lustre/d1.sanity/d2
mkdir: cannot create directory '/mnt/lustre/d1.sanity/d2': File exists
/mnt/lustre/d1.sanity/d2 has type dir OK
rmdir: failed to remove '/mnt/lustre/d1.sanity/d2': Invalid argument
rmdir: failed to remove '/mnt/lustre/d1.sanity': Directory not empty
/mnt/lustre/d1.sanity exists
sanity test_1: @@@@@@ FAIL: d1.sanity was not removed
Trace dump:
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error()
= /home/centos/work/lustre-release/lustre/tests/sanity.sh:280:test_1()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6743:run_one()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6790:run_one_logged()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6616:run_test()
= /home/centos/work/lustre-release/lustre/tests/sanity.sh:282:main()
Dumping lctl log to /home/centos/log-26/sanity.test_1.*.1656294260.log
Dumping logs only on local client.
FAIL 1 (2s)
resend_count is set to 4 4
resend_count is set to 4 4
resend_count is set to 4 4
resend_count is set to 4 4
resend_count is set to 4 4
== sanity test complete, duration 7 sec ================== 01:44:23 (1656294263)
sanity: FAIL: test_1 d1.sanity was not removed
rm: cannot remove '/mnt/lustre/d1.sanity': Directory not empty
sanity test_904: @@@@@@ FAIL: remove sub-test dirs failed
Trace dump:
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error()
= /home/centos/work/lustre-release/lustre/tests/test-framework.sh:5892:check_and_cleanup_lustre()
= /home/centos/work/lustre-release/lustre/tests/sanity.sh:28307:main()
Dumping lctl log to /home/centos/log-26/sanity.test_904.*.1656294263.log
Dumping logs only on local client.
sanity returned 1
...

Xinliang Liu made changes - 28/Jun/22 9:14 AM

Description

Original: Directory deletion fails on multiple MDS for aarch64 64K PAGE_SIZE.

It fails on many tests which creating directories on multi MDS. E.g. sanity test_1
{code:java}
sudo MDSCOUNT=2 PTLDEBUG=-1 RUNAS_ID=1000 ~/work/lustre-release/lustre/tests/auster -D ~/log-28 -rv sanity --only 1
...
-----============= acceptance-small: sanity ============----- Mon Jun 27 01:44:16 UTC 2022Running: bash /home/centos/work/lustre-release/lustre/tests/sanity.shexcepting tests: 42a 42c 42b 277 312 407 45 400a 400b 817 411skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300oliuxl-centos-aio.novalocal: executing check_config_client /mnt/lustreliuxl-centos-aio.novalocal: Checking config lustre mounted on /mnt/lustreChecking servers environmentsChecking clients liuxl-centos-aio.novalocal environmentsUsing TIMEOUT=20osc.lustre-OST0000-osc-ffffa6b777fc5000.idle_timeout=debugosc.lustre-OST0001-osc-ffffa6b777fc5000.idle_timeout=debugdisable quota as requiredosd-ldiskfs.track_declares_assert=1running as uid/gid/euid/egid 1000/1000/1000/1000, groups: [true]running as uid/gid/euid/egid 1000/1000/1000/1000, groups: [touch] [/mnt/lustre/d0_runas_test/f25961]preparing for tests involving mountsmke2fs 1.46.2.wc3 (18-Jun-2021)
debug=-1
== sanity test 1: mkdir; remkdir; rmdir ================== 01:44:19 (1656294259)striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d1.sanitystriped dir -i1 -c2 -H all_char /mnt/lustre/d1.sanity/d2mkdir: cannot create directory '/mnt/lustre/d1.sanity/d2': File exists/mnt/lustre/d1.sanity/d2 has type dir OKrmdir: failed to remove '/mnt/lustre/d1.sanity/d2': Invalid argumentrmdir: failed to remove '/mnt/lustre/d1.sanity': Directory not empty/mnt/lustre/d1.sanity exists sanity test_1: @@@@@@ FAIL: d1.sanity was not removed Trace dump: = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:280:test_1() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6743:run_one() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6790:run_one_logged() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6616:run_test() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:282:main()Dumping lctl log to /home/centos/log-26/sanity.test_1.*.1656294260.logDumping logs only on local client.FAIL 1 (2s)resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4== sanity test complete, duration 7 sec ================== 01:44:23 (1656294263)sanity: FAIL: test_1 d1.sanity was not removedrm: cannot remove '/mnt/lustre/d1.sanity': Directory not empty sanity test_904: @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:5892:check_and_cleanup_lustre() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:28307:main()Dumping lctl log to /home/centos/log-26/sanity.test_904.*.1656294263.logDumping logs only on local client.sanity returned 1
...{code}

New: Directory deletion fails on multiple MDS for aarch64 64K PAGE_SIZE.

It fails on many tests which creating directories on multi MDS. E.g. sanity test_1

{code:bash}
sudo MDSCOUNT=2 PTLDEBUG=-1 RUNAS_ID=1000 ~/work/lustre-release/lustre/tests/auster -D ~/log-28 -rv sanity --only 1
== sanity test 1: mkdir; remkdir; rmdir ================== 01:44:19 (1656294259)striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d1.sanitystriped dir -i1 -c2 -H all_char /mnt/lustre/d1.sanity/d2mkdir: cannot create directory '/mnt/lustre/d1.sanity/d2': File exists/mnt/lustre/d1.sanity/d2 has type dir OKrmdir: failed to remove '/mnt/lustre/d1.sanity/d2': Invalid argumentrmdir: failed to remove '/mnt/lustre/d1.sanity': Directory not empty/mnt/lustre/d1.sanity exists sanity test_1: @@@@@@ FAIL: d1.sanity was not removed Trace dump: = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:280:test_1() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6743:run_one() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6790:run_one_logged() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6616:run_test() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:282:main()Dumping lctl log to /home/centos/log-26/sanity.test_1.*.1656294260.logDumping logs only on local client.FAIL 1 (2s)resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4resend_count is set to 4 4== sanity test complete, duration 7 sec ================== 01:44:23 (1656294263)sanity: FAIL: test_1 d1.sanity was not removedrm: cannot remove '/mnt/lustre/d1.sanity': Directory not empty sanity test_904: @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:6408:error() = /home/centos/work/lustre-release/lustre/tests/test-framework.sh:5892:check_and_cleanup_lustre() = /home/centos/work/lustre-release/lustre/tests/sanity.sh:28307:main()Dumping lctl log to /home/centos/log-26/sanity.test_904.*.1656294263.logDumping logs only on local client.sanity returned 1{code}

Xinliang Liu made changes - 28/Jun/22 9:11 AM

Labels

New: arm ppc

Xinliang Liu created issue - 28/Jun/22 9:05 AM

People

Assignee:: Xinliang Liu

Reporter:: Xinliang Liu

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 28/Jun/22 9:05 AM

Updated:: 12/Jan/24 4:04 AM

Resolved:: 19/Aug/23 12:08 PM