[LU-15706] "lctl pool_destroy" can't work correctly due to "SKIP" records in llog

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.2
• Labels: llog
• Severity: 3
• Assignee: Emoly Liu
• Reporter: Emoly Liu

Description

"lctl pool_destroy" can't work correctly when MGS and MDT0 is from different node.

Then I did the following test on two nodes(centos7-2 and centos7-4) and found it failed because it didn't process "SKIP" pool llog records properly.

centos7-2 (MDT+2OST+client):

[root@centos7-2 tests]# MGSDEV=/tmp/lustre-mgs mgs_HOST=centos7-4 mdt_HOST=centos7-2 ost1_HOST=centos7-2 ost2_HOST=centos7-2 PDSH="pdsh -S -Rssh -w" sh llmount.sh

centos7-4(MGS):

[root@centos7-4 tests]# lctl pool_new lustre.testpool
Pool lustre.testpool created
[root@centos7-4 tests]# lctl pool_add lustre.testpool OST0000
OST lustre-OST0000_UUID added to pool lustre.testpool
[root@centos7-4 tests]# lctl pool_add lustre.testpool OST0001
OST lustre-OST0001_UUID added to pool lustre.testpool
[root@centos7-4 tests]# lctl pool_destroy lustre.testpool
Pool lustre.testpool not empty, please remove all members
pool_destroy: Directory not empty
[root@centos7-4 tests]# lctl pool_remove lustre.testpool OST0000
OST lustre-OST0000_UUID removed from pool lustre.testpool
[root@centos7-4 tests]# lctl pool_destroy lustre.testpool
Pool lustre.testpool destroyed

"testpool" was destroyed wrongly even OST0001 was still in the pool.

After checking the llog records, I found "SKIP pool add 0:lustre-clilov  1:lustre  2:testpool  3:lustre-OST0000_UUID" was skipped.

#43 (112)SKIP pool new 0:lustre-clilov  1:lustre  2:testpool  
#44 (224)SKIP END   marker  29 (flags=0x06, v2.15.0.0) lustre-clilov   'new lustre.testpool' Wed Mar 30 11:02:17 2022-Wed Mar 30 11:02:51 2022
#45 (224)SKIP START marker  31 (flags=0x05, v2.15.0.0) lustre-clilov   'add lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:28 2022-Wed Mar 30 11:02:49 2022
#46 (136)SKIP pool add 0:lustre-clilov  1:lustre  2:testpool  3:lustre-OST0000_UUID  
#48 (224)SKIP END   marker  31 (flags=0x06, v2.15.0.0) lustre-clilov   'add lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:28 2022-Wed Mar 30 11:02:49 2022
#49 (224)marker  33 (flags=0x01, v2.15.0.0) lustre-clilov   'add lustre.testpool.lustre-OST0001_UUID' Wed Mar 30 11:02:30 2022-
#50 (136)pool add 0:lustre-clilov  1:lustre  2:testpool  3:lustre-OST0001_UUID  
#51 (224)END   marker  33 (flags=0x02, v2.15.0.0) lustre-clilov   'add lustre.testpool.lustre-OST0001_UUID' Wed Mar 30 11:02:30 2022-
#52 (224)marker  35 (flags=0x01, v2.15.0.0) lustre-clilov   'rem lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:49 2022-
#53 (136)pool remove 0:lustre-clilov  1:lustre  2:testpool  3:lustre-OST0000_UUID  
#54 (224)END   marker  35 (flags=0x02, v2.15.0.0) lustre-clilov   'rem lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:49 2022-
#55 (224)marker  37 (flags=0x01, v2.15.0.0) lustre-clilov   'del lustre.testpool' Wed Mar 30 11:02:51 2022-
#56 (112)pool destroy 0:lustre-clilov  1:lustre  2:testpool  
#57 (224)END   marker  37 (flags=0x02, v2.15.0.0) lustre-clilov   'del lustre.testpool' Wed Mar 30 11:02:51 2022-

I will look into this issue.

Issue Links

• is related to: LU-16324 conf-sanity test_123ae: old conf_param wasn't stored in params log
• is related to: LU-15142 LCTL: permanent parameter deletion and llog_print issues