[LU-9154] MDS Nodemap filset will be clear to empty after add new nodemap

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: sebg-crd-pm (Inactive)
• Assignee: Emoly Liu
• Resolution: Duplicate

Environment

Centos 7.2.3.10.0-327.22.2.el7.x86_64
Lustre 2.8.59_25_g40e3986
ZFS 0.6.5.7

• Severity: 3

Description

I have one nodemapA and set fileset by "lctl set_param -P xxxx".
The nodemapA fileset in MDS will be clear to empty when do
"lctl nodemap_add_range --name nodemapA --range xxx" or "lctl nodemap_add nodemapB"
But the nodemapA fileset still exist in MGS. Does anyone have the same problem? Thanks.

Issue Links

• is duplicated by: LU-10703 All mds Nodemap filset will be cleare...

Comments

[Peter Jones]

Emoly

Could you please assist with this issue?

Thanks

Peter

[sebg-crd-pm (Inactive)]

Does anyone have the same issue?

[Emoly Liu]

Could you show me your commands in details? Thanks.

[sebg-crd-pm (Inactive)]

[test script ]
echo "create nodemapA"
pdsh -S -Rssh -w mgs lctl nodemap_add testnodemapA
sleep 10
echo "check mds nodemapA"
pdsh -S -Rssh -w mds lctl get_param nodemap.testnodemapA
echo "set nodemapA fileset"
pdsh -S -Rssh -w mgs lctl set_param -P nodemap.testnodemapA.fileset=/testnodemapA
sleep 10
echo "check mgs nodemapA fileset"
pdsh -S -Rssh -w mgs lctl get_param nodemap.testnodemapA.* | grep fileset
echo "check mds nodemapA fileset"
pdsh -S -Rssh -w mds lctl get_param nodemap.testnodemapA.* | grep fileset
echo "create nodemapB"
pdsh -S -Rssh -w mgs lctl nodemap_add testnodemapB
sleep 10
echo "check mgs nodemapA fileset"
pdsh -S -Rssh -w mgs lctl get_param nodemap.testnodemapA.* | grep fileset
echo "check mds nodemapA fileset"
pdsh -S -Rssh -w mds lctl get_param nodemap.testnodemapA.* | grep fileset

[result]
create nodemapA
check mds nodemapA
set nodemapA fileset
check mgs nodemapA fileset
mgs: nodemap.testnodemapA.fileset=/testnodemapA
check mds nodemapA fileset
mds: nodemap.testnodemapA.fileset=/testnodemapA
create nodemapB
check mgs nodemapA fileset
mgs: nodemap.testnodemapA.fileset=/testnodemapA
check mds nodemapA fileset
mds: nodemap.testnodemapA.fileset=

=>the testnodemapA.fileset will be clear to empty

[Emoly Liu]

Here is my local output:

[root@centos6-2 tests]# sh lu-9154.sh 
create nodemapA
check mds nodemapA
set nodemapA fileset
check mgs nodemapA fileset
nodemap.testnodemapA.fileset=/testnodemapA
check mds nodemapA fileset
nodemap.testnodemapA.fileset=/testnodemapA
create nodemapB
check mgs nodemapA fileset
nodemap.testnodemapA.fileset=/testnodemapA
check mds nodemapA fileset
nodemap.testnodemapA.fileset=/testnodemapA

"testnodemapA" is still there. I will try with multiple nodes and the same lustre version to yours(Lustre 2.8.59_25_g40e3986) later.

[Emoly Liu]

Just did a test on multiple nodes. The output of my first run is

[root@trevis-73vm6 tests]# sh lu-9154.patch
create nodemapA
check mds nodemapA
set nodemapA fileset
check mgs nodemapA fileset
trevis-73vm3: nodemap.testnodemapA.fileset=
check mds nodemapA fileset
trevis-73vm4: nodemap.testnodemapA.fileset=/testnodemapA
create nodemapB
check mgs nodemapA fileset
trevis-73vm3: nodemap.testnodemapA.fileset=
check mds nodemapA fileset
trevis-73vm4: nodemap.testnodemapA.fileset=

Then, I found the patch http://review.whamcloud.com/18783, which introduced fileset to nodemap, the description says:

commit 25420c75e45a91c6131d52a674869d6d30ab36bf
Author: Sebastien Buisson <[e-mail]>
Date:   Fri Feb 5 16:44:40 2016 +0100

    LU-7846 nodemap: add fileset info to nodemap
    
    Give the ability to set a fileset information on a nodemap.
    When set, a client pertaining to this nodemap will be allowed to
    mount only the subdirectory represented by this fileset.
    
    Fileset info of nodemap is stored permanently by using
    'lctl set_param -P' commands. It means MDS and OSS will be able to
    retrieve fileset info of nodemap after a restart (when they read
    params llog).
    MGS will not see fileset info after restart, but this does not prevent
    the feature from working.

After I restarted mds, I can see fileset info there:

[root@trevis-73vm4 ~]# umount /mnt/lustre-mds1
[root@trevis-73vm4 ~]# mount -t lustre -o loop /tmp/lustre-mdt1 /mnt/lustre-mds1
[root@trevis-73vm4 ~]# lctl get_param nodemap.testnodemapA.fileset
nodemap.testnodemapA.fileset=/testnodemapA

Can you have a try to restart your mds and see if fileset info is there?

[sebg-crd-pm (Inactive)]

I have tried to restart mds then get nodemap.testnodemapA.fileset=/testnodemapA after mds started.

Emoly, Could you tell me what's your lustre version (test nodemap script)? Thanks.

[Emoly Liu]

I used 2.9.53 in my local test and 2.9.0 in multiple nodes test.

[sebg-crd-pm (Inactive)]

I have test 2.9.0 and it also have the same problem.
Did you set nodemap.active ? I test it with "lctl set_param nodemap.active=1"

[sebg-crd-pm (Inactive)]

Emoly,

I have set lctl set_param debug=-1 then test it again.
Thereis somethiong wrong. (This log get form mds. ) Should I get something else?

Jun 16 22:01:28 mdsb2 kernel: Lustre: 31848:0:(nodemap_handler.c:1020:nodemap_create()) adding nodemap 'nodemap1' to config without default nodemap
Jun 16 22:01:28 mdsb2 kernel: Lustre: 31848:0:(nodemap_handler.c:1020:nodemap_create()) Skipped 1 previous similar message
Jun 16 22:05:01 mdsb2 kernel: LustreError: 32302:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) cannot add nodemap config to non-existing MGS.

[Emoly Liu]

If possible, please post all the steps you did to reproduce this issue. Thanks.

[sebg-crd-pm (Inactive)]

nodemap-LU9154.sh build-LU9154.sh
Hi Emoly,
I simplified the my build lustre steps and nodemap test scripts.
The issue can be reprodced in my site only execute these 2 scripts.

build-LU9154.sh
create 3 nodes lustre system(based on zfs) , mgs:mdsb1, mds:mdsb2, oss:ossb1
nodemap-LU915.sh
create nodemapA( set fileset) then create nodemapB =>mds nodemapA fileset missing

[sebg-crd-pm (Inactive)]

Hi Emoly,
Is it ok to reproduce this issue by build-LU9154.sh/nodemap-LU915.sh ?

[Malcolm Haak - NCI (Inactive)]

I am seeing this issue in production with a Lustre 2.10.x system

ON MDS0: lctl set_param -P nodemap.test.fileset='/filesystem/test/path'

ON ADMIN NODE: cexec -p mds: 'lctl get_param nodemap.test.fileset'
mds0: }}{{nodemap.test.fileset=/filesystem/test/path
mds1: }}{{nodemap.test.fileset=/filesystem/test/path
mds2: }}{{nodemap.test.fileset=/filesystem/test/path
mds3: }}{{nodemap.test.fileset=/filesystem/test/path

ON ADMIN NODE: cexec -p oss: 'lctl get_param nodemap.test.fileset'
oss0: }}{{nodemap.test.fileset=/filesystem/test/path
oss1: }}{{nodemap.test.fileset=/filesystem/test/path
.............

ON MDS0: lctl set_param nodemap.active=1

ON ADMIN NODE: cexec -p mds: 'lctl get_param nodemap.test.fileset'
mds0: }}{{nodemap.test.fileset=
mds1: }}{{nodemap.test.fileset=
mds2: }}{{nodemap.test.fileset=
mds3: }}{{nodemap.test.fileset=

ON ADMIN NODE: cexec -p oss: 'lctl get_param nodemap.test.fileset'
oss0: }}{{nodemap.test.fileset=
oss1: }}{{nodemap.test.fileset=
.............

Hope this helps.

It looks like its quite broken.

Also I'm about to lodge another case because setting fileset twice on the same nodemap kills the MGS.

We found that because we tried to enable nodemaps, it cleared the fileset value and we tried setting it again.

[sebg-crd-pm (Inactive)]

Hi mhaakddn ,

*Is there *any further update?

[Emoly Liu]

This issue is a duplicate of LU-10703. I created a patch at https://review.whamcloud.com/31450 .

[Andreas Dilger]

Close as a duplicate of LU-10703, since that has a patch.