|
I attached full debug logs from the MDS/MGS.
|
|
Dropping severity because I take it that this is not a production service interruption
|
|
Just our test bed is busted 
|
|
James
Could you please elaborate as to the exact commit you are seeing this with and the last commit that you did not see this problem?
Peter
|
|
James,
Could you be more specific about the Lustre version here? (Saying 'latest' does not age well.) Also can you give a recent Lustre version where you didn't see this issue?
Dose this still happen if you set lnet_peer_discovery_disabled=1 in the lnet module parameters?
|
|
James, can you also try: https://review.whamcloud.com/#/c/29007/
|
|
I did and it still fail to bring up the file system John was right. When I set lnet_peer_discovery_disabled=1 everything worked as usual.
|
|
in the dump-mds.log I see:
(lib-move.c:3199:LNetGet()) Error sending GET to 12345-10.37.202.59@o2ib1: -113
Is that issue resolved?
Can you also turn on net and neterror when you're mounting and failing, and attach the output.
|
|
That is due to patch LU-9810 lnet: prefer Fast Reg. I had to revert it locally to make things work better. I will post new logs shortly.
|
|
With the revert of LU-9810 prefer Fast Reg and the fix from LU-9933 + LU-9992 everything seems to work now.
|
|
Ah, I found what caused this issue. I have a reproducer. Its very simple, add routes that are all down to your yaml config file and you will not be able to mount your file system. We have Cray routers and they have been done recently. So add to your yaml config file something like this:
route:
- net: gni1
gateway: 10.37.202.60@o2ib1
hop: 1
priority: 0
state: down
- net: gni1
gateway: 10.37.202.59@o2ib1
hop: 1
priority: 0
state: down
- net: gni1
gateway: 10.37.202.61@o2ib1
hop: 1
priority: 0
state: down
And you will see problems.
|
|
I couldn't reproduce this on my local setup. I'm assuming that the downed gateways are not needed for the FS to be mounted, correct? IE do you need those routes for communication? If that's not the case, can you please enable net and neterror and dump the logs after the problem happens to so I can take a look.
lctl set_param debug=+net
lctl set_param debug=+neterror
lctl dk > log
|
|
James,
can you confirm - LU-9180 isn't caused it bug? if no, can you test with patches from LU-9810 ?
thanks.
|
|
I have been testing without LU-9810 patches. This bug is still their. Its just when I run with LU-9810 patches new bugs show up Amir I will get new debug logs for you Friday.
|
|
James,
what HW you use for testing? if it MLX5, these patches do nothing for you. MLX5 uses a FastReg only model, while MLX4 support a both Fast and FMR.
I don't have access to HW IB for now, so may check only with VM's + M-OFED 4.1+ Soft IB
|
|
Sorry I was having a hard time reproducing this problem. Its not the route configuration that breaks lnet but the numa node setting in my lnet.conf that did. I ended up removing the numa stuff from my config file. If you add
numa:
range: 0
to your lnet yaml config file you will see this breakage.
|
|
I believe the numa range defaults to 0. When you remove it and you do "lnetctl numa show" do you see a different value for the range?
|
|
[root@ninja34 ~]# lnetctl global show
global:
numa_range: 0
max_intf: 200
discovery: 1
Does the numa_range have to be "under" global: in the YAML config file?
|
|
so the output is a little strange. It looks like you're using the latest master. But the configuration you're feeding in seems to have been generated from 2.10. numa should be fed in under the global, as shown in the global show output you pasted above. Now thinking about it, this is a backwards compatibility issue since 2.10 is already out. I'll have to make lnetctl handle the older configuration as well for master.
Do you see "call back for 'numa' not found" error when you configure with the numa block?
I think I know what the problem is. When the parser encounters a problem in the YAML file, it'll quit and simply stop configuring the rest of the items. So it could be that when it hits this error it doesn't finish the configuration leading to the problem you're seeing.
Are you calling "lnetctl import" from a script? If so, I think you should be checking if the command succeeds or fails. If it fails you should assume that the node is not configured properly.
Can you verify if my theory is correct?
|
|
Yes I do see a "call back from 'num' not found error when I start up. Also I was using lnet.conf from my 2.10 setup. I'm running the lnetctl import from the command line not script.
|
|
ok. I'll make a change to handle "numa" entry in YAML file so I can make master backwards compatible with 2.10. In the meantime, if there is an error configuring, you should assume that the configuration is not complete, and the node is not really usable.
|
|
Amir Shehata (amir.shehata@intel.com) uploaded a new patch: https://review.whamcloud.com/29333
Subject: LU-9990 lnet: add backwards compatibility for YAML config
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e72cdc1373dfb930eccbc5d9afba215d8368b331
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/29333/
Subject: LU-9990 lnet: add backwards compatibility for YAML config
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3187d551d538bd8203c7156daaa617620c6569ab
|
|
Landed for 2.11
|
|
ashehata said we don't need this for LTS
|
Generated at Sat Feb 10 02:31:05 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
dump-mds.log