[LU-3845] Traffic doesn't run over the route with priority 1

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.5.0
• Type: Bug
• Priority: Minor
• Reporter: Sarah Liu
• Assignee: WC Triage
• Resolution: Not a Bug

Environment

lustre-master build #1627

• Severity: 3

Description

1.Configure 2 routes between servers(running o2ib) and clients(running tcp), one with priority of 0, the other is 1.

2.Run sanity on client wtm-77

3.Run Tcpdump on wtm-77, shows that the route with default priority(0) have the most traffic

----------
On client:

[root@wtm-77 ~]# cat /proc/sys/lnet/routes
Routing disabled
net      hops priority   state router
o2ib        1        1      up 10.10.18.199@tcp
o2ib        1        0      up 10.10.18.190@tcp

for route with priority=1

[root@wtm-77 ~]# tcpdump host 10.10.18.199
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:20:07.183349 ARP, Request who-has wbm-4.rosso.whamcloud.com tell wtm-76.rosso.whamcloud.com, length 46
17:20:08.335248 ARP, Request who-has wbm-4.rosso.whamcloud.com tell wtm-76.rosso.whamcloud.com, length 46
17:20:09.335391 ARP, Request who-has wbm-4.rosso.whamcloud.com tell wtm-76.rosso.whamcloud.com, length 46
17:20:10.335487 ARP, Request who-has wbm-4.rosso.whamcloud.com tell wtm-76.rosso.whamcloud.com, length 46
17:20:15.961815 IP wtm-77.rosso.whamcloud.com.1023 > wtm-76.rosso.whamcloud.com.7988: Flags [.], ack 2307569068, win 501, options [nop,nop,TS val 20459199 ecr 21318863], length 0
17:20:15.961954 IP wtm-76.rosso.whamcloud.com.7988 > wtm-77.rosso.whamcloud.com.1023: Flags [.], ack 1, win 114, options [nop,nop,TS val 21348863 ecr 20429199], length 0
17:20:15.962407 IP wtm-76.rosso.whamcloud.com.7988 > wtm-77.rosso.whamcloud.com.1023: Flags [.], ack 1, win 114, options [nop,nop,TS val 21348864 ecr 20429199], length 0
17:20:15.962420 IP wtm-77.rosso.whamcloud.com.1023 > wtm-76.rosso.whamcloud.com.7988: Flags [.], ack 1, win 501, options [nop,nop,TS val 20459199 ecr 21348863], length 0
17:20:16.962051 IP wtm-77.rosso.whamcloud.com.1023 > wtm-76.rosso.whamcloud.com.7988: Flags [P.], seq 1:97, ack 1, win 501, options [nop,nop,TS val 20460199 ecr 21348863], length 96
17:20:16.962306 IP wtm-76.rosso.whamcloud.com.7988 > wtm-77.rosso.whamcloud.com.1023: Flags [P.], seq 1:161, ack 97, win 114, options [nop,nop,TS val 21349864 ecr 20460199], length 160
17:20:16.962336 IP wtm-77.rosso.whamcloud.com.1023 > wtm-76.rosso.whamcloud.com.7988: Flags [.], ack 161, win 501, options [nop,nop,TS val 20460199 ecr 21349864], length 0
17:20:19.060737 IP wtm-77.rosso.whamcloud.com.exp1 > wtm-76.rosso.whamcloud.com.7988: Flags [.], ack 766608616, win 115, options [nop,nop,TS val 20462298 ecr 21321963], length 0
17:20:19.060867 IP wtm-76.rosso.whamcloud.com.7988 > wtm-77.rosso.whamcloud.com.exp1: Flags [.], ack 1, win 114, options [nop,nop,TS val 21351962 ecr 20432298], length 0
17:20:19.061348 IP wtm-76.rosso.whamcloud.com.7988 > wtm-77.rosso.whamcloud.com.exp1: Flags [.], ack 1, win 114, options [nop,nop,TS val 21351963 ecr 20432298], length 0
17:20:19.061366 IP wtm-77.rosso.whamcloud.com.exp1 > wtm-76.rosso.whamcloud.com.7988: Flags [.], ack 1, win 115, options [nop,nop,TS val 20462298 ecr 21351962], length 0
17:20:20.410311 ARP, Request who-has wbm-4.rosso.whamcloud.com tell wtm-76.rosso.whamcloud.com, length 46
^C
16 packets captured
19 packets received by filter
0 packets dropped by kernel

for route with priority=0

[root@wtm-77 ~]# tcpdump host 10.10.18.190
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:20:28.866280 ARP, Request who-has wbm-4.rosso.whamcloud.com tell wtm-75.rosso.whamcloud.com, length 46
17:20:30.941570 IP wtm-77.rosso.whamcloud.com.1023 > wtm-75.rosso.whamcloud.com.7988: Flags [P.], seq 4204049531:4204050211, ack 54217498, win 6642, options [nop,nop,TS val 20474178 ecr 21295024], length 680
17:20:30.943684 IP wtm-75.rosso.whamcloud.com.7988 > wtm-77.rosso.whamcloud.com.1023: Flags [P.], seq 1:641, ack 680, win 12342, options [nop,nop,TS val 21299096 ecr 20474178], length 640
17:20:30.943718 IP wtm-77.rosso.whamcloud.com.1023 > wtm-75.rosso.whamcloud.com.7988: Flags [.], ack 641, win 6642, options [nop,nop,TS val 20474181 ecr 21299096], length 0

.....

17:20:32.361700 IP wtm-77.rosso.whamcloud.com.exp1 > wtm-75.rosso.whamcloud.com.7988: Flags [.], ack 97, win 115, options [nop,nop,TS val 20475599 ecr 21300475], length 0
17:20:32.370706 IP wtm-77.rosso.whamcloud.com.1023 > wtm-75.rosso.whamcloud.com.7988: Flags [.], ack 26713, win 6642, options [nop,nop,TS val 20475608 ecr 21300484], length 0
^C
555 packets captured
555 packets received by filter
0 packets dropped by kernel

Comments

[Doug Oucharek (Inactive)]

The updated test plan for this feature indicates this change in behaviour. Fujitsu decided that the "lower" the priority number the higher the priority. Our implementation was updated to match theirs. Priority 0 has the highest priority so what you are seeing is correct.