[LU-17938] Client evicted during stress test

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.15.5
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Minh Diep

Description

During soak testing, we see tests started to fail when the load was high, clients being evicted

 

[Tue Jun 11 23:47:20 2024] LustreError: 11-0: testfs-MDT0000-mdc-ffff8b5a60d1f000: operation ldlm_enqueue to node 172.16.5.1@tcp failed: rc7
[Tue Jun 11 23:47:20 2024] Lustre: testfs-MDT0000-mdc-ffff8b5a60d1f000: Connection to testfs-MDT0000 (at 172.16.5.1@tcp) was lost; in progre
[Tue Jun 11 23:47:20 2024] LustreError: Skipped 7 previous similar messages
[Tue Jun 11 23:47:20 2024] LustreError: 167-0: testfs-MDT0000-mdc-ffff8b5a60d1f000: This client was evicted by testfs-MDT0000; in progress .
[Tue Jun 11 23:47:20 2024] LustreError: 1815697:0:(file.c:5189:ll_inode_revalidate_fini()) testfs: revalidate FID [0x2000915a3:0xd56:0x0] e5
[Tue Jun 11 23:47:20 2024] LustreError: 1820420:0:(file.c:242:ll_close_inode_openhandle()) testfs-clilmv-ffff8b5a60d1f000: inode [0x20009158
[Tue Jun 11 23:47:20 2024] LustreError: 1815697:0:(file.c:5189:ll_inode_revalidate_fini()) Skipped 16 previous similar messages
[Tue Jun 11 23:47:20 2024] LustreError: 1820420:0:(file.c:242:ll_close_inode_openhandle()) Skipped 93 previous similar messages
[Tue Jun 11 23:47:20 2024] LustreError: 1817163:0:(vvp_io.c:1836:vvp_io_init()) testfs: refresh file layout [0x20009159d:0x96a5:0x0] error .
[Tue Jun 11 23:47:20 2024] LustreError: 1817163:0:(vvp_io.c:1836:vvp_io_init()) Skipped 209 previous similar messages
[Tue Jun 11 23:47:20 2024] LustreError: 1815843:0:(mdc_request.c:1484:mdc_read_page()) testfs-MDT0000-mdc-ffff8b5a60d1f000: [0x200091568:0x8
[Tue Jun 11 23:47:20 2024] LustreError: 1820420:0:(ldlm_resource.c:1126:ldlm_resource_complain()) testfs-MDT0000-mdc-ffff8b5a60d1f000: name.
[Tue Jun 11 23:47:20 2024] LustreError: 1815843:0:(mdc_request.c:1484:mdc_read_page()) Skipped 28 previous similar messages
[Tue Jun 11 23:47:20 2024] LustreError: 1820420:0:(ldlm_resource.c:1126:ldlm_resource_complain()) Skipped 1 previous similar message
[Tue Jun 11 23:47:20 2024] Lustre: testfs-MDT0000-mdc-ffff8b5a60d1f000: Connection restored to 172.16.5.1@tcp (at 172.16.5.1@tcp)
[Tue Jun 11 23:47:20 2024] Lustre: dir [0x24008903f:0xaa0b:0x0] stripe 3 readdir failed: -108, directory is partially accessed!
[Tue Jun 11 23:47:20 2024] Lustre: Skipped 28 previous similar messages
[Tue Jun 11 23:47:21 2024] soak stop fio-random:58542:6 on co-es-pm-247
 

Issue Links

• is related to: LU-17197 Performance regression with "LU-15947 obdclass: improve precision of wakeups for mod_rpcs"
• is related to: LU-16633 obd_get_mod_rpc_slot() is vulnerable to races