[LU-16287] replay-single: test_102d timeout

Thu, 3 Nov 2022 00:51:39 +0000

This issue was created by maloo for Lai Siyao <lai.siyao@whamcloud.com>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/aec823a8-3961-47ad-b2b5-d8a97f1a242f

[Wed Nov  2 05:35:20 2022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 05:35:36 \(1667367336\)
[Wed Nov  2 05:35:21 2022] Lustre: DEBUG MARKER: == replay-single test 102d: check replay
[Wed Nov  2 05:35:21 2022] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a
[Wed Nov  2 05:35:21 2022] Lustre: *** cfs_fail_loc=15a, val=0***
[Wed Nov  2 05:35:21 2022] Lustre: Skipped 5 previous similar messages
[Wed Nov  2 05:35:23 2022] Lustre: DEBUG MARKER: sync; sync; sync
[Wed Nov  2 05:35:24 2022] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0
[Wed Nov  2 05:35:24 2022] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
[Wed Nov  2 05:35:25 2022] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds4
[Wed Nov  2 05:35:25 2022] Lustre: lustre-MDT0003: Not available for connect from 10.240.29.171@tcp (stopping)
[Wed Nov  2 05:35:25 2022] Lustre: Skipped 7 previous similar messages
[Wed Nov  2 05:35:29 2022] Lustre: lustre-MDT0003: Not available for connect from 10.240.29.168@tcp (stopping)
[Wed Nov  2 05:35:29 2022] Lustre: Skipped 10 previous similar messages
[Wed Nov  2 05:35:34 2022] Lustre: lustre-MDT0003: Not available for connect from 10.240.29.168@tcp (stopping)
[Wed Nov  2 05:35:34 2022] Lustre: Skipped 12 previous similar messages
[Wed Nov  2 05:35:42 2022] Lustre: lustre-MDT0003: Not available for connect from 0@lo (stopping)
[Wed Nov  2 05:35:42 2022] Lustre: Skipped 24 previous similar messages
[Wed Nov  2 05:35:45 2022] LustreError: 190662:0:(import.c:355:ptlrpc_invalidate_import()) lustre-MDT0001_UUID: timeout waiting for callback (1 != 0)
[Wed Nov  2 05:35:45 2022] LustreError: 190662:0:(import.c:383:ptlrpc_invalidate_import()) @@@ still on delayed list  req@00000000e4286085 x1748352857944832/t0(0) o41->lustre-MDT0001-osp-MDT0003@0@lo:24/4 lens 224/224 e 0 to 0 dl 1670132019 ref 1 fl Rpc:RESQU/0/0 rc -5/-107 job:'osp-pre-1-3.0'
[Wed Nov  2 05:35:45 2022] LustreError: 190662:0:(import.c:389:ptlrpc_invalidate_import()) lustre-MDT0001_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out.

The log shows umount /mnt/lustre-mdt4 is stuck in ptlrpc_invalidate_import(), and the remaining request is statfs between MDTs.

Whamcloud Community JIRA

[LU-16287] replay-single: test_102d timeout