[LU-14906] sanity-pcc test_15: test_15 returned 32

• Status: Open
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

This issue was created by maloo for Qian Yingjin <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/345df1ff-e05e-4447-a15a-ded082a0f81b

test_15 failed with the following error:

test_15 returned 32

CMD: trevis-53vm2 /usr/bin/lfs pcc detach -k /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/bin/lfs pcc state /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 cat /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007161:0xe:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d=
Waiting 200s for 'SUCCEED'
CMD: trevis-53vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007161:0xe:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d=
Changed after 2s: from 'WAITING' to 'STARTED'
CMD: trevis-53vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007161:0xe:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d=
CMD: trevis-53vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007161:0xe:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d=
CMD: trevis-53vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007161:0xe:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d=
Updated after 6s: want 'SUCCEED' got 'SUCCEED'
CMD: trevis-53vm2 /usr/bin/lfs pcc state /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
Verify auto attach at open for RO-PCC
CMD: trevis-53vm2 /usr/bin/lfs pcc attach -i 2 -r /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/bin/lfs pcc state /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/sbin/lctl set_param ldlm.namespaces.*mdc*.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff94cfd98b8000.lru_size=clear
ldlm.namespaces.lustre-MDT0000-mdc-ffff94cfe4d1b800.lru_size=clear
ldlm.namespaces.lustre-MDT0001-mdc-ffff94cfd98b8000.lru_size=clear
ldlm.namespaces.lustre-MDT0001-mdc-ffff94cfe4d1b800.lru_size=clear
ldlm.namespaces.lustre-MDT0002-mdc-ffff94cfd98b8000.lru_size=clear
ldlm.namespaces.lustre-MDT0002-mdc-ffff94cfe4d1b800.lru_size=clear
ldlm.namespaces.lustre-MDT0003-mdc-ffff94cfd98b8000.lru_size=clear
ldlm.namespaces.lustre-MDT0003-mdc-ffff94cfe4d1b800.lru_size=clear
CMD: trevis-53vm2 cat /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/bin/lfs pcc state /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/bin/lfs pcc detach -k /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/bin/lfs pcc state /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 cat /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
CMD: trevis-53vm2 /usr/bin/lfs pcc detach /mnt/lustre/d15.sanity-pcc/f15.sanity-pcc
Resetting fail_loc on all nodes...CMD: trevis-53vm1.trevis.whamcloud.com,trevis-53vm2,trevis-53vm3,trevis-53vm4,trevis-53vm5 lctl set_param -n fail_loc=0 	    fail_val=0 2>/dev/null
done.
CMD: trevis-53vm1.trevis.whamcloud.com /usr/sbin/lctl get_param catastrophe 2>&1
CMD: trevis-53vm2 /usr/sbin/lctl get_param catastrophe 2>&1
CMD: trevis-53vm3 /usr/sbin/lctl get_param catastrophe 2>&1
CMD: trevis-53vm4 /usr/sbin/lctl get_param catastrophe 2>&1
CMD: trevis-53vm5 /usr/sbin/lctl get_param catastrophe 2>&1
CMD: trevis-53vm1.trevis.whamcloud.com,trevis-53vm2,trevis-53vm3,trevis-53vm4,trevis-53vm5 dmesg
Cleanup PCC backend on /mnt/lustre
CMD: trevis-53vm2 /usr/sbin/lctl pcc clear /mnt/lustre
CMD: trevis-53vm2 pkill --pidfile=/var/run/lhsmtool_posix.pid --signal=TERM hsmtool
CMD: trevis-53vm2 rm -rf /mnt/pcc.d15.sanity-pcc/d15.sanity-pcc
CMD: trevis-53vm2 lsof /mnt/pcc.d15.sanity-pcc
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
lhsmtool_ 26391 root    3r   DIR    7,0        0   12 /mnt/pcc.d15.sanity-pcc/d15.sanity-pcc (deleted)
CMD: trevis-53vm2 umount -d /mnt/pcc.d15.sanity-pcc
trevis-53vm2: umount: /mnt/pcc.d15.sanity-pcc: target is busy.
trevis-53vm2:         (In some cases useful info about processes that use
trevis-53vm2:          the device is found by lsof(8) or fuser(1))





VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-pcc test_15 - test_15 returned 32

Comments

[Sergey Cheremencev]

See the same but only at sanity-pcc test_2c: https://testing.whamcloud.com/test_sets/4939eb8b-082d-4b76-aba5-695dbfb6b314

I guess LU-12840 has the same roots.

 

[Sergey Cheremencev]

+1 sanity-pcc test_2c on master: https://testing.whamcloud.com/test_sets/9320ea9f-3458-40cf-971a-bb94e034050b

[Jian Yu]

+1 on sanity-pcc test_11: https://testing.whamcloud.com/test_sets/b156c3d8-9ac5-44bb-8d7e-c6a4e8ec53fd

[Arshad Hussain]

+1 on sanity-pcc test_1d https://testing.whamcloud.com/test_sets/ee93fc33-33c6-467b-9408-2de4107873aa (zfs)