[LU-711] lfsck : @@@@@@ FAIL: removing objects failed

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Minh Diep
• Assignee: WC Triage
• Resolution: Won't Fix

Environment

Lustre Clients:
Tag: v2_1_0_0_RC2
Distro/Arch: Sles11/x86_64 (kernel version: 2.6.32.43-0.4-default)
Build: https://build.whamcloud.com/job/lustre-master/arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/
Network: TCP
ENABLE_QUOTA=yes

Lustre Servers:
Tag: v2_1_0_0_RC2
Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.6.1.el6)
Build: https://build.whamcloud.com/job/lustre-master/283/arch=x86_64,build_type=cserver,distro=el6,ib_stack=inkernel/
Network: TCP

• Severity: 3

Description

v2_1_0_0_RC2 testing

Report: https://maloo.whamcloud.com/test_sets/18d6652a-e58c-11e0-9909-52540025f9af

08:23:11:----============= acceptance-small: lfsck ============---- Thu Sep 22 08:23:10 PDT 2011
08:23:12:Running: bash /usr/lib64/lustre/tests/lfsck.sh
08:23:12:which: no lfsck in (/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/mpi/gcc/openmpi/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin
08:23:12:client-22vm1: Checking config lustre mounted on /mnt/lustre
08:23:12:client-22vm2: Checking config lustre mounted on /mnt/lustre
08:23:12:Checking servers environments
08:23:13:Checking clients client-22vm1,client-22vm2 environments
08:23:14:Using TIMEOUT=20
08:23:15:enable quota as required
08:23:15:[HOST:client-22vm1] [old_quota_type:3] [new_quota_type:]
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:16:client-22vm3: error: conf_param: No such device
08:23:17:Total disk size: 49867944 block-softlimit: 49868968 block-hardlimit:
08:23:17: 52362416 inode-softlimit: 5000040 inode-hardlimit: 5250042
08:23:17:debug=0x33f0404
08:23:17:subsystem_debug=0xffb7e3ff
08:23:17:debug_mb=32
08:23:18:client-22vm2: debug=0x33f0404
08:23:18:client-22vm2: subsystem_debug=0xffb7e3ff
08:23:18:client-22vm2: debug_mb=32
08:23:18:client-22vm3: debug=0x33f0404
08:23:18:client-22vm4: debug=0x33f0404
08:23:18:client-22vm3: subsystem_debug=0xffb7e3ff
08:23:18:client-22vm3: debug_mb=32
08:23:18:client-22vm4: subsystem_debug=0xffb7e3ff
08:23:18:client-22vm4: debug_mb=32
08:23:18:setting all flavor to null
08:23:20:already have total 23 null connections
08:23:20:1+0 records in
08:23:20:1+0 records out
08:23:20:1048576 bytes (1.0 MB) copied, 0.160869 s, 6.5 MB/s
08:23:21:creating files in /mnt/lustre/d0.lfsck/d1
08:23:21:creating files in /mnt/lustre/d0.lfsck/d2
08:23:21:creating files in /mnt/lustre/d0.lfsck/d3
08:23:21:creating files in /mnt/lustre/d0.lfsck/d4
08:23:21:creating /mnt/lustre/d0.lfsck/testfile.1
08:23:21:creating /mnt/lustre/d0.lfsck/testfile.2
08:23:21:creating /mnt/lustre/d0.lfsck/testfile.3
08:23:21:creating /mnt/lustre/d0.lfsck/testfile.4
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.5
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.6
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.7
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.8
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.9
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.10
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.11
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.12
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.13
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.14
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.15
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.16
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.17
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.18
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.19
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.20
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.21
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.22
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.23
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.24
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.25
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.26
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.27
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.28
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.29
08:23:22:creating /mnt/lustre/d0.lfsck/testfile.30
08:23:22:creating files in /mnt/lustre/d0.lfsck/d9
08:23:22:creating files in /mnt/lustre/d0.lfsck/d10
08:23:22:creating files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:linking files in /mnt/lustre/d0.lfsck/d11
08:23:23:Stopping clients: client-22vm1,client-22vm2 /mnt/lustre (opts:-f)
08:23:23:Stopping client client-22vm1 /mnt/lustre opts:-f
08:23:23:Stopping client client-22vm2 /mnt/lustre opts:-f
08:23:23:Stopping clients: client-22vm1,client-22vm2 /mnt/lustre2 (opts:-f)
08:23:23:Stopping /mnt/mds1 (opts:-f)
08:23:35:Stopping /mnt/ost1 (opts:-f)
08:23:39:Stopping /mnt/ost2 (opts:-f)
08:23:45:Stopping /mnt/ost3 (opts:-f)
08:23:49:Stopping /mnt/ost4 (opts:-f)
08:23:54:Stopping /mnt/ost5 (opts:-f)
08:24:00:Stopping /mnt/ost6 (opts:-f)
08:24:04:Stopping /mnt/ost7 (opts:-f)
08:24:20:removing objects from /dev/mapper/lvm-P0 on : 42 43
08:24:21:client-22vm4: debugfs 1.41.90.wc4 (01-Sep-2011)
08:24:21:client-22vm4: /tmp/debugfs.YC6um7v6FJ: No such file or directory
08:24:21: lfsck : @@@@@@ FAIL: removing objects failed
08:24:21:Dumping lctl log to /logdir/test_logs/2011-09-22/lustre-master-el6-x86_64-sles11-x86_64_283_-7f86ff513a78/lfsck..*.1316705060.log
08:24:25:lfsck returned 0
08:24:25:running: liblustre
08:24:26:run_suite liblustre /usr/lib64/lustre/tests/liblustre.sh

Comments

[Jian Yu]

Lustre Tag: v2_1_0_0_RC2
Lustre Build: http://newbuild.whamcloud.com/job/lustre-master/283/
Distro/Arch: RHEL6/x86_64(server, 2.6.32-131.6.1.el6_lustre), SLES11/x86_64(client, 2.6.32.36-0.5-default)
Network: TCP (1GigE)
ENABLE_QUOTA=yes

Manually running lfsck test failed with TT-177:
https://maloo.whamcloud.com/test_sets/26d65256-e5dd-11e0-9909-52540025f9af

[Johann Lombardi (Inactive)]

From this last report, it seems that lfsck was not installed on the node, no?

-----============= acceptance-small: lfsck ============----- Fri Sep 23 05:04:33 PDT 2011
Running: bash /usr/lib64/lustre/tests/lfsck.sh
which: no lfsck in (/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/mpi/gcc/openmpi/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:)
client-1: Checking config lustre mounted on /mnt/lustre
client-2: Checking config lustre mounted on /mnt/lustre
[...]

lfsck -c -l --mdsdb /home/yujian/test_logs/mdsdb --ostdb /home/yujian/test_logs/ostdb-0 /home/yujian/test_logs/ostdb-1 /home/yujian/test_logs/ostdb-2 /home/yujian/test_logs/ostdb-3 /home/yujian/test_logs/ostdb-4 /home/yujian/test_logs/ostdb-5 /home/yujian/test_logs/ostdb-6 /mnt/lustre
/usr/lib64/lustre/tests/test-framework.sh: line 2564: lfsck: command not found
 lfsck : @@@@@@ FAIL: lfsck -c -l --mdsdb /home/yujian/test_logs/mdsdb --ostdb  /home/yujian/test_logs/ostdb-0 /home/yujian/test_logs/ostdb-1 /home/yujian/test_logs/ostdb-2 /home/yujian/test_logs/ostdb-3 /home/yujian/test_logs/ostdb-4 /home/yujian/test_logs/ostdb-5 /home/yujian/test_logs/ostdb-6 /mnt/lustre returned 127, should be <= 1 
Dumping lctl log to /home/yujian/test_logs/2011-09-23/050355/lfsck..*.1316779576.log

Yujian, am i missing something?

[Jian Yu]

Yujian, am i missing something?

Yes, "Manually running lfsck test failed with TT-177".

[Johann Lombardi (Inactive)]

hm, so why is this problem reported here? TT-177 is quite different for this bug.
Please advise.

[Jian Yu]

hm, so why is this problem reported here? TT-177 is quite different for this bug. Please advise.

The original issue was hit in autotest run. In order to reproduce the issue, I manually ran the test but hit TT-177. So, I think it's better to describe this in this ticket to make a comparison between the results of autotest and manual runs.

[Andreas Dilger]

Old lfsck is no longer supported.