[LU-3326] recovery-mds-scale test_failover_ost: tar: Cannot open: No space left on device

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.3
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1, Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Labels: mn1 mn4
• Environment: Lustre Branch: master; Lustre Build: http://build.whamcloud.com/job/lustre-master/1486; Distro/Arch: RHEL6.4/x86_64; Test Group: failover
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Jian Yu

Description

After running recovery-mds-scale test_failover_ost for 1.5 hours (OST failed over 6 times), client load on one of the clients failed as follows:

<snip>
tar: etc/mail/submit.cf: Cannot open: No space left on device
tar: etc/mail/trusted-users: Cannot open: No space left on device
tar: etc/mail/virtusertable: Cannot open: No space left on device
tar: etc/mail/access: Cannot open: No space left on device
tar: etc/mail/aliasesdb-stamp: Cannot open: No space left on device
tar: etc/gssapi_mech.conf: Cannot open: No space left on device
tar: Exiting with failure status due to previous errors

Console log on the client (client-32vm6) showed that:

19:40:31:INFO: task tar:2790 blocked for more than 120 seconds.
19:40:31:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
19:40:31:tar           D 0000000000000000     0  2790   2788 0x00000080
19:40:31: ffff88004eb73a28 0000000000000082 ffff88004eb739d8 ffff88007c24fe50
19:40:31: 0000000000000286 0000000000000003 0000000000000001 0000000000000286
19:40:31: ffff88007bcb3ab8 ffff88004eb73fd8 000000000000fb88 ffff88007bcb3ab8
19:40:31:Call Trace:
19:40:31: [<ffffffffa03d775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
19:40:31: [<ffffffff8150ea05>] schedule_timeout+0x215/0x2e0
19:40:31: [<ffffffffa068517c>] ? ptlrpc_request_bufs_pack+0x5c/0x80 [ptlrpc]
19:40:31: [<ffffffffa069a770>] ? lustre_swab_ost_body+0x0/0x10 [ptlrpc]
19:40:31: [<ffffffff8150e683>] wait_for_common+0x123/0x180
19:40:31: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
19:40:31: [<ffffffff8150e79d>] wait_for_completion+0x1d/0x20
19:40:31: [<ffffffffa08cbf6c>] osc_io_setattr_end+0xbc/0x190 [osc]
19:40:31: [<ffffffffa095cde0>] ? lov_io_end_wrapper+0x0/0x100 [lov]
19:40:31: [<ffffffffa055cf30>] cl_io_end+0x60/0x150 [obdclass]
19:40:31: [<ffffffffa055d7e0>] ? cl_io_start+0x0/0x140 [obdclass]
19:40:31: [<ffffffffa095ced1>] lov_io_end_wrapper+0xf1/0x100 [lov]
19:40:31: [<ffffffffa095c86e>] lov_io_call+0x8e/0x130 [lov]
19:40:31: [<ffffffffa095e3bc>] lov_io_end+0x4c/0xf0 [lov]
19:40:31: [<ffffffffa055cf30>] cl_io_end+0x60/0x150 [obdclass]
19:40:31: [<ffffffffa0561f92>] cl_io_loop+0xc2/0x1b0 [obdclass]
19:40:31: [<ffffffffa0a2aa08>] cl_setattr_ost+0x208/0x2c0 [lustre]
19:40:31: [<ffffffffa09f8b0e>] ll_setattr_raw+0x9ce/0x1000 [lustre]
19:40:31: [<ffffffffa09f919b>] ll_setattr+0x5b/0xf0 [lustre]
19:40:31: [<ffffffff8119e708>] notify_change+0x168/0x340
19:40:31: [<ffffffff811b284c>] utimes_common+0xdc/0x1b0
19:40:31: [<ffffffff811828d1>] ? __fput+0x1a1/0x210
19:40:31: [<ffffffff811b29fe>] do_utimes+0xde/0xf0
19:40:31: [<ffffffff811b2b12>] sys_utimensat+0x32/0x90
19:40:31: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Maloo report: https://maloo.whamcloud.com/test_sets/053120d2-bb19-11e2-8824-52540035b04c

Issue Links

• is duplicated by: LU-5235 Failure on test suite recovery-mds-scale test_failover_mds: No space left on device
• is related to: LU-5526 recovery-mds-scale test failover_mds: dd: No space left on device

Comments

[Jian Yu]

The new ticket is LU-5526.

[Peter Jones]

Test fixes landed for 2.5.3 and 2.7. Residual patch landing will be tracked under a new ticket.

[Jian Yu]

The back-ported patch for http://review.whamcloud.com/8071 on Lustre b2_5 branch is in http://review.whamcloud.com/11425.

[Hongchao Zhang]

updated the two patches of this ticket

(1) http://review.whamcloud.com/8671
as per the comments in Lustre, the LDLM_FL_AST_DISCARD_DATA is also designed to indicate "FL_DISCARD" flag to the blocking ASTs

/**
 * These are flags that are mapped into the flags and ASTs of blocking
 * locks Add FL_DISCARD to blocking ASTs */
#define LDLM_FL_AST_DISCARD_DATA        0x0000000080000000ULL // bit  31

(2) http://review.whamcloud.com/8071
wait the deletion of objects to complete before doing various tests at client node.

[Jian Yu]

Hi Hongchao,

Could you please drive patches of http://review.whamcloud.com/8671 (needs address the comment from Andreas) and http://review.whamcloud.com/8071 to be landed? Thanks. LU-3326 has been blocking MDS/OST hard failover testing in Lustre release testing cycles.

[Jian Yu]

This is blocking recovery-mds-scale testing.

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/39/ (2.5.1 RC1)
Distro/Arch: RHEL6.5/x86_64
Test Group: failover

The same failure occurred:
https://maloo.whamcloud.com/test_sets/36c8c4fe-a657-11e3-a191-52540035b04c

Many sub-tests in replay-single in failover test group also hit the "No space left on device" failure:
https://maloo.whamcloud.com/test_sets/c55e55ee-a657-11e3-a191-52540035b04c

[Jian Yu]

More instances on Lustre b2_5 branch:
https://maloo.whamcloud.com/test_sets/fcbcabd0-770e-11e3-b181-52540035b04c
https://maloo.whamcloud.com/test_sets/e669c6a8-8643-11e3-9f3f-52540035b04c
https://maloo.whamcloud.com/test_sets/fc5c9556-8505-11e3-8da9-52540035b04c
https://maloo.whamcloud.com/test_sets/f16b8720-9922-11e3-83d7-52540035b04c
https://maloo.whamcloud.com/test_sets/7d4fc910-956b-11e3-936f-52540035b04c

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/5/
Test Group: failover

The same failure occurred:
https://maloo.whamcloud.com/test_sets/c86198c4-7505-11e3-95ae-52540035b04c

[Hongchao Zhang]

local test with two nodes, the server node runs master and client runs b2_1, b2_3, b2_4 or master, the pages were not discarded after the lock's blocking AST
is received, and will discard the pages after adding "LDLM_FL_DISCARD_DATA" instead of "LDLM_FL_AST_DISCARD_DATA" into "LDLM_FL_AST_MASK".

LDLM_AST_DISCARD_DATA (changed to LDLM_FL_AST_DISCARD_DATA in master) is used to indicate ldlm to sent blocking AST with LDLM_FL_DISCARD_DATA flag,
but it self is not on the wire, then the LDLM_FL_DISCARD_DATA will be ignored for it will be masked by "LDLM_FL_AST_MASK".

code snippet in master

void ldlm_add_bl_work_item(struct ldlm_lock *lock, struct ldlm_lock *new,
                           cfs_list_t *work_list)
{
        if (!ldlm_is_ast_sent(lock)) {
                LDLM_DEBUG(lock, "lock incompatible; sending blocking AST.");
                ldlm_set_ast_sent(lock);
                /* If the enqueuing client said so, tell the AST recipient to
                 * discard dirty data, rather than writing back. */
                if (ldlm_is_ast_discard_data(new))         <---- check LDLM_FL_AST_DISCARD_DATA
                        ldlm_set_discard_data(lock);       <---- set LDLM_FL_DISCARD_DATA
                LASSERT(cfs_list_empty(&lock->l_bl_ast));
                cfs_list_add(&lock->l_bl_ast, work_list);
                LDLM_LOCK_GET(lock);
                LASSERT(lock->l_blocking_lock == NULL);
                lock->l_blocking_lock = LDLM_LOCK_GET(new);
        }
}

the corresponding patch is tracked at http://review.whamcloud.com/#/c/8671/