[LU-7368] e2fsck unsafe to interrupt with quota enabled

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Andreas Dilger

Description

It looks there is a bug in how e2fsck handles being interrupted by CTRL-C. If CTRL-C is pressed to kill e2fsck rather than e.g. kill -9, then the interrupt handler sets E2F_FLAG_CANCEL in the context but doesn't actually kill the process. Instead, e2fsck_pass1() checks this flag before processing the next inode.

If a filesystem is running in fix mode (e2fsck -fy) is interrupted, and the quota feature is enabled, then the quota file will still be written to disk even though the inode scan was not complete and the quota information is totally inaccurate. Even worse, if the Pass 1 inode and block scan was not finished, then the in-memory block bitmaps (which are used for block allocation during e2fsck) are also invalid, so any blocks allocated to the quota files may corrupt other files if those blocks were actually used.

      e2fsck 1.42.13.wc3 (28-Aug-2015)
      Pass 1: Checking inodes, blocks, and sizes
      ^C[QUOTA WARNING] Usage inconsistent for ID 0:
          actual (6455296, 168) != expected (8568832, 231)
      [QUOTA WARNING] Usage inconsistent for ID 695:
          actual (614932320256, 63981) != expected (2102405386240, 176432)
      Update quota info for quota type 0? yes
    
      [QUOTA WARNING] Usage inconsistent for ID 0:
          actual (6455296, 168) != expected (8568832, 231)
      [QUOTA WARNING] Usage inconsistent for ID 538:
          actual (614932320256, 63981) != expected (2102405386240, 176432)
      Update quota info for quota type 1? yes
    
      myth-OST0001: e2fsck canceled.
      myth-OST0001: ***** FILE SYSTEM WAS MODIFIED *****

It also looks like the journal may also be recreated after e2fsck is interrupted, if it was deleted during pass 1 because of corruption.

static void signal_cancel(int sig EXT2FS_ATTR((unused)))
{
       e2fsck_t ctx = e2fsck_global_ctx;

       if (!ctx)
               exit(FSCK_CANCELED);

       ctx->flags |= E2F_FLAG_CANCEL;
}

int main()
{
        :
	sa.sa_handler = signal_cancel;
	sigaction(SIGINT, &sa, 0);
	sigaction(SIGTERM, &sa, 0);
	:
	:
       run_result = e2fsck_run(ctx);
       e2fsck_clear_progbar(ctx);

       if (!ctx->invalid_bitmaps &&
           (ctx->flags & E2F_FLAG_JOURNAL_INODE)) {
		if (fix_problem(ctx, PR_6_RECREATE_JOURNAL, &pctx)) {
			:
			:
			retval = ext2fs_add_journal_inode(fs, journal_size, 0);
		}
	}

no_journal:
	if (ctx->qctx) {
		for (i = 0; i < MAXQUOTAS; i++) {
			retval = quota_compare_and_update(ctx->qctx, i, &needs_writeout);
		}
	}

	if (run_result & E2F_FLAG_ABORT)
		fatal_error(ctx, _("aborted"));

This should have a cancel check right after the return from e2fsck_run() rather than trying to recover the journal and quota files? I can imagine that there is a desire to flush out modified inodes and such that have been repaired, so that restarting an interrupted e2fsck will make progress, but the quota file update is plain wrong unless at least pass1 has finished, and the journal recreation is also dangerous if the block bitmaps have not been fully updated.

The quota problem was hit in on a system, but the journal problem is only a theory at this point. I'm working on a patch but wanted to solicit input in case there is something that I'm missing.

Issue Links

• is related to: LU-8706 e2fsck -fDy running forever
• is related to: LU-7381 "e2fsck -fD" on directory may cause extent tree corruption
• is related to: LU-7428 conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!