From patchwork Fri Jun 14 11:45:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Li Dongyang X-Patchwork-Id: 1115930 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=ddn.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=ddn.com header.i=@ddn.com header.b="qpuiTPxa"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 45QJhm3Ftkz9s1c for ; Fri, 14 Jun 2019 21:46:00 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727389AbfFNLqA (ORCPT ); Fri, 14 Jun 2019 07:46:00 -0400 Received: from mail-eopbgr730068.outbound.protection.outlook.com ([40.107.73.68]:33120 "EHLO NAM05-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727164AbfFNLp7 (ORCPT ); Fri, 14 Jun 2019 07:45:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ddn.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PlqwACHmrnJmflKHGNOGSJQbEjMXXUJz3ttmS/1NKVw=; b=qpuiTPxaapsstRe8+pRTLXCaAFE0Ywf44goVAIVM4LjL3KHPzvxM7xS3FoFMpnEAvI0hMATqRJ5HqzCZgvHm8HynJwsRuI8nDkLtcALz1MgaEsg4wf2n0RXdBqQ3t+rTTu2/XCX29ZLa44m8Ia/iARfk0alKP0LictzuE4VJb3c= Received: from MN2PR19MB2592.namprd19.prod.outlook.com (20.179.82.160) by MN2PR19MB2573.namprd19.prod.outlook.com (20.179.80.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1965.15; Fri, 14 Jun 2019 11:45:55 +0000 Received: from MN2PR19MB2592.namprd19.prod.outlook.com ([fe80::f19e:764d:aba7:c6c]) by MN2PR19MB2592.namprd19.prod.outlook.com ([fe80::f19e:764d:aba7:c6c%5]) with mapi id 15.20.1987.013; Fri, 14 Jun 2019 11:45:55 +0000 From: Dongyang Li To: "linux-ext4@vger.kernel.org" CC: "adilger@dilger.ca" Subject: [PATCH] e2fsck: allow deleting or zeroing shared blocks Thread-Topic: [PATCH] e2fsck: allow deleting or zeroing shared blocks Thread-Index: AQHVIqa5WgjG1URFrEmS3agW9WZdAA== Date: Fri, 14 Jun 2019 11:45:54 +0000 Message-ID: <20190614114530.5663-1-dongyangli@ddn.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: SYBPR01CA0113.ausprd01.prod.outlook.com (2603:10c6:10:1::29) To MN2PR19MB2592.namprd19.prod.outlook.com (2603:10b6:208:102::32) authentication-results: spf=none (sender IP is ) smtp.mailfrom=dongyangli@ddn.com; x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 2.22.0 x-originating-ip: [220.233.199.190] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b2f5e452-b0a9-4a65-c281-08d6f0bddba0 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:MN2PR19MB2573; x-ms-traffictypediagnostic: MN2PR19MB2573: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2803; x-forefront-prvs: 0068C7E410 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(136003)(396003)(376002)(39850400004)(199004)(189003)(4326008)(86362001)(52116002)(5660300002)(71200400001)(186003)(2906002)(476003)(14454004)(99286004)(316002)(2616005)(36756003)(26005)(5640700003)(486006)(6506007)(3846002)(2501003)(386003)(71190400001)(66446008)(53936002)(6512007)(6436002)(6116002)(6486002)(102836004)(305945005)(50226002)(478600001)(1076003)(66066001)(7736002)(64756008)(6916009)(14444005)(68736007)(81166006)(256004)(66476007)(66946007)(81156014)(30864003)(25786009)(73956011)(66556008)(2351001)(8936002)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR19MB2573; H:MN2PR19MB2592.namprd19.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: ddn.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: JtMADAYkCLmTW2Be7hzu5bqj8bm9TwciO/mBmLk24fLKUFgGfQYlOUTnagTLVAzaFtc/W982oZfrYYmpmM+wCM0ucjiCsuAMvdzws1JkGTMtG7J8zYsED1CmijBwKo8xlR51IN9uZcqDRznp2f1+174EM4oX34Iy3iherAFw9snpIGx/dAbwIMRAZ2tmT+ifLagO1MTV1I7Yf+A3uit4KjyhOhVIUhHoawD69Rn2BQhGxSKF5FyhQr0ucY1XIlEz7IVL9Adb85rv/2hniqmooolmZPbMqt+Po4H8YV86ix3Lr3UCsik5rr3QmaA5d0k7Izv2Fv/VKZuSzjdClr5bCwCt2PfwWy1fbcgy3bOiRUJrZBrcsNQv8dAgPIcsSz/WBTgRWU3oNc9BjM543ycHBSOSoSuXCVIy9Ev5PFlN+YU= MIME-Version: 1.0 X-OriginatorOrg: ddn.com X-MS-Exchange-CrossTenant-Network-Message-Id: b2f5e452-b0a9-4a65-c281-08d6f0bddba0 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jun 2019 11:45:54.9201 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 753b6e26-6fd3-43e6-8248-3f1735d59bb4 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: dli@ddn.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR19MB2573 Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Jim Garlick E2fsck fixes files that are found to be sharing blocks by cloning the shared blocks and giving each file a private copy in pass 1D. Allowing all files claiming the shared blocks to have copies can inadvertantly bypass access restrictions. Deleting all the files, zeroing the cloned blocks, or placing the files in the /lost+found directory after cloning may be preferable in some secure environments. The following patches implement config file and command line options in e2fsck that allow pass 1D behavior to be tuned according to site policy. It adds two extended options and config file counterparts. On the command line: -E clone=dup|zero Select the block cloning method. "dup" is old behavior, and is the default. "zero" is a new method that substitutes zero-filled blocks for the shared blocks in all the files that claim them. -E shared=preserve|lost+found|delete Select the disposition of files containing shared blocks. "preserve" is the old behavior which remains the default. "lost+found" causes files to be unlinked after cloning so they will be reconnected to /lost+found in pass 3. "delete" skips cloning entirely and simply deletes the files. In the config file: [options] clone=dup|zero shared=preserve|lost+found|delete Change-Id: I1d0fe6f476e50257e5c34e46f1a1cdcd426516ec Signed-off-by: Jim Garlick Signed-off-by: Andreas Dilger Signed-off-by: Li Dongyang --- e2fsck/e2fsck.8.in | 15 +++++++ e2fsck/e2fsck.conf.5.in | 13 +++++++ e2fsck/e2fsck.h | 13 +++++++ e2fsck/pass1b.c | 48 ++++++++++++++++++----- e2fsck/problem.c | 8 ++++ e2fsck/problem.h | 7 ++++ e2fsck/unix.c | 86 +++++++++++++++++++++++++++++++++++++++++ 7 files changed, 181 insertions(+), 9 deletions(-) diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in index 4e3890b2..7a8e961c 100644 --- a/e2fsck/e2fsck.8.in +++ b/e2fsck/e2fsck.8.in @@ -198,6 +198,21 @@ separated, and may take an argument using the equals ('=') sign. The following options are supported: .RS 1.2i .TP +.BI clone= dup|zero +Resolve files with shared blocks in pass 1D by giving each file a private +copy of the blocks (dup); +or replacing the shared blocks with private, zero-filled blocks (zero). +The default is dup. zero option is incompatible with -E unshare_blocks +option. +.TP +.BI shared= preserve|lost+found|delete +Files with shared blocks discovered in pass 1D are cloned and then left +in place (preserve); +cloned and then disconnected from their parent directory, +then reconnected to /lost+found in pass 3 (lost+found); +or simply deleted (delete). The default is preserve. +lost+found and delete are incompatible with -E unshare_blocks option. +.TP .BI ea_ver= extended_attribute_version Set the version of the extended attribute blocks which .B e2fsck diff --git a/e2fsck/e2fsck.conf.5.in b/e2fsck/e2fsck.conf.5.in index 48ad0fde..90ddba15 100644 --- a/e2fsck/e2fsck.conf.5.in +++ b/e2fsck/e2fsck.conf.5.in @@ -147,6 +147,19 @@ will offer to clear the test_fs flag if the ext4 filesystem is available on the system. It defaults to true. .TP +.I clone +This string relation controls the default handling of shared blocks in pass 1D. +It can be set to dup or zero. See the +.I "-E clone" +option description in e2fsck(8). +.TP +.I shared +This string relation controls the default disposition of files discovered to +have shared blocks in pass 1D. It can be set to preserve, lost+found, +or delete. See the +.I "-E shared" +option description in e2fsck(8). +.TP .I defer_check_on_battery This boolean relation controls whether or not the interval between filesystem checks (either based on time or number of mounts) should diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 2d359b38..2c64a145 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -211,6 +211,17 @@ struct resource_track { #define E2F_PASS_5 5 #define E2F_PASS_1B 6 +enum shared_opt { + E2F_SHARED_PRESERVE = 0, + E2F_SHARED_DELETE, + E2F_SHARED_LPF +}; + +enum clone_opt { + E2F_CLONE_DUP = 0, + E2F_CLONE_ZERO +}; + /* * Define the extended attribute refcount structure */ @@ -387,6 +398,8 @@ struct e2fsck_struct { time_t now; time_t time_fudge; /* For working around buggy init scripts */ int ext_attr_ver; + enum shared_opt shared; + enum clone_opt clone; profile_t profile; int blocks_per_page; ext2_u32_list encrypted_dirs; diff --git a/e2fsck/pass1b.c b/e2fsck/pass1b.c index 5693b9cf..f41bd913 100644 --- a/e2fsck/pass1b.c +++ b/e2fsck/pass1b.c @@ -541,6 +541,9 @@ static void pass1d(e2fsck_t ctx, char *block_buf) q = (struct dup_cluster *) dnode_get(m); if (q->num_bad > 1) file_ok = 0; + if (q->num_bad == 1 && (ctx->clone == E2F_CLONE_ZERO || + ctx->shared != E2F_SHARED_PRESERVE)) + file_ok = 0; if (check_if_fs_cluster(ctx, s->cluster)) { file_ok = 0; meta_data = 1; @@ -601,13 +604,26 @@ static void pass1d(e2fsck_t ctx, char *block_buf) continue; } if ((ctx->options & E2F_OPT_UNSHARE_BLOCKS) || - fix_problem(ctx, PR_1D_CLONE_QUESTION, &pctx)) { + (ctx->shared != E2F_SHARED_DELETE && + fix_problem(ctx, PR_1D_CLONE_QUESTION, &pctx))) { pctx.errcode = clone_file(ctx, ino, p, block_buf); - if (pctx.errcode) + if (pctx.errcode) { fix_problem(ctx, PR_1D_CLONE_ERROR, &pctx); - else - continue; + goto delete; + } + if (ctx->shared == E2F_SHARED_LPF && + fix_problem(ctx, PR_1D_DISCONNECT_QUESTION, &pctx)){ + pctx.errcode = ext2fs_unlink(fs, p->dir, + NULL, ino, 0); + if (pctx.errcode) { + fix_problem(ctx, PR_1D_DISCONNECT_ERROR, + &pctx); + goto delete; + } + } + continue; } +delete: /* * Note: When unsharing blocks, we don't prompt to delete * files. If the clone operation fails than the unshare @@ -631,7 +647,8 @@ static void decrement_badcount(e2fsck_t ctx, blk64_t block, { p->num_bad--; if (p->num_bad <= 0 || - (p->num_bad == 1 && !check_if_fs_block(ctx, block))) { + (p->num_bad == 1 && !check_if_fs_block(ctx, block) && + ctx->clone == E2F_CLONE_DUP)) { if (check_if_fs_cluster(ctx, EXT2FS_B2C(ctx->fs, block))) return; ext2fs_unmark_block_bitmap2(ctx->block_dup_map, block); @@ -830,6 +847,14 @@ static int clone_file_block(ext2_filsys fs, p = (struct dup_cluster *) dnode_get(n); + if (!is_meta) { + if (ctx->clone == E2F_CLONE_ZERO && p->num_bad == 0) { + ext2fs_unmark_block_bitmap2(ctx->block_found_map, + *block_nr); + ext2fs_block_alloc_stats(fs, *block_nr, -1); + } + } + cs->dup_cluster = c; /* * Let's try an implied cluster allocation. If we get the same @@ -876,10 +901,15 @@ cluster_alloc_ok: printf("Cloning block #%lld from %llu to %llu\n", blockcnt, *block_nr, new_block); #endif - retval = io_channel_read_blk64(fs->io, *block_nr, 1, cs->buf); - if (retval) { - cs->errcode = retval; - return BLOCK_ABORT; + if (ctx->clone == E2F_CLONE_ZERO) { + memset(cs->buf, 0, fs->blocksize); + } else { + retval = io_channel_read_blk64(fs->io, *block_nr, 1, + cs->buf); + if (retval) { + cs->errcode = retval; + return BLOCK_ABORT; + } } if (should_write) { retval = io_channel_write_blk64(fs->io, new_block, 1, cs->buf); diff --git a/e2fsck/problem.c b/e2fsck/problem.c index 75b49ba3..1679cfae 100644 --- a/e2fsck/problem.c +++ b/e2fsck/problem.c @@ -1281,6 +1281,14 @@ static struct e2fsck_problem problem_table[] = { { PR_1D_CLONE_ERROR, N_("Couldn't clone file: %m\n"), PROMPT_NONE, 0, 0, 0, 0 }, + /* File with shared blocks found */ + { PR_1D_DISCONNECT_QUESTION, + N_("File with shared blocks found\n"), PROMPT_CONNECT, 0 }, + + /* Couldn't unlink file (error) */ + { PR_1D_DISCONNECT_ERROR, + N_("Couldn't unlink file: %m\n"), PROMPT_NONE, 0 }, + /* Pass 1E Extent tree optimization */ /* Pass 1E: Optimizing extent trees */ diff --git a/e2fsck/problem.h b/e2fsck/problem.h index 2c79169e..efdb8d54 100644 --- a/e2fsck/problem.h +++ b/e2fsck/problem.h @@ -756,6 +756,13 @@ struct problem_context { /* Couldn't clone file (error) */ #define PR_1D_CLONE_ERROR 0x013008 +/* File with shared blocks found */ +#define PR_1D_DISCONNECT_QUESTION 0x013009 + +/* Couldn't unlink file (error) */ +#define PR_1D_DISCONNECT_ERROR 0x01300A + + /* * Pass 1e --- rebuilding extent trees */ diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 68f49873..425ea44b 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -657,6 +657,49 @@ static void signal_cancel(int sig EXT2FS_ATTR((unused))) } #endif +static void initialize_profile_options(e2fsck_t ctx) +{ + char *tmp; + + /* [options] shared=preserve|lost+found|delete */ + tmp = NULL; + ctx->shared = E2F_SHARED_PRESERVE; + profile_get_string(ctx->profile, "options", "shared", 0, + "preserve", &tmp); + if (tmp) { + if (strcmp(tmp, "preserve") == 0) + ctx->shared = E2F_SHARED_PRESERVE; + else if (strcmp(tmp, "delete") == 0) + ctx->shared = E2F_SHARED_DELETE; + else if (strcmp(tmp, "lost+found") == 0) + ctx->shared = E2F_SHARED_LPF; + else { + com_err(ctx->program_name, 0, + _("configuration error: 'shared=%s'"), tmp); + fatal_error(ctx, 0); + } + free(tmp); + } + + /* [options] clone=dup|zero */ + tmp = NULL; + ctx->clone = E2F_CLONE_DUP; + profile_get_string(ctx->profile, "options", "clone", 0, + "dup", &tmp); + if (tmp) { + if (strcmp(tmp, "dup") == 0) + ctx->clone = E2F_CLONE_DUP; + else if (strcmp(tmp, "zero") == 0) + ctx->clone = E2F_CLONE_ZERO; + else { + com_err(ctx->program_name, 0, + _("configuration error: 'clone=%s'"), tmp); + fatal_error(ctx, 0); + } + free(tmp); + } +} + static void parse_extended_opts(e2fsck_t ctx, const char *opts) { char *buf, *token, *next, *p, *arg; @@ -707,6 +750,36 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts) } else if (strcmp(token, "fragcheck") == 0) { ctx->options |= E2F_OPT_FRAGCHECK; continue; + /* -E shared=preserve|lost+found|delete */ + } else if (strcmp(token, "shared") == 0) { + if (!arg) { + extended_usage++; + continue; + } + if (strcmp(arg, "preserve") == 0) { + ctx->shared = E2F_SHARED_PRESERVE; + } else if (strcmp(arg, "lost+found") == 0) { + ctx->shared = E2F_SHARED_LPF; + } else if (strcmp(arg, "delete") == 0) { + ctx->shared = E2F_SHARED_DELETE; + } else { + extended_usage++; + continue; + } + /* -E clone=dup|zero */ + } else if (strcmp(token, "clone") == 0) { + if (!arg) { + extended_usage++; + continue; + } + if (strcmp(arg, "dup") == 0) { + ctx->clone = E2F_CLONE_DUP; + } else if (strcmp(arg, "zero") == 0) { + ctx->clone = E2F_CLONE_ZERO; + } else { + extended_usage++; + continue; + } } else if (strcmp(token, "journal_only") == 0) { if (arg) { extended_usage++; @@ -771,6 +844,8 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts) fputs("\tjournal_only\n", stderr); fputs("\tdiscard\n", stderr); fputs("\tnodiscard\n", stderr); + fputs(_("\tshared=\n"), stderr); + fputs(_("\tclone=\n"), stderr); fputs("\toptimize_extents\n", stderr); fputs("\tno_optimize_extents\n", stderr); fputs("\tinode_count_fullmap\n", stderr); @@ -840,6 +915,8 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) else ctx->program_name = "e2fsck"; + initialize_profile_options(ctx); + phys_mem_kb = get_memory_size() / 1024; ctx->readahead_kb = ~0ULL; while ((c = getopt(argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) @@ -1032,6 +1109,15 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) fatal_error(ctx, 0); } + if ((ctx->options & E2F_OPT_UNSHARE_BLOCKS) && + (ctx->clone != E2F_CLONE_DUP || + ctx->shared != E2F_SHARED_PRESERVE)) { + com_err(ctx->program_name, 0, "%s", + _("The -E unshare_blocks and specified clone/shared " + "options are incompatible")); + fatal_error(ctx, 0); + } + if ((cp = getenv("E2FSCK_CONFIG")) != NULL) config_fn[0] = cp; profile_set_syntax_err_cb(syntax_err_report);