License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2006-10-11 08:20:50 +00:00
|
|
|
/*
|
2006-10-11 08:20:53 +00:00
|
|
|
* linux/fs/ext4/balloc.c
|
2006-10-11 08:20:50 +00:00
|
|
|
*
|
|
|
|
* Copyright (C) 1992, 1993, 1994, 1995
|
|
|
|
* Remy Card (card@masi.ibp.fr)
|
|
|
|
* Laboratoire MASI - Institut Blaise Pascal
|
|
|
|
* Universite Pierre et Marie Curie (Paris VI)
|
|
|
|
*
|
|
|
|
* Enhanced block allocation by Stephen Tweedie (sct@redhat.com), 1993
|
|
|
|
* Big-endian to little-endian byte-swapping/bitmaps by
|
|
|
|
* David S. Miller (davem@caip.rutgers.edu), 1995
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/time.h>
|
|
|
|
#include <linux/capability.h>
|
|
|
|
#include <linux/fs.h>
|
|
|
|
#include <linux/quotaops.h>
|
|
|
|
#include <linux/buffer_head.h>
|
2008-04-29 22:13:32 +00:00
|
|
|
#include "ext4.h"
|
|
|
|
#include "ext4_jbd2.h"
|
2009-01-06 02:36:02 +00:00
|
|
|
#include "mballoc.h"
|
2008-04-29 22:13:32 +00:00
|
|
|
|
2011-03-22 01:38:05 +00:00
|
|
|
#include <trace/events/ext4.h>
|
|
|
|
|
2012-01-05 03:33:28 +00:00
|
|
|
static unsigned ext4_num_base_meta_clusters(struct super_block *sb,
|
|
|
|
ext4_group_t block_group);
|
2006-10-11 08:20:50 +00:00
|
|
|
/*
|
|
|
|
* balloc.c contains the blocks allocation and deallocation routines
|
|
|
|
*/
|
|
|
|
|
2013-04-04 03:32:34 +00:00
|
|
|
/*
|
|
|
|
* Calculate block group number for a given block number
|
|
|
|
*/
|
|
|
|
ext4_group_t ext4_get_group_number(struct super_block *sb,
|
|
|
|
ext4_fsblk_t block)
|
|
|
|
{
|
|
|
|
ext4_group_t group;
|
|
|
|
|
|
|
|
if (test_opt2(sb, STD_GROUP_SIZE))
|
2013-07-06 03:11:16 +00:00
|
|
|
group = (block -
|
|
|
|
le32_to_cpu(EXT4_SB(sb)->s_es->s_first_data_block)) >>
|
2013-04-04 03:32:34 +00:00
|
|
|
(EXT4_BLOCK_SIZE_BITS(sb) + EXT4_CLUSTER_BITS(sb) + 3);
|
|
|
|
else
|
|
|
|
ext4_get_group_no_and_offset(sb, block, &group, NULL);
|
|
|
|
return group;
|
|
|
|
}
|
|
|
|
|
2006-10-11 08:21:18 +00:00
|
|
|
/*
|
2011-09-09 22:46:51 +00:00
|
|
|
* Calculate the block group number and offset into the block/cluster
|
|
|
|
* allocation bitmap, given a block number
|
2006-10-11 08:21:18 +00:00
|
|
|
*/
|
|
|
|
void ext4_get_group_no_and_offset(struct super_block *sb, ext4_fsblk_t blocknr,
|
2008-01-29 04:58:27 +00:00
|
|
|
ext4_group_t *blockgrpp, ext4_grpblk_t *offsetp)
|
2006-10-11 08:21:18 +00:00
|
|
|
{
|
2007-05-24 17:04:54 +00:00
|
|
|
struct ext4_super_block *es = EXT4_SB(sb)->s_es;
|
2006-10-11 08:21:18 +00:00
|
|
|
ext4_grpblk_t offset;
|
|
|
|
|
2007-05-24 17:04:54 +00:00
|
|
|
blocknr = blocknr - le32_to_cpu(es->s_first_data_block);
|
2011-09-09 22:46:51 +00:00
|
|
|
offset = do_div(blocknr, EXT4_BLOCKS_PER_GROUP(sb)) >>
|
|
|
|
EXT4_SB(sb)->s_cluster_bits;
|
2006-10-11 08:21:18 +00:00
|
|
|
if (offsetp)
|
|
|
|
*offsetp = offset;
|
|
|
|
if (blockgrpp)
|
2007-05-24 17:04:54 +00:00
|
|
|
*blockgrpp = blocknr;
|
2006-10-11 08:21:18 +00:00
|
|
|
|
|
|
|
}
|
|
|
|
|
2013-04-04 02:12:52 +00:00
|
|
|
/*
|
|
|
|
* Check whether the 'block' lives within the 'block_group'. Returns 1 if so
|
|
|
|
* and 0 otherwise.
|
|
|
|
*/
|
|
|
|
static inline int ext4_block_in_group(struct super_block *sb,
|
|
|
|
ext4_fsblk_t block,
|
|
|
|
ext4_group_t block_group)
|
2008-06-03 18:07:29 +00:00
|
|
|
{
|
|
|
|
ext4_group_t actual_group;
|
2013-04-04 02:12:52 +00:00
|
|
|
|
2013-04-04 03:32:34 +00:00
|
|
|
actual_group = ext4_get_group_number(sb, block);
|
2013-04-04 02:12:52 +00:00
|
|
|
return (actual_group == block_group) ? 1 : 0;
|
2008-06-03 18:07:29 +00:00
|
|
|
}
|
|
|
|
|
2011-09-09 22:44:51 +00:00
|
|
|
/* Return the number of clusters used for file system metadata; this
|
|
|
|
* represents the overhead needed by the file system.
|
|
|
|
*/
|
2014-05-12 14:50:23 +00:00
|
|
|
static unsigned ext4_num_overhead_clusters(struct super_block *sb,
|
|
|
|
ext4_group_t block_group,
|
|
|
|
struct ext4_group_desc *gdp)
|
2008-06-03 18:07:29 +00:00
|
|
|
{
|
2011-09-09 22:44:51 +00:00
|
|
|
unsigned num_clusters;
|
|
|
|
int block_cluster = -1, inode_cluster = -1, itbl_cluster = -1, i, c;
|
|
|
|
ext4_fsblk_t start = ext4_group_first_block_no(sb, block_group);
|
|
|
|
ext4_fsblk_t itbl_blk;
|
2008-06-03 18:07:29 +00:00
|
|
|
struct ext4_sb_info *sbi = EXT4_SB(sb);
|
|
|
|
|
2011-09-09 22:44:51 +00:00
|
|
|
/* This is the number of clusters used by the superblock,
|
|
|
|
* block group descriptors, and reserved block group
|
|
|
|
* descriptor blocks */
|
|
|
|
num_clusters = ext4_num_base_meta_clusters(sb, block_group);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* For the allocation bitmaps and inode table, we first need
|
|
|
|
* to check to see if the block is in the block group. If it
|
|
|
|
* is, then check to see if the cluster is already accounted
|
|
|
|
* for in the clusters used for the base metadata cluster, or
|
|
|
|
* if we can increment the base metadata cluster to include
|
|
|
|
* that block. Otherwise, we will have to track the cluster
|
|
|
|
* used for the allocation bitmap or inode table explicitly.
|
|
|
|
* Normally all of these blocks are contiguous, so the special
|
|
|
|
* case handling shouldn't be necessary except for *very*
|
|
|
|
* unusual file system layouts.
|
|
|
|
*/
|
|
|
|
if (ext4_block_in_group(sb, ext4_block_bitmap(sb, gdp), block_group)) {
|
2012-06-07 22:56:06 +00:00
|
|
|
block_cluster = EXT4_B2C(sbi,
|
|
|
|
ext4_block_bitmap(sb, gdp) - start);
|
2011-09-09 22:44:51 +00:00
|
|
|
if (block_cluster < num_clusters)
|
|
|
|
block_cluster = -1;
|
|
|
|
else if (block_cluster == num_clusters) {
|
|
|
|
num_clusters++;
|
|
|
|
block_cluster = -1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (ext4_block_in_group(sb, ext4_inode_bitmap(sb, gdp), block_group)) {
|
|
|
|
inode_cluster = EXT4_B2C(sbi,
|
2012-06-07 22:56:06 +00:00
|
|
|
ext4_inode_bitmap(sb, gdp) - start);
|
2011-09-09 22:44:51 +00:00
|
|
|
if (inode_cluster < num_clusters)
|
|
|
|
inode_cluster = -1;
|
|
|
|
else if (inode_cluster == num_clusters) {
|
|
|
|
num_clusters++;
|
|
|
|
inode_cluster = -1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
itbl_blk = ext4_inode_table(sb, gdp);
|
|
|
|
for (i = 0; i < sbi->s_itb_per_group; i++) {
|
|
|
|
if (ext4_block_in_group(sb, itbl_blk + i, block_group)) {
|
2012-06-07 22:56:06 +00:00
|
|
|
c = EXT4_B2C(sbi, itbl_blk + i - start);
|
2011-09-09 22:44:51 +00:00
|
|
|
if ((c < num_clusters) || (c == inode_cluster) ||
|
|
|
|
(c == block_cluster) || (c == itbl_cluster))
|
|
|
|
continue;
|
|
|
|
if (c == num_clusters) {
|
|
|
|
num_clusters++;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
num_clusters++;
|
|
|
|
itbl_cluster = c;
|
2008-06-03 18:07:29 +00:00
|
|
|
}
|
|
|
|
}
|
2011-09-09 22:44:51 +00:00
|
|
|
|
|
|
|
if (block_cluster != -1)
|
|
|
|
num_clusters++;
|
|
|
|
if (inode_cluster != -1)
|
|
|
|
num_clusters++;
|
|
|
|
|
|
|
|
return num_clusters;
|
2008-06-03 18:07:29 +00:00
|
|
|
}
|
2008-10-10 13:40:52 +00:00
|
|
|
|
2011-09-09 22:44:51 +00:00
|
|
|
static unsigned int num_clusters_in_group(struct super_block *sb,
|
|
|
|
ext4_group_t block_group)
|
2011-09-09 22:40:51 +00:00
|
|
|
{
|
2011-09-09 22:44:51 +00:00
|
|
|
unsigned int blocks;
|
|
|
|
|
2011-09-09 22:40:51 +00:00
|
|
|
if (block_group == ext4_get_groups_count(sb) - 1) {
|
|
|
|
/*
|
|
|
|
* Even though mke2fs always initializes the first and
|
|
|
|
* last group, just in case some other tool was used,
|
|
|
|
* we need to make sure we calculate the right free
|
|
|
|
* blocks.
|
|
|
|
*/
|
2011-09-09 22:44:51 +00:00
|
|
|
blocks = ext4_blocks_count(EXT4_SB(sb)->s_es) -
|
2011-09-09 22:40:51 +00:00
|
|
|
ext4_group_first_block_no(sb, block_group);
|
|
|
|
} else
|
2011-09-09 22:44:51 +00:00
|
|
|
blocks = EXT4_BLOCKS_PER_GROUP(sb);
|
|
|
|
return EXT4_NUM_B2C(EXT4_SB(sb), blocks);
|
2011-09-09 22:40:51 +00:00
|
|
|
}
|
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
/* Initializes an uninitialized block bitmap */
|
2014-10-13 07:42:12 +00:00
|
|
|
static int ext4_init_block_bitmap(struct super_block *sb,
|
2014-05-12 14:50:23 +00:00
|
|
|
struct buffer_head *bh,
|
|
|
|
ext4_group_t block_group,
|
|
|
|
struct ext4_group_desc *gdp)
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
{
|
2011-09-09 22:44:51 +00:00
|
|
|
unsigned int bit, bit_max;
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
struct ext4_sb_info *sbi = EXT4_SB(sb);
|
2011-09-09 22:42:51 +00:00
|
|
|
ext4_fsblk_t start, tmp;
|
|
|
|
int flex_bg = 0;
|
2013-08-28 22:46:56 +00:00
|
|
|
struct ext4_group_info *grp;
|
2011-09-09 22:42:51 +00:00
|
|
|
|
|
|
|
J_ASSERT_BH(bh, buffer_locked(bh));
|
|
|
|
|
|
|
|
/* If checksum is bad mark all blocks used to prevent allocation
|
|
|
|
* essentially implementing a per-group read-only flag. */
|
2012-04-29 22:45:10 +00:00
|
|
|
if (!ext4_group_desc_csum_verify(sb, block_group, gdp)) {
|
2013-08-28 22:46:56 +00:00
|
|
|
grp = ext4_get_group_info(sb, block_group);
|
2014-06-26 14:11:53 +00:00
|
|
|
if (!EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
|
|
|
|
percpu_counter_sub(&sbi->s_freeclusters_counter,
|
|
|
|
grp->bb_free);
|
2013-08-28 22:46:56 +00:00
|
|
|
set_bit(EXT4_GROUP_INFO_BBITMAP_CORRUPT_BIT, &grp->bb_state);
|
2014-06-26 14:11:53 +00:00
|
|
|
if (!EXT4_MB_GRP_IBITMAP_CORRUPT(grp)) {
|
|
|
|
int count;
|
|
|
|
count = ext4_free_inodes_count(sb, gdp);
|
|
|
|
percpu_counter_sub(&sbi->s_freeinodes_counter,
|
|
|
|
count);
|
|
|
|
}
|
2013-08-28 22:46:56 +00:00
|
|
|
set_bit(EXT4_GROUP_INFO_IBITMAP_CORRUPT_BIT, &grp->bb_state);
|
2015-10-17 20:16:04 +00:00
|
|
|
return -EFSBADCRC;
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
}
|
2011-09-09 22:42:51 +00:00
|
|
|
memset(bh->b_data, 0, sb->s_blocksize);
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
|
2011-09-09 22:44:51 +00:00
|
|
|
bit_max = ext4_num_base_meta_clusters(sb, block_group);
|
2016-07-06 00:01:52 +00:00
|
|
|
if ((bit_max >> 3) >= bh->b_size)
|
|
|
|
return -EFSCORRUPTED;
|
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
for (bit = 0; bit < bit_max; bit++)
|
|
|
|
ext4_set_bit(bit, bh->b_data);
|
2008-04-17 14:38:59 +00:00
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
start = ext4_group_first_block_no(sb, block_group);
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
|
2015-10-17 20:18:43 +00:00
|
|
|
if (ext4_has_feature_flex_bg(sb))
|
2011-09-09 22:42:51 +00:00
|
|
|
flex_bg = 1;
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
/* Set bits for block and inode bitmaps, and inode table */
|
|
|
|
tmp = ext4_block_bitmap(sb, gdp);
|
|
|
|
if (!flex_bg || ext4_block_in_group(sb, tmp, block_group))
|
2011-09-09 22:44:51 +00:00
|
|
|
ext4_set_bit(EXT4_B2C(sbi, tmp - start), bh->b_data);
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
tmp = ext4_inode_bitmap(sb, gdp);
|
|
|
|
if (!flex_bg || ext4_block_in_group(sb, tmp, block_group))
|
2011-09-09 22:44:51 +00:00
|
|
|
ext4_set_bit(EXT4_B2C(sbi, tmp - start), bh->b_data);
|
2008-06-03 18:07:29 +00:00
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
tmp = ext4_inode_table(sb, gdp);
|
|
|
|
for (; tmp < ext4_inode_table(sb, gdp) +
|
|
|
|
sbi->s_itb_per_group; tmp++) {
|
2008-06-03 18:07:29 +00:00
|
|
|
if (!flex_bg || ext4_block_in_group(sb, tmp, block_group))
|
2011-09-09 22:44:51 +00:00
|
|
|
ext4_set_bit(EXT4_B2C(sbi, tmp - start), bh->b_data);
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
}
|
2011-09-09 22:44:51 +00:00
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
/*
|
|
|
|
* Also if the number of blocks within the group is less than
|
|
|
|
* the blocksize * 8 ( which is the size of bitmap ), set rest
|
|
|
|
* of the block bitmap to 1
|
|
|
|
*/
|
2011-09-09 22:44:51 +00:00
|
|
|
ext4_mark_bitmap_end(num_clusters_in_group(sb, block_group),
|
2011-09-09 22:42:51 +00:00
|
|
|
sb->s_blocksize * 8, bh->b_data);
|
2014-10-13 07:42:12 +00:00
|
|
|
return 0;
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
}
|
|
|
|
|
2011-09-09 22:42:51 +00:00
|
|
|
/* Return the number of free blocks in a block group. It is used when
|
|
|
|
* the block bitmap is uninitialized, so we can't just count the bits
|
|
|
|
* in the bitmap. */
|
2011-09-09 23:12:51 +00:00
|
|
|
unsigned ext4_free_clusters_after_init(struct super_block *sb,
|
|
|
|
ext4_group_t block_group,
|
|
|
|
struct ext4_group_desc *gdp)
|
2011-09-09 22:42:51 +00:00
|
|
|
{
|
2011-09-09 22:44:51 +00:00
|
|
|
return num_clusters_in_group(sb, block_group) -
|
|
|
|
ext4_num_overhead_clusters(sb, block_group, gdp);
|
2011-09-09 22:42:51 +00:00
|
|
|
}
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
|
2006-10-11 08:20:50 +00:00
|
|
|
/*
|
|
|
|
* The free blocks are managed by bitmaps. A file system contains several
|
|
|
|
* blocks groups. Each group contains 1 bitmap block for blocks, 1 bitmap
|
|
|
|
* block for inodes, N blocks for the inode table and data blocks.
|
|
|
|
*
|
|
|
|
* The file system contains group descriptors which are located after the
|
|
|
|
* super block. Each descriptor contains the number of the bitmap block and
|
|
|
|
* the free blocks count in the block. The descriptors are loaded in memory
|
2007-02-20 21:57:58 +00:00
|
|
|
* when a file system is mounted (see ext4_fill_super).
|
2006-10-11 08:20:50 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
2006-10-11 08:20:53 +00:00
|
|
|
* ext4_get_group_desc() -- load group descriptor from disk
|
2006-10-11 08:20:50 +00:00
|
|
|
* @sb: super block
|
|
|
|
* @block_group: given block group
|
|
|
|
* @bh: pointer to the buffer head to store the block
|
|
|
|
* group descriptor
|
|
|
|
*/
|
2008-09-09 02:25:24 +00:00
|
|
|
struct ext4_group_desc * ext4_get_group_desc(struct super_block *sb,
|
2008-01-29 04:58:27 +00:00
|
|
|
ext4_group_t block_group,
|
2008-09-09 02:25:24 +00:00
|
|
|
struct buffer_head **bh)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2008-11-05 05:14:04 +00:00
|
|
|
unsigned int group_desc;
|
|
|
|
unsigned int offset;
|
2009-05-01 12:50:38 +00:00
|
|
|
ext4_group_t ngroups = ext4_get_groups_count(sb);
|
2008-09-09 02:25:24 +00:00
|
|
|
struct ext4_group_desc *desc;
|
2006-10-11 08:20:53 +00:00
|
|
|
struct ext4_sb_info *sbi = EXT4_SB(sb);
|
2006-10-11 08:20:50 +00:00
|
|
|
|
2009-05-01 12:50:38 +00:00
|
|
|
if (block_group >= ngroups) {
|
2010-02-15 19:19:27 +00:00
|
|
|
ext4_error(sb, "block_group >= groups_count - block_group = %u,"
|
|
|
|
" groups_count = %u", block_group, ngroups);
|
2006-10-11 08:20:50 +00:00
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2006-10-11 08:20:53 +00:00
|
|
|
group_desc = block_group >> EXT4_DESC_PER_BLOCK_BITS(sb);
|
|
|
|
offset = block_group & (EXT4_DESC_PER_BLOCK(sb) - 1);
|
2006-10-11 08:20:50 +00:00
|
|
|
if (!sbi->s_group_desc[group_desc]) {
|
2010-02-15 19:19:27 +00:00
|
|
|
ext4_error(sb, "Group descriptor not loaded - "
|
2008-11-05 05:14:04 +00:00
|
|
|
"block_group = %u, group_desc = %u, desc = %u",
|
2008-09-09 02:25:24 +00:00
|
|
|
block_group, group_desc, offset);
|
2006-10-11 08:20:50 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2006-10-11 08:21:14 +00:00
|
|
|
desc = (struct ext4_group_desc *)(
|
|
|
|
(__u8 *)sbi->s_group_desc[group_desc]->b_data +
|
|
|
|
offset * EXT4_DESC_SIZE(sb));
|
2006-10-11 08:20:50 +00:00
|
|
|
if (bh)
|
|
|
|
*bh = sbi->s_group_desc[group_desc];
|
2006-10-11 08:21:14 +00:00
|
|
|
return desc;
|
2006-10-11 08:20:50 +00:00
|
|
|
}
|
|
|
|
|
2012-08-10 17:57:52 +00:00
|
|
|
/*
|
|
|
|
* Return the block number which was discovered to be invalid, or 0 if
|
|
|
|
* the block bitmap is valid.
|
|
|
|
*/
|
|
|
|
static ext4_fsblk_t ext4_valid_block_bitmap(struct super_block *sb,
|
|
|
|
struct ext4_group_desc *desc,
|
2013-08-28 19:59:51 +00:00
|
|
|
ext4_group_t block_group,
|
2012-08-10 17:57:52 +00:00
|
|
|
struct buffer_head *bh)
|
2008-01-29 04:58:27 +00:00
|
|
|
{
|
2014-05-12 14:17:55 +00:00
|
|
|
struct ext4_sb_info *sbi = EXT4_SB(sb);
|
2008-01-29 04:58:27 +00:00
|
|
|
ext4_grpblk_t offset;
|
|
|
|
ext4_grpblk_t next_zero_bit;
|
2012-08-10 17:57:52 +00:00
|
|
|
ext4_fsblk_t blk;
|
2008-01-29 04:58:27 +00:00
|
|
|
ext4_fsblk_t group_first_block;
|
|
|
|
|
2015-10-17 20:18:43 +00:00
|
|
|
if (ext4_has_feature_flex_bg(sb)) {
|
2008-01-29 04:58:27 +00:00
|
|
|
/* with FLEX_BG, the inode/block bitmaps and itable
|
|
|
|
* blocks may not be in the group at all
|
|
|
|
* so the bitmap validation will be skipped for those groups
|
|
|
|
* or it has to also read the block group where the bitmaps
|
|
|
|
* are located to verify they are set.
|
|
|
|
*/
|
2012-08-10 17:57:52 +00:00
|
|
|
return 0;
|
2008-01-29 04:58:27 +00:00
|
|
|
}
|
|
|
|
group_first_block = ext4_group_first_block_no(sb, block_group);
|
|
|
|
|
|
|
|
/* check whether block bitmap block number is set */
|
2012-08-10 17:57:52 +00:00
|
|
|
blk = ext4_block_bitmap(sb, desc);
|
|
|
|
offset = blk - group_first_block;
|
2014-05-12 14:17:55 +00:00
|
|
|
if (!ext4_test_bit(EXT4_B2C(sbi, offset), bh->b_data))
|
2008-01-29 04:58:27 +00:00
|
|
|
/* bad block bitmap */
|
2012-08-10 17:57:52 +00:00
|
|
|
return blk;
|
2008-01-29 04:58:27 +00:00
|
|
|
|
|
|
|
/* check whether the inode bitmap block number is set */
|
2012-08-10 17:57:52 +00:00
|
|
|
blk = ext4_inode_bitmap(sb, desc);
|
|
|
|
offset = blk - group_first_block;
|
2014-05-12 14:17:55 +00:00
|
|
|
if (!ext4_test_bit(EXT4_B2C(sbi, offset), bh->b_data))
|
2008-01-29 04:58:27 +00:00
|
|
|
/* bad block bitmap */
|
2012-08-10 17:57:52 +00:00
|
|
|
return blk;
|
2008-01-29 04:58:27 +00:00
|
|
|
|
|
|
|
/* check whether the inode table block number is set */
|
2012-08-10 17:57:52 +00:00
|
|
|
blk = ext4_inode_table(sb, desc);
|
|
|
|
offset = blk - group_first_block;
|
2008-01-29 04:58:27 +00:00
|
|
|
next_zero_bit = ext4_find_next_zero_bit(bh->b_data,
|
2018-01-11 18:17:49 +00:00
|
|
|
EXT4_B2C(sbi, offset + sbi->s_itb_per_group),
|
2014-05-12 14:17:55 +00:00
|
|
|
EXT4_B2C(sbi, offset));
|
|
|
|
if (next_zero_bit <
|
2018-01-11 18:17:49 +00:00
|
|
|
EXT4_B2C(sbi, offset + sbi->s_itb_per_group))
|
2012-08-10 17:57:52 +00:00
|
|
|
/* bad bitmap for inode tables */
|
|
|
|
return blk;
|
2008-01-29 04:58:27 +00:00
|
|
|
return 0;
|
|
|
|
}
|
2012-04-29 22:35:10 +00:00
|
|
|
|
2015-10-18 01:33:24 +00:00
|
|
|
static int ext4_validate_block_bitmap(struct super_block *sb,
|
|
|
|
struct ext4_group_desc *desc,
|
|
|
|
ext4_group_t block_group,
|
|
|
|
struct buffer_head *bh)
|
2012-04-29 22:35:10 +00:00
|
|
|
{
|
2012-08-10 17:57:52 +00:00
|
|
|
ext4_fsblk_t blk;
|
ext4: mark block group as corrupt on block bitmap error
When we notice a block-bitmap corruption (because of device failure or
something else), we should mark this group as corrupt and prevent
further block allocations/deallocations from it. Currently, we end up
generating one error message for every block in the bitmap. This
potentially could make the system unstable as noticed in some
bugs. With this patch, the error will be printed only the first time
and mark the entire block group as corrupted. This prevents future
access allocations/deallocations from it.
Also tested by corrupting the block
bitmap and forcefully introducing the mb_free_blocks error:
(1) create a largefile (2Gb)
$ dd if=/dev/zero of=largefile oflag=direct bs=10485760 count=200
(2) umount filesystem. use dumpe2fs to see which block-bitmaps
are in use by largefile and note their block numbers
(3) use dd to zero-out the used block bitmaps
$ dd if=/dev/zero of=/dev/hdc4 bs=4096 seek=14 count=8 oflag=direct
(4) mount the FS and delete the largefile.
(5) recreate the largefile. verify that the new largefile does not
get any blocks from the groups marked as bad.
Without the patch, we will see mb_free_blocks error for each bit in
each zero'ed out bitmap at (4). With the patch, we only see the error
once per blockgroup:
[ 309.706803] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 15: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.720824] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 14: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.732858] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.748321] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 13: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.760331] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.769695] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 12: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.781721] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.798166] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 11: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.810184] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.819532] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 10: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
Google-Bug-Id: 7258357
[darrick.wong@oracle.com]
Further modifications (by Darrick) to make more obvious that this corruption
bit applies to blocks only. Set the corruption flag if the block group bitmap
verification fails.
Original-author: Aditya Kali <adityakali@google.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
2013-08-28 21:35:51 +00:00
|
|
|
struct ext4_group_info *grp = ext4_get_group_info(sb, block_group);
|
2014-06-26 14:11:53 +00:00
|
|
|
struct ext4_sb_info *sbi = EXT4_SB(sb);
|
2012-08-10 17:57:52 +00:00
|
|
|
|
2015-10-18 01:33:24 +00:00
|
|
|
if (buffer_verified(bh))
|
|
|
|
return 0;
|
|
|
|
if (EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
|
|
|
|
return -EFSCORRUPTED;
|
2012-04-29 22:35:10 +00:00
|
|
|
|
|
|
|
ext4_lock_group(sb, block_group);
|
2015-10-18 01:33:24 +00:00
|
|
|
if (unlikely(!ext4_block_bitmap_csum_verify(sb, block_group,
|
|
|
|
desc, bh))) {
|
2012-08-10 17:57:52 +00:00
|
|
|
ext4_unlock_group(sb, block_group);
|
2015-10-18 01:33:24 +00:00
|
|
|
ext4_error(sb, "bg %u: bad block bitmap checksum", block_group);
|
2014-06-26 14:11:53 +00:00
|
|
|
if (!EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
|
|
|
|
percpu_counter_sub(&sbi->s_freeclusters_counter,
|
|
|
|
grp->bb_free);
|
ext4: mark block group as corrupt on block bitmap error
When we notice a block-bitmap corruption (because of device failure or
something else), we should mark this group as corrupt and prevent
further block allocations/deallocations from it. Currently, we end up
generating one error message for every block in the bitmap. This
potentially could make the system unstable as noticed in some
bugs. With this patch, the error will be printed only the first time
and mark the entire block group as corrupted. This prevents future
access allocations/deallocations from it.
Also tested by corrupting the block
bitmap and forcefully introducing the mb_free_blocks error:
(1) create a largefile (2Gb)
$ dd if=/dev/zero of=largefile oflag=direct bs=10485760 count=200
(2) umount filesystem. use dumpe2fs to see which block-bitmaps
are in use by largefile and note their block numbers
(3) use dd to zero-out the used block bitmaps
$ dd if=/dev/zero of=/dev/hdc4 bs=4096 seek=14 count=8 oflag=direct
(4) mount the FS and delete the largefile.
(5) recreate the largefile. verify that the new largefile does not
get any blocks from the groups marked as bad.
Without the patch, we will see mb_free_blocks error for each bit in
each zero'ed out bitmap at (4). With the patch, we only see the error
once per blockgroup:
[ 309.706803] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 15: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.720824] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 14: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.732858] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.748321] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 13: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.760331] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.769695] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 12: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.781721] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.798166] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 11: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.810184] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.819532] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 10: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
Google-Bug-Id: 7258357
[darrick.wong@oracle.com]
Further modifications (by Darrick) to make more obvious that this corruption
bit applies to blocks only. Set the corruption flag if the block group bitmap
verification fails.
Original-author: Aditya Kali <adityakali@google.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
2013-08-28 21:35:51 +00:00
|
|
|
set_bit(EXT4_GROUP_INFO_BBITMAP_CORRUPT_BIT, &grp->bb_state);
|
2015-10-18 01:33:24 +00:00
|
|
|
return -EFSBADCRC;
|
2012-08-10 17:57:52 +00:00
|
|
|
}
|
2015-10-18 01:33:24 +00:00
|
|
|
blk = ext4_valid_block_bitmap(sb, desc, block_group, bh);
|
|
|
|
if (unlikely(blk != 0)) {
|
2012-08-10 17:57:52 +00:00
|
|
|
ext4_unlock_group(sb, block_group);
|
2015-10-18 01:33:24 +00:00
|
|
|
ext4_error(sb, "bg %u: block %llu: invalid block bitmap",
|
|
|
|
block_group, blk);
|
2014-06-26 14:11:53 +00:00
|
|
|
if (!EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
|
|
|
|
percpu_counter_sub(&sbi->s_freeclusters_counter,
|
|
|
|
grp->bb_free);
|
ext4: mark block group as corrupt on block bitmap error
When we notice a block-bitmap corruption (because of device failure or
something else), we should mark this group as corrupt and prevent
further block allocations/deallocations from it. Currently, we end up
generating one error message for every block in the bitmap. This
potentially could make the system unstable as noticed in some
bugs. With this patch, the error will be printed only the first time
and mark the entire block group as corrupted. This prevents future
access allocations/deallocations from it.
Also tested by corrupting the block
bitmap and forcefully introducing the mb_free_blocks error:
(1) create a largefile (2Gb)
$ dd if=/dev/zero of=largefile oflag=direct bs=10485760 count=200
(2) umount filesystem. use dumpe2fs to see which block-bitmaps
are in use by largefile and note their block numbers
(3) use dd to zero-out the used block bitmaps
$ dd if=/dev/zero of=/dev/hdc4 bs=4096 seek=14 count=8 oflag=direct
(4) mount the FS and delete the largefile.
(5) recreate the largefile. verify that the new largefile does not
get any blocks from the groups marked as bad.
Without the patch, we will see mb_free_blocks error for each bit in
each zero'ed out bitmap at (4). With the patch, we only see the error
once per blockgroup:
[ 309.706803] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 15: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.720824] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 14: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.732858] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.748321] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 13: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.760331] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.769695] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 12: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.781721] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.798166] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 11: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
[ 309.810184] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
[ 309.819532] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 10: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
Google-Bug-Id: 7258357
[darrick.wong@oracle.com]
Further modifications (by Darrick) to make more obvious that this corruption
bit applies to blocks only. Set the corruption flag if the block group bitmap
verification fails.
Original-author: Aditya Kali <adityakali@google.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
2013-08-28 21:35:51 +00:00
|
|
|
set_bit(EXT4_GROUP_INFO_BBITMAP_CORRUPT_BIT, &grp->bb_state);
|
2015-10-18 01:33:24 +00:00
|
|
|
return -EFSCORRUPTED;
|
2012-08-10 17:57:52 +00:00
|
|
|
}
|
|
|
|
set_buffer_verified(bh);
|
2012-04-29 22:35:10 +00:00
|
|
|
ext4_unlock_group(sb, block_group);
|
2015-10-18 01:33:24 +00:00
|
|
|
return 0;
|
2012-04-29 22:35:10 +00:00
|
|
|
}
|
|
|
|
|
2006-10-11 08:20:50 +00:00
|
|
|
/**
|
2013-01-12 21:33:25 +00:00
|
|
|
* ext4_read_block_bitmap_nowait()
|
2006-10-11 08:20:50 +00:00
|
|
|
* @sb: super block
|
|
|
|
* @block_group: given block group
|
|
|
|
*
|
2008-01-29 04:58:27 +00:00
|
|
|
* Read the bitmap for a given block_group,and validate the
|
|
|
|
* bits for block/inode/inode tables are set in the bitmaps
|
2006-10-11 08:20:50 +00:00
|
|
|
*
|
|
|
|
* Return buffer_head on success or NULL in case of failure.
|
|
|
|
*/
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
struct buffer_head *
|
2012-02-20 22:52:46 +00:00
|
|
|
ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2008-09-09 02:25:24 +00:00
|
|
|
struct ext4_group_desc *desc;
|
2012-02-20 22:52:46 +00:00
|
|
|
struct buffer_head *bh;
|
2007-10-17 06:27:02 +00:00
|
|
|
ext4_fsblk_t bitmap_blk;
|
2015-10-18 01:33:24 +00:00
|
|
|
int err;
|
2006-10-11 08:20:50 +00:00
|
|
|
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
desc = ext4_get_group_desc(sb, block_group, NULL);
|
2006-10-11 08:20:50 +00:00
|
|
|
if (!desc)
|
2015-10-18 01:33:24 +00:00
|
|
|
return ERR_PTR(-EFSCORRUPTED);
|
2007-10-17 06:27:02 +00:00
|
|
|
bitmap_blk = ext4_block_bitmap(sb, desc);
|
2008-01-29 04:58:27 +00:00
|
|
|
bh = sb_getblk(sb, bitmap_blk);
|
|
|
|
if (unlikely(!bh)) {
|
2012-02-20 22:52:46 +00:00
|
|
|
ext4_error(sb, "Cannot get buffer for block bitmap - "
|
|
|
|
"block_group = %u, block_bitmap = %llu",
|
|
|
|
block_group, bitmap_blk);
|
2015-10-18 01:33:24 +00:00
|
|
|
return ERR_PTR(-ENOMEM);
|
2008-01-29 04:58:27 +00:00
|
|
|
}
|
2009-01-06 02:49:55 +00:00
|
|
|
|
|
|
|
if (bitmap_uptodate(bh))
|
2012-04-29 22:35:10 +00:00
|
|
|
goto verify;
|
2008-01-29 04:58:27 +00:00
|
|
|
|
ext4: fix initialization of UNINIT bitmap blocks
This fixes a bug which caused on-line resizing of filesystems with a
1k blocksize to fail. The root cause of this bug was the fact that if
an uninitalized bitmap block gets read in by userspace (which
e2fsprogs does try to avoid, but can happen when the blocksize is less
than the pagesize and an adjacent blocks is read into memory)
ext4_read_block_bitmap() was erroneously depending on the buffer
uptodate flag to decide whether it needed to initialize the bitmap
block in memory --- i.e., to set the standard set of blocks in use by
a block group (superblock, bitmaps, inode table, etc.). Essentially,
ext4_read_block_bitmap() assumed it was the only routine that might
try to read a block containing a block bitmap, which is simply not
true.
To fix this, ext4_read_block_bitmap() and ext4_read_inode_bitmap()
must always initialize uninitialized bitmap blocks. Once a block or
inode is allocated out of that bitmap, it will be marked as
initialized in the block group descriptor, so in general this won't
result any extra unnecessary work.
Signed-off-by: Frederic Bohe <frederic.bohe@bull.net>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
2008-10-10 12:09:18 +00:00
|
|
|
lock_buffer(bh);
|
2009-01-06 02:49:55 +00:00
|
|
|
if (bitmap_uptodate(bh)) {
|
|
|
|
unlock_buffer(bh);
|
2012-04-29 22:35:10 +00:00
|
|
|
goto verify;
|
2009-01-06 02:49:55 +00:00
|
|
|
}
|
2009-05-03 00:35:09 +00:00
|
|
|
ext4_lock_group(sb, block_group);
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
if (desc->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
|
2014-10-13 07:42:12 +00:00
|
|
|
err = ext4_init_block_bitmap(sb, bh, block_group, desc);
|
2009-01-06 02:49:55 +00:00
|
|
|
set_bitmap_uptodate(bh);
|
2008-01-29 04:58:27 +00:00
|
|
|
set_buffer_uptodate(bh);
|
2018-02-19 19:16:47 +00:00
|
|
|
set_buffer_verified(bh);
|
2009-05-03 00:35:09 +00:00
|
|
|
ext4_unlock_group(sb, block_group);
|
2009-01-04 03:33:39 +00:00
|
|
|
unlock_buffer(bh);
|
2016-02-12 04:15:12 +00:00
|
|
|
if (err) {
|
|
|
|
ext4_error(sb, "Failed to init block bitmap for group "
|
|
|
|
"%u: %d", block_group, err);
|
2015-10-18 01:33:24 +00:00
|
|
|
goto out;
|
2016-02-12 04:15:12 +00:00
|
|
|
}
|
2015-06-08 15:18:52 +00:00
|
|
|
goto verify;
|
Ext4: Uninitialized Block Groups
In pass1 of e2fsck, every inode table in the fileystem is scanned and checked,
regardless of whether it is in use. This is this the most time consuming part
of the filesystem check. The unintialized block group feature can greatly
reduce e2fsck time by eliminating checking of uninitialized inodes.
With this feature, there is a a high water mark of used inodes for each block
group. Block and inode bitmaps can be uninitialized on disk via a flag in the
group descriptor to avoid reading or scanning them at e2fsck time. A checksum
of each group descriptor is used to ensure that corruption in the group
descriptor's bit flags does not cause incorrect operation.
The feature is enabled through a mkfs option
mke2fs /dev/ -O uninit_groups
A patch adding support for uninitialized block groups to e2fsprogs tools has
been posted to the linux-ext4 mailing list.
The patches have been stress tested with fsstress and fsx. In performance
tests testing e2fsck time, we have seen that e2fsck time on ext3 grows
linearly with the total number of inodes in the filesytem. In ext4 with the
uninitialized block groups feature, the e2fsck time is constant, based
solely on the number of used inodes rather than the total inode count.
Since typical ext4 filesystems only use 1-10% of their inodes, this feature can
greatly reduce e2fsck time for users. With performance improvement of 2-20
times, depending on how full the filesystem is.
The attached graph shows the major improvements in e2fsck times in filesystems
with a large total inode count, but few inodes in use.
In each group descriptor if we have
EXT4_BG_INODE_UNINIT set in bg_flags:
Inode table is not initialized/used in this group. So we can skip
the consistency check during fsck.
EXT4_BG_BLOCK_UNINIT set in bg_flags:
No block in the group is used. So we can skip the block bitmap
verification for this group.
We also add two new fields to group descriptor as a part of
uninitialized group patch.
__le16 bg_itable_unused; /* Unused inodes count */
__le16 bg_checksum; /* crc16(sb_uuid+group+desc) */
bg_itable_unused:
If we have EXT4_BG_INODE_UNINIT not set in bg_flags
then bg_itable_unused will give the offset within
the inode table till the inodes are used. This can be
used by fsck to skip list of inodes that are marked unused.
bg_checksum:
Now that we depend on bg_flags and bg_itable_unused to determine
the block and inode usage, we need to make sure group descriptor
is not corrupt. We add checksum to group descriptor to
detect corruption. If the descriptor is found to be corrupt, we
mark all the blocks and inodes in the group used.
Signed-off-by: Avantika Mathur <mathur@us.ibm.com>
Signed-off-by: Andreas Dilger <adilger@clusterfs.com>
Signed-off-by: Mingming Cao <cmm@us.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
2007-10-16 22:38:25 +00:00
|
|
|
}
|
2009-05-03 00:35:09 +00:00
|
|
|
ext4_unlock_group(sb, block_group);
|
2009-01-06 02:49:55 +00:00
|
|
|
if (buffer_uptodate(bh)) {
|
|
|
|
/*
|
|
|
|
* if not uninit if bh is uptodate,
|
|
|
|
* bitmap is also uptodate
|
|
|
|
*/
|
|
|
|
set_bitmap_uptodate(bh);
|
|
|
|
unlock_buffer(bh);
|
2012-04-29 22:35:10 +00:00
|
|
|
goto verify;
|
2009-01-06 02:49:55 +00:00
|
|
|
}
|
|
|
|
/*
|
2012-02-20 22:52:46 +00:00
|
|
|
* submit the buffer_head for reading
|
2009-01-06 02:49:55 +00:00
|
|
|
*/
|
2012-02-20 22:52:46 +00:00
|
|
|
set_buffer_new(bh);
|
2011-03-22 01:38:05 +00:00
|
|
|
trace_ext4_read_block_bitmap_load(sb, block_group);
|
2012-02-20 22:52:46 +00:00
|
|
|
bh->b_end_io = ext4_end_bitmap_read;
|
|
|
|
get_bh(bh);
|
2016-06-05 19:31:43 +00:00
|
|
|
submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh);
|
2012-02-20 22:52:46 +00:00
|
|
|
return bh;
|
2012-04-29 22:35:10 +00:00
|
|
|
verify:
|
2015-10-18 01:33:24 +00:00
|
|
|
err = ext4_validate_block_bitmap(sb, desc, block_group, bh);
|
|
|
|
if (err)
|
|
|
|
goto out;
|
|
|
|
return bh;
|
|
|
|
out:
|
2013-08-28 19:35:27 +00:00
|
|
|
put_bh(bh);
|
2015-10-18 01:33:24 +00:00
|
|
|
return ERR_PTR(err);
|
2012-02-20 22:52:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Returns 0 on success, 1 on error */
|
|
|
|
int ext4_wait_block_bitmap(struct super_block *sb, ext4_group_t block_group,
|
|
|
|
struct buffer_head *bh)
|
|
|
|
{
|
|
|
|
struct ext4_group_desc *desc;
|
|
|
|
|
|
|
|
if (!buffer_new(bh))
|
|
|
|
return 0;
|
|
|
|
desc = ext4_get_group_desc(sb, block_group, NULL);
|
|
|
|
if (!desc)
|
2015-10-18 01:33:24 +00:00
|
|
|
return -EFSCORRUPTED;
|
2012-02-20 22:52:46 +00:00
|
|
|
wait_on_buffer(bh);
|
|
|
|
if (!buffer_uptodate(bh)) {
|
2010-02-15 19:19:27 +00:00
|
|
|
ext4_error(sb, "Cannot read block bitmap - "
|
2012-02-20 22:52:46 +00:00
|
|
|
"block_group = %u, block_bitmap = %llu",
|
2012-02-20 22:57:24 +00:00
|
|
|
block_group, (unsigned long long) bh->b_blocknr);
|
2015-10-18 01:33:24 +00:00
|
|
|
return -EIO;
|
2008-01-29 04:58:27 +00:00
|
|
|
}
|
2012-02-20 22:52:46 +00:00
|
|
|
clear_buffer_new(bh);
|
|
|
|
/* Panic or remount fs read-only if block bitmap is invalid */
|
2015-10-18 01:33:24 +00:00
|
|
|
return ext4_validate_block_bitmap(sb, desc, block_group, bh);
|
2012-02-20 22:52:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
struct buffer_head *
|
|
|
|
ext4_read_block_bitmap(struct super_block *sb, ext4_group_t block_group)
|
|
|
|
{
|
|
|
|
struct buffer_head *bh;
|
2015-10-18 01:33:24 +00:00
|
|
|
int err;
|
2012-02-20 22:52:46 +00:00
|
|
|
|
|
|
|
bh = ext4_read_block_bitmap_nowait(sb, block_group);
|
2015-10-18 01:33:24 +00:00
|
|
|
if (IS_ERR(bh))
|
|
|
|
return bh;
|
|
|
|
err = ext4_wait_block_bitmap(sb, block_group, bh);
|
|
|
|
if (err) {
|
2012-02-20 22:52:46 +00:00
|
|
|
put_bh(bh);
|
2015-10-18 01:33:24 +00:00
|
|
|
return ERR_PTR(err);
|
2012-02-20 22:52:46 +00:00
|
|
|
}
|
2006-10-11 08:20:50 +00:00
|
|
|
return bh;
|
|
|
|
}
|
|
|
|
|
2008-10-28 04:08:12 +00:00
|
|
|
/**
|
2011-09-09 23:16:51 +00:00
|
|
|
* ext4_has_free_clusters()
|
2008-10-28 04:08:12 +00:00
|
|
|
* @sbi: in-core super block structure.
|
2011-09-09 23:16:51 +00:00
|
|
|
* @nclusters: number of needed blocks
|
|
|
|
* @flags: flags from ext4_mb_new_blocks()
|
2008-10-28 04:08:12 +00:00
|
|
|
*
|
2011-09-09 23:16:51 +00:00
|
|
|
* Check if filesystem has nclusters free & available for allocation.
|
2008-10-28 04:08:12 +00:00
|
|
|
* On success return 1, return 0 on failure.
|
|
|
|
*/
|
2011-09-09 23:16:51 +00:00
|
|
|
static int ext4_has_free_clusters(struct ext4_sb_info *sbi,
|
|
|
|
s64 nclusters, unsigned int flags)
|
2008-10-09 14:56:23 +00:00
|
|
|
{
|
2013-04-10 02:11:22 +00:00
|
|
|
s64 free_clusters, dirty_clusters, rsv, resv_clusters;
|
2011-09-09 22:56:51 +00:00
|
|
|
struct percpu_counter *fcc = &sbi->s_freeclusters_counter;
|
2011-09-09 23:16:51 +00:00
|
|
|
struct percpu_counter *dcc = &sbi->s_dirtyclusters_counter;
|
2008-10-09 14:56:23 +00:00
|
|
|
|
2011-09-09 23:16:51 +00:00
|
|
|
free_clusters = percpu_counter_read_positive(fcc);
|
|
|
|
dirty_clusters = percpu_counter_read_positive(dcc);
|
2013-04-10 02:11:22 +00:00
|
|
|
resv_clusters = atomic64_read(&sbi->s_resv_clusters);
|
2013-02-22 20:27:52 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* r_blocks_count should always be multiple of the cluster ratio so
|
|
|
|
* we are safe to do a plane bit shift only.
|
|
|
|
*/
|
2013-04-10 02:11:22 +00:00
|
|
|
rsv = (ext4_r_blocks_count(sbi->s_es) >> sbi->s_cluster_bits) +
|
|
|
|
resv_clusters;
|
2008-10-09 14:56:23 +00:00
|
|
|
|
2013-04-10 02:11:22 +00:00
|
|
|
if (free_clusters - (nclusters + rsv + dirty_clusters) <
|
2011-09-09 23:16:51 +00:00
|
|
|
EXT4_FREECLUSTERS_WATERMARK) {
|
2013-02-22 20:27:52 +00:00
|
|
|
free_clusters = percpu_counter_sum_positive(fcc);
|
2011-09-09 23:16:51 +00:00
|
|
|
dirty_clusters = percpu_counter_sum_positive(dcc);
|
2008-10-10 13:39:00 +00:00
|
|
|
}
|
2011-09-09 23:16:51 +00:00
|
|
|
/* Check whether we have space after accounting for current
|
|
|
|
* dirty clusters & root reserved clusters.
|
2008-10-10 13:39:00 +00:00
|
|
|
*/
|
2013-04-10 02:11:22 +00:00
|
|
|
if (free_clusters >= (rsv + nclusters + dirty_clusters))
|
2008-10-28 04:08:17 +00:00
|
|
|
return 1;
|
2008-10-09 14:56:23 +00:00
|
|
|
|
2011-09-09 23:16:51 +00:00
|
|
|
/* Hm, nope. Are (enough) root reserved clusters available? */
|
2012-02-07 23:41:49 +00:00
|
|
|
if (uid_eq(sbi->s_resuid, current_fsuid()) ||
|
|
|
|
(!gid_eq(sbi->s_resgid, GLOBAL_ROOT_GID) && in_group_p(sbi->s_resgid)) ||
|
2011-05-25 11:41:26 +00:00
|
|
|
capable(CAP_SYS_RESOURCE) ||
|
2013-04-10 02:11:22 +00:00
|
|
|
(flags & EXT4_MB_USE_ROOT_BLOCKS)) {
|
2011-05-25 11:41:26 +00:00
|
|
|
|
2013-04-10 02:11:22 +00:00
|
|
|
if (free_clusters >= (nclusters + dirty_clusters +
|
|
|
|
resv_clusters))
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
/* No free blocks. Let's see if we can dip into reserved pool */
|
|
|
|
if (flags & EXT4_MB_USE_RESERVED) {
|
2011-09-09 23:16:51 +00:00
|
|
|
if (free_clusters >= (nclusters + dirty_clusters))
|
2008-10-28 04:08:17 +00:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
2008-10-09 14:56:23 +00:00
|
|
|
}
|
|
|
|
|
2011-09-09 23:14:51 +00:00
|
|
|
int ext4_claim_free_clusters(struct ext4_sb_info *sbi,
|
|
|
|
s64 nclusters, unsigned int flags)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2011-09-09 23:16:51 +00:00
|
|
|
if (ext4_has_free_clusters(sbi, nclusters, flags)) {
|
2011-09-09 23:14:51 +00:00
|
|
|
percpu_counter_add(&sbi->s_dirtyclusters_counter, nclusters);
|
2008-08-20 01:16:54 +00:00
|
|
|
return 0;
|
2008-10-28 04:08:12 +00:00
|
|
|
} else
|
|
|
|
return -ENOSPC;
|
2008-10-09 14:56:23 +00:00
|
|
|
}
|
2008-07-11 23:27:31 +00:00
|
|
|
|
2006-10-11 08:20:50 +00:00
|
|
|
/**
|
2006-10-11 08:20:53 +00:00
|
|
|
* ext4_should_retry_alloc()
|
2006-10-11 08:20:50 +00:00
|
|
|
* @sb: super block
|
|
|
|
* @retries number of attemps has been made
|
|
|
|
*
|
2006-10-11 08:20:53 +00:00
|
|
|
* ext4_should_retry_alloc() is called when ENOSPC is returned, and if
|
2006-10-11 08:20:50 +00:00
|
|
|
* it is profitable to retry the operation, this function will wait
|
2011-03-31 01:57:33 +00:00
|
|
|
* for the current or committing transaction to complete, and then
|
2017-10-01 21:59:54 +00:00
|
|
|
* return TRUE. We will only retry once.
|
2006-10-11 08:20:50 +00:00
|
|
|
*/
|
2006-10-11 08:20:53 +00:00
|
|
|
int ext4_should_retry_alloc(struct super_block *sb, int *retries)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2011-09-09 23:16:51 +00:00
|
|
|
if (!ext4_has_free_clusters(EXT4_SB(sb), 1, 0) ||
|
2017-10-01 21:59:54 +00:00
|
|
|
(*retries)++ > 1 ||
|
2009-02-26 05:57:35 +00:00
|
|
|
!EXT4_SB(sb)->s_journal)
|
2006-10-11 08:20:50 +00:00
|
|
|
return 0;
|
|
|
|
|
2016-06-26 22:24:01 +00:00
|
|
|
smp_mb();
|
2017-10-01 21:59:54 +00:00
|
|
|
if (EXT4_SB(sb)->s_mb_free_pending == 0)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
jbd_debug(1, "%s: retrying operation after ENOSPC\n", sb->s_id);
|
|
|
|
jbd2_journal_force_commit_nested(EXT4_SB(sb)->s_journal);
|
2016-05-13 04:42:40 +00:00
|
|
|
return 1;
|
2006-10-11 08:20:50 +00:00
|
|
|
}
|
|
|
|
|
2008-07-11 23:27:31 +00:00
|
|
|
/*
|
2008-07-14 21:52:37 +00:00
|
|
|
* ext4_new_meta_blocks() -- allocate block for meta data (indexing) blocks
|
2008-07-11 23:27:31 +00:00
|
|
|
*
|
|
|
|
* @handle: handle to this transaction
|
|
|
|
* @inode: file inode
|
|
|
|
* @goal: given target block(filesystem wide)
|
2011-09-09 23:04:51 +00:00
|
|
|
* @count: pointer to total number of clusters needed
|
2008-07-11 23:27:31 +00:00
|
|
|
* @errp: error code
|
|
|
|
*
|
2008-12-12 17:41:28 +00:00
|
|
|
* Return 1st allocated block number on success, *count stores total account
|
2008-07-14 21:52:37 +00:00
|
|
|
* error stores in errp pointer
|
2008-07-11 23:27:31 +00:00
|
|
|
*/
|
2008-07-14 21:52:37 +00:00
|
|
|
ext4_fsblk_t ext4_new_meta_blocks(handle_t *handle, struct inode *inode,
|
2011-05-25 11:41:26 +00:00
|
|
|
ext4_fsblk_t goal, unsigned int flags,
|
|
|
|
unsigned long *count, int *errp)
|
2008-07-11 23:27:31 +00:00
|
|
|
{
|
2008-12-12 17:41:28 +00:00
|
|
|
struct ext4_allocation_request ar;
|
2008-07-14 21:52:37 +00:00
|
|
|
ext4_fsblk_t ret;
|
2008-12-12 17:41:28 +00:00
|
|
|
|
|
|
|
memset(&ar, 0, sizeof(ar));
|
|
|
|
/* Fill with neighbour allocated blocks */
|
|
|
|
ar.inode = inode;
|
|
|
|
ar.goal = goal;
|
|
|
|
ar.len = count ? *count : 1;
|
2011-05-25 11:41:26 +00:00
|
|
|
ar.flags = flags;
|
2008-12-12 17:41:28 +00:00
|
|
|
|
|
|
|
ret = ext4_mb_new_blocks(handle, &ar, errp);
|
|
|
|
if (count)
|
|
|
|
*count = ar.len;
|
2008-07-14 21:52:37 +00:00
|
|
|
/*
|
2010-05-16 15:00:00 +00:00
|
|
|
* Account for the allocated meta blocks. We will never
|
|
|
|
* fail EDQUOT for metdata, but we do account for it.
|
2008-07-14 21:52:37 +00:00
|
|
|
*/
|
2014-09-04 22:07:25 +00:00
|
|
|
if (!(*errp) && (flags & EXT4_MB_DELALLOC_RESERVED)) {
|
2011-09-09 23:04:51 +00:00
|
|
|
dquot_alloc_block_nofail(inode,
|
|
|
|
EXT4_C2B(EXT4_SB(inode->i_sb), ar.len));
|
2008-07-14 21:52:37 +00:00
|
|
|
}
|
|
|
|
return ret;
|
2008-07-11 23:27:31 +00:00
|
|
|
}
|
|
|
|
|
2006-10-11 08:20:50 +00:00
|
|
|
/**
|
2011-09-09 23:10:51 +00:00
|
|
|
* ext4_count_free_clusters() -- count filesystem free clusters
|
2006-10-11 08:20:50 +00:00
|
|
|
* @sb: superblock
|
|
|
|
*
|
2011-09-09 23:10:51 +00:00
|
|
|
* Adds up the number of free clusters from each block group.
|
2006-10-11 08:20:50 +00:00
|
|
|
*/
|
2011-09-09 23:10:51 +00:00
|
|
|
ext4_fsblk_t ext4_count_free_clusters(struct super_block *sb)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2006-10-11 08:20:53 +00:00
|
|
|
ext4_fsblk_t desc_count;
|
|
|
|
struct ext4_group_desc *gdp;
|
2008-01-29 04:58:27 +00:00
|
|
|
ext4_group_t i;
|
2009-05-01 12:50:38 +00:00
|
|
|
ext4_group_t ngroups = ext4_get_groups_count(sb);
|
2013-10-31 15:46:31 +00:00
|
|
|
struct ext4_group_info *grp;
|
2006-10-11 08:20:53 +00:00
|
|
|
#ifdef EXT4FS_DEBUG
|
|
|
|
struct ext4_super_block *es;
|
|
|
|
ext4_fsblk_t bitmap_count;
|
2008-11-05 05:14:04 +00:00
|
|
|
unsigned int x;
|
2006-10-11 08:20:50 +00:00
|
|
|
struct buffer_head *bitmap_bh = NULL;
|
|
|
|
|
2006-10-11 08:20:53 +00:00
|
|
|
es = EXT4_SB(sb)->s_es;
|
2006-10-11 08:20:50 +00:00
|
|
|
desc_count = 0;
|
|
|
|
bitmap_count = 0;
|
|
|
|
gdp = NULL;
|
|
|
|
|
|
|
|
for (i = 0; i < ngroups; i++) {
|
2006-10-11 08:20:53 +00:00
|
|
|
gdp = ext4_get_group_desc(sb, i, NULL);
|
2006-10-11 08:20:50 +00:00
|
|
|
if (!gdp)
|
|
|
|
continue;
|
2013-10-31 15:46:31 +00:00
|
|
|
grp = NULL;
|
|
|
|
if (EXT4_SB(sb)->s_group_info)
|
|
|
|
grp = ext4_get_group_info(sb, i);
|
|
|
|
if (!grp || !EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
|
|
|
|
desc_count += ext4_free_group_clusters(sb, gdp);
|
2006-10-11 08:20:50 +00:00
|
|
|
brelse(bitmap_bh);
|
2008-07-11 23:27:31 +00:00
|
|
|
bitmap_bh = ext4_read_block_bitmap(sb, i);
|
2015-10-18 01:33:24 +00:00
|
|
|
if (IS_ERR(bitmap_bh)) {
|
|
|
|
bitmap_bh = NULL;
|
2006-10-11 08:20:50 +00:00
|
|
|
continue;
|
2015-10-18 01:33:24 +00:00
|
|
|
}
|
2006-10-11 08:20:50 +00:00
|
|
|
|
2012-06-30 23:14:57 +00:00
|
|
|
x = ext4_count_free(bitmap_bh->b_data,
|
2014-04-15 03:36:15 +00:00
|
|
|
EXT4_CLUSTERS_PER_GROUP(sb) / 8);
|
2009-01-27 00:26:26 +00:00
|
|
|
printk(KERN_DEBUG "group %u: stored = %d, counted = %u\n",
|
2011-09-09 23:08:51 +00:00
|
|
|
i, ext4_free_group_clusters(sb, gdp), x);
|
2006-10-11 08:20:50 +00:00
|
|
|
bitmap_count += x;
|
|
|
|
}
|
|
|
|
brelse(bitmap_bh);
|
2011-09-09 23:10:51 +00:00
|
|
|
printk(KERN_DEBUG "ext4_count_free_clusters: stored = %llu"
|
|
|
|
", computed = %llu, %llu\n",
|
2013-03-02 22:18:58 +00:00
|
|
|
EXT4_NUM_B2C(EXT4_SB(sb), ext4_free_blocks_count(es)),
|
2008-09-09 03:00:52 +00:00
|
|
|
desc_count, bitmap_count);
|
2006-10-11 08:20:50 +00:00
|
|
|
return bitmap_count;
|
|
|
|
#else
|
|
|
|
desc_count = 0;
|
|
|
|
for (i = 0; i < ngroups; i++) {
|
2006-10-11 08:20:53 +00:00
|
|
|
gdp = ext4_get_group_desc(sb, i, NULL);
|
2006-10-11 08:20:50 +00:00
|
|
|
if (!gdp)
|
|
|
|
continue;
|
2013-10-31 15:46:31 +00:00
|
|
|
grp = NULL;
|
|
|
|
if (EXT4_SB(sb)->s_group_info)
|
|
|
|
grp = ext4_get_group_info(sb, i);
|
|
|
|
if (!grp || !EXT4_MB_GRP_BBITMAP_CORRUPT(grp))
|
|
|
|
desc_count += ext4_free_group_clusters(sb, gdp);
|
2006-10-11 08:20:50 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return desc_count;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2008-01-29 04:58:27 +00:00
|
|
|
static inline int test_root(ext4_group_t a, int b)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2013-06-06 15:40:37 +00:00
|
|
|
while (1) {
|
|
|
|
if (a < b)
|
|
|
|
return 0;
|
|
|
|
if (a == b)
|
|
|
|
return 1;
|
|
|
|
if ((a % b) != 0)
|
|
|
|
return 0;
|
|
|
|
a = a / b;
|
|
|
|
}
|
2006-10-11 08:20:50 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2006-10-11 08:20:53 +00:00
|
|
|
* ext4_bg_has_super - number of blocks used by the superblock in group
|
2006-10-11 08:20:50 +00:00
|
|
|
* @sb: superblock for filesystem
|
|
|
|
* @group: group number to check
|
|
|
|
*
|
|
|
|
* Return the number of blocks used by the superblock (primary or backup)
|
|
|
|
* in this group. Currently this will be only 0 or 1.
|
|
|
|
*/
|
2008-01-29 04:58:27 +00:00
|
|
|
int ext4_bg_has_super(struct super_block *sb, ext4_group_t group)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2014-05-12 14:16:06 +00:00
|
|
|
struct ext4_super_block *es = EXT4_SB(sb)->s_es;
|
|
|
|
|
|
|
|
if (group == 0)
|
|
|
|
return 1;
|
2015-10-17 20:18:43 +00:00
|
|
|
if (ext4_has_feature_sparse_super2(sb)) {
|
2014-05-12 14:16:06 +00:00
|
|
|
if (group == le32_to_cpu(es->s_backup_bgs[0]) ||
|
|
|
|
group == le32_to_cpu(es->s_backup_bgs[1]))
|
|
|
|
return 1;
|
|
|
|
return 0;
|
|
|
|
}
|
2015-10-17 20:18:43 +00:00
|
|
|
if ((group <= 1) || !ext4_has_feature_sparse_super(sb))
|
2014-05-12 14:16:06 +00:00
|
|
|
return 1;
|
|
|
|
if (!(group & 1))
|
2006-10-11 08:20:50 +00:00
|
|
|
return 0;
|
2014-05-12 14:16:06 +00:00
|
|
|
if (test_root(group, 3) || (test_root(group, 5)) ||
|
|
|
|
test_root(group, 7))
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
return 0;
|
2006-10-11 08:20:50 +00:00
|
|
|
}
|
|
|
|
|
2008-01-29 04:58:27 +00:00
|
|
|
static unsigned long ext4_bg_num_gdb_meta(struct super_block *sb,
|
|
|
|
ext4_group_t group)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2006-10-11 08:20:53 +00:00
|
|
|
unsigned long metagroup = group / EXT4_DESC_PER_BLOCK(sb);
|
2008-01-29 04:58:27 +00:00
|
|
|
ext4_group_t first = metagroup * EXT4_DESC_PER_BLOCK(sb);
|
|
|
|
ext4_group_t last = first + EXT4_DESC_PER_BLOCK(sb) - 1;
|
2006-10-11 08:20:50 +00:00
|
|
|
|
|
|
|
if (group == first || group == first + 1 || group == last)
|
|
|
|
return 1;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-01-29 04:58:27 +00:00
|
|
|
static unsigned long ext4_bg_num_gdb_nometa(struct super_block *sb,
|
|
|
|
ext4_group_t group)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
2009-11-23 12:24:38 +00:00
|
|
|
if (!ext4_bg_has_super(sb, group))
|
|
|
|
return 0;
|
|
|
|
|
2015-10-17 20:18:43 +00:00
|
|
|
if (ext4_has_feature_meta_bg(sb))
|
2009-11-23 12:24:38 +00:00
|
|
|
return le32_to_cpu(EXT4_SB(sb)->s_es->s_first_meta_bg);
|
|
|
|
else
|
|
|
|
return EXT4_SB(sb)->s_gdb_count;
|
2006-10-11 08:20:50 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2006-10-11 08:20:53 +00:00
|
|
|
* ext4_bg_num_gdb - number of blocks used by the group table in group
|
2006-10-11 08:20:50 +00:00
|
|
|
* @sb: superblock for filesystem
|
|
|
|
* @group: group number to check
|
|
|
|
*
|
|
|
|
* Return the number of blocks used by the group descriptor table
|
|
|
|
* (primary or backup) in this group. In the future there may be a
|
|
|
|
* different number of descriptor blocks in each group.
|
|
|
|
*/
|
2008-01-29 04:58:27 +00:00
|
|
|
unsigned long ext4_bg_num_gdb(struct super_block *sb, ext4_group_t group)
|
2006-10-11 08:20:50 +00:00
|
|
|
{
|
|
|
|
unsigned long first_meta_bg =
|
2006-10-11 08:20:53 +00:00
|
|
|
le32_to_cpu(EXT4_SB(sb)->s_es->s_first_meta_bg);
|
|
|
|
unsigned long metagroup = group / EXT4_DESC_PER_BLOCK(sb);
|
2006-10-11 08:20:50 +00:00
|
|
|
|
2015-10-17 20:18:43 +00:00
|
|
|
if (!ext4_has_feature_meta_bg(sb) || metagroup < first_meta_bg)
|
2008-09-09 02:25:24 +00:00
|
|
|
return ext4_bg_num_gdb_nometa(sb, group);
|
2006-10-11 08:20:50 +00:00
|
|
|
|
2006-10-11 08:20:53 +00:00
|
|
|
return ext4_bg_num_gdb_meta(sb,group);
|
2006-10-11 08:20:50 +00:00
|
|
|
|
|
|
|
}
|
2008-10-10 13:40:52 +00:00
|
|
|
|
2011-09-09 22:40:51 +00:00
|
|
|
/*
|
2011-09-09 22:44:51 +00:00
|
|
|
* This function returns the number of file system metadata clusters at
|
2011-09-09 22:40:51 +00:00
|
|
|
* the beginning of a block group, including the reserved gdt blocks.
|
|
|
|
*/
|
2012-01-05 03:33:28 +00:00
|
|
|
static unsigned ext4_num_base_meta_clusters(struct super_block *sb,
|
2011-09-09 22:44:51 +00:00
|
|
|
ext4_group_t block_group)
|
2011-09-09 22:40:51 +00:00
|
|
|
{
|
|
|
|
struct ext4_sb_info *sbi = EXT4_SB(sb);
|
2011-09-09 22:44:51 +00:00
|
|
|
unsigned num;
|
2011-09-09 22:40:51 +00:00
|
|
|
|
|
|
|
/* Check for superblock and gdt backups in this group */
|
|
|
|
num = ext4_bg_has_super(sb, block_group);
|
|
|
|
|
2015-10-17 20:18:43 +00:00
|
|
|
if (!ext4_has_feature_meta_bg(sb) ||
|
2011-09-09 22:40:51 +00:00
|
|
|
block_group < le32_to_cpu(sbi->s_es->s_first_meta_bg) *
|
|
|
|
sbi->s_desc_per_block) {
|
|
|
|
if (num) {
|
|
|
|
num += ext4_bg_num_gdb(sb, block_group);
|
|
|
|
num += le16_to_cpu(sbi->s_es->s_reserved_gdt_blocks);
|
|
|
|
}
|
|
|
|
} else { /* For META_BG_BLOCK_GROUPS */
|
|
|
|
num += ext4_bg_num_gdb(sb, block_group);
|
|
|
|
}
|
2011-09-09 22:44:51 +00:00
|
|
|
return EXT4_NUM_B2C(sbi, num);
|
2011-09-09 22:40:51 +00:00
|
|
|
}
|
2011-06-28 14:01:31 +00:00
|
|
|
/**
|
|
|
|
* ext4_inode_to_goal_block - return a hint for block allocation
|
|
|
|
* @inode: inode for block allocation
|
|
|
|
*
|
|
|
|
* Return the ideal location to start allocating blocks for a
|
|
|
|
* newly created inode.
|
|
|
|
*/
|
|
|
|
ext4_fsblk_t ext4_inode_to_goal_block(struct inode *inode)
|
|
|
|
{
|
|
|
|
struct ext4_inode_info *ei = EXT4_I(inode);
|
|
|
|
ext4_group_t block_group;
|
|
|
|
ext4_grpblk_t colour;
|
|
|
|
int flex_size = ext4_flex_bg_size(EXT4_SB(inode->i_sb));
|
|
|
|
ext4_fsblk_t bg_start;
|
|
|
|
ext4_fsblk_t last_block;
|
|
|
|
|
|
|
|
block_group = ei->i_block_group;
|
|
|
|
if (flex_size >= EXT4_FLEX_SIZE_DIR_ALLOC_SCHEME) {
|
|
|
|
/*
|
|
|
|
* If there are at least EXT4_FLEX_SIZE_DIR_ALLOC_SCHEME
|
|
|
|
* block groups per flexgroup, reserve the first block
|
|
|
|
* group for directories and special files. Regular
|
|
|
|
* files will start at the second block group. This
|
|
|
|
* tends to speed up directory access and improves
|
|
|
|
* fsck times.
|
|
|
|
*/
|
|
|
|
block_group &= ~(flex_size-1);
|
|
|
|
if (S_ISREG(inode->i_mode))
|
|
|
|
block_group++;
|
|
|
|
}
|
|
|
|
bg_start = ext4_group_first_block_no(inode->i_sb, block_group);
|
|
|
|
last_block = ext4_blocks_count(EXT4_SB(inode->i_sb)->s_es) - 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we are doing delayed allocation, we don't need take
|
|
|
|
* colour into account.
|
|
|
|
*/
|
|
|
|
if (test_opt(inode->i_sb, DELALLOC))
|
|
|
|
return bg_start;
|
|
|
|
|
|
|
|
if (bg_start + EXT4_BLOCKS_PER_GROUP(inode->i_sb) <= last_block)
|
|
|
|
colour = (current->pid % 16) *
|
|
|
|
(EXT4_BLOCKS_PER_GROUP(inode->i_sb) / 16);
|
|
|
|
else
|
|
|
|
colour = (current->pid % 16) * ((last_block - bg_start) / 16);
|
|
|
|
return bg_start + colour;
|
|
|
|
}
|
|
|
|
|