The UFS1 integrity checks added in FreeBSD 14 were too aggressive
for UFS1 filesystems created in FreeBSD 4 and 9 systems. This patch
removes those tests which can be done safely since they are not
relevant to the current implementation of UFS1.
This is a follow-on report to bug report 264450 (comments 21-28).
Reported by: slb@sonnet.com
Tested by: slb@sonnet.com
PR: 264450
MFC after: 1 week
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.
Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/
Sponsored by: Netflix
We've ifdef'd out the copyright strings for some time now. Go ahead and
remove the ifdefs. Plus whatever other detritis was left over from other
recent removals. These copyright strings are present in the comments and
are largely from CSRG's attempt at adding their copyright to every
binary file (which modern interpretations of the license doesn't
require).
Sponsored by: Netflix
Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.
Sponsored by: Netflix
Various disk controllers require their buffers to be aligned to a
cache-line size (128 bytes). For buffers allocated in structures,
ensure that they are 128-byte aligned. Use aligned_malloc to allocate
memory to ensure that the returned memory is 128-byte aligned.
While we are here, we replace the dynamically allocated inode buffer
with a buffer allocated in the uufsd structure just as the superblock
and cylinder group buffers do.
This can be removed if/when the kernel is fixed. Because this problem
has existed on one I/O subsystem or another since the 1990's, we
are probably stuck with dealing with it forever.
The problem most recent showed up in Azure, see:
https://reviews.freebsd.org/D41728https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267654
Before these fixes were applied, it was confirmed that the changes
in this commit also fixed the issue in Azure.
Reviewed-by: Warner Losh, kib
Tested-by: Souradeep Chakrabarti of Microsoft (earlier version)
PR: 267654
Differential Revision: https://reviews.freebsd.org/D41724
When fsck_ffs(8) runs in background, it creates a snapshot named
fsck_snapshot in the filesystem's .snap directory. The fsck_snapshot
file was removed when the background fsck finished. If the system
crashed or the fsck exited unexpectedly, the fsck_snapshot file
would remain. The snapshot would consume ever more space as the
filesystem changed over time until it was removed by a system
administrator or a future run of background fsck removed it to
create a new snapshot file.
This commit unlinks the .snap/fsck_snapshot file immediately after
opening it so that it will be reclaimed when fsck closes it at the
conclusion of its run. After a system crash, it will be removed as
part of the filesystem cleanup because of its zero reference count.
As only a few milliseconds pass between its creation and unlinking,
there is far less opportunity for it to be accidentally left behind.
PR: 106107
MFC-after: 1 week
An earlier addition of code to fsck_ffs(8) allowed it to support
snapshots when running with journalled soft updates. Further
functionality has now been added to fsck_ffs(8) to allow it to use
snapshots to run in background on live filesystems running with
journaled soft updates. This commit enables the use of this functionality.
Tested-by: Peter Holm
Sponsored-by: The FreeBSD Foundation
MFC-after: 2 weeks
When loading the root directory ensure that it is a directory
and has a size greater than the minimum directory size. If an
invalid root directory is found, fall back to full fsck.
Reported-by: Robert Morris
PR: 271414
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation
When checking an inode ensure that it does not have a negative size.
Stop scaning a directory when an unallocated block is found.
Fully clear an inode when it is first allocated.
Ensure that an inode is marked dirty whenever it is updated and that
it has a correct check hash when it is released.
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation
Check for valid block numbers while loading journal entries that
contain block numbers. If an invalid block number is found, fall
back to full fsck.
Reported-by: Robert Morris
PR: 271383
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation
Check for valid file size before processing journal entries for it.
Done by extracting the file size check from pass1.c into chkfilesize()
then using it in the journal code in suj.c
Reported-by: Robert Morris
PR: 271378
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation
The last valid inode in the filesystem is maxino - 1, not maxino.
Thus validity checks should ino < maxino, not ino <= maxino.
Reported-by: Robert Morris
PR: 271312
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation
Always create a directory inode structure when a directory inode is
found in Pass 1 as it is not known whether it will be saved or removed
in later passes. If it is to be saved the directory inode structure
is needed to track its status and fsck_ffs(8) will segment fault if
it does not exist.
Reported-by: Robert Morris
PR: 271310
PR: 271354
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation
Commit fe5e6e2 improved FFS directory placement when creating new
directories. It is done by keeping track of the depth of directories
in the filesystem and placing those lower in the tree closer together
while spreading out those higher in the tree.
Fsck_ffs(8) checks these depths and if incorrect adjusts them to
their correct value. When running in background fsck_ffs(8) needs
to be able to make an adjustment to the depth. This commit adds
the sysctl to make such an adjustment and adds the code to fsck_ffs(8)
to use the new sysctl.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
The cylinder group header structure ended with `u_int8_t cg_space[1]'
representing the beginning of the inode bitmap array. Some architectures
like the i386 rounded this up to a 4-byte boundry while other
architectures like the amd64 rounded it up to an 8-byte boundry.
Thus sizeof(struct cg) was four bytes bigger on an amd64 machine
than on an i386 machine. If a filesystem created on an i386 machine
was moved to an amd64 machine, the size of the cylinder group
calculated by the CGSIZE macro would appear to grow by four bytes.
Filesystems whose cylinder groups were exactly equal to the block
size on an i386 machine would appear to have a cylinder group that
was four bytes too big when moved to an amd64 machine. Note that
although the structure appears to be too big, it in fact is fine.
It is just the calaculation of its size that is in error.
The fix is to remove the cg_space element from the cylinder-group
structure so that the calculated size of the structure is the same
size on all architectures.
Reported by: Tijl Coosemans
Tested by: Tijl Coosemans and Peter Holm
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
On an amd64-CURRENT machine with an i-node that refers to a block
number that is one too large will cause a core dump, due to writing
beyond the end of blockmap[] and corrupting the next heap block,
which happens to contain a struct inoinfo in inphash[]. Note that
valgrind catches the blockmap[] access.
Reported by: Robert Morris
PR: 271289
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
The previous change involved calling check_cgmagic() twice in a row
for the same CG in order to differentiate when the CG was already ok vs.
when the CG was rebuilt, but that doesn't work because the second call
(which was supposed to rebuild the CG) returns 0 (indicating that
the CG was not rebuilt) due to the prevfailcg check causing an early
failure return. Fix this by moving the rebuild part of check_cgmagic()
out into a separate function which is called by pass1() when it wants to
rebuild a CG.
Fixes: da86e7a20d
Reported by: pho
Discussed with: mckusick
Sponsored by: Netflix
Pass 1 of fsck_ffs checks the integrity of all the cylinder groups.
If any are found to have been corrupted it offers to rebuild them.
Pass 5 then makes a second pass over the cylinder groups to validate
their block and inode maps. Pass 5 assumes that the cylinder groups
are not corrupted and can segment fault if they are corrupted. Rather
than rerunning the corruption checks a second time in pass 5, this
fix keeps track whether any corrupt cylinder groups were found but not
fixed in pass 1 either due to running with the -n flag or by explicitly
answering `no' when asked whether to fix a corrupted cylinder group.
If any corrupted cylinder groups remain after pass 1, fsck_ffs will
decline to run pass 5. Instead it marks the filesystem as unclean
so that fsck_ffs will need to be run again before the filesystem can
be mounted.
This patch cleans up and documents the return value from check_cgmagic().
It also renames the variable / parameter "rebuildcg" to "rebuiltcg".
This parameter describes whether the cylinder group has been rebuilt
rather than whether it should be rebuilt.
Reported by: Chuck Silvers
Reviewed by: Chuck Silvers
MFC after: 1 week
Increment a reference count when returning a zero'ed out buffer
after a failed read.
Zero out a structure before using it.
Only dirty a buffer that has been modified.
Submitted by: Chuck Silvers
Sponsored by: Netflix
MFC after: 1 week
The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In those days
large disks had at most 100 cylinder groups and often as few as 10-20.
Modern multi-terrabyte disks have thousands of cylinder groups. The
original algorithm does not handle these large sizes well. This change
attempts to expand the scope of the original algorithm to work well
with these much larger disks while still retaining the properties
of the original algorithm for small disks.
The filesystem implementation is divided into policy routines and
implementation routines. The policy routines can be changed in any
way desired without risk of corrupting the filesystem. The policy
requests are handled by the implementation layer. If the policy
asks for an available resource, it is granted. But if it asks for
an already in-use resource, then the implementation will provide
an available one nearby the request. Thus it is impossible for a
policy to double allocate. This change is limited to the policy
implementation.
This change updates the ffs_dirpref() routine which is responsible
for selecting the cylinder group into which a new directory should
be placed. If we are near the root of the filesystem we aim to
spread them out as much as possible. As we descend deeper from the
root we cluster them closer together around their parent as we
expect them to be more closely interactive. Higher-level directories
like usr/src/sys and usr/src/bin should be separated while the
directories in these areas are more likely to be accessed together
so should be closer. And directories within commands or kernel
subsystems should be closer still.
We pick a range of cylinder groups around the cylinder group of the
directory in which we are being created. The size of the range for
our search is based on our depth from the root of our filesystem.
We then probe that range based on how many directories are already
present. The first new directory is at 1/2 (middle) of the range;
the second is in the first 1/4 of the range, then at 3/4, 1/8, 3/8,
5/8, 7/8, 1/16, 3/16, 5/16, etc.
It is desirable to store the depth of a directory in its on-disk
inode so that it is available when we need it. We add a new field
di_dirdepth to track the depth of each directory. Because there are
few spare fields left in the inode, we choose to share an existing
field in the inode rather than having one of our own. Specifically
we create a union with the di_freelink field. The di_freelink field
is used to track inodes that have been unlinked but remain referenced.
It is not needed until a rmdir(2) operation has been done on a
directory. At that point, the directory has no contents and even
if it is kept active as a current directory is no longer able to
have any new directories or files created in it. Thus the use of
di_dirdepth and di_freelink will never coincide.
Reported by: Timo Voelker
Reviewed by: kib
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39246
If a directory entry has an illegal inode number (less than zero
or greater than the last inode in the filesystem) the entry is removed.
If a directory '.' or '..' entry had an illegal inode number they
were being removed. Since fsck_ffs knows what the correct value is
for these two entries fix them rather deleting them.
Add much more extensive cylinder group checks and use them to be
more careful about rebuilding a cylinder group.
Check for out-of-range block numbers before trying to free them.
When a directory is deleted also remove its cache entry created
in pass1 so that later passes do not try to operate on a deleted
directory.
Check for ctime(3) returning NULL before trying to use its return.
When freeing a directory inode, do not try to interpret it as a
directory.
Reserve space in the inostatlist to have room to allocate a
lost+found directory.
If an invalid block number is found past the end of an inode simply
remove it rather than clearing and removing the inode.
Modernize the inoinfo structure to use queue(3) LIST rather than a
handrolled linked list implementation.
Reported by: Bob Prohaska, John-Mark Gurney, and Mark Millard
Tested by: Peter Holm
Reviewed by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38668
Use it instead of defining IN_RTLD by base sources that want P_OSREL_
defines in userspace, but are not rtld.
This allows to remove abuse of IN_RTLD from userspace.
Reviewed by: dchagin, markj, imp
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D38585
The mntopts(3) functions support operations associated with a mount
point. The main purpose of this commit is to document the mntopts(3)
functions that now appear in 18 utilities in the base system. See
mntopts(3) for the documentation details.
The getmntopts() function appeared in 4.4BSD. The build_iovec(),
build_iovec_argf(), free_iovec(), checkpath(), and rmslashes()
functions were added with nmount(8) in FreeBSD 5.0. The getmntpoint()
and chkdoreload() functions are being added in this commit.
These functions should be in a library but for historic reasons are
in a file in the sources for the mount(8) program. Thus, to access
them the following lines need to be added to the Makefile of the
program wanting to use them:
SRCS+= getmntopts.c
MOUNT= ${SRCTOP}/sbin/mount
CFLAGS+= -I${MOUNT}
.PATH: ${MOUNT}
Once these changes have been MFC'ed to 13 they may be made into
a library.
Reviewed by: kib, gbe
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37907
The kernel handles the managment of UFS/FFS snapshots. Since UFS/FFS
updates filesystem data (rather than always writing changes to new
locations like ZFS), the kernel must check every filesystem write
to see if the block being written is part of a snapshot. If it is
part of a snapshot, then the kernel must make a copy of the old
block value into a newly allocated block for the snapshot before
allowing the write to be done. Similarly, if a block is being freed,
the kernel must check to see if it is part of a snapshot and let
the snapshot claim the block rather than freeing it for future use.
When a snapshot is freed, its blocks need to be offered to older
snapshots and freed only if no older snapshots wish to claim them.
When snapshots were added to UFS/FFS they were integrated into soft
updates and just a small part of the management of snapshots needed
to be added to fsck_ffs(8) as soft updates minimized the set of
snapshot changes that might need correction. When journaling was
added to soft updates a much more complete knowledge of snapshots
needed to be added to fsck_ffs(8) for it to be able to properly
handle the filesystem changes that a journal rollback needs to do
(specifically the freeing and allocation of blocks). Since this
functionality was unavailable, the use of snapshots was disabled
when running with journaled soft updates.
This set of changes imports the kernel code for the management of
snapshots to fsck_ffs(8). With this code in place it will become
possible to enable snapshots when running with journalled soft
updates. The most immediate benefit will be the ability to use
snapshots to take consistent filesystem dumps on live filesystems.
Future work will be done to update fsck_ffs(8) to be able to use
snapshots to run in background on live filesystems running with
journaled soft updates.
Reviewed by: kib
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36491
It's not really useful in a jail or in a mdroot or even if a users
wants to do a full zfs machine.
Reviewed by: mckusick
Differential Revision: https://reviews.freebsd.org/D36227
When something was found wrong with an inode the error message
was always "UNKNOWN FILE TYPE". This error is now used only when
the file type field is wrong. Other errors have their own messages:
"BAD FILE SIZE", "NEGATIVE FILE SIZE", "BAD SPECIAL-FILE RDEV",
"INVALID DIRECT BLOCK", and "INVALID INDIRECT BLOCK".
More complete information about the inode is also provided.
Sponsored by: The FreeBSD Foundation
