jmarya/knowledge
jmarya/knowledge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

init

Browse source
This commit is contained in:
JMARyA 2023-12-04 11:02:23 +01:00
commit c5cd492449
Signed by: jmarya
GPG key ID: 901B2ADDF27C2263
475 changed files with 27928 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

105
technology/linux/filesystems/Btrfs.md Normal file
View file

@ -0,0 +1,105 @@
---
arch-wiki: https://wiki.archlinux.org/title/Btrfs
obj: filesystem
---
# Btrfs
Btrfs is a modern copy on write (CoW) filesystem for [Linux](../Linux.md) aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Btrfs is licensed under the GPL and open for contribution from anyone.
## Create the filesystem
**Single Device**:
``` shell
mkfs.btrfs -L _mylabel_ /dev/partition
```
**Multiple Devices (similiar to [RAID](RAID.md))**:
```shell
mkfs.btrfs -d single -m raid1 /dev/part1 /dev/part2 ...
```
## Subvolumes
Subvolumes are their own seperated namespaces and work like [ZFS](ZFS.md) Datasets. They can be mounted seperately. Each Btrfs file system has a top-level subvolume with ID 5.
**Create a subvolume**:
```shell
btrfs subvolume create /path/to/subvolume
```
**List suvolumes**:
```shell
btrfs subvolume list -p path
```
**Delete subvolume**:
```shell
btrfs subvolume delete /path/to/subvolume
```
## Using Swapfiles
Swapfiles can be used by disabling CoW:
```shell
chattr +C /path/to/swapsubvolume
```
**Use swapfile:**
```shell
touch /swapfile
chattr +C /swapfile
fallocate -l 8G /swapfile
chmod 0600 /swapfile
mkswap /swapfile
swapon /swapfile
```
## Filesystem Usage
**See Usage**:
```shell
btrfs filesystem usage /
```
**Available Space:**
```shell
btrfs filesystem df /
```
## Scrub
Btrfs can self heal itself or check for errors with scrubbing.
**Start:**
```shell
btrfs scrub start /
```
**See Status:**
```shell
btrfs scrub status /
```
## Balance
Rebalance the filesystem.
```shell
btrfs balance start --bg /
btrfs balance status /
```
## Snapshots
Btrfs can use snapshots
```shell
btrfs subvolume snapshot source [dest/]name
````
## Resizing
Btrfs can be resized.
**To extend the file system size to the maximum available size of the device:**
```shell
btrfs filesystem resize max /
```
**To extend the file system to a specific size:**
```shell
btrfs filesystem resize size /
```
**Add or remove size**
```shell
btrfs filesystem resize +size /
btrfs filesystem resize -size /
```

23
technology/linux/filesystems/Ext4.md Normal file
View file

@ -0,0 +1,23 @@
---
arch-wiki: "https://wiki.archlinux.org/title/Ext4"
obj: filesystem
---
# Ext4
The Extended Filesystem is a robust and currently default filesystem for [Linux](../Linux.md).
## Create Filesystem
```shell
mkfs.ext4 /dev/device
```
## Resize Filesystem:
```shell
resize2fs /dev/device
```
## Reserved Blocks
**Set reserved space to NUM% of space:**
```shell
tune2fs -m NUM /dev/device
```

13
technology/linux/filesystems/Filesystems.md Normal file
View file

@ -0,0 +1,13 @@
---
arch-wiki: https://wiki.archlinux.org/title/File_systems
tags: ["meta"]
obj: meta/collection
---
## Filesystems
- [ZFS](ZFS.md)
- [Btrfs](Btrfs.md)
- [Ext4](Ext4.md)
- [SquashFS](SquashFS.md)
- [MergerFS](MergerFS.md)
- [exFAT](exFAT.md)
- [SSHFS](SSHFS.md)

42
technology/linux/filesystems/LUKS.md Normal file
View file

@ -0,0 +1,42 @@
---
aliases: ["dm-crypt"]
obj: filesystem
---
# LUKS
a transparent disk encryption subsystem in the [Linux](../Linux.md) kernel. It is implemented as a device mapper target and may be stacked on top of other device mapper transformations. It can thus encrypt whole disks (including removable media), partitions, software [RAID](RAID.md) volumes, logical volumes, as well as files. It appears as a block device, which can be used to back file systems, swap or as an [LVM](LVM.md) physical volume
## Usage
Initialize device:
```shell
cryptsetup luksFormat device
```
Open device:
```shell
cryptsetup open device dmname
```
Device will be mapped in `/dev/mapper/dmname`
Close device:
```shell
cryptsetup close dmname
```
Show device info:
```shell
cryptsetup luksDump device
```
Resize device:
```shell
cryptsetup resize device
```
## Crypttab
Define crypto targets in `/etc/crypttab`
```
name underlying device passphrase cryptsetup options
backup /dev/sdb1 /key/backup.key
```
After devices are unlocked they can be mounted using fstab.

141
technology/linux/filesystems/LVM.md Normal file
View file

@ -0,0 +1,141 @@
---
obj: filesystem
---
# Logical Volume Management (LVM)
Logical Volume Management (LVM) is a method of managing disk drives in a way that is more flexible and scalable than traditional partitioning. LVM provides a layer of abstraction between the physical storage devices and the operating system, allowing for dynamic resizing and efficient management of storage space.
## Key Concepts
### 1. **Physical Volumes (PVs):**
- Physical volumes are the actual storage devices, such as hard drives or SSDs, that contribute storage space to the LVM.
### 2. **Volume Groups (VGs):**
- Volume groups are created by combining one or more physical volumes. They represent a pool of storage from which logical volumes can be allocated.
### 3. **Logical Volumes (LVs):**
- Logical volumes are created within volume groups. They act as flexible partitions that can be resized dynamically. They are what the operating system interacts with as if they were physical partitions.
## Advantages of LVM
1. **Dynamic Volume Resizing:**
- LVM allows for the resizing of logical volumes while the file systems they contain are mounted and active.
2. **Snapshot and Backup:**
- LVM supports the creation of snapshots, which are point-in-time copies of a logical volume. These snapshots can be used for backup purposes.
3. **Improved Storage Management:**
- LVM simplifies storage management by providing a flexible and dynamic way to allocate and manage disk space.
4. **Striping and Mirroring:**
- LVM supports data striping (improves performance) and mirroring (provides redundancy) for logical volumes.
## Usage
### `pvcreate`
Initialize physical volume(s) for use by LVM
Usage: `pvcreate /dev/device`
### `pvdisplay`
Display various attributes of physical volume(s)
Usage: `pvdisplay /dev/device`
### `pvscan`
List all physical volumes
Usage: `pvscan`
### `vgcreate`
Create volume groups combining multiple mass-storage devices.
Usage: `vgcreate vg1 /dev/sda1`
### `vgdisplay`
Display information about Logical Volume Manager (LVM) volume groups.
Usage: `vgdisplay`
### `vgextend`
Add physical volumes to a volume group
Usage: `vgextend VG PV`
### `vgreduce`
Remove physical volume(s) from a volume group
Usage: `vgreduce VG PV`
### `lvcreate`
Creates a logical volume in an existing volume group.
Usage:
```shell
# Create a logical volume of 10 gigabytes in the volume group vg1:
lvcreate -L 10G vg1
# Create a 1500 megabyte linear logical volume named mylv in the volume group vg1:
lvcreate -L 1500 -n mylv vg1
# Create a logical volume called mylv that uses 60% of the total space in the volume group vg1:
lvcreate -l 60%VG -n mylv vg1
# Create a logical volume called mylv that uses all of the unallocated space in the volume group vg1:
lvcreate -l 100%FREE -n mylv vg1
```
### `lvdisplay`
Display information about Logical Volume Manager (LVM) logical volumes.
Usage:
```shell
# Display information about all logical volumes:
lvdisplay
# Display information about all logical volumes in volume group vg1:
lvdisplay vg1
# Display information about logical volume lv1 in volume group vg1:
lvdisplay vg1/lv1
```
### `lvextend`
Extends a logical volume in an existing volume group.
Usage:
```shell
# Extend volume in volume mylv in group vg0
# (defined by volume path /dev/vg0/mylv)
# to 12 gigabyte:
lvextend -L 12G /dev/vg0/mylv
# Extend volume in volume mylv in group vg0 by 1 gigabyte:
lvextend -L +1G /dev/vg0/mylv
# Extend volume in volume mylv in group vg0 to use all of the unallocated space in the volume group vg0:
lvextend -l +100%FREE /dev/vg0/mylv
# Extend ext4 filesystem after changing a logical volume (takes volume path as parameter):
ext2resize /dev/vg0/mylv
# Extend btrfs filesystem after changing a logical volume (takes the mount point as parameter, not the volume path):
btrfs filesystem resize max /mylv
# Extend xfs filesystem after changing a logical volume (takes the mount point as parameter, not the volume path):
xfs_growfs /mylv
```
### `lvreduce`
Reduce the size of a logical volume.
Usage: `lvreduce --size 120G logical_volume`
### `lvremove`
Remove one or more logical volumes.
Usage: `lvremove volume_group/logical_volume`
### `lvrename`
Rename a logical volume
Usage: `lvrename VG LV LV_new`
### `lvresize`
Change the size of a logical volume.
```shell
# Change the size of a logical volume to 120 GB:
lvresize --size 120G volume_group/logical_volume
# Extend the size of a logical volume as well as the underlying filesystem by 120 GB:
lvresize --size +120G --resizefs volume_group/logical_volume
# Extend the size of a logical volume to 100% of the free physical volume space:
lvresize --size 100%FREE volume_group/logical_volume
# Reduce the size of a logical volume as well as the underlying filesystem by 120 GB:
lvresize --size -120G --resizefs volume_group/logical_volume
```
### `lvscan`
List all logical volumes in all volume groups
Usage: `lvscan`

179
technology/linux/filesystems/MergerFS Tools.md Normal file
View file

@ -0,0 +1,179 @@
---
obj: application
repo: https://github.com/trapexit/mergerfs-tools
---
# [MergerFS](MergerFS.md) Tools
## mergerfs.ctl
A wrapper around mergerfs.
```shell
$ mergerfs.ctl -h
usage: mergerfs.ctl [-h] [-m MOUNT] {add,remove,list,get,set,info} ...
positional arguments:
{add,remove,list,get,set,info}
optional arguments:
-h, --help show this help message and exit
-m MOUNT, --mount MOUNT
mergerfs mount to act on
$ mergerfs.ctl info
- mount: /storage
version: 2.14.0
pid: 1234
srcmounts:
- /mnt/drive0
- /mnt/drive1
$ mergerfs.ctl -m /storage add path /mnt/drive2
$ mergerfs.ctl info
- mount: /storage
version: 2.14.0
pid: 1234
srcmounts:
- /mnt/drive0
- /mnt/drive1
- /mnt/drive2
```
## mergerfs.fsck
Audits permissions and ownership of files and directories in a mergerfs mount and allows for manual and automatic fixing of them.
```shell
$ mergerfs.fsck -h
usage: mergerfs.fsck [-h] [-v] [-s] [-f {manual,newest,nonroot}] dir
audit a mergerfs mount for inconsistencies
positional arguments:
dir starting directory
optional arguments:
-h, --help show this help message and exit
-v, --verbose print details of audit item
-s, --size only consider if the size is the same
-f {manual,newest,nonroot}, --fix {manual,newest,nonroot}
fix policy
$ mergerfs.fsck -v -f manual /path/to/dir
```
## mergerfs.dup
Duplicates files & directories across branches in a pool. The file selected for duplication is picked by the `dup` option. Files will be copied to drives with the most free space. Deleted from others if `prune` is enabled.
```shell
usage: mergerfs.dup [<options>] <dir>
Duplicate files & directories across multiple drives in a pool.
Will print out commands for inspection and out of band use.
positional arguments:
dir starting directory
optional arguments:
-c, --count= Number of copies to create. (default: 2)
-d, --dup= Which file (if more than one exists) to choose to
duplicate. Each one falls back to `mergerfs` if
all files have the same value. (default: newest)
* newest : file with largest mtime
* oldest : file with smallest mtime
* smallest : file with smallest size
* largest : file with largest size
* mergerfs : file chosen by mergerfs' getattr
-p, --prune Remove files above `count`. Without this enabled
it will update all existing files.
-e, --execute Execute `rsync` and `rm` commands. Not just
print them.
-I, --include= fnmatch compatible filter to include files.
Can be used multiple times.
-E, --exclude= fnmatch compatible filter to exclude files.
Can be used multiple times.
```
## mergerfs.dedup
Finds and removes duplicate files across mergerfs pool's branches. Use the `ignore`, `dedup`, and `strict` options to target specific use cases.
```shell
usage: mergerfs.dedup [<options>] <dir>
Remove duplicate files across branches of a mergerfs pool. Provides
multiple algos for determining which file to keep and what to skip.
positional arguments:
dir Starting directory
optional arguments:
-v, --verbose Once to print `rm` commands
Twice for status info
Three for file info
-i, --ignore= Ignore files if... (default: none)
* same-size : have the same size
* different-size : have different sizes
* same-time : have the same mtime
* different-time : have different mtimes
* same-hash : have the same md5sum
* different-hash : have different md5sums
-d, --dedup= What file to *keep* (default: newest)
* manual : ask user
* oldest : file with smallest mtime
* newest : file with largest mtime
* largest : file with largest size
* smallest : file with smallest size
* mostfreespace : file on drive with most free space
-s, --strict Skip dedup if all files have same value.
Only applies to oldest, newest, largest, smallest.
-e, --execute Will not perform file removal without this.
-I, --include= fnmatch compatible filter to include files.
Can be used multiple times.
-E, --exclude= fnmatch compatible filter to exclude files.
Can be used multiple times.
# mergerfs.dedup /path/to/dir
# Total savings: 10.0GB
# mergerfs.dedup -e -d newest /path/to/dir
mergerfs.dedup -v -d newest /media/tmp/test
rm -vf /mnt/drive0/test/foo
rm -vf /mnt/drive1/test/foo
rm -vf /mnt/drive2/test/foo
rm -vf /mnt/drive3/test/foo
# Total savings: 10.0B
```
## mergerfs.balance
Will move files from the most filled drive (percentage wise) to the least filled drive. Will do so till the most and least filled drives come within a user defined percentage range (defaults to 2%).
```shell
usage: mergerfs.balance [-h] [-p PERCENTAGE] [-i INCLUDE] [-e EXCLUDE]
[-I INCLUDEPATH] [-E EXCLUDEPATH] [-s EXCLUDELT]
[-S EXCLUDEGT]
dir
balance files on a mergerfs mount based on percentage drive filled
positional arguments:
dir starting directory
optional arguments:
-h, --help show this help message and exit
-p PERCENTAGE percentage range of freespace (default 2.0)
-i INCLUDE, --include INCLUDE
fnmatch compatible file filter (can use multiple
times)
-e EXCLUDE, --exclude EXCLUDE
fnmatch compatible file filter (can use multiple
times)
-I INCLUDEPATH, --include-path INCLUDEPATH
fnmatch compatible path filter (can use multiple
times)
-E EXCLUDEPATH, --exclude-path EXCLUDEPATH
fnmatch compatible path filter (can use multiple
times)
-s EXCLUDELT exclude files smaller than <int>[KMGT] bytes
-S EXCLUDEGT exclude files larger than <int>[KMGT] bytes
# mergerfs.balance /media
from: /mnt/drive1/foo/bar
to: /mnt/drive2/foo/bar
rsync ...
```

192
technology/linux/filesystems/MergerFS.md Normal file
View file

@ -0,0 +1,192 @@
---
repo: https://github.com/trapexit/mergerfs
obj: filesystem
---
# MergerFS
**mergerfs** is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to **mhddfs**, **unionfs**, and **aufs**.
Usage: `mergerfs -o<options> <branches> <mountpoint>`
mergerfs logically merges multiple paths together. Think a union of sets. The file/s or directory/s acted on or presented through mergerfs are based on the policy chosen for that particular action
See [MergerFS Tools](MergerFS%20Tools.md) for managing mergerfs.
## Terminology
- branch: A base path used in the pool.
- pool: The mergerfs mount. The union of the branches.
- relative path: The path in the pool relative to the branch and mount.
- function: A filesystem call (open, unlink, create, getattr, rmdir, etc.)
- category: A collection of functions based on basic behavior (action, create, search).
- policy: The algorithm used to select a file when performing a function.
- path preservation: Aspect of some policies which includes checking the path for which a file would be created.
## Basic Setup
Command Line:
```shell
mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media
```
/etc/fstab:
```
/mnt/hdd0:/mnt/hdd1 /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0
```
[Systemd](../Systemd.md):
```toml
[Unit]
Description=mergerfs service
[Service]
Type=simple
KillMode=none
ExecStart=/usr/bin/mergerfs \
-f \
-o cache.files=partial \
-o dropcacheonclose=true \
-o category.create=mfs \
/mnt/hdd0:/mnt/hdd1 \
/media
ExecStop=/bin/fusermount -uz /media
Restart=on-failure
[Install]
WantedBy=default.target
```
## Options
These options are the same regardless of whether you use them with the `mergerfs` commandline program, in fstab, or in a config file.
| mount option | description |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| config | Path to a config file. Same arguments as below in key=val / ini style format. |
| branches | Colon delimited list of branches. |
| minfreespace=SIZE | The minimum space value used for creation policies. Can be overridden by branch specific option. Understands 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte respectively. (default: 4G) |
| moveonenospc=BOOL\|POLICY | When enabled if a write fails with ENOSPC (no space left on device) or EDQUOT (disk quota exceeded) the policy selected will run to find a new location for the file. An attempt to move the file to that branch will occur (keeping all metadata possible) and if successful the original is unlinked and the write retried. (default: false, true = mfs) |
| inodecalc=passthrough\|path-hash\|devino-hash\|hybrid-hash | Selects the inode calculation algorithm. (default: hybrid-hash) |
| dropcacheonclose=BOOL | When a file is requested to be closed call posix_fadvise on it first to instruct the kernel that we no longer need the data and it can drop its cache. Recommended when cache.files=partial\|full\|auto-full\|per-process to limit double caching. (default: false) |
| symlinkify=BOOL | When enabled and a file is not writable and its mtime or ctime is older than symlinkify_timeout files will be reported as symlinks to the original files. Please read more below before using. (default: false) |
| symlinkify_timeout=UINT | Time to wait, in seconds, to activate the symlinkify behavior. (default: 3600) |
| nullrw=BOOL | Turns reads and writes into no-ops. The request will succeed but do nothing. Useful for benchmarking mergerfs. (default: false) |
| lazy-umount-mountpoint=BOOL | mergerfs will attempt to "lazy umount" the mountpoint before mounting itself. Useful when performing live upgrades of mergerfs. (default: false) |
| ignorepponrename=BOOL | Ignore path preserving on rename. Typically rename and link act differently depending on the policy of create (read below). Enabling this will cause rename and link to always use the non-path preserving behavior. This means files, when renamed or linked, will stay on the same filesystem. (default: false) |
| security_capability=BOOL | If false return ENOATTR when xattr security.capability is queried. (default: true) |
| xattr=passthrough\|noattr\|nosys | Runtime control of xattrs. Default is to passthrough xattr requests. 'noattr' will short circuit as if nothing exists. 'nosys' will respond with ENOSYS as if xattrs are not supported or disabled. (default: passthrough) |
| link_cow=BOOL | When enabled if a regular file is opened which has a link count > 1 it will copy the file to a temporary file and rename over the original. Breaking the link and providing a basic copy-on-write function similar to cow-shell. (default: false) |
| statfs=base\|full | Controls how statfs works. 'base' means it will always use all branches in statfs calculations. 'full' is in effect path preserving and only includes branches where the path exists. (default: base) |
| statfs_ignore=none\|ro\|nc | 'ro' will cause statfs calculations to ignore available space for branches mounted or tagged as 'read-only' or 'no create'. 'nc' will ignore available space for branches tagged as 'no create'. (default: none) |
| nfsopenhack=off\|git\|all | A workaround for exporting mergerfs over NFS where there are issues with creating files for write while setting the mode to read-only. (default: off) |
| branches-mount-timeout=UINT | Number of seconds to wait at startup for branches to be a mount other than the mountpoint's filesystem. (default: 0) |
| follow-symlinks=never\|directory\|regular\|all | Turns symlinks into what they point to. (default: never) |
| link-exdev=passthrough\|rel-symlink\|abs-base-symlink\|abs-pool-symlink | When a link fails with EXDEV optionally create a symlink to the file instead. |
| rename-exdev=passthrough\|rel-symlink\|abs-symlink | When a rename fails with EXDEV optionally move the file to a special directory and symlink to it. |
| readahead=UINT | Set readahead (in kilobytes) for mergerfs and branches if greater than 0. (default: 0) |
| posix_acl=BOOL | Enable POSIX ACL support (if supported by kernel and underlying filesystem). (default: false) |
| async_read=BOOL | Perform reads asynchronously. If disabled or unavailable the kernel will ensure there is at most one pending read request per file handle and will attempt to order requests by offset. (default: true) |
| fuse_msg_size=UINT | Set the max number of pages per FUSE message. Only available on Linux >= 4.20 and ignored otherwise. (min: 1; max: 256; default: 256) |
| threads=INT | Number of threads to use. When used alone (process-thread-count=-1) it sets the number of threads reading and processing FUSE messages. When used together it sets the number of threads reading from FUSE. When set to zero it will attempt to discover and use the number of logical cores. If the thread count is set negative it will look up the number of cores then divide by the absolute value. ie. threads=-2 on an 8 core machine will result in 8 / 2 = 4 threads. There will always be at least 1 thread. If set to -1 in combination with process-thread-count then it will try to pick reasonable values based on CPU thread count. NOTE: higher number of threads increases parallelism but usually decreases throughput. (default: 0) |
| read-thread-count=INT | Alias for threads. |
| process-thread-count=INT | Enables separate thread pool to asynchronously process FUSE requests. In this mode read-thread-count refers to the number of threads reading FUSE messages which are dispatched to process threads. -1 means disabled otherwise acts like read-thread-count. (default: -1) |
| process-thread-queue-depth=UINT | Sets the number of requests any single process thread can have queued up at one time. Meaning the total memory usage of the queues is queue depth multiplied by the number of process threads plus read thread count. 0 sets the depth to the same as the process thread count. (default: 0) |
| pin-threads=STR | Selects a strategy to pin threads to CPUs (default: unset) |
| scheduling-priority=INT | Set mergerfs' scheduling priority. Valid values range from -20 to 19. See setpriority man page for more details. (default: -10) |
| fsname=STR | Sets the name of the filesystem as seen in mount, df, etc. Defaults to a list of the source paths concatenated together with the longest common prefix removed. |
| func.FUNC=POLICY | Sets the specific FUSE function's policy. See below for the list of value types. Example: func.getattr=newest |
| func.readdir=seq\|cosr\|cor\|cosr | INT\|cor:INT: Sets readdir policy. INT value sets the number of threads to use for concurrency. (default: seq) |
| category.action=POLICY | Sets policy of all FUSE functions in the action category. (default: epall) |
| category.create=POLICY | Sets policy of all FUSE functions in the create category. (default: epmfs) |
| category.search=POLICY | Sets policy of all FUSE functions in the search category. (default: ff) |
| cache.open=UINT | 'open' policy cache timeout in seconds. (default: 0) |
| cache.statfs=UINT | 'statfs' cache timeout in seconds. (default: 0) |
| cache.attr=UINT | File attribute cache timeout in seconds. (default: 1) |
| cache.entry=UINT | File name lookup cache timeout in seconds. (default: 1) |
| cache.negative_entry=UINT | Negative file name lookup cache timeout in seconds. (default: 0) |
| cache.files=libfuse\|off\|partial\|full\|auto-full\|per-process | File page caching mode (default: libfuse) |
| cache.files.process-names=LIST | A pipe \| delimited list of process comm names to enable page caching for when cache.files=per-process. (default: "rtorrent\|qbittorrent-nox") |
| cache.writeback=BOOL | Enable kernel writeback caching (default: false) |
| cache.symlinks=BOOL | Cache symlinks (if supported by kernel) (default: false) |
| cache.readdir=BOOL | Cache readdir (if supported by kernel) (default: false) |
| parallel-direct-writes=BOOL | Allow the kernel to dispatch multiple, parallel (non-extending) write requests for files opened with cache.files=per-process (if the process is not in process-names) or cache.files=off. (This requires kernel support, and was added in v6.2) |
| direct_io | deprecated - Bypass page cache. Use cache.files=off instead. (default: false) |
| kernel_cache | deprecated - Do not invalidate data cache on file open. Use cache.files=full instead. (default: false) |
| auto_cache | deprecated - Invalidate data cache if file mtime or size change. Use cache.files=auto-full instead. (default: false) |
| async_read | deprecated - Perform reads asynchronously. Use async_read=true instead. |
| sync_read | deprecated - Perform reads synchronously. Use async_read=false instead. |
| splice_read | deprecated - Does nothing. |
| splice_write | deprecated - Does nothing. |
| splice_move | deprecated - Does nothing. |
| allow_other | deprecated - mergerfs always sets this FUSE option as normal permissions can be used to limit access. |
| use_ino | deprecated - mergerfs should always control inode calculation so this is enabled all the time. |
### Value Types
| Type | Value |
| -------- | ------------------------------------------------------------------ |
| BOOL | 'true' \| 'false' |
| INT | \[MIN_INT,MAX_INT] |
| UINT | \[0,MAX_INT] |
| SIZE | 'NNM'; NN = INT, M = 'K' \| 'M' \| 'G' \| 'T' |
| STR | string (may refer to an enumerated value, see details of argument) |
| FUNC | filesystem function |
| CATEGORY | function category |
| POLICY | mergerfs function policy |
### branches
The 'branches' argument is a colon (':') delimited list of paths to be pooled together. It does not matter if the paths are on the same or different filesystems nor does it matter the filesystem type (within reason). Used and available space will not be duplicated for paths on the same filesystem and any features which aren't supported by the underlying filesystem (such as file attributes or extended attributes) will return the appropriate errors.
Branches currently have two options which can be set. A type which impacts whether or not the branch is included in a policy calculation and a individual minfreespace value. The values are set by prepending an `=` at the end of a branch designation and using commas as delimiters. Example: `/mnt/drive=RW,1234`
#### branch mode
- RW: (read/write) - Default behavior. Will be eligible in all policy categories.
- RO: (read-only) - Will be excluded from `create` and `action` policies. Same as a read-only mounted filesystem would be (though faster to process).
- NC: (no-create) - Will be excluded from `create` policies. You can't create on that branch but you can change or delete.
#### globbing
To make it easier to include multiple branches mergerfs supports [globbing](http://linux.die.net/man/7/glob). **The globbing tokens MUST be escaped when using via the shell else the shell itself will apply the glob itself.**
```
# mergerfs /mnt/hdd\*:/mnt/ssd /media
```
The above line will use all mount points in /mnt prefixed with **hdd** and **ssd**.
To have the pool mounted at boot or otherwise accessible from related tools use **/etc/fstab**.
```
# <file system> <mount point> <type> <options> <dump> <pass>
/mnt/hdd*:/mnt/ssd /media fuse.mergerfs minfreespace=16G 0 0
```
## Functions, Categories and Policies
The POSIX filesystem API is made up of a number of functions. **creat**, **stat**, **chown**, etc. For ease of configuration in mergerfs most of the core functions are grouped into 3 categories: **action**, **create**, and **search**. These functions and categories can be assigned a policy which dictates which branch is chosen when performing that function.
### Functions and their Category classifications
| Category | FUSE Functions |
| -------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| action | chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens |
| create | create, mkdir, mknod, symlink |
| search | access, getattr, getxattr, ioctl (directories), listxattr, open, readlink |
| N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range |
#### Policies
A policy is the algorithm used to choose a branch or branches for a function to work on or generally how the function behaves.
A policy's behavior differs, as mentioned above, based on the function it is used with. Sometimes it really might not make sense to even offer certain policies because they are literally the same as others but it makes things a bit more uniform.
| Policy | Description |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. |
| epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). |
| epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. |
| eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the branch with the least free space. |
| eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the branch with the least used space. |
| epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the branch with the most free space. |
| eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. |
| eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. |
| ff (first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found. |
| lfs (least free space) | Pick the branch with the least available free space. |
| lus (least used space) | Pick the branch with the least used space. |
| mfs (most free space) | Pick the branch with the most available free space. |
| msplfs (most shared path, least free space) | Like **eplfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
| msplus (most shared path, least used space) | Like **eplus** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
| mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
| msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
| newest | Pick the file / directory with the largest mtime. |
| pfrd (percentage free random distribution) | Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. |
| rand (random) | Calls **all** and then randomizes. Returns 1 branch. |

44
technology/linux/filesystems/RAID.md Normal file
View file

@ -0,0 +1,44 @@
---
arch-wiki: https://wiki.archlinux.org/title/RAID
obj: concept
---
# RAID
Redundant Array of Independent Disks (RAID) is a storage technology that combines multiple disk drive components (typically disk drives or partitions thereof) into a logical unit. Depending on the RAID implementation, this logical unit can be a file system or an additional transparent layer that can hold several partitions. Data is distributed across the drives in one of several ways called RAID levels, depending on the level of redundancy and performance required. The RAID level chosen can thus prevent data loss in the event of a hard disk failure, increase performance or be a combination of both.
## RAID Levels
### RAID 0
Uses striping to combine disks. Even though it _does not provide redundancy_, it is still considered RAID. It does, however, _provide a big speed benefit_. If the speed increase is worth the possibility of data loss, choose this RAID level. On a server, RAID 1 and RAID 5 arrays are more appropriate. The size of a RAID 0 array block device is the size of the smallest component partition times the number of component partitions.
### RAID 1
The most straightforward RAID level: straight mirroring. As with other RAID levels, it only makes sense if the partitions are on different physical disk drives. If one of those drives fails, the block device provided by the RAID array will continue to function as normal. The example will be using RAID 1 for everything except swap and temporary data. Please note that with a software implementation, the RAID 1 level is the only option for the boot partition, because bootloaders reading the boot partition do not understand RAID, but a RAID 1 component partition can be read as a normal partition. The size of a RAID 1 array block device is the size of the smallest component partition.
### RAID 5
Requires 3 or more physical drives, and provides the redundancy of RAID 1 combined with the speed and size benefits of RAID 0. RAID 5 uses striping, like RAID 0, but also stores parity blocks _distributed across each member disk_. In the event of a failed disk, these parity blocks are used to reconstruct the data on a replacement disk. RAID 5 can withstand the loss of one member disk.
### RAID 6
Requires 4 or more physical drives, and provides the benefits of RAID 5 but with security against two drive failures. RAID 6 also uses striping, like RAID 5, but stores two distinct parity blocks _distributed across each member disk_. In the event of a failed disk, these parity blocks are used to reconstruct the data on a replacement disk. RAID 6 can withstand the loss of two member disks. The robustness against unrecoverable read errors is somewhat better, because the array still has parity blocks when rebuilding from a single failed drive. However, given the overhead, RAID 6 is costly and in most settings RAID 10 in far2 layout (see below) provides better speed benefits and robustness, and is therefore preferred.
## Using RAID
Software RAID is the easiest implementation as it does not rely on obscure proprietary firmware and software to be used. The array is managed by the operating system either by:
- an abstraction layer (e.g. mdadm);
- a logical volume manager (e.g. [LVM](LVM.md));
- a component of a file system (e.g. [ZFS](ZFS.md), [Btrfs](Btrfs.md))
## Installation
_mdadm_ is used for administering pure software RAID using plain block devices: the underlying hardware does not provide any RAID logic, just a supply of disks. _mdadm_ will work with any collection of block devices. Even if unusual. For example, one can thus make a RAID array from a collection of thumb drives.
**Erase disks**
```shell
mdadm --misc --zero-superblock /dev/drive
```
**The following example shows building a 2-device RAID1 array:**
```shell
mdadm --create --verbose --level=1 --metadata=1.2 --raid-devices=2 /dev/md/MyRAID1Array /dev/sdb1 /dev/sdc1
```
**The following example shows building a RAID5 array with 4 active devices and 1 spare device:**
```shell
mdadm --create --verbose --level=5 --metadata=1.2 --chunk=256 --raid-devices=4 /dev/md/MyRAID5Array /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 --spare-devices=1 /dev/sdf1
```

26
technology/linux/filesystems/SSHFS.md Normal file
View file

@ -0,0 +1,26 @@
---
arch-wiki: https://wiki.archlinux.org/title/SSHFS
obj: filesystem
---
# SSHFS
SSHFS is a FUSE Filesystem based on the [SSH](../../applications/SSH.md) and SFTP Protocol.
## Mounting
```shell
sshfs [user@]host:[dir] mountpoint [options]
```
## Automount
**fstab entry:**
```
user@host:/remote/path /local/path fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/user/.ssh/id_rsa,allow_other,reconnect 0 0
```
## Options
```
-p [PORT] - SSH Port
-C - Compression
-o reconnect - reconnect
-o idmap=TYPE - TYPE = none, user, file | Map UID and GIDs
-o follow_symlinks - Symlinks
```

25
technology/linux/filesystems/SquashFS.md Normal file
View file

@ -0,0 +1,25 @@
---
obj: filesystem
---
# SquashFS
**Squashfs** is a compressed read-only [file system](Filesystems.md) for [Linux](../Linux.md).
## Creation
```shell
mksquashfs source1 source2 ... destination [options]
```
**Use alternate compressions:**
```shell
mksquashfs source1 destination -comp <compression>
```
**Highly compressed:**
```shell
mksquashfs source1 destination -comp xz -Xdict-size 100%
```
## Extraction
```shell
unsquashfs [options] target [files/directories to extract]
```

177
technology/linux/filesystems/ZFS.md Normal file
View file

@ -0,0 +1,177 @@
---
arch-wiki: https://wiki.archlinux.org/title/ZFS
website: https://openzfs.org/wiki/Main_Page
repo: https://github.com/openzfs/zfs
obj: filesystem
---
#refactor
# ZFS
[ZFS](https://en.wikipedia.org/wiki/ZFS "wikipedia:ZFS") is an advanced filesystem.
## ZPool
**Create a pool***
```shell
zpool create -f -m <mount> <pool> [raidz(2|3)|mirror] <ids>
```
**Pool Status**
```shell
zpool status -v
```
**Import Pool**
```shell
zpool import -l -d /dev/disk/by-id <pool> # Import pool
zpool import -a # Import all available pools
zpool import -a -f # Import all available pools with force option
```
**Export Pool**
```shell
zpool export <pool>
```
**Extend Pool**
```shell
zpool add <pool> <device-id>
```
**Destroy pool / dataset**
```shell
zpool destroy <pool>
zfs destroy <pool>/<dataset>
```
**Rename a pool**
```shell
zpool export oldname
zpool import oldname newname
```
**Upgrade pool**
```shell
zpool upgrade <pool>
```
### Automount
Specify this for `zfs-import-cache.service` to pick up the pool
```shell
zpool set cachefile=/etc/zfs/zpool.cache <pool>
```
## Datasets
### Creation
**Create Dataset**
```shell
zfs create <nameofzpool>/<nameofdataset>
```
**Create encrypted dataset:**
```shell
dd if=/dev/random of=/path/to/key bs=1 count=32 # Generate key
zfs create -o encryption=on -o keyformat=raw -o keylocation=file://$KEYLOCATION "$DATASET"
```
**Load all encryption keys**
```shell
zfs load-key -a
```
**Load all keys at boot with systemd**
__/etc/systemd/system/zfs-load-key.service__:
``` ini
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/zfs load-key -a
StandardInput=tty-force
[Install]
WantedBy=zfs-mount.service
```
### Snapshots
**Create snapshot:**
```shell
zfs snapshot <pool>/<dataset>@<snapshot>
```
**Send and receive:**
```shell
zfs send -v -w <pool>/<dataset>@<snapshot> | zfs recv <pool>/<newdataset>
```
### Properties
**Get properties:**
```shell
zfs get <property> <pool>/<dataset>
```
**Set properties**
```shell
zfs set <property>=<value> <pool>/<dataset>
```
### Available Properties
- `compressratio` : Shows information on compression
- `creation` : Get creation date
- `mounted` : Status on mount
- `used` : Information on usage
- `canmount` : Forbid or enable mounting
- `casesensitivity` : Choose sensitivity
- `compression` : Choose compression method
- `encryption` : Encryption Information
- `mountpoint` : Specify the mountpoint
- `quota` : Filesystem Quota
- `readonly` : Readonly or Readwrite
- `reservation` : Reserved Space
### Scrub
Like [Btrfs](Btrfs.md) ZFS can heal itself with scrubbing
**Start a scrub:**
```shell
zpool scrub <pool>
```
**Cancel a scrub:**
```shell
zpool scrub -s <pool>
```
**Autoscrubbing with systemd:**
__/etc/systemd/system/zfs-scrub@.timer__
```ini
[Unit]
Description=Monthly zpool scrub on %i
[Timer]
OnCalendar=monthly
AccuracySec=1h
Persistent=true
[Install]
WantedBy=multi-user.target
```
__/etc/systemd/system/zfs-scrub@.service__
```ini
[Unit]
Description=zpool scrub on %i
[Service]
Nice=19
IOSchedulingClass=idle
KillSignal=SIGINT
ExecStart=/usr/bin/zpool scrub %i
[Install]
WantedBy=multi-user.target
```

9
technology/linux/filesystems/exFAT.md Normal file
View file

@ -0,0 +1,9 @@
---
obj: filesystem
---
# exFAT
exFAT (Extensible File Allocation Table) is a file system introduced by Microsoft in 2006 and optimized for flash memory such as USB flash drives and SD cards. exFAT was proprietary until 28 August 2019, when Microsoft published its specification. Microsoft owns patents on several elements of its design.
exFAT can be used where NTFS is not a feasible solution (due to data-structure overhead), but a greater file-size limit than the standard FAT32 file system (i.e. 4 GB) is required.
exFAT has been adopted by the SD Association as the default file system for SDXC cards larger than 32 GB.