network/rclone
network/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone synced 2024-09-29 20:55:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
rclone/fs
History
nielash 9b4b3033da fs/cache: fix parent not getting pinned when remote is a file 
Before this change, when cache.GetFn was called on a file rather than a
directory, two cache entries would be added (the file + its parent) but only one
of them would get pinned if the caller then called Pin(f). This left the other
one exposed to expiration if the ci.FsCacheExpireDuration was reached. This was
problematic because both entries point to the same Fs, and if one entry expires
while the other is pinned, the Shutdown method gets erroneously called on an Fs
that is still in use.

An example of the problem showed up in the Hasher backend, which uses the
Shutdown method to stop the bolt db used to store hashes. If a command was run
on a Hasher file (ex. `rclone md5sum --download hasher:somelargefile.zip`) and
hashing the file took longer than the --fs-cache-expire-duration (5m by default), the
bolt db was stopped before the hashing operation completed, resulting in an
error.

This change fixes the issue by ensuring that:
1. only one entry is added to the cache (the file's parent, not the file).
2. future lookups correctly find the entry regardless of whether they are called
	with the parent name or one of its children.
3. fs.ErrorIsFile is returned when (and only when) fsString points to a file
	(preserving the fix from 8d5bc7f28b).

Note that f.Root() should always point to the parent dir as of c69eb84573
2024-09-28 13:49:56 +01:00
..
accounting accounting: write the current bwlimit to the log on SIGUSR2 2024-09-26 18:01:18 +01:00
asyncreader Replace deprecated ioutil 2022-11-07 11:41:47 +00:00
cache fs/cache: fix parent not getting pinned when remote is a file 2024-09-28 13:49:56 +01:00
chunkedreader chunkedreader: add --vfs-read-chunk-streams to parallel read chunks 2024-08-14 21:13:09 +01:00
chunksize azureblob,b2,s3: fix chunksize calculations producing too many parts 2022-08-09 12:57:38 +01:00
config fs: fix setting stringArray config values from environment variables 2024-09-13 15:52:51 +01:00
dirtree dirtree: fix performance with large directories of directories and --fast-list 2023-07-03 14:09:21 +01:00
driveletter build: remove build constraint syntax for go 1.16 and older 2024-04-18 16:53:55 +02:00
filter build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
fserrors build: fix linting issues reported by running golangci-lint with different GOOS 2024-04-22 19:29:12 +02:00
fshttp build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
fspath build: remove build constraint syntax for go 1.16 and older 2024-04-18 16:53:55 +02:00
hash build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
list fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
log build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
logger fs/logger: make the tests deterministic 2024-06-12 16:39:30 +01:00
march march: Fix excessive parallelism when using --no-traverse 2023-11-20 17:36:31 +00:00
object operations: ensure SrcFsType is set correctly when using --metadata-mapper 2024-05-20 18:16:54 +01:00
operations build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
rc fs/cache: fix parent not getting pinned when remote is a file 2024-09-28 13:49:56 +01:00
sync local: add server-side copy with xattrs on macOS (part-fix #1710) 2024-08-15 15:36:38 +01:00
walk fs/test: fix erratic test 2024-07-15 11:09:54 +01:00
backend_config.go fs: fix typos found by codespell 2023-03-25 12:51:04 +01:00
backend_config_test.go
bits.go fs: fix docs for Bits 2023-10-23 15:43:55 +01:00
bits_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
bwtimetable.go fs: Allow semicolons as well as spaces in --bwlimit timetable parsing - fixes #7595 2024-07-17 11:04:01 +01:00
bwtimetable_test.go fs: Allow semicolons as well as spaces in --bwlimit timetable parsing - fixes #7595 2024-07-17 11:04:01 +01:00
config.go help: make help command output less distracting 2024-08-15 18:45:12 +02:00
config_list.go
config_list_test.go
config_test.go build: add lint ignore comment required for golangci-staticcheck in addition to stand-alone staticcheck 2023-03-26 14:28:15 +02:00
configmap.go fs: fix the defaults overriding the actual config 2024-07-15 11:09:54 +01:00
countsuffix.go build: fix gocritic lint issue dupbranchbody 2024-08-15 22:08:34 +01:00
countsuffix_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
cutoffmode.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
cutoffmode_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
daemon_other.go build: remove build constraint syntax for go 1.16 and older 2024-04-18 16:53:55 +02:00
daemon_unix.go build: remove build constraint syntax for go 1.16 and older 2024-04-18 16:53:55 +02:00
deletemode.go
dir.go fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
dir_wrapper.go combine: fix CopyDirMetadata error on upstream root 2024-03-07 11:09:07 +00:00
direntries.go
direntries_test.go
dump.go fs: implement --metadata-mapper to transform metatadata with a user supplied program 2023-11-18 17:49:35 +00:00
dump_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
enum.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
enum_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
features.go fs: add Directory Metadata flags for backends and interfaces 2024-02-27 10:56:19 +00:00
fingerprint.go
fingerprint_test.go mockfs: make it so it can be registered as an Fs 2023-04-28 14:31:05 +01:00
fs.go sync: update docs and error messages to reflect fixes to overlap checks 2022-07-13 16:04:53 +01:00
fs_test.go rc: add options/info call to enumerate options 2024-07-15 11:09:54 +01:00
log.go fs: add non-format variants of log functions to avoid non-constant format string warnings 2024-09-06 17:04:18 +01:00
log_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
metadata.go fs: allow Metadata calls to be called with Directory or Object 2024-02-27 10:56:19 +00:00
metadata_mapper_code.go build: remove build constraint syntax for go 1.16 and older 2024-04-18 16:53:55 +02:00
metadata_test.go fs: implement --metadata-mapper to transform metatadata with a user supplied program 2023-11-18 17:49:35 +00:00
mimetype.go dlna: set more correct mime type 2024-08-15 22:01:52 +01:00
mount_helper.go build: update logging statements to make json log work - fixes #6038 2024-09-06 17:04:18 +01:00
mount_helper_test.go
newfs.go fs: refactor fs.ConfigMap to take a prefix and Options rather than an fs.RegInfo 2024-07-15 11:09:53 +01:00
newfs_test.go mount: fix mount failure on macOS with on the fly remote 2023-06-23 14:12:03 +01:00
open_options.go fs: add MetadataAsOpenOptions 2024-03-07 14:44:45 +00:00
open_options_test.go fs: add --metadata-set flag to specify metadata for uploads 2022-06-29 14:29:36 +01:00
override.go fs: when creating new fs.OverrideRemotes don't layer overrides if not needed 2023-05-09 16:28:10 +01:00
override_dir.go fs: add OverrideDirectory for overriding path of directory 2023-05-14 11:22:57 +01:00
override_dir_test.go fs: add OverrideDirectory for overriding path of directory 2023-05-14 11:22:57 +01:00
override_test.go fs: move operations.NewOverrideRemote to fs.NewOverrideRemote 2022-11-21 08:02:09 +00:00
pacer.go pacer: fix b2 deadlock by defaulting max connections to unlimited 2023-10-01 12:30:26 +01:00
parseduration.go fs: improve JSON Unmarshalling for Duration 2024-03-13 18:08:59 +00:00
parseduration_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
parsetime.go rc: fix output of Time values in options/get 2023-05-04 15:04:11 +01:00
parsetime_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
registry.go fs: fix setting stringArray config values from environment variables 2024-09-13 15:52:51 +01:00
sizesuffix.go build: fix gocritic lint issue dupbranchbody 2024-08-15 22:08:34 +01:00
sizesuffix_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
terminalcolormode.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
terminalcolormode_test.go fs: re-implement CutoffMode, LogLevel, TerminalColorMode with Enum 2023-10-03 15:14:24 +01:00
tristate.go fs: fix tristate conversion to JSON 2023-03-27 16:17:49 +01:00
tristate_test.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
types.go fs: make Flagger and FlaggerNP interfaces public so we can test flags elsewhere 2024-07-15 11:09:54 +01:00
version.go
versioncheck.go build: update to go1.23rc1 and make go1.21 the minimum required version 2024-07-20 10:54:47 +01:00
versionsuffix.go
versiontag.go Start v1.69.0-DEV development 2024-09-08 17:22:19 +01:00