FastCDC chunking #26

Merged

jmarya merged 5 commits from fastcdc into main 2025-12-26 16:10:03 +00:00

Conversation 1 Commits 5 Files changed 9 +219 -36

jmarya commented 2025-12-16 20:01:41 +00:00

Owner

Copy link

This switches blob storage again to FastCDC for content-defined chunking, so small changes in large files don’t waste space. Old archives still work fine because reading hasn’t changed, but to actually get the new chunking benefits, the records should be copied to a new archive. That’s why there’s now a copy command, which duplicates all files and records from one archive to another. copy needs to be fixed up though, it's pretty basic. Future improvements could make copy even smarter, maybe only copy certain domains or paths over.

This switches blob storage again to FastCDC for content-defined chunking, so small changes in large files don’t waste space. Old archives still work fine because reading hasn’t changed, but to actually get the new chunking benefits, the records should be copied to a new archive. That’s why there’s now a `copy` command, which duplicates all files and records from one archive to another. `copy` needs to be fixed up though, it's pretty basic. Future improvements could make `copy` even smarter, maybe only copy certain domains or paths over.

jmarya added 1 commit 2025-12-16 20:01:45 +00:00

feat: fastcdc chunking 1be7edd2fd

jmarya added 2 commits 2025-12-21 02:14:35 +00:00

fix: verbose logging c5fcb7ec72

feat: copy cmd c2aef66b17

jmarya added 1 commit 2025-12-21 02:20:02 +00:00

feat: copy progress bar 75d39f3dfc

jmarya added 1 commit 2025-12-26 00:51:16 +00:00

perf: 8 async copies 1dd5283724

jmarya changed title from WIP: FastCDC chunking to FastCDC chunking 2025-12-26 00:53:47 +00:00

jmarya commented 2025-12-26 16:09:36 +00:00

Author

Owner

Copy link

The copy command is now implemented as a logical copy over all records stored in the archive.
Copying over the prod archive into the new fastcdc based archives yielded another massive improvement:
The original archive turned from roughly 117G to 44G, or roughly 37% the original size.

The copy command is now implemented as a logical copy over all records stored in the archive. Copying over the prod archive into the new fastcdc based archives yielded another massive improvement: The original archive turned from roughly 117G to 44G, or roughly 37% the original size.

jmarya merged commit 12ed3cbc8a into main 2025-12-26 16:10:03 +00:00

jmarya deleted branch fastcdc 2025-12-26 16:10:04 +00:00

jmarya referenced this pull request from a commit 2025-12-26 16:10:05 +00:00

Merge pull request 'FastCDC chunking' (#26) from fastcdc into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

jmarya/webarc!26

No description provided.