📜 Website Archive
- Rust 98.5%
- Nix 1.1%
- Shell 0.4%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
JMARyA
3a9554604c
Some checks failed
moira/threads moira/threads — definitions compiled
moira/flake/env.path.678f50d24738eb94@x86_64-linux env.path.678f50d24738eb94 — built
moira/flake/packages.containerImage@x86_64-linux packages.containerImage — queued
moira/flake/packages.default@x86_64-linux packages.default — queued
moira/flake/packages.dev-services@x86_64-linux packages.dev-services — built
moira/flake/packages.docs@x86_64-linux packages.docs — queued
moira/flake/packages.docsContainer@x86_64-linux packages.docsContainer — queued
moira/flake/packages.webarc@x86_64-linux packages.webarc — queued
moira/flake/checks.webarc@x86_64-linux checks.webarc — queued
moira/docs moira/docs — failed
moira/test moira/test — succeeded
moira/container moira/container — succeeded
All background work runs as jobs in Postgres instead of in-memory tasks and timer loops: indexing captures, catching up on older records (backfill), crawls, recapture schedules, garbage collection and favicons. - workers claim with FOR UPDATE SKIP LOCKED, per-kind concurrency limits, LISTEN/NOTIFY wake-up with polling fallback - retries with exponential backoff; failed jobs stay until retried or discarded; heartbeats hand abandoned jobs to another worker after 2 min - running jobs are released on SIGTERM so a rollout continues at once - keyed jobs are deduplicated; fresh captures are indexed ahead of backfill - /ops shows queues, indexer progress, running, retrying and failed jobs (retry/discard), periodic jobs and workers; refreshes every 3 s - metrics webarc_jobs, webarc_job_duration_seconds, webarc_job_failures_total - GET /api/v1/jobs - crawls index their own pages instead of waiting for the indexer Performance: - coverage queries only the scope's host (3.9 s → 0.23 s on the live data) - the retention plan is one streamed query instead of one per domain - index on records (domain, port) for lookups that include private records - overlong link URLs are skipped instead of failing a page's whole batch OpenSpec change jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
| .claude | ||
| docs | ||
| migrations | ||
| nix | ||
| openspec | ||
| src | ||
| tests | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| cog.toml | ||
| config.yml | ||
| devenv.lock | ||
| devenv.nix | ||
| docker-compose.yml | ||
| flake.lock | ||
| flake.nix | ||
| README.md | ||
| renovate.json | ||
| result | ||
| zensical.toml | ||
WebArc
webarc is a local-first web archiving system designed to capture, preserve, and replay HTTP content in an extensible way.
Unlike traditional crawlers that fetch pages in isolation, WebArc focuses on recording real HTTP traffic, storing it in an archive, and making that archive usable through multiple interfaces.
See the documentation.