feat: add a persisted effect cache and an sccache toolchain recipe
commit
982f380feat: add a persisted effect cache and an sccache toolchain recipe
An effect’s sandbox is otherwise stateless build-output-wise beyond the
Sprite’s own filesystem; a cargo-build effect wants a build cache that
survives independent of that Sprite’s lifetime, git-native like everything
else here.
feat: add Effect::cache, restored into the sandbox before a run and snapshotted to refs/meta/cache/<name> after
feat: add --cache to git ents effect add
feat: add an sccache toolchain recipe (--from sccache) alongside rustup
docs: document checks.cache and the sccache recipe in checks.adoc, cli.adoc, meta-ref.adoc, conformance.adoc
Assisted-by: Claude:claude-sonnet-5
Reviews
No reviews of this commit yet — record a verdict below.
Start a review
crates/git-effect/Cargo.toml
@@ -11,12 +11,12 @@
git-toolchain = { workspace = true }
gix-hash = { workspace = true }
portable-pty = "0.9.0"
+tempfile = { workspace = true }
tokio = { workspace = true }
uuid = { workspace = true }
[dev-dependencies]
git-store = { workspace = true, features = ["test-support"] }
-tempfile = { workspace = true }
[lints]
workspace = true
docs/spec/checks.adoc
@@ -12,8 +12,8 @@
The configured checks for a repository MUST be stored one ref per check at
`refs/meta/effects/<name>`, each holding an optional shell command, an
optional sandbox image, an optional list of dependencies (names of sibling
-checks that must pass first), and an optional list of toolchain names
-(<<checks.toolchains>>).
+checks that must pass first), an optional list of toolchain names
+(<<checks.toolchains>>), and an optional cache name (<<checks.cache>>).
A check with no command is a composite: it runs nothing itself and derives
its outcome from its dependencies alone.
A check's definition living on a meta ref means a branch under check cannot
@@ -25,9 +25,9 @@
The dependency graph is fully static — no conditional edges, no runtime
expansion — and MUST be validated when the set is written: a dependency
naming no configured check, a duplicate or self edge, a check with neither
-a command nor dependencies, any dependency cycle, and a toolchain name that
-is not a valid ref-path segment MUST each be rejected before the set is
-stored.
+a command nor dependencies, any dependency cycle, and a toolchain or cache
+name that is not a valid ref-path segment MUST each be rejected before the
+set is stored.
A check that sets an image MUST be rejected until the sandbox can honor
one; the field is reserved in the format so honoring it later is not a data
migration.
@@ -44,6 +44,25 @@
configured.
--
+[role="requirement", id="checks.cache"]
+.Effect Cache
+--
+An effect's definition MAY name a persisted cache: a directory restored into
+the sandbox at a fixed path before the command runs and snapshotted back to
+`refs/meta/cache/<name>` after, unlike a toolchain (<<checks.toolchains>>),
+which is immutable and extracted once. The worker MUST restore a named
+cache's most recent snapshot (creating an empty directory when none exists
+yet) before any effect naming it runs, and MUST snapshot the directory back
+after the command finishes, regardless of its outcome, so a partial cache
+from a failed run is still available to the next one.
+The command MUST be able to find the restored directory's path via the
+`EFFECT_CACHE_DIR` environment variable.
+Persisting the cache through `git-store` rather than relying solely on the
+Sprite's own persistent filesystem means the cache survives a Sprite reset
+or migration, and is auditable and portable like every other meta-ref
+document.
+--
+
[role="requirement", id="checks.toolchains"]
.Check Toolchains
--
docs/spec/cli.adoc
@@ -66,7 +66,9 @@
(<<account.ref>>) and print the account's genesis identity
(<<account.genesis>>).
`git ents effect` MUST provide `list`, `add`, `remove`, `debug`, and `log`
-over the `refs/meta/effects/<name>` set (<<checks.definition>>).
+over the `refs/meta/effects/<name>` set (<<checks.definition>>). `add` MUST
+accept a `--cache <name>` naming a persisted cache (<<checks.cache>>) for
+the effect to restore before, and snapshot after, its command runs.
--
[role="requirement", id="cli.toolchains"]
@@ -80,14 +82,16 @@
`bin` MUST be non-empty; `license` MUST be a valid SPDX expression;
`version` MUST be valid semver; `platform` MUST be a valid target triple.
* `import --from <recipe>` — derive the fields from a local toolchain
- install via a named recipe (currently only `rustup`, selected further by
- `--spec <name>`, e.g. `stable`); explicitly passed fields override the
- recipe.
+ install, or a hosted release, via a named recipe (`rustup` or `sccache`,
+ selected further by `--spec <name>`, e.g. `stable` for `rustup`, a release
+ tag for `sccache`); explicitly passed fields override the recipe.
A recipe capable of pointing at a distributor's own hosted, hash-pinned
archives (rust-lang's dist tarballs, for `rustup`) records those as the
toolchain's `bin` by default, sparing the repository the bytes; `--embed`
forces importing the local install's actual `bin` bytes, as `import`
- without `--from` always does.
+ without `--from` always does. `sccache` has no hash manifest to pin a
+ hosted download against, so it always imports the downloaded bytes
+ directly regardless of `--embed`.
* `list` — render every toolchain on a remote with its `bin` (a tree id,
or a component count when hosted externally), version, platform, and
license.
docs/spec/conformance.adoc
@@ -43,6 +43,7 @@
|`account.genesis` |`git-ents/src/account.rs` (`genesis`) | _(exercised via `members add --account`; no dedicated unit test)_
|`config.ref` |`git-ents-core/src/config.rs` |`config::store_then_load_round_trips_the_config`, `default_when_the_config_ref_is_absent`
|`checks.definition` |`git-effect/src/definition.rs` (`load`/`store`, `EFFECTS_NS`, `order`) |`definition::tests::store_then_load_round_trips_an_effect`, `store_then_load_round_trips_image_and_depends`, `order_runs_dependencies_first`, `order_rejects_a_cycle`, `order_rejects_an_unknown_dependency`, `order_rejects_self_and_duplicate_edges`, `order_rejects_an_empty_effect`
+|`checks.cache` |`git-effect/src/cache.rs` (`CACHE_NS`, `cache_ref`, `cache_dir`, `restore`, `snapshot`); `git-effect/src/engine.rs` (`with_cache_env`) |`definition::tests::store_then_load_round_trips_cache`, `order_rejects_an_invalid_cache_name`; `engine::tests::with_cache_env_leaves_a_cache_free_command_unchanged`, `with_cache_env_exports_the_restored_directory` _(`restore`/`snapshot` themselves require a live Sprite; not covered by unit tests)_
|`checks.post-receive` |`git-effect/src/engine.rs` (`post_receive`, `enqueue`) | _(hook-level, exercised manually; `enqueue` writes tmp+rename)_
|`checks.worker` |`git-effect/src/engine.rs` (`worker`, `pending_jobs`, `drain_repo`, `process_job`, `derive_composite`) |`engine::tests::pending_jobs_groups_by_repo_and_drops_malformed`, `composite_status_derives_from_its_dependencies` _(dependency-ordered execution requires a live Sprite; `order` is unit-tested in `git-effect`)_
|`checks.sandbox` |`git-effect/src/engine.rs` (`ensure_auth`, `ensure_sprite`, `sync_tree`) | _(requires a live Sprite; not covered by unit tests)_
docs/spec/meta-ref.adoc
@@ -60,7 +60,8 @@
`revoked`, `issue-number`): `Store::load`/`store`.
A *named collection* is one-ref-per-item under a namespace prefix
(`member/<username>`, `toolchains/<name>`, `effects/<name>`,
- `results/<effect>/<short-oid>`) or a scalar-keyed map on a single ref
+ `results/<effect>/<short-oid>`, `cache/<name>`) or a scalar-keyed map on a
+ single ref
(`revoked/<fingerprint>`): `Store::load_item`/`store_item`,
`load_map`/`store_map`.
A *content-addressed* collection is keyed by the hash of its own content
crates/git-effect/src/definition.rs
@@ -52,6 +52,10 @@
/// activated on `PATH` before the command runs. Stored as `None` when
/// empty, like `depends`.
toolchains: Option<Vec<String>>,
+ /// Name of a persisted cache (`refs/meta/cache/<name>`) restored into the
+ /// sandbox before the command runs and snapshotted back after, or `None`
+ /// for an effect with no cache.
+ cache: Option<String>,
}
impl component::Collection for EffectBody {
@@ -77,6 +81,9 @@
pub depends: Vec<String>,
/// Names of toolchains activated on `PATH` before the command runs.
pub toolchains: Vec<String>,
+ /// Name of a persisted cache (`refs/meta/cache/<name>`) restored into the
+ /// sandbox before the command runs and snapshotted back after.
+ pub cache: Option<String>,
}
impl component::Component for Effect {
@@ -91,6 +98,7 @@
image: body.image,
depends: body.depends.unwrap_or_default(),
toolchains: body.toolchains.unwrap_or_default(),
+ cache: body.cache,
}
}
@@ -108,6 +116,7 @@
} else {
Some(effect.toolchains.clone())
},
+ cache: effect.cache.clone(),
}
}
@@ -151,7 +160,8 @@
/// now so supporting it later is not a data migration. Whether a named
/// toolchain actually exists is checked server-side at job time, not here —
/// unlike `depends`, `toolchains` cross-references a different ref
-/// namespace this function has no set of configured names to check against.
+/// namespace this function has no set of configured names to check against. A
+/// `cache` naming an invalid ref-path segment is rejected the same way.
///
/// ## Requirements
///
@@ -185,6 +195,14 @@
));
}
}
+ if let Some(cache) = &effect.cache
+ && !git_store::ref_segment_ok(cache)
+ {
+ return Err(format!(
+ "effect {} names an invalid cache {cache:?}",
+ effect.name
+ ));
+ }
let mut seen = std::collections::BTreeSet::new();
for dep in &effect.depends {
if !by_name.contains_key(dep.as_str()) {
@@ -259,6 +277,7 @@
image: None,
depends: Vec::new(),
toolchains: Vec::new(),
+ cache: None,
}
}
@@ -269,6 +288,7 @@
image: None,
depends: depends.iter().map(|dep| (*dep).to_owned()).collect(),
toolchains: Vec::new(),
+ cache: None,
}
}
@@ -453,4 +473,28 @@
assert_eq!(load(&repo, "build").unwrap(), Some(written));
let _ = std::fs::remove_dir_all(&repo);
}
+
+ // @relation(checks.cache, role=Verifies)
+ #[test]
+ fn order_rejects_an_invalid_cache_name() {
+ let effects = vec![Effect {
+ cache: Some("not/valid".to_owned()),
+ ..effect("build", "cargo build")
+ }];
+ let err = order(&effects).unwrap_err();
+ assert!(err.contains("invalid cache"), "unexpected error: {err}");
+ }
+
+ // @relation(checks.cache, role=Verifies)
+ #[test]
+ fn store_then_load_round_trips_cache() {
+ let repo = unique_repo();
+ let written = Effect {
+ cache: Some("sccache".to_owned()),
+ ..effect("build", "cargo build --workspace")
+ };
+ store(&repo, &written).unwrap();
+ assert_eq!(load(&repo, "build").unwrap(), Some(written));
+ let _ = std::fs::remove_dir_all(&repo);
+ }
}
crates/git-effect/src/engine.rs
@@ -34,6 +34,7 @@
use portable_pty::{CommandBuilder, PtySize, native_pty_system};
use tokio::sync::Mutex;
+use crate::cache;
use crate::definition::{self, Effect};
use crate::results::{self, RunOutcome, Status};
@@ -314,6 +315,19 @@
}
};
+ let mut cache_names: Vec<&str> = runnable
+ .iter()
+ .filter_map(|effect| effect.cache.as_deref())
+ .collect();
+ cache_names.sort_unstable();
+ cache_names.dedup();
+ for name in cache_names {
+ if let Err(e) = cache::restore(&job.repo, &sprite, name) {
+ finalize_error(&job.repo, job.new, &mut outcomes);
+ return Err(e);
+ }
+ }
+
for index in ordered {
let Some(effect) = runnable.get(index) else {
continue;
@@ -333,10 +347,16 @@
match &effect.command {
Some(command) if all_pass => {
let command = activate(command, &effect.toolchains, &toolchain_dirs);
+ let command = with_cache_env(&command, effect.cache.as_deref());
let key: LiveKey = (job.repo.clone(), job.new, effect.name.clone());
let buffer = live_start(live, key.clone());
let result = run_one(&sprite, &effect.name, &command, &buffer);
live_finish(live, &key);
+ if let Some(name) = &effect.cache
+ && let Err(e) = cache::snapshot(&job.repo, &sprite, name)
+ {
+ eprintln!("effects: could not snapshot cache {name}: {e}");
+ }
if let Some(outcome) = outcomes.get_mut(index) {
outcome.status = result.status;
outcome.duration_secs = Some(result.duration_secs);
@@ -679,6 +699,24 @@
format!("export PATH={path}:$PATH; {command}")
}
+/// Prefix `command` with an `EFFECT_CACHE_DIR` export pointing at `cache`'s
+/// restored directory (see [`cache::restore`]), so the command can point a
+/// tool (`sccache`, ...) at it; an effect with no cache is returned
+/// unchanged.
+///
+/// ## Requirements
+///
+/// @relation(checks.cache)
+fn with_cache_env(command: &str, cache: Option<&str>) -> String {
+ match cache {
+ Some(name) => format!(
+ "export EFFECT_CACHE_DIR={}; {command}",
+ cache::cache_dir(name)
+ ),
+ None => command.to_owned(),
+ }
+}
+
/// Extract the toolchain tree `tree` into the Sprite at
/// `{TOOLCHAINS_DIR}/<tree>`, once — a directory already there from an
/// earlier push is left alone rather than re-extracted, since the Sprite's
@@ -1045,6 +1083,21 @@
);
}
+ // @relation(checks.cache, role=Verifies)
+ #[test]
+ fn with_cache_env_leaves_a_cache_free_command_unchanged() {
+ assert_eq!(with_cache_env("cargo build", None), "cargo build");
+ }
+
+ // @relation(checks.cache, role=Verifies)
+ #[test]
+ fn with_cache_env_exports_the_restored_directory() {
+ assert_eq!(
+ with_cache_env("cargo build", Some("sccache")),
+ "export EFFECT_CACHE_DIR=/cache/sccache; cargo build"
+ );
+ }
+
// @relation(checks.worker, role=Verifies)
#[test]
fn composite_status_derives_from_its_dependencies() {
crates/git-effect/src/lib.rs
@@ -2,21 +2,24 @@
//! linting, versioning gates), decomposed into two pieces.
//!
//! [`definition`] holds an effect's static shape — its command, dependencies,
-//! and toolchains, one ref per effect at `refs/meta/effects/<name>`.
+//! toolchains, and cache, one ref per effect at `refs/meta/effects/<name>`.
//! [`results`] holds what running an effect against a commit produced, one
//! ref per effect per commit at `refs/meta/results/<effect>/<short-oid>`.
//! [`engine`] runs the effects a `post-receive` hook queues, in a Sprite
-//! sandbox, and records their outcomes through `results`.
+//! sandbox, and records their outcomes through `results`. [`cache`] persists
+//! a read-write cache directory an effect's command can build up across runs.
//!
//! This crate used to be `checks` (definitions in `git-ents-core`, execution
//! in `git-ents-server`) — see each module's migration note for the storage
//! rename that came with the split into its own crate.
+pub mod cache;
pub mod definition;
pub mod engine;
pub mod results;
#[cfg(test)]
mod testutil;
+pub use cache::{CACHE_NS, cache_dir, cache_ref};
pub use definition::{EFFECTS_NS, Effect, effect_ref, load, load_all, order, store};
pub use results::{CommitRuns, RESULTS_NS, Run, RunOutcome, Status, record, runs, update_run};
crates/git-ents/src/main.rs
@@ -223,6 +223,11 @@
/// before the command runs (repeatable).
#[facet(args::named, args::label = "TOOLCHAIN", default)]
toolchains: Vec<String>,
+ /// Persisted cache (`refs/meta/cache/<name>`) restored into the
+ /// sandbox before the command runs and snapshotted back after; the
+ /// command finds its path in `$EFFECT_CACHE_DIR`.
+ #[facet(args::named, default)]
+ cache: Option<String>,
},
/// Remove an effect from a remote's set and push the update.
Remove {
@@ -483,7 +488,8 @@
image,
depends,
toolchains,
- } => effect_add(name, command, image, depends, toolchains, remote),
+ cache,
+ } => effect_add(name, command, image, depends, toolchains, cache, remote),
EffectAction::Remove { name } => effect_remove(&name, remote),
EffectAction::Debug => effect_debug(remote),
EffectAction::Log => effect_log(remote),
@@ -1014,6 +1020,7 @@
image: Option<String>,
depends: Vec<String>,
toolchains: Vec<String>,
+ cache: Option<String>,
remote: &str,
) -> Result<(), String> {
let name = interactive::text_or(name, "Effect name")?;
@@ -1046,6 +1053,7 @@
image,
depends,
toolchains,
+ cache,
};
effects.push(effect.clone());
let _ordered = git_effect::order(&effects)?;
@@ -1662,6 +1670,25 @@
}
}
+/// GET `url`, returning the raw response bytes — [`http_get`]'s counterpart
+/// for a binary download (an archive, ...) rather than text.
+fn http_get_bytes(url: &str) -> Result<Vec<u8>, String> {
+ let mut response = ureq::get(url)
+ .config()
+ .http_status_as_error(false)
+ .build()
+ .call()
+ .map_err(|error| format!("GET {url} failed: {error}"))?;
+ let status = response.status();
+ if !status.is_success() {
+ return Err(format!("GET {url} returned {status}"));
+ }
+ response
+ .body_mut()
+ .read_to_vec()
+ .map_err(|error| format!("could not read the response: {error}"))
+}
+
/// POST an `application/x-www-form-urlencoded` `body` to `url`, returning the
/// response body, or its body text as the error on a non-2xx status.
fn http_post_form(url: &str, body: &str) -> Result<String, String> {
crates/git-ents/src/registry.rs
@@ -59,13 +59,23 @@
/// `resolve`'s own error message — a plain list rather than a trait registry,
/// since each recipe is one function with its own selector semantics, not a
/// uniform interface worth abstracting over for a list of one.
-pub const RECIPES: &[RecipeInfo] = &[RecipeInfo {
- name: "rustup",
- spec: "a channel or version, e.g. stable, nightly, 1.75.0",
- summary: "Resolves a rustup-managed toolchain via `rustc +<spec> -vV`; \
- by default points at rust-lang's own hosted, hash-pinned \
- component archives instead of importing local bytes.",
-}];
+pub const RECIPES: &[RecipeInfo] = &[
+ RecipeInfo {
+ name: "rustup",
+ spec: "a channel or version, e.g. stable, nightly, 1.75.0",
+ summary: "Resolves a rustup-managed toolchain via `rustc +<spec> -vV`; \
+ by default points at rust-lang's own hosted, hash-pinned \
+ component archives instead of importing local bytes.",
+ },
+ RecipeInfo {
+ name: "sccache",
+ spec: "a mozilla/sccache release tag, e.g. v0.8.2, or empty for latest",
+ summary: "Downloads a prebuilt sccache release for this machine's \
+ platform from GitHub and imports it directly (always \
+ embedded — the binary is small and GitHub publishes no \
+ hash manifest to pin a hosted download against).",
+ },
+];
/// Resolve `recipe` against `spec` (a recipe-specific selector, e.g. a
/// rustup toolchain name). See [`RECIPES`] for what's known.
@@ -81,6 +91,7 @@
pub fn resolve(recipe: &str, spec: &str, embed: bool) -> Result<Resolved, String> {
match recipe {
"rustup" => rustup(spec, embed),
+ "sccache" => sccache(spec),
other => Err(format!(
"unknown toolchain recipe {other:?} (known: {})",
RECIPES
@@ -172,6 +183,115 @@
})
}
+/// Resolve a prebuilt `sccache` release named `spec` (a GitHub release tag,
+/// e.g. `v0.8.2`), or the latest release when `spec` is empty, from
+/// `mozilla/sccache`'s GitHub releases — the archive matching this machine's
+/// own OS/architecture, the same "what's already usable here" convention
+/// [`rustup`] follows via its local `rustc`.
+///
+/// Unlike `rustup`, GitHub publishes no manifest of hashes alongside a
+/// release to pin a hosted download against, and the archive is a single
+/// ~15 MB binary, so this recipe always imports it directly (`Bin::Dir`)
+/// rather than offering [`Bin::Components`] — `embed` has nothing to toggle
+/// here.
+///
+/// ## Requirements
+///
+/// @relation(cli.toolchains)
+fn sccache(spec: &str) -> Result<Resolved, String> {
+ let tag = if spec.is_empty() {
+ latest_sccache_tag()?
+ } else {
+ spec.to_owned()
+ };
+ let version = tag.strip_prefix('v').unwrap_or(&tag).to_owned();
+ let target = sccache_target()?;
+ let url = format!(
+ "https://github.com/mozilla/sccache/releases/download/{tag}/sccache-{tag}-{target}.tar.gz"
+ );
+ let bytes = crate::http_get_bytes(&url)?;
+
+ let staging = tempfile::tempdir()
+ .map_err(|error| format!("could not create a staging directory: {error}"))?;
+ stage_sccache(&bytes, &tag, target, staging.path())?;
+
+ Ok(Resolved {
+ bin: Bin::Dir(staging.path().to_owned()),
+ src: None,
+ license: "MPL-2.0".to_owned(),
+ version,
+ platform: target.to_owned(),
+ _staging: Some(staging),
+ })
+}
+
+/// The latest `mozilla/sccache` release's tag name, from GitHub's "latest
+/// release" API.
+fn latest_sccache_tag() -> Result<String, String> {
+ let body = crate::http_get("https://api.github.com/repos/mozilla/sccache/releases/latest")?;
+ json_string_field(&body, "tag_name")
+ .ok_or_else(|| "GitHub's latest sccache release response carried no tag_name".to_owned())
+}
+
+/// Extract `"<key>": "value"` from a flat JSON response — a hand-rolled
+/// reader for the one field this recipe needs from GitHub's release API,
+/// rather than a full JSON parser for a format this is the only caller of.
+fn json_string_field(body: &str, key: &str) -> Option<String> {
+ let prefix = format!("\"{key}\": \"");
+ let rest = body.split_once(&prefix)?.1;
+ let end = rest.find('"')?;
+ rest.get(..end).map(str::to_owned)
+}
+
+/// This machine's OS/architecture as an `mozilla/sccache` release asset
+/// name's platform segment (e.g. `x86_64-unknown-linux-musl`).
+fn sccache_target() -> Result<&'static str, String> {
+ match (std::env::consts::OS, std::env::consts::ARCH) {
+ ("linux", "x86_64") => Ok("x86_64-unknown-linux-musl"),
+ ("linux", "aarch64") => Ok("aarch64-unknown-linux-musl"),
+ ("macos", "x86_64") => Ok("x86_64-apple-darwin"),
+ ("macos", "aarch64") => Ok("aarch64-apple-darwin"),
+ (os, arch) => Err(format!(
+ "the sccache recipe does not know a release asset for {os}/{arch}"
+ )),
+ }
+}
+
+/// Unpack `bytes` (an `sccache-<tag>-<target>.tar.gz` release archive) and
+/// copy its `sccache` binary to the top level of `staging`, executable —
+/// where `git-toolchain`'s `Bin::Embedded` extraction expects an embedded
+/// toolchain's binaries to live.
+fn stage_sccache(bytes: &[u8], tag: &str, target: &str, staging: &Path) -> Result<(), String> {
+ let scratch =
+ tempfile::tempdir().map_err(|error| format!("could not create a temp dir: {error}"))?;
+ let archive_path = scratch.path().join("sccache.tar.gz");
+ fs::write(&archive_path, bytes)
+ .map_err(|error| format!("could not write the downloaded archive: {error}"))?;
+ let status = Command::new("tar")
+ .arg("-xzf")
+ .arg(&archive_path)
+ .arg("-C")
+ .arg(scratch.path())
+ .status()
+ .map_err(|error| format!("could not run tar: {error}"))?;
+ if !status.success() {
+ return Err("could not extract the sccache archive".to_owned());
+ }
+ let binary = scratch
+ .path()
+ .join(format!("sccache-{tag}-{target}"))
+ .join("sccache");
+ let dest = staging.join("sccache");
+ fs::copy(&binary, &dest)
+ .map_err(|error| format!("could not copy {}: {error}", binary.display()))?;
+ let mut perms = fs::metadata(&dest)
+ .map_err(|error| format!("could not read {}: {error}", dest.display()))?
+ .permissions();
+ perms.set_mode(0o755);
+ fs::set_permissions(&dest, perms)
+ .map_err(|error| format!("could not set permissions on {}: {error}", dest.display()))
+}
+
/// The three components of rust-lang's channel manifest that together make
/// a working toolchain (compiler, cargo, and the target's standard library),
/// resolved for `target` against the manifest for `version` (or the shared
crates/git-effect/src/cache.rs
@@ -1,0 +1,190 @@
+//! A read-write cache directory persisted at `refs/meta/cache/<name>`,
+//! restored into the sandbox before an effect that names it runs and
+//! snapshotted back after — unlike a toolchain (`git-toolchain`, extract-once
+//! and immutable), a cache's contents change on every run, so it is written
+//! back rather than only ever read.
+//!
+//! The persisted snapshot survives independent of the Sprite's own lifetime:
+//! a Sprite reset or migration loses nothing a cache-using effect built up,
+//! since the cache lives in the object database under [`CACHE_NS`] like
+//! everything else `git-store` holds — not just on the Sprite's own
+//! persistent filesystem, which the toolchain extraction cache leans on
+//! instead.
+
+use std::io::Write as _;
+use std::path::Path;
+use std::process::{Command, Stdio};
+
+use gix_hash::ObjectId;
+
+/// The ref namespace holding cache snapshots, one ref per cache:
+/// `refs/meta/cache/<name>`.
+pub const CACHE_NS: &str = "refs/meta/cache";
+
+/// The ref holding the cache named `name`.
+#[must_use]
+pub fn cache_ref(name: &str) -> String {
+ format!("{CACHE_NS}/{name}")
+}
+
+/// Where cache `name` is restored inside the sandbox, exported to an
+/// effect's command as `$EFFECT_CACHE_DIR`.
+#[must_use]
+pub fn cache_dir(name: &str) -> String {
+ format!("/cache/{name}")
+}
+
+/// Restore `name`'s persisted snapshot (if any) into the sandbox at
+/// [`cache_dir`], so an effect using it picks up where the last run against
+/// this cache left off. The directory is created even when there is no prior
+/// snapshot, so the tool populating it (sccache, ...) always finds it there
+/// on a cold start.
+///
+/// ## Requirements
+///
+/// @relation(checks.cache)
+pub fn restore(repo: &Path, sprite: &str, name: &str) -> Result<(), String> {
+ let dir = cache_dir(name);
+ let mkdir = Command::new("sprite")
+ .args([
+ "exec",
+ "-s",
+ sprite,
+ "--",
+ "sh",
+ "-c",
+ &format!("mkdir -p {dir}"),
+ ])
+ .status()
+ .map_err(|e| format!("could not run the sprite CLI: {e}"))?;
+ if !mkdir.success() {
+ return Err(format!(
+ "could not create cache directory {dir} in the sprite"
+ ));
+ }
+
+ let store = git_store::Store::open(repo).map_err(|e| format!("could not open store: {e}"))?;
+ let Ok(tree) = store.ref_tree(&cache_ref(name)) else {
+ // No snapshot yet — a fresh cache, populated by whatever the command runs.
+ return Ok(());
+ };
+
+ let archive = Command::new("git")
+ .arg("-C")
+ .arg(repo)
+ .args(["archive", "--format=tar", &tree.to_string()])
+ .output()
+ .map_err(|e| format!("could not run git archive: {e}"))?;
+ if !archive.status.success() {
+ return Err(format!("git archive failed for cache {name}"));
+ }
+
+ let mut child = Command::new("sprite")
+ .args([
+ "exec",
+ "-s",
+ sprite,
+ "--",
+ "sh",
+ "-c",
+ &format!("tar -x -C {dir}"),
+ ])
+ .stdin(Stdio::piped())
+ .spawn()
+ .map_err(|e| format!("could not run the sprite CLI: {e}"))?;
+ child
+ .stdin
+ .take()
+ .ok_or("sprite exec did not accept stdin")?
+ .write_all(&archive.stdout)
+ .map_err(|e| format!("could not stream the cache into the sprite: {e}"))?;
+ let status = child
+ .wait()
+ .map_err(|e| format!("sprite exec did not complete: {e}"))?;
+ if status.success() {
+ Ok(())
+ } else {
+ Err(format!("could not restore cache {name} in the sprite"))
+ }
+}
+
+/// Snapshot the sandbox's [`cache_dir`] for `name` back to [`cache_ref`],
+/// replacing any prior snapshot as a new commit — the state [`restore`] will
+/// pick up on this cache's next use.
+///
+/// ## Requirements
+///
+/// @relation(checks.cache)
+pub fn snapshot(repo: &Path, sprite: &str, name: &str) -> Result<(), String> {
+ let dir = cache_dir(name);
+ let archive = Command::new("sprite")
+ .args([
+ "exec",
+ "-s",
+ sprite,
+ "--",
+ "sh",
+ "-c",
+ &format!("tar -C {dir} -cf - ."),
+ ])
+ .output()
+ .map_err(|e| format!("could not run the sprite CLI: {e}"))?;
+ if !archive.status.success() {
+ return Err(format!("could not archive cache {name} from the sprite"));
+ }
+
+ let extracted = tempfile::tempdir().map_err(|e| format!("could not create temp dir: {e}"))?;
+ let mut child = Command::new("tar")
+ .args(["-x", "-C"])
+ .arg(extracted.path())
+ .stdin(Stdio::piped())
+ .spawn()
+ .map_err(|e| format!("could not run tar: {e}"))?;
+ child
+ .stdin
+ .take()
+ .ok_or("tar did not accept stdin")?
+ .write_all(&archive.stdout)
+ .map_err(|e| format!("could not extract the cache archive: {e}"))?;
+ let status = child
+ .wait()
+ .map_err(|e| format!("tar did not complete: {e}"))?;
+ if !status.success() {
+ return Err(format!("could not extract cache {name}'s archive"));
+ }
+
+ // A scratch index and an explicit work tree, so this builds a tree from
+ // the extracted directory without disturbing the repository's own
+ // (nonexistent, since it is bare) index.
+ let index = extracted.path().join(".git-index");
+ let add = Command::new("git")
+ .arg("-C")
+ .arg(repo)
+ .env("GIT_INDEX_FILE", &index)
+ .env("GIT_WORK_TREE", extracted.path())
+ .args(["add", "-A", "."])
+ .status()
+ .map_err(|e| format!("could not stage the cache tree: {e}"))?;
+ if !add.success() {
+ return Err(format!("could not stage cache {name}'s tree"));
+ }
+ let write_tree = Command::new("git")
+ .arg("-C")
+ .arg(repo)
+ .env("GIT_INDEX_FILE", &index)
+ .env("GIT_WORK_TREE", extracted.path())
+ .args(["write-tree"])
+ .output()
+ .map_err(|e| format!("could not write the cache tree: {e}"))?;
+ if !write_tree.status.success() {
+ return Err(format!("could not write cache {name}'s tree"));
+ }
+ let tree = String::from_utf8_lossy(&write_tree.stdout);
+ let tree = ObjectId::from_hex(tree.trim().as_bytes())
+ .map_err(|e| format!("git write-tree returned an invalid tree oid: {e}"))?;
+
+ let store = git_store::Store::open(repo).map_err(|e| format!("could not open store: {e}"))?;
+ store
+ .store_tree(&cache_ref(name), tree, "Update cache")
+ .map_err(|e| format!("could not store cache {name}: {e}"))
+}