feat: parameterize toolchain recipes by target platform
commit
e010378feat: parameterize toolchain recipes by target platform
A downloaded toolchain pin could previously only be authored from a machine already running the target platform’s binaries. --platform now drives recipe resolution itself: rustup resolves version and components from the channel manifest alone, and sccache/a new url recipe fetch the archive once at import time solely to compute its sha256 (trust on first use), so any machine can author a hash-pinned toolchain for any platform without holding its bytes.
feat: add strip/dest layout metadata to Component feat: add url toolchain recipe for pinning arbitrary hosted archives feat: add --strip/--dest CLI flags for the url recipe’s layout refactor: extract downloaded_script as a pure, unit-tested function docs: document --platform, the url recipe, and TOFU pinning Assisted-by: Claude:claude-sonnet-5
Reviews
No reviews of this commit yet — record a verdict below.
Start a review
docs/abstractions.adoc
@@ -127,7 +127,8 @@
subsystems.
* *Toolchains* — typed trees under `refs/meta/toolchains/*`; a resource
effects declare, not a trigger. The repo carries its own execution
-environment with provenance. They keep a subcommand only because
+environment with provenance, as ~1KB hash-pinned manifests; only the
+sandbox touches the bytes. They keep a subcommand only because
import/activation logic is nontrivial; if that shrinks, the subcommand
dies.
* *Members, accounts, comments* — typed trees behind meta-refs (1+2),
docs/spec/checks.adoc
@@ -117,10 +117,16 @@
Before running a check that names a toolchain, the worker MUST resolve each
distinct toolchain named across the job's checks (<<checks.toolchains>>) to
-its typed `bin` tree (`Toolchain::bin`, not the whole document) and extract
-it into the Sprite at a hash-keyed directory, skipping the extraction when
-that directory already exists — the Sprite's persistent filesystem is the
-cache.
+its typed `bin` (`Toolchain::bin`, not the whole document) and materialize
+it into the Sprite at a hash-keyed directory, skipping the work when that
+directory already exists — the Sprite's persistent filesystem is the cache.
+An embedded toolchain's tree is extracted from the object database; a
+downloaded toolchain's components MUST be fetched, sha256-verified against
+their recorded pins, and extracted *inside* the Sprite itself, each per its
+recorded layout (leading path segments stripped, landed in its recorded
+subdirectory, compression auto-detected), under a cache key derived from
+the component hashes plus their layout — the repository never proxies the
+bytes it deliberately does not store.
A check's command MUST then run with each of its named toolchains'
extracted `bin` directory prefixed onto `PATH`, in declaration order, so an
earlier toolchain wins a name collision.
docs/spec/cli.adoc
@@ -82,24 +82,35 @@
`bin` MUST be non-empty; `license` MUST be a valid SPDX expression;
`version` MUST be valid semver; `platform` MUST be a valid target triple.
* `import --from <recipe>` — derive the fields from a local toolchain
- install, or a hosted release, via a named recipe (`rustup` or `sccache`,
- selected further by `--spec <name>`, e.g. `stable` for `rustup`, a release
- tag for `sccache`); explicitly passed fields override the recipe.
- A recipe capable of pointing at a distributor's own hosted, hash-pinned
- archives (rust-lang's dist tarballs, for `rustup`) records those as the
- toolchain's `bin` by default, sparing the repository the bytes; `--embed`
- forces importing the local install's actual `bin` bytes, as `import`
- without `--from` always does. `sccache` has no hash manifest to pin a
- hosted download against, so it always imports the downloaded bytes
- directly regardless of `--embed`.
+ install, or a hosted release, via a named recipe (`rustup`, `sccache`, or
+ `url`, selected further by `--spec <name>`, e.g. `stable` for `rustup`, a
+ release tag for `sccache`, the archive's URL for `url`); explicitly passed
+ fields override the recipe. A recipe capable of pointing at a
+ distributor's own hosted, hash-pinned archives (rust-lang's dist tarballs,
+ for `rustup`) records those as the toolchain's `bin` by default, sparing
+ the repository the bytes; `--embed` forces importing the local install's
+ actual `bin` bytes, as `import` without `--from` always does.
+* `import --from <recipe> --platform <triple>` — `--platform` MUST
+ additionally parameterize the recipe's own resolution, pinning the named
+ target's hosted archives without the local machine ever holding that
+ platform's binaries (`rustup` resolves version and components from the
+ channel manifest alone), and MUST be rejected together with `--embed`.
+ A recipe whose distributor publishes no hash manifest (`sccache`, `url`)
+ MUST instead fetch the archive once at import time solely to compute the
+ sha256 recorded as its pin — trust on first use, with the recipe string
+ and the ref's commit history as the audit trail; every later fetch
+ verifies against that pin. `url` pins any archive (`--spec <url>`,
+ required) with its extraction layout from `--strip`/`--dest`, and
+ supplies no version, platform, or license of its own.
* `list` — render every toolchain on a remote with its `bin` (a tree id,
or a component count when hosted externally), version, platform, and
license.
* `export` — recreate a remote toolchain's `bin` (and `src`, if present)
under a local destination, restoring the executable bit and symlinks —
- fetching and sha256-verifying externally hosted components first — and
- print the version, platform, and license; read-only, per
- <<cli.remote-admin>>.
+ fetching, sha256-verifying, and extracting externally hosted components
+ first, each per its recorded layout (leading segments stripped, landed in
+ its recorded subdirectory, compression auto-detected) — and print the
+ version, platform, and license; read-only, per <<cli.remote-admin>>.
* `remove` — delete the toolchain's ref on a remote.
--
docs/spec/conformance.adoc
@@ -39,6 +39,7 @@
|`cli.comments` |`git-ents/src/main.rs` (`CommentAction`, `run_comment`) | _(manual/CLI UX, no automated test)_
|`cli.login` |`git-ents/src/main.rs` (`login`); `git-ents-server/src/main.rs` (`/login/cli` routes) | _(manual/CLI UX, no automated test)_
|`cli.key-resolution` |`git-ents/src/main.rs` (`fingerprint`, MD5 colon form) | _(manual/CLI UX, no automated test)_
+|`cli.toolchains` |`git-ents/src/main.rs` (`ToolchainAction`, `toolchain_import`); `git-ents/src/registry.rs` (`resolve`, `rustup`, `sccache`, `url_archive`) | _(manual/CLI UX, no automated test)_
|`account.ref` |`git-ents/src/account.rs` |`account::store_then_load_round_trips_the_account`, `the_account_ref_marks_an_account_repo`
|`account.genesis` |`git-ents/src/account.rs` (`genesis`) | _(exercised via `members add --account`; no dedicated unit test)_
|`config.ref` |`git-ents-core/src/config.rs` |`config::store_then_load_round_trips_the_config`, `default_when_the_config_ref_is_absent`
@@ -46,7 +47,7 @@
|`checks.cache` |`git-effect/src/cache.rs` (`CACHE_NS`, `cache_ref`, `cache_dir`, `restore`, `snapshot`); `git-effect/src/engine.rs` (`with_cache_env`) |`definition::tests::store_then_load_round_trips_cache`, `order_rejects_an_invalid_cache_name`; `engine::tests::with_cache_env_leaves_a_cache_free_command_unchanged`, `with_cache_env_exports_the_restored_directory` _(`restore`/`snapshot` themselves require a live Sprite; not covered by unit tests)_
|`checks.post-receive` |`git-effect/src/engine.rs` (`post_receive`, `enqueue`) | _(hook-level, exercised manually; `enqueue` writes tmp+rename)_
|`checks.worker` |`git-effect/src/engine.rs` (`worker`, `pending_jobs`, `drain_repo`, `process_job`, `derive_composite`) |`engine::tests::pending_jobs_groups_by_repo_and_drops_malformed`, `composite_status_derives_from_its_dependencies` _(dependency-ordered execution requires a live Sprite; `order` is unit-tested in `git-effect`)_
-|`checks.sandbox` |`git-effect/src/engine.rs` (`ensure_auth`, `ensure_sprite`, `sync_tree`) | _(requires a live Sprite; not covered by unit tests)_
+|`checks.sandbox` |`git-effect/src/engine.rs` (`ensure_auth`, `ensure_sprite`, `sync_tree`, `resolve_toolchains`, `downloaded_script`, `components_key`, `activate`) |`engine::tests::downloaded_script_extracts_each_component_per_its_layout`, `components_key_includes_the_layout`, `activate_prefixes_path_in_declared_order` _(live-Sprite paths not covered by unit tests)_
|`checks.outcomes` |`git-effect/src/results.rs` (`Status`, `record`/`update_run`); `git-effect/src/engine.rs` (`CHECK_TIMEOUT`, `finalize_error`) |`results::tests::update_run_advances_in_place_rather_than_appending`, `round_trips_an_outcomes_duration_and_recording`
|`checks.debug` |`git-ents-server/src/web/debug.rs` (`handshake`, broker); `git-ents/src/main.rs` (`effect debug`) | _(requires a live Sprite; manually verified)_
|`checks.admin-only` |`git-signed-push/src/lib.rs` (`pre_receive`, `EFFECTS_NS_PREFIX`) |`git-ents-server::pre_receive rejects_a_push_to_effects_from_a_self_attested_member`
crates/git-effect/src/engine.rs
@@ -656,8 +656,10 @@
sync_downloaded_toolchain(sprite, &key, components)?;
// Unlike an embedded toolchain's tree (already flattened to
// put executables at its own top level), each component
- // archive extracts its own `bin/` alongside `lib/`, so `PATH`
- // must point one level deeper.
+ // extracts per its recorded layout — its own `bin/` top level
+ // (rustup) or straight into a `bin` dest (a flat archive) —
+ // landing executables at `<key>/bin` either way, so `PATH`
+ // points one level deeper.
format!("{TOOLCHAINS_DIR}/{key}/bin")
}
};
@@ -667,13 +669,20 @@
}
/// A stable, filesystem-safe cache key for a [`git_toolchain::Bin::Downloaded`]
-/// toolchain: its components' sha256s, joined in extraction order — there is
-/// no tree oid to key the extraction cache by, since nothing is written to
-/// the object database for a downloaded toolchain's `bin`.
+/// toolchain: each component's sha256 plus its recorded layout
+/// (`strip`/`dest` — the same bytes extracted differently are a different
+/// toolchain on disk), joined in extraction order — there is no tree oid to
+/// key the extraction cache by, since nothing is written to the object
+/// database for a downloaded toolchain's `bin`.
fn components_key(components: &[git_toolchain::Component]) -> String {
components
.iter()
- .map(|component| component.sha256.as_str())
+ .map(|component| {
+ format!(
+ "{}.{}.{}",
+ component.sha256, component.strip, component.dest
+ )
+ })
.collect::<Vec<_>>()
.join("-")
}
@@ -811,17 +820,7 @@
return Ok(());
}
- let mut script = format!("mkdir -p {dir}");
- for component in components {
- script.push_str(&format!(
- " && curl -fsSL '{url}' -o /tmp/component.tar.gz \
- && [ \"$(sha256sum /tmp/component.tar.gz | cut -d' ' -f1)\" = '{sha256}' ] \
- && tar -xz --strip-components=2 -C {dir} -f /tmp/component.tar.gz \
- && rm -f /tmp/component.tar.gz",
- url = component.url,
- sha256 = component.sha256,
- ));
- }
+ let script = downloaded_script(&dir, components);
let status = Command::new("sprite")
.args(["exec", "-s", sprite, "--", "sh", "-c", &script])
.status()
@@ -835,6 +834,40 @@
}
}
+/// The `sh` script fetching, verifying, and extracting `components` into
+/// `dir` — pure, so the exact extraction semantics the Sprite runs are unit
+/// tested against `git_toolchain::export`'s local equivalent (the
+/// local/hosted parity anchor). Each component lands in `dir`/its `dest`,
+/// stripped of its leading `strip` path segments, compression auto-detected
+/// by `tar` (rust-lang ships gzip, zig ships xz). Interpolation is safe by
+/// construction: `git_toolchain::import_downloaded` refuses a component
+/// whose fields could escape the single quotes.
+///
+/// ## Requirements
+///
+/// @relation(checks.sandbox)
+fn downloaded_script(dir: &str, components: &[git_toolchain::Component]) -> String {
+ let mut script = format!("mkdir -p {dir}");
+ for component in components {
+ let dest = if component.dest.is_empty() {
+ dir.to_owned()
+ } else {
+ format!("{dir}/{}", component.dest)
+ };
+ script.push_str(&format!(
+ " && mkdir -p {dest} \
+ && curl -fsSL '{url}' -o /tmp/component.archive \
+ && [ \"$(sha256sum /tmp/component.archive | cut -d' ' -f1)\" = '{sha256}' ] \
+ && tar -x --strip-components={strip} -C {dest} -f /tmp/component.archive \
+ && rm -f /tmp/component.archive",
+ url = component.url,
+ sha256 = component.sha256,
+ strip = component.strip,
+ ));
+ }
+ script
+}
+
/// How long a single effect may run before the worker abandons it. A runaway
/// effect that outlived this — a hung build, a command blocked on input — is
/// killed and recorded `error` rather than wedging the worker (and with it every
@@ -1083,6 +1116,58 @@
);
}
+ // @relation(checks.sandbox, role=Verifies)
+ #[test]
+ fn downloaded_script_extracts_each_component_per_its_layout() {
+ let components = vec![
+ git_toolchain::Component {
+ url: "https://static.rust-lang.org/dist/rustc.tar.gz".to_owned(),
+ sha256: "aaa".to_owned(),
+ strip: 2,
+ dest: String::new(),
+ },
+ git_toolchain::Component {
+ url: "https://example.com/flat.tar.xz".to_owned(),
+ sha256: "bbb".to_owned(),
+ strip: 1,
+ dest: "bin".to_owned(),
+ },
+ ];
+ assert_eq!(
+ downloaded_script("/toolchains/key", &components),
+ "mkdir -p /toolchains/key \
+ && mkdir -p /toolchains/key \
+ && curl -fsSL 'https://static.rust-lang.org/dist/rustc.tar.gz' -o /tmp/component.archive \
+ && [ \"$(sha256sum /tmp/component.archive | cut -d' ' -f1)\" = 'aaa' ] \
+ && tar -x --strip-components=2 -C /toolchains/key -f /tmp/component.archive \
+ && rm -f /tmp/component.archive \
+ && mkdir -p /toolchains/key/bin \
+ && curl -fsSL 'https://example.com/flat.tar.xz' -o /tmp/component.archive \
+ && [ \"$(sha256sum /tmp/component.archive | cut -d' ' -f1)\" = 'bbb' ] \
+ && tar -x --strip-components=1 -C /toolchains/key/bin -f /tmp/component.archive \
+ && rm -f /tmp/component.archive"
+ );
+ }
+
+ // @relation(checks.sandbox, role=Verifies)
+ #[test]
+ fn components_key_includes_the_layout() {
+ let component = git_toolchain::Component {
+ url: "https://example.com/a.tar.gz".to_owned(),
+ sha256: "aaa".to_owned(),
+ strip: 2,
+ dest: String::new(),
+ };
+ let mut flat = component.clone();
+ flat.strip = 1;
+ flat.dest = "bin".to_owned();
+ assert_eq!(components_key(std::slice::from_ref(&component)), "aaa.2.");
+ assert_ne!(
+ components_key(std::slice::from_ref(&component)),
+ components_key(&[flat])
+ );
+ }
+
// @relation(checks.cache, role=Verifies)
#[test]
fn with_cache_env_leaves_a_cache_free_command_unchanged() {
crates/git-ents/src/main.rs
@@ -276,11 +276,12 @@
#[facet(args::named, default)]
platform: Option<String>,
/// Recipe to derive `bin`/`src`/`license`/`version`/`platform` from
- /// instead of supplying them by hand (currently only `rustup`).
+ /// instead of supplying them by hand (`rustup`, `sccache`, `url`).
#[facet(args::named)]
from: Option<String>,
/// Recipe-specific selector (for `--from rustup`, the toolchain
- /// name `rustup` itself knows, e.g. `stable`; defaults to `stable`).
+ /// name `rustup` itself knows, e.g. `stable`; defaults to `stable`.
+ /// For `--from url`, the archive URL — required).
#[facet(args::named)]
spec: Option<String>,
/// With `--from`, import the recipe's actual `bin` bytes instead of
@@ -288,6 +289,16 @@
/// (see `git_toolchain::Bin::Downloaded`).
#[facet(args::named, default)]
embed: bool,
+ /// With `--from url`: leading path segments to strip when the
+ /// sandbox extracts the archive (default 1, a flat
+ /// `<pkg>-<version>/…` release tarball).
+ #[facet(args::named, default)]
+ strip: Option<u8>,
+ /// With `--from url`: subdirectory of the toolchain to extract the
+ /// archive into (default `bin`, putting a flat archive's payload on
+ /// `PATH`).
+ #[facet(args::named, default)]
+ dest: Option<String>,
},
/// List the toolchains configured on a remote.
List,
@@ -540,8 +551,10 @@
from,
spec,
embed,
+ strip,
+ dest,
} => toolchain_import(
- name, bin, src, license, version, platform, from, spec, embed, remote,
+ name, bin, src, license, version, platform, from, spec, embed, strip, dest, remote,
),
ToolchainAction::List => toolchain_list(remote),
ToolchainAction::Recipes => toolchain_recipes(),
@@ -556,7 +569,10 @@
/// `remote` and push it. Prompts for any field left unset when run at an
/// interactive terminal, unless `from` names a recipe (`registry::resolve`)
/// to derive `bin`/`src`/`license`/`version`/`platform` from instead;
-/// explicit flags still win over a recipe's values.
+/// explicit flags still win over a recipe's values, and `platform`
+/// additionally parameterizes the recipe's own resolution — pinning another
+/// platform's hosted archives without this machine ever holding its
+/// binaries.
///
/// ## Requirements
///
@@ -572,15 +588,26 @@
from: Option<String>,
spec: Option<String>,
embed: bool,
+ strip: Option<u8>,
+ dest: Option<String>,
remote: &str,
) -> Result<(), String> {
let name = interactive::text_or(name, "Toolchain name")?;
+ if from.as_deref() == Some("url") && spec.is_none() {
+ return Err("the url recipe needs --spec <archive-url>".to_owned());
+ }
let recipe_desc = from
.as_deref()
.map(|from| registry::describe(from, spec.as_deref().unwrap_or("stable")));
+ let opts = registry::RecipeOptions {
+ embed,
+ platform: platform.clone(),
+ strip,
+ dest,
+ };
let recipe = from
- .map(|recipe| registry::resolve(&recipe, spec.as_deref().unwrap_or("stable"), embed))
+ .map(|recipe| registry::resolve(&recipe, spec.as_deref().unwrap_or("stable"), &opts))
.transpose()?;
let bin_plan = match bin {
@@ -603,15 +630,15 @@
} else {
interactive::optional_text_or(None, "Directory of source to import (optional)")?
};
- let license = match license.or_else(|| recipe.as_ref().map(|r| r.license.clone())) {
+ let license = match license.or_else(|| recipe.as_ref().and_then(|r| r.license.clone())) {
Some(license) => license,
None => interactive::text_or(None, "License (SPDX expression)")?,
};
- let version = match version.or_else(|| recipe.as_ref().map(|r| r.version.clone())) {
+ let version = match version.or_else(|| recipe.as_ref().and_then(|r| r.version.clone())) {
Some(version) => version,
None => interactive::text_or(None, "Version (semver)")?,
};
- let platform = match platform.or_else(|| recipe.as_ref().map(|r| r.platform.clone())) {
+ let platform = match platform.or_else(|| recipe.as_ref().and_then(|r| r.platform.clone())) {
Some(platform) => platform,
None => interactive::text_or(None, "Platform (target triple)")?,
};
crates/git-ents/src/registry.rs
@@ -27,7 +27,10 @@
}
/// What a recipe resolved from a local toolchain install, ready to hand to
-/// `git_toolchain::import`/`import_downloaded`.
+/// `git_toolchain::import`/`import_downloaded`. A `None` metadata field is
+/// one the recipe cannot know (the `url` recipe knows nothing about what an
+/// arbitrary archive contains); the CLI's own flag-else-prompt chain covers
+/// it.
///
/// `_staging`, when `bin` is [`Bin::Dir`] pointing into a temporary
/// directory, is kept alive only so the directory survives until the
@@ -35,12 +38,31 @@
pub struct Resolved {
pub bin: Bin,
pub src: Option<PathBuf>,
- pub license: String,
- pub version: String,
- pub platform: String,
+ pub license: Option<String>,
+ pub version: Option<String>,
+ pub platform: Option<String>,
_staging: Option<TempDir>,
}
+/// How [`resolve`] should resolve, beyond the recipe's own `spec` selector.
+#[derive(Default)]
+pub struct RecipeOptions {
+ /// Import the recipe's actual local `bin` bytes instead of recording
+ /// hosted, hash-pinned archives. Incompatible with `platform`.
+ pub embed: bool,
+ /// Resolve for this target triple instead of the local machine's — the
+ /// recipe then never touches local binaries, only the distributor's
+ /// hosted metadata and archives, so a toolchain for the effect worker's
+ /// sandbox can be pinned from any machine.
+ pub platform: Option<String>,
+ /// `url` recipe only: leading path segments to strip at extraction
+ /// (default 1, a flat `<pkg>-<version>/…` release tarball).
+ pub strip: Option<u8>,
+ /// `url` recipe only: subdirectory of the toolchain to extract into
+ /// (default `bin`, so a flat archive's payload lands on `PATH`).
+ pub dest: Option<String>,
+}
+
/// A recipe `git ents toolchain import --from` accepts, described richly
/// enough to render on its own via `facet_pretty` (see `git ents toolchain
/// recipes`) rather than as a bare name.
@@ -70,28 +92,51 @@
RecipeInfo {
name: "sccache",
spec: "a mozilla/sccache release tag, e.g. v0.8.2, or empty for latest",
- summary: "Downloads a prebuilt sccache release for this machine's \
- platform from GitHub and imports it directly (always \
- embedded — the binary is small and GitHub publishes no \
- hash manifest to pin a hosted download against).",
+ summary: "Resolves a prebuilt sccache release from GitHub; with \
+ --platform it records the release archive as a hash pin \
+ computed at import time (trust on first use — GitHub \
+ publishes no hash manifest), otherwise it downloads this \
+ machine's archive and imports the binary directly.",
+ },
+ RecipeInfo {
+ name: "url",
+ spec: "an archive URL, e.g. https://ziglang.org/download/.../zig-x86_64-linux-0.15.2.tar.xz",
+ summary: "Pins any hosted archive as a downloaded toolchain: fetches \
+ it once at import time only to compute its sha256 (trust \
+ on first use), records url+hash+layout (--strip, --dest), \
+ and lets the sandbox fetch the bytes itself. Version, \
+ platform, and license must be supplied explicitly.",
},
];
/// Resolve `recipe` against `spec` (a recipe-specific selector, e.g. a
/// rustup toolchain name). See [`RECIPES`] for what's known.
///
-/// `embed` forces the old behavior of staging and importing `bin`'s actual
-/// bytes; by default the recipe instead points at its distributor's own
-/// hosted, hash-verified archives (see [`Bin::Components`]), sparing the
-/// repository the toolchain's own bytes.
+/// `opts.embed` forces the old behavior of staging and importing `bin`'s
+/// actual bytes; by default the recipe instead points at its distributor's
+/// own hosted, hash-verified archives (see [`Bin::Components`]), sparing the
+/// repository the toolchain's own bytes. `opts.platform` resolves for a
+/// foreign target without touching local binaries at all, and is therefore
+/// rejected together with `embed`.
///
/// ## Requirements
///
/// @relation(cli.toolchains)
-pub fn resolve(recipe: &str, spec: &str, embed: bool) -> Result<Resolved, String> {
+pub fn resolve(recipe: &str, spec: &str, opts: &RecipeOptions) -> Result<Resolved, String> {
+ if opts.embed && opts.platform.is_some() {
+ return Err(
+ "--embed imports this machine's bytes; it cannot target another platform".to_owned(),
+ );
+ }
+ if (opts.strip.is_some() || opts.dest.is_some()) && recipe != "url" {
+ return Err(format!(
+ "--strip/--dest are layout hints for the url recipe; {recipe} records its own layout"
+ ));
+ }
match recipe {
- "rustup" => rustup(spec, embed),
- "sccache" => sccache(spec),
+ "rustup" => rustup(spec, opts),
+ "sccache" => sccache(spec, opts.platform.as_deref()),
+ "url" => url_archive(spec, opts),
other => Err(format!(
"unknown toolchain recipe {other:?} (known: {})",
RECIPES
@@ -144,10 +189,31 @@
/// component is installed, else omitted, regardless of `embed`. Rust's own
/// toolchain is dual-licensed `MIT OR Apache-2.0`.
///
+/// With `opts.platform`, no local toolchain is consulted at all: the channel
+/// manifest for `spec` (`stable`, `nightly`, a version) is the sole source —
+/// it names its own version (`[pkg.rustc].version`) and hosts hash-pinned
+/// archives for every target, so a toolchain for a foreign platform (the
+/// effect worker's sandbox) can be pinned from any machine. `src` is omitted
+/// there: there is no local sysroot to point at.
+///
/// ## Requirements
///
/// @relation(cli.toolchains)
-fn rustup(spec: &str, embed: bool) -> Result<Resolved, String> {
+fn rustup(spec: &str, opts: &RecipeOptions) -> Result<Resolved, String> {
+ if let Some(platform) = &opts.platform {
+ let manifest = fetch_manifest(spec)?;
+ let version = manifest_version(&manifest)
+ .ok_or_else(|| format!("channel-rust-{spec}.toml has no [pkg.rustc].version"))?;
+ return Ok(Resolved {
+ bin: Bin::Components(manifest_components(&manifest, platform)?),
+ src: None,
+ license: Some("MIT OR Apache-2.0".to_owned()),
+ version: Some(version),
+ platform: Some(platform.clone()),
+ _staging: None,
+ });
+ }
+
let toolchain_arg = format!("+{spec}");
let sysroot = rustc(&toolchain_arg, &["--print", "sysroot"])?;
let sysroot = PathBuf::from(sysroot.trim());
@@ -161,14 +227,15 @@
let src = sysroot.join("lib/rustlib/src/rust");
let src = src.is_dir().then_some(src);
- let (bin, staging) = if embed {
+ let (bin, staging) = if opts.embed {
let staging = tempfile::tempdir()
.map_err(|error| format!("could not create a staging directory: {error}"))?;
stage_bin(&sysroot.join("bin"), &sysroot.join("lib"), staging.path())?;
(Bin::Dir(staging.path().to_owned()), Some(staging))
} else {
+ let manifest = fetch_manifest(&channel_for(&version))?;
(
- Bin::Components(manifest_components(&version, &platform)?),
+ Bin::Components(manifest_components(&manifest, &platform)?),
None,
)
};
@@ -176,9 +243,9 @@
Ok(Resolved {
bin,
src,
- license: "MIT OR Apache-2.0".to_owned(),
- version,
- platform,
+ license: Some("MIT OR Apache-2.0".to_owned()),
+ version: Some(version),
+ platform: Some(platform),
_staging: staging,
})
}
@@ -190,41 +257,101 @@
/// [`rustup`] follows via its local `rustc`.
///
/// Unlike `rustup`, GitHub publishes no manifest of hashes alongside a
-/// release to pin a hosted download against, and the archive is a single
-/// ~15 MB binary, so this recipe always imports it directly (`Bin::Dir`)
-/// rather than offering [`Bin::Components`] — `embed` has nothing to toggle
-/// here.
+/// release to pin a hosted download against. Without a `platform` override
+/// this recipe therefore imports this machine's archive's binary directly
+/// (`Bin::Dir`). With `platform`, it instead records the release archive as
+/// a downloaded component whose sha256 it computes itself, once, at import
+/// time — trust on first use: the trust decision is taken exactly here, is
+/// audited via the recipe string on the document and the ref's commit
+/// history, and every later fetch (local export, the sandbox) verifies
+/// against the pinned hash. The archive is flat
+/// (`sccache-<tag>-<target>/sccache`), hence `strip: 1, dest: "bin"`.
///
/// ## Requirements
///
/// @relation(cli.toolchains)
-fn sccache(spec: &str) -> Result<Resolved, String> {
+fn sccache(spec: &str, platform: Option<&str>) -> Result<Resolved, String> {
let tag = if spec.is_empty() {
latest_sccache_tag()?
} else {
spec.to_owned()
};
let version = tag.strip_prefix('v').unwrap_or(&tag).to_owned();
- let target = sccache_target()?;
+ let target = match platform {
+ Some(platform) => platform.to_owned(),
+ None => sccache_target()?.to_owned(),
+ };
let url = format!(
"https://github.com/mozilla/sccache/releases/download/{tag}/sccache-{tag}-{target}.tar.gz"
);
let bytes = crate::http_get_bytes(&url)?;
+ if platform.is_some() {
+ let sha256 = git_toolchain::sha256_hex(&bytes)
+ .map_err(|error| format!("could not hash: {error}"))?;
+ return Ok(Resolved {
+ bin: Bin::Components(vec![Component {
+ url,
+ sha256,
+ strip: 1,
+ dest: "bin".to_owned(),
+ }]),
+ src: None,
+ license: Some("MPL-2.0".to_owned()),
+ version: Some(version),
+ platform: Some(target),
+ _staging: None,
+ });
+ }
+
let staging = tempfile::tempdir()
.map_err(|error| format!("could not create a staging directory: {error}"))?;
- stage_sccache(&bytes, &tag, target, staging.path())?;
+ stage_sccache(&bytes, &tag, &target, staging.path())?;
Ok(Resolved {
bin: Bin::Dir(staging.path().to_owned()),
src: None,
- license: "MPL-2.0".to_owned(),
- version,
- platform: target.to_owned(),
+ license: Some("MPL-2.0".to_owned()),
+ version: Some(version),
+ platform: Some(target),
_staging: Some(staging),
})
}
+/// Pin any hosted archive as a one-component downloaded toolchain —
+/// `http_archive`, in Bazel terms. `spec` is the archive's URL; it is
+/// fetched once, here, only to compute the sha256 every later verification
+/// pins against (trust on first use, audited exactly like [`sccache`]'s
+/// pin). The layout hints come from `--strip`/`--dest` (default: a flat
+/// `<pkg>-<version>/…` tarball whose payload belongs on `PATH`). This recipe
+/// knows nothing about what the archive contains, so version, platform, and
+/// license all stay `None` for the caller to supply.
+///
+/// ## Requirements
+///
+/// @relation(cli.toolchains)
+fn url_archive(spec: &str, opts: &RecipeOptions) -> Result<Resolved, String> {
+ if spec.is_empty() {
+ return Err("the url recipe needs --spec <archive-url>".to_owned());
+ }
+ let bytes = crate::http_get_bytes(spec)?;
+ let sha256 =
+ git_toolchain::sha256_hex(&bytes).map_err(|error| format!("could not hash: {error}"))?;
+ Ok(Resolved {
+ bin: Bin::Components(vec![Component {
+ url: spec.to_owned(),
+ sha256,
+ strip: opts.strip.unwrap_or(1),
+ dest: opts.dest.clone().unwrap_or_else(|| "bin".to_owned()),
+ }]),
+ src: None,
+ license: None,
+ version: None,
+ platform: None,
+ _staging: None,
+ })
+}
+
/// The latest `mozilla/sccache` release's tag name, from GitHub's "latest
/// release" API.
fn latest_sccache_tag() -> Result<String, String> {
@@ -292,31 +419,52 @@
.map_err(|error| format!("could not set permissions on {}: {error}", dest.display()))
}
-/// The three components of rust-lang's channel manifest that together make
-/// a working toolchain (compiler, cargo, and the target's standard library),
-/// resolved for `target` against the manifest for `version` (or the shared
-/// `nightly` channel, when `version` names one — rust-lang does not publish
-/// a stable per-version manifest name for nightly builds).
-fn manifest_components(version: &str, target: &str) -> Result<Vec<Component>, String> {
- let channel = if version.contains("nightly") {
+/// The manifest name for a version rustc reported: nightly builds collapse
+/// to the shared `nightly` channel, since rust-lang publishes no stable
+/// per-version manifest name for them.
+fn channel_for(version: &str) -> String {
+ if version.contains("nightly") {
"nightly".to_owned()
} else {
version.to_owned()
- };
- let url = format!("https://static.rust-lang.org/dist/channel-rust-{channel}.toml");
- let manifest = crate::http_get(&url)?;
+ }
+}
+/// Fetch rust-lang's channel manifest for `channel` (`stable`, `nightly`, or
+/// a version) — the one authoritative document naming the channel's version
+/// and every target's hash-pinned component archives.
+fn fetch_manifest(channel: &str) -> Result<String, String> {
+ let url = format!("https://static.rust-lang.org/dist/channel-rust-{channel}.toml");
+ crate::http_get(&url)
+}
+
+/// The channel's own version, from `[pkg.rustc] version = "1.88.0 (hash
+/// date)"` — the first whitespace-separated token, valid semver for stable
+/// (`1.88.0`) and nightly (`1.90.0-nightly`) alike.
+fn manifest_version(manifest: &str) -> Option<String> {
+ let raw = manifest_field(manifest, "pkg.rustc", "version")?;
+ raw.split_whitespace().next().map(str::to_owned)
+}
+
+/// The three components of rust-lang's channel manifest that together make
+/// a working toolchain (compiler, cargo, and the target's standard library),
+/// resolved for `target`. Every rust-lang dist archive unpacks to
+/// `<package>-<version>-<target>/<component>/…`, hence `strip: 2` with no
+/// `dest` — the payload carries its own `bin/`/`lib/` top level.
+fn manifest_components(manifest: &str, target: &str) -> Result<Vec<Component>, String> {
["rustc", "cargo", "rust-std"]
.into_iter()
.map(|package| {
let section = format!("pkg.{package}.target.{target}");
- let component_url = manifest_field(&manifest, §ion, "url")
- .ok_or_else(|| format!("{url} has no [{section}].url"))?;
- let sha256 = manifest_field(&manifest, §ion, "hash")
- .ok_or_else(|| format!("{url} has no [{section}].hash"))?;
+ let component_url = manifest_field(manifest, §ion, "url")
+ .ok_or_else(|| format!("the channel manifest has no [{section}].url"))?;
+ let sha256 = manifest_field(manifest, §ion, "hash")
+ .ok_or_else(|| format!("the channel manifest has no [{section}].hash"))?;
Ok(Component {
url: component_url,
sha256,
+ strip: 2,
+ dest: String::new(),
})
})
.collect()
crates/git-toolchain/src/lib.rs
@@ -100,15 +100,15 @@
/// [`export`] (or a Sprite, at check-activation time) instead of stored
/// in the object database — spares the repository the toolchain's own
/// bytes when a stable, content-hashed origin (a distributor's release
- /// archives) already exists. Each component is extracted with its outer
- /// two path segments (`<package>-<version>-<target>/<component>/`)
- /// stripped, the layout rust-lang's (and most other distributors')
- /// dist archives use.
+ /// archives) already exists. Each component records its own archive
+ /// layout ([`Component::strip`], [`Component::dest`]), so a rust-lang
+ /// dist archive and a flat single-binary release tarball can coexist in
+ /// one toolchain.
Downloaded(Vec<Component>),
}
/// One archive making up a [`Bin::Downloaded`] toolchain: fetched from `url`
-/// and checked against `sha256` before being extracted.
+/// and checked against `sha256` before being extracted per `strip`/`dest`.
#[derive(Debug, Clone, PartialEq, Facet)]
pub struct Component {
/// Where to fetch the archive from.
@@ -116,6 +116,17 @@
/// The archive's expected sha256, hex-encoded — checked before
/// extraction; a mismatch is refused rather than extracted anyway.
pub sha256: String,
+ /// Leading path segments stripped at extraction (`tar
+ /// --strip-components`): 2 for rust-lang dist archives
+ /// (`<package>-<version>-<target>/<component>/…`), 1 for a flat release
+ /// tarball (`<package>-<version>/<binary>`).
+ pub strip: u8,
+ /// Subdirectory under the toolchain's extraction root to extract into:
+ /// empty for archives already carrying their own `bin/`/`lib/`/… top
+ /// level (rustup components), `bin` for flat archives whose payload
+ /// should itself land on `PATH` — keeping `<root>/bin` the one activation
+ /// convention either way.
+ pub dest: String,
}
/// A failure importing, resolving, listing, exporting, or removing a
@@ -153,6 +164,11 @@
/// activates nothing on `PATH` is not a toolchain.
#[error("a downloaded toolchain must list at least one component")]
NoComponents,
+ /// A [`Component`] carried a field unsafe to interpolate into the shell
+ /// script that fetches and extracts it: a `dest` that is not empty or a
+ /// single safe path segment, or a `url`/`sha256` containing a quote.
+ #[error("invalid component: {0}")]
+ InvalidComponent(String),
/// A [`Bin::Downloaded`] component could not be fetched or extracted.
#[error("could not fetch {0}: {1}")]
Fetch(String, String),
@@ -242,6 +258,9 @@
if components.is_empty() {
return Err(Error::NoComponents);
}
+ for component in &components {
+ validate_component(component)?;
+ }
validate_metadata(license, version, platform)?;
let odb = odb_at(repo)?;
let collector = PackCollector::default();
@@ -259,6 +278,26 @@
store_toolchain(repo, name, toolchain, &odb)
}
+/// A [`Component`]'s fields end up interpolated into a shell script (the
+/// Sprite-side fetch-and-extract in `git-effect`), so refuse anything that
+/// could escape it: `dest` must be empty or one safe path segment, and no
+/// field may contain a single quote.
+fn validate_component(component: &Component) -> Result<(), Error> {
+ if !component.dest.is_empty() && !git_store::ref_segment_ok(&component.dest) {
+ return Err(Error::InvalidComponent(format!(
+ "dest {:?} is not empty or a single safe path segment",
+ component.dest
+ )));
+ }
+ if component.url.contains('\'') || component.sha256.contains('\'') {
+ return Err(Error::InvalidComponent(format!(
+ "{:?} contains a quote",
+ component.url
+ )));
+ }
+ Ok(())
+}
+
/// `license` MUST be a valid SPDX license expression, `version` a valid
/// semver version, and `platform` a valid target triple — shared by
/// [`import`] and [`import_downloaded`].
@@ -347,10 +386,9 @@
///
/// [`Bin::Embedded`] writes its (already self-contained: executables plus a
/// sibling `lib/`) tree straight under `dest/bin`. [`Bin::Downloaded`]'s
-/// components already carry their own `bin/`/`lib/`/... top-level
-/// directories once their outer two path segments are stripped, so they are
-/// extracted directly under `dest` instead, landing at the same `dest/bin/…`
-/// shape by construction.
+/// components are each fetched, verified, and extracted per their own
+/// recorded layout ([`Component::strip`], [`Component::dest`]) relative to
+/// `dest`, landing at the same `dest/bin/…` shape by construction.
pub fn export(repo: &Path, name: &str, dest: &Path) -> Result<Toolchain, Error> {
let toolchain = resolve(repo, name)?;
let odb = odb_at(repo)?;
@@ -729,7 +767,7 @@
actual,
));
}
- extract_stripped(&archive, dest)?;
+ extract_component(&archive, component, dest)?;
}
Ok(())
}
@@ -752,8 +790,9 @@
/// Hex-encoded sha256 of `bytes`, via the system `shasum` (macOS) or
/// `sha256sum` (Linux) — shells out rather than adding a hashing dependency
-/// to this crate.
-fn sha256_hex(bytes: &[u8]) -> Result<String, Error> {
+/// to this crate. Public so a recipe pinning a hosted archive (trust on
+/// first use) computes its hash the same way every later verification does.
+pub fn sha256_hex(bytes: &[u8]) -> Result<String, Error> {
let (program, args): (&str, &[&str]) = match std::env::consts::OS {
"macos" => ("shasum", &["-a", "256"]),
_ => ("sha256sum", &[]),
@@ -786,13 +825,20 @@
.ok_or_else(|| Error::Fetch(program.to_owned(), "no hash in output".to_owned()))
}
-/// Extract a gzipped tar `archive` into `dest`, stripping the outer two path
-/// segments every rust-lang dist archive (and most other distributors')
-/// wraps its payload in (`<package>-<version>-<target>/<component>/`).
-fn extract_stripped(archive: &[u8], dest: &Path) -> Result<(), Error> {
+/// Extract a tar `archive` (compression auto-detected — gzip, xz, ...) per
+/// `component`'s recorded layout: strip its leading [`Component::strip`]
+/// path segments and land it in `dest_root`/[`Component::dest`].
+fn extract_component(archive: &[u8], component: &Component, dest_root: &Path) -> Result<(), Error> {
+ let dest = if component.dest.is_empty() {
+ dest_root.to_owned()
+ } else {
+ dest_root.join(&component.dest)
+ };
+ fs::create_dir_all(&dest).map_err(|error| Error::Io(dest.clone(), error))?;
+ let strip = format!("--strip-components={}", component.strip);
let mut child = Command::new("tar")
- .args(["-xz", "--strip-components=2", "-C"])
- .arg(dest)
+ .args(["-x", &strip, "-C"])
+ .arg(&dest)
.stdin(Stdio::piped())
.spawn()
.map_err(|error| Error::Fetch("tar".to_owned(), error.to_string()))?;
@@ -1022,6 +1068,34 @@
Component {
url: format!("file://{}", archive.display()),
sha256: sha256_hex(&bytes).unwrap(),
+ strip: 2,
+ dest: String::new(),
+ }
+ }
+
+ /// Build a flat single-binary `.tar.xz` release archive
+ /// (`<pkg>-<version>/<binary>`), the shape a GitHub release or a zig
+ /// tarball ships — one leading segment, no `bin/` of its own.
+ fn build_flat_component(staging: &Path, name: &str, contents: &[u8]) -> Component {
+ let root = staging.join("pkg-1.0.0");
+ fs::create_dir_all(&root).unwrap();
+ fs::write(root.join(name), contents).unwrap();
+ let archive = staging.join("flat.tar.xz");
+ let status = Command::new("tar")
+ .args(["-cJf"])
+ .arg(&archive)
+ .args(["-C"])
+ .arg(staging)
+ .arg("pkg-1.0.0")
+ .status()
+ .unwrap();
+ assert!(status.success());
+ let bytes = fs::read(&archive).unwrap();
+ Component {
+ url: format!("file://{}", archive.display()),
+ sha256: sha256_hex(&bytes).unwrap(),
+ strip: 1,
+ dest: "bin".to_owned(),
}
}
@@ -1053,6 +1127,66 @@
);
}
+ #[test]
+ fn export_extracts_a_flat_xz_component_into_its_dest() {
+ let repo_dir = repo();
+ let staging = tempfile::tempdir().unwrap();
+ let component = build_flat_component(staging.path(), "tool", b"#!/bin/sh\n");
+
+ import_downloaded(
+ repo_dir.path(),
+ "flat",
+ vec![component],
+ None,
+ "MIT",
+ VERSION,
+ PLATFORM,
+ None,
+ )
+ .unwrap();
+
+ let dest = tempfile::tempdir().unwrap();
+ let dest_path = dest.path().join("out");
+ export(repo_dir.path(), "flat", &dest_path).unwrap();
+ assert_eq!(
+ fs::read(dest_path.join("bin/tool")).unwrap(),
+ b"#!/bin/sh\n"
+ );
+ }
+
+ #[test]
+ fn import_downloaded_rejects_an_unsafe_component() {
+ let repo_dir = repo();
+ let staging = tempfile::tempdir().unwrap();
+ let mut component = build_component(staging.path(), &[("bin/tool", b"#!/bin/sh\n")]);
+ component.dest = "a/b".to_owned();
+ let result = import_downloaded(
+ repo_dir.path(),
+ "bad-dest",
+ vec![component],
+ None,
+ "MIT",
+ VERSION,
+ PLATFORM,
+ None,
+ );
+ assert!(matches!(result, Err(Error::InvalidComponent(_))));
+
+ let mut component = build_component(staging.path(), &[("bin/tool", b"#!/bin/sh\n")]);
+ component.url = "https://example.com/x' rm -rf'".to_owned();
+ let result = import_downloaded(
+ repo_dir.path(),
+ "bad-url",
+ vec![component],
+ None,
+ "MIT",
+ VERSION,
+ PLATFORM,
+ None,
+ );
+ assert!(matches!(result, Err(Error::InvalidComponent(_))));
+ }
+
#[test]
fn import_downloaded_rejects_an_empty_component_list() {
let repo_dir = repo();