feat: store a toolchain's bin as a downloadable manifest by default
commit
adad66ffeat: store a toolchain's bin as a downloadable manifest by default
Importing a rustup toolchain the old way wrote its full sysroot (~700MB) as
loose git objects on every import. rust-lang already hosts every release’s
component archives at a stable, sha256-pinned URL (its own channel manifest
TOML), so there is no need to re-host those bytes: a toolchain’s bin is now
either Bin::Embedded (the old RawTree capture) or Bin::Downloaded, a list
of {url, sha256} components fetched and verified fresh at export or
check-activation time. The rustup import recipe uses the manifest by
default; --embed forces the old local-bytes behavior for toolchains with no
independent hosted origin.
feat: add Bin/Component to Toolchain, replacing bin: RawTree
feat: add git_toolchain::import_downloaded and teach export to
fetch/verify/extract a downloaded toolchain’s components
feat: derive Bin::Downloaded components from rust-lang’s channel manifest
in the rustup recipe; add --embed to git ents toolchain import
feat: teach check activation to fetch and cache a downloaded toolchain in
the Sprite, keyed by its components' hashes
docs: describe the Bin split and --embed in storage.adoc and cli.adoc
Assisted-by: Claude:claude-sonnet-5
Reviews
No reviews of this commit yet — record a verdict below.
Start a review
docs/spec/cli.adoc
@@ -78,13 +78,20 @@
`platform` from a local toolchain install instead of supplying them by
hand, via a named recipe (currently only `rustup`, selected further by
`--spec <name>`, e.g. `stable`); any of the fields also passed explicitly
- override the recipe's value.
+ override the recipe's value. By default a recipe capable of pointing at a
+ distributor's own hosted, hash-pinned archives (rust-lang's dist tarballs,
+ for `rustup`) records those as the toolchain's `bin` instead of importing
+ local bytes, sparing the repository the toolchain's own bytes; `--embed`
+ forces the recipe to import the local install's actual `bin` bytes
+ instead, as `import` without `--from` always does.
* `list` — render every toolchain configured on a remote with its `bin`
- tree id, version, platform, and license.
+ (a tree id, or a component count when hosted externally), version,
+ platform, and license.
* `export` — recreate a remote's named toolchain's `bin` (and `src`, if
present) under a local destination directory, restoring the executable bit
- and symlinks, and print the version, platform, and license; read-only, per
- <<cli.remote-admin>>.
+ and symlinks — fetching and sha256-verifying `bin`'s components first when
+ hosted externally rather than embedded — and print the version, platform,
+ and license; read-only, per <<cli.remote-admin>>.
* `remove` — delete the toolchain's ref on a remote.
--
docs/spec/storage.adoc
@@ -65,19 +65,27 @@
(origin-or-content-hash), so filing an item never contends a counter and
the identity rule cannot drift between collections. `toolchains/<name>`
(`git-toolchain`) is a named collection like `member/<username>`, and its
- item is a `Facet` document like any other — a `bin` directory tree, an
- optional `src` directory tree, an SPDX license expression, a semver
- version, and a target-triple platform — but `bin` and `src` are each
- captured whole as a `facet_git_tree::RawTree`, a raw-passthrough field
- wrapping an already-written tree's object id rather than a directory
- layout `Facet` could model field-by-field. `license`, `version`, and
- `platform` are plain strings validated against a real parser (`spdx`,
- `semver`, `target-lexicon`) at import time rather than carried as a parsed
- type, since nothing downstream needs more than the canonical string back.
+ item is a `Facet` document like any other — a `bin`, an optional `src`
+ directory tree, an SPDX license expression, a semver version, and a
+ target-triple platform. `src` is always captured whole as a
+ `facet_git_tree::RawTree`, a raw-passthrough field wrapping an
+ already-written tree's object id rather than a directory layout `Facet`
+ could model field-by-field. `bin` is a `Bin` enum over two
+ representations: `Bin::Embedded`, the same `RawTree` capture, for a
+ toolchain with no stable independent origin; or `Bin::Downloaded`, a list
+ of `{url, sha256}` components pointing at a distributor's own hosted,
+ hash-pinned archives (rust-lang's dist tarballs, for the `rustup` import
+ recipe's default) instead of storing the toolchain's bytes at all —
+ fetched, sha256-verified, and extracted fresh each time a Sprite activates
+ the toolchain or `git ents toolchain export` runs locally, sparing the
+ repository the bytes entirely. `license`, `version`, and `platform` are
+ plain strings validated against a real parser (`spdx`, `semver`,
+ `target-lexicon`) at import time rather than carried as a parsed type,
+ since nothing downstream needs more than the canonical string back.
`Store::store_tree`/`ref_tree` write and read the document's root tree
- directly, since the two directories underneath must be written into the
- object database before the document that references them can be
- assembled.
+ directly, since a `src` (or `Bin::Embedded` `bin`) directory underneath
+ must be written into the object database before the document that
+ references it can be assembled.
Authored collections::
A collection whose documents treat the commit as the record
crates/git-ents-server/src/checks.rs
@@ -573,13 +573,39 @@
for name in names {
let toolchain = git_toolchain::resolve(repo, name)
.map_err(|e| format!("could not resolve toolchain {name}: {e}"))?;
- let bin = toolchain.bin.oid();
- sync_toolchain(repo, sprite, bin)?;
- dirs.insert(name.to_owned(), format!("{TOOLCHAINS_DIR}/{bin}"));
+ let dir = match &toolchain.bin {
+ git_toolchain::Bin::Embedded(tree) => {
+ let tree = tree.oid();
+ sync_toolchain(repo, sprite, tree)?;
+ format!("{TOOLCHAINS_DIR}/{tree}")
+ }
+ git_toolchain::Bin::Downloaded(components) => {
+ let key = components_key(components);
+ sync_downloaded_toolchain(sprite, &key, components)?;
+ // Unlike an embedded toolchain's tree (already flattened to
+ // put executables at its own top level), each component
+ // archive extracts its own `bin/` alongside `lib/`, so `PATH`
+ // must point one level deeper.
+ format!("{TOOLCHAINS_DIR}/{key}/bin")
+ }
+ };
+ dirs.insert(name.to_owned(), dir);
}
Ok(dirs)
}
+/// A stable, filesystem-safe cache key for a [`git_toolchain::Bin::Downloaded`]
+/// toolchain: its components' sha256s, joined in extraction order — there is
+/// no tree oid to key the extraction cache by, since nothing is written to
+/// the object database for a downloaded toolchain's `bin`.
+fn components_key(components: &[git_toolchain::Component]) -> String {
+ components
+ .iter()
+ .map(|component| component.sha256.as_str())
+ .collect::<Vec<_>>()
+ .join("-")
+}
+
/// Prefix `command` with a `PATH` export activating `toolchains`' extracted
/// `bin` directories, declared order first (so the first-listed toolchain's
/// `bin` wins on a name collision); a check with no toolchains is returned
@@ -653,6 +679,60 @@
}
}
+/// Fetch, sha256-verify, and extract a [`git_toolchain::Bin::Downloaded`]
+/// toolchain's components into the Sprite at `{TOOLCHAINS_DIR}/<key>`, once —
+/// same cache-once discipline as [`sync_toolchain`], keyed by
+/// [`components_key`] since there is no tree oid to key by. Verification and
+/// extraction both happen inside the Sprite via `curl`/`sha256sum`/`tar`,
+/// mirroring `git_toolchain::export`'s local equivalent: downloading through
+/// the server first and streaming the bytes in would defeat the point of not
+/// storing them.
+fn sync_downloaded_toolchain(
+ sprite: &str,
+ key: &str,
+ components: &[git_toolchain::Component],
+) -> Result<(), String> {
+ let dir = format!("{TOOLCHAINS_DIR}/{key}");
+ let cached = Command::new("sprite")
+ .args([
+ "exec",
+ "-s",
+ sprite,
+ "--",
+ "sh",
+ "-c",
+ &format!("[ -d {dir} ]"),
+ ])
+ .status()
+ .map_err(|e| format!("could not run the sprite CLI: {e}"))?;
+ if cached.success() {
+ return Ok(());
+ }
+
+ let mut script = format!("mkdir -p {dir}");
+ for component in components {
+ script.push_str(&format!(
+ " && curl -fsSL '{url}' -o /tmp/component.tar.gz \
+ && [ \"$(sha256sum /tmp/component.tar.gz | cut -d' ' -f1)\" = '{sha256}' ] \
+ && tar -xz --strip-components=2 -C {dir} -f /tmp/component.tar.gz \
+ && rm -f /tmp/component.tar.gz",
+ url = component.url,
+ sha256 = component.sha256,
+ ));
+ }
+ let status = Command::new("sprite")
+ .args(["exec", "-s", sprite, "--", "sh", "-c", &script])
+ .status()
+ .map_err(|e| format!("could not run the sprite CLI: {e}"))?;
+ if status.success() {
+ Ok(())
+ } else {
+ Err(format!(
+ "could not fetch and extract downloaded toolchain {key} in the sprite"
+ ))
+ }
+}
+
/// How long a single check may run before the worker abandons it. A runaway
/// check that outlived this — a hung build, a command blocked on input — is
/// killed and recorded `error` rather than wedging the worker (and with it every
crates/git-ents/src/main.rs
@@ -263,6 +263,11 @@
/// name `rustup` itself knows, e.g. `stable`; defaults to `stable`).
#[facet(args::named)]
spec: Option<String>,
+ /// With `--from`, import the recipe's actual `bin` bytes instead of
+ /// its default of pointing at the distributor's own hosted archives
+ /// (see `git_toolchain::Bin::Downloaded`).
+ #[facet(args::named, default)]
+ embed: bool,
},
/// List the toolchains configured on a remote.
List,
@@ -481,8 +486,9 @@
platform,
from,
spec,
+ embed,
} => toolchain_import(
- name, bin, src, license, version, platform, from, spec, remote,
+ name, bin, src, license, version, platform, from, spec, embed, remote,
),
ToolchainAction::List => toolchain_list(remote),
ToolchainAction::Export { name, dest } => toolchain_export(&name, &dest, remote),
@@ -505,17 +511,24 @@
platform: Option<String>,
from: Option<String>,
spec: Option<String>,
+ embed: bool,
remote: &str,
) -> Result<(), String> {
let name = interactive::text_or(name, "Toolchain name")?;
let recipe = from
- .map(|recipe| registry::resolve(&recipe, spec.as_deref().unwrap_or("stable")))
+ .map(|recipe| registry::resolve(&recipe, spec.as_deref().unwrap_or("stable"), embed))
.transpose()?;
- let bin = match bin.or_else(|| recipe.as_ref().map(|r| r.bin.display().to_string())) {
- Some(bin) => bin,
- None => interactive::text_or(None, "Directory of executables to import")?,
+ let bin_plan = match bin {
+ Some(bin) => registry::Bin::Dir(PathBuf::from(bin)),
+ None => match recipe.as_ref().map(|r| r.bin.clone()) {
+ Some(bin) => bin,
+ None => registry::Bin::Dir(PathBuf::from(interactive::text_or(
+ None,
+ "Directory of executables to import",
+ )?)),
+ },
};
let src = src.or_else(|| {
recipe
@@ -543,15 +556,26 @@
let refname = format!("{TOOLCHAINS_NS}/{name}");
let expected = sync(remote, &refname)?;
let repo = repo()?;
- git_toolchain::import(
- &repo,
- &name,
- Path::new(&bin),
- src.as_deref().map(Path::new),
- &license,
- &version,
- &platform,
- )
+ match bin_plan {
+ registry::Bin::Dir(bin) => git_toolchain::import(
+ &repo,
+ &name,
+ &bin,
+ src.as_deref().map(Path::new),
+ &license,
+ &version,
+ &platform,
+ ),
+ registry::Bin::Components(components) => git_toolchain::import_downloaded(
+ &repo,
+ &name,
+ components,
+ src.as_deref().map(Path::new),
+ &license,
+ &version,
+ &platform,
+ ),
+ }
.map_err(|error| error.to_string())?;
push_signed(remote, &refname, expected.as_deref())?;
println!("imported toolchain {name}");
@@ -569,12 +593,15 @@
return Ok(());
}
for (name, toolchain) in toolchains {
+ let bin = match &toolchain.bin {
+ git_toolchain::Bin::Embedded(tree) => short_id(&tree.oid().to_string()).to_owned(),
+ git_toolchain::Bin::Downloaded(components) => {
+ format!("{} components", components.len())
+ }
+ };
println!(
- "{name} {} {} {} {}",
- short_id(&toolchain.bin.oid().to_string()),
- toolchain.version,
- toolchain.platform,
- toolchain.license
+ "{name} {bin} {} {} {}",
+ toolchain.version, toolchain.platform, toolchain.license
);
}
Ok(())
crates/git-ents/src/registry.rs
@@ -4,23 +4,35 @@
//! toolchain install a user already has (rustup, ...) instead of requiring
//! them to hand-supply paths and metadata `git-toolchain` itself has no way
//! to discover. This module only locates and describes what's already on
-//! disk; it never installs a toolchain.
+//! disk (or, for `bin`, what a distributor already hosts); it never installs
+//! a toolchain.
use std::fs;
use std::os::unix::fs::PermissionsExt as _;
use std::path::{Path, PathBuf};
use std::process::Command;
+use git_toolchain::Component;
use tempfile::TempDir;
+/// How a recipe resolved `bin`: either a local directory to import as-is (the
+/// embedded path, `--embed`), or a list of externally-hosted components to
+/// record as a [`git_toolchain::Bin::Downloaded`] manifest instead of
+/// importing local bytes.
+#[derive(Clone)]
+pub enum Bin {
+ Dir(PathBuf),
+ Components(Vec<Component>),
+}
+
/// What a recipe resolved from a local toolchain install, ready to hand to
-/// `git_toolchain::import`.
+/// `git_toolchain::import`/`import_downloaded`.
///
-/// `_staging`, when present, is a temporary directory `bin` (and `src`, if
-/// under it) points into; it is kept alive only so the directory survives
-/// until the caller's `import()` call has read it, and is deleted on drop.
+/// `_staging`, when `bin` is [`Bin::Dir`] pointing into a temporary
+/// directory, is kept alive only so the directory survives until the
+/// caller's `import()` call has read it, and is deleted on drop.
pub struct Resolved {
- pub bin: PathBuf,
+ pub bin: Bin,
pub src: Option<PathBuf>,
pub license: String,
pub version: String,
@@ -30,9 +42,14 @@
/// Resolve `recipe` against `spec` (a recipe-specific selector, e.g. a
/// rustup toolchain name). The only recipe today is `rustup`.
-pub fn resolve(recipe: &str, spec: &str) -> Result<Resolved, String> {
+///
+/// `embed` forces the old behavior of staging and importing `bin`'s actual
+/// bytes; by default the recipe instead points at its distributor's own
+/// hosted, hash-verified archives (see [`Bin::Components`]), sparing the
+/// repository the toolchain's own bytes.
+pub fn resolve(recipe: &str, spec: &str, embed: bool) -> Result<Resolved, String> {
match recipe {
- "rustup" => rustup(spec),
+ "rustup" => rustup(spec, embed),
other => Err(format!(
"unknown toolchain recipe {other:?} (known: rustup)"
)),
@@ -44,9 +61,19 @@
/// toolchain's own `release` (its version) and `host` (its target platform)
/// without needing rustup's own metadata format.
///
-/// A rustup sysroot's `bin/*` binaries are linked against `lib/*.dylib` (or
-/// `.so`) via an rpath relative to `bin`'s own parent (`@loader_path/../lib`
-/// on macOS, `$ORIGIN/../lib` on Linux) — but `git-toolchain` activates a
+/// By default `bin` is resolved as [`Bin::Components`]: the `rustc`,
+/// `cargo`, and `rust-std` entries of rust-lang's own published channel
+/// manifest for `version` (or the `nightly` channel manifest, which has no
+/// stable per-version name, when `version` is a nightly), each already
+/// hash-pinned by rust-lang. These are real rustup-installer archives: every
+/// one unpacks to `<package>-<version>-<target>/<component>/...`, so
+/// `git_toolchain::export`'s extraction strips exactly that two-segment
+/// prefix rather than needing this recipe to relocate anything.
+///
+/// With `embed`, `bin` is resolved the old way instead: a rustup sysroot's
+/// `bin/*` binaries are linked against `lib/*.dylib` (or `.so`) via an rpath
+/// relative to `bin`'s own parent (`@loader_path/../lib` on macOS,
+/// `$ORIGIN/../lib` on Linux) — but `git-toolchain` activates an embedded
/// toolchain by extracting `bin` as-is and putting *that* directory straight
/// on `PATH`, with no sibling `lib` beside it. Passing `sysroot/bin` alone
/// therefore produces a `rustc` that can neither load its own shared runtime
@@ -56,10 +83,12 @@
/// `sysroot/lib` copied under a `lib/` subdirectory inside it, with each
/// binary's rpath rewritten from `../lib` to `lib` so it resolves relative
/// to wherever the toolchain ends up extracted, not relative to `bin`'s
-/// original location. `src` is `<sysroot>/lib/rustlib/src/rust`, unstaged,
-/// when the `rust-src` component is installed, else omitted. Rust's own
+/// original location.
+///
+/// `src` is `<sysroot>/lib/rustlib/src/rust`, unstaged, when the `rust-src`
+/// component is installed, else omitted, regardless of `embed`. Rust's own
/// toolchain is dual-licensed `MIT OR Apache-2.0`.
-fn rustup(spec: &str) -> Result<Resolved, String> {
+fn rustup(spec: &str, embed: bool) -> Result<Resolved, String> {
let toolchain_arg = format!("+{spec}");
let sysroot = rustc(&toolchain_arg, &["--print", "sysroot"])?;
let sysroot = PathBuf::from(sysroot.trim());
@@ -70,23 +99,83 @@
let platform = verbose_field(&verbose, "host")
.ok_or_else(|| format!("rustc +{spec} -vV did not report a host"))?;
- let staging = tempfile::tempdir()
- .map_err(|error| format!("could not create a staging directory: {error}"))?;
- stage_bin(&sysroot.join("bin"), &sysroot.join("lib"), staging.path())?;
-
let src = sysroot.join("lib/rustlib/src/rust");
let src = src.is_dir().then_some(src);
+ let (bin, staging) = if embed {
+ let staging = tempfile::tempdir()
+ .map_err(|error| format!("could not create a staging directory: {error}"))?;
+ stage_bin(&sysroot.join("bin"), &sysroot.join("lib"), staging.path())?;
+ (Bin::Dir(staging.path().to_owned()), Some(staging))
+ } else {
+ (
+ Bin::Components(manifest_components(&version, &platform)?),
+ None,
+ )
+ };
+
Ok(Resolved {
- bin: staging.path().to_owned(),
+ bin,
src,
license: "MIT OR Apache-2.0".to_owned(),
version,
platform,
- _staging: Some(staging),
+ _staging: staging,
})
}
+/// The three components of rust-lang's channel manifest that together make
+/// a working toolchain (compiler, cargo, and the target's standard library),
+/// resolved for `target` against the manifest for `version` (or the shared
+/// `nightly` channel, when `version` names one — rust-lang does not publish
+/// a stable per-version manifest name for nightly builds).
+fn manifest_components(version: &str, target: &str) -> Result<Vec<Component>, String> {
+ let channel = if version.contains("nightly") {
+ "nightly".to_owned()
+ } else {
+ version.to_owned()
+ };
+ let url = format!("https://static.rust-lang.org/dist/channel-rust-{channel}.toml");
+ let manifest = crate::http_get(&url)?;
+
+ ["rustc", "cargo", "rust-std"]
+ .into_iter()
+ .map(|package| {
+ let section = format!("pkg.{package}.target.{target}");
+ let component_url = manifest_field(&manifest, §ion, "url")
+ .ok_or_else(|| format!("{url} has no [{section}].url"))?;
+ let sha256 = manifest_field(&manifest, §ion, "hash")
+ .ok_or_else(|| format!("{url} has no [{section}].hash"))?;
+ Ok(Component {
+ url: component_url,
+ sha256,
+ })
+ })
+ .collect()
+}
+
+/// Extract `<key> = "value"` from `manifest`'s `[section]` table.
+///
+/// A hand-rolled reader for the one shape this recipe needs from rust-lang's
+/// channel manifest TOML (a flat `key = "value"` line under a `[section]`
+/// header), rather than a full TOML parser for a format this is the only
+/// caller of.
+fn manifest_field(manifest: &str, section: &str, key: &str) -> Option<String> {
+ let prefix = format!("{key} = \"");
+ let mut in_section = false;
+ for line in manifest.lines() {
+ let line = line.trim();
+ if let Some(name) = line.strip_prefix('[').and_then(|s| s.strip_suffix(']')) {
+ in_section = name == section;
+ continue;
+ }
+ if in_section && let Some(rest) = line.strip_prefix(&prefix) {
+ return rest.strip_suffix('"').map(str::to_owned);
+ }
+ }
+ None
+}
+
/// Copy `bin_src`'s executables flat into `staging`, relink each one's rpath
/// from `bin`-relative (`../lib`) to `staging`-relative (`lib`), then copy
/// the whole of `lib_src` under `staging/lib`.
crates/git-toolchain/src/lib.rs
@@ -22,8 +22,10 @@
//! operational follow-up, not something this crate does.
use std::fs;
+use std::io::Write as _;
use std::os::unix::fs::PermissionsExt as _;
use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
use std::str::FromStr as _;
use facet::Facet;
@@ -43,9 +45,11 @@
/// its license, version, and target platform — the document stored at the
/// tip of `refs/meta/toolchains/<name>`.
///
-/// `bin` and `src` are [`RawTree`]: each is captured as a single opaque git
-/// tree by [`import`], not walked field-by-field, since a toolchain's
-/// on-disk layout has no fixed shape for `Facet` to model.
+/// `src` is [`RawTree`]: captured as a single opaque git tree by [`import`],
+/// not walked field-by-field, since a toolchain's on-disk layout has no fixed
+/// shape for `Facet` to model. `bin` is either the same ([`Bin::Embedded`])
+/// or a set of externally-hosted archives fetched fresh at activation or
+/// export time ([`Bin::Downloaded`]) — see [`Bin`].
///
/// `license`, `version`, and `platform` are stored as plain strings — like
/// `license` before them, `version` and `platform` are validated against a
@@ -56,7 +60,7 @@
pub struct Toolchain {
/// The toolchain's executables, activated on `PATH` when a check
/// requests it.
- pub bin: RawTree,
+ pub bin: Bin,
/// The toolchain's source, if imported — not activated on `PATH`, kept
/// only for provenance.
pub src: Option<RawTree>,
@@ -72,6 +76,36 @@
pub platform: String,
}
+/// How a toolchain's `bin` is provisioned.
+#[derive(Debug, Clone, PartialEq, Facet)]
+#[repr(u8)]
+pub enum Bin {
+ /// `bin`'s directory tree, captured whole in the object database by
+ /// [`import`] — the only representation for a toolchain with no stable,
+ /// independently-hosted origin (an in-house build).
+ Embedded(RawTree),
+ /// A set of archives fetched, sha256-verified, and merged onto disk by
+ /// [`export`] (or a Sprite, at check-activation time) instead of stored
+ /// in the object database — spares the repository the toolchain's own
+ /// bytes when a stable, content-hashed origin (a distributor's release
+ /// archives) already exists. Each component is extracted with its outer
+ /// two path segments (`<package>-<version>-<target>/<component>/`)
+ /// stripped, the layout rust-lang's (and most other distributors')
+ /// dist archives use.
+ Downloaded(Vec<Component>),
+}
+
+/// One archive making up a [`Bin::Downloaded`] toolchain: fetched from `url`
+/// and checked against `sha256` before being extracted.
+#[derive(Debug, Clone, PartialEq, Facet)]
+pub struct Component {
+ /// Where to fetch the archive from.
+ pub url: String,
+ /// The archive's expected sha256, hex-encoded — checked before
+ /// extraction; a mismatch is refused rather than extracted anyway.
+ pub sha256: String,
+}
+
/// A failure importing, resolving, listing, exporting, or removing a
/// toolchain.
#[derive(Debug, thiserror::Error)]
@@ -103,6 +137,17 @@
/// activates nothing on `PATH` is not a toolchain.
#[error("{0} contains nothing importable; a toolchain's bin directory must not be empty")]
EmptyBin(PathBuf),
+ /// [`import_downloaded`]'s component list was empty. A toolchain that
+ /// activates nothing on `PATH` is not a toolchain.
+ #[error("a downloaded toolchain must list at least one component")]
+ NoComponents,
+ /// A [`Bin::Downloaded`] component could not be fetched or extracted.
+ #[error("could not fetch {0}: {1}")]
+ Fetch(String, String),
+ /// A [`Bin::Downloaded`] component's fetched content did not match its
+ /// recorded sha256.
+ #[error("{0}: expected sha256 {1}, got {2}")]
+ HashMismatch(String, String, String),
/// [`import`]'s `license` argument was not a valid SPDX license
/// expression.
#[error("{0:?} is not a valid SPDX license expression: {1}")]
@@ -135,26 +180,15 @@
if !git_store::ref_segment_ok(name) {
return Err(Error::InvalidName(name.to_owned()));
}
- spdx::Expression::parse(license)
- .map_err(|error| Error::InvalidLicense(license.to_owned(), error))?;
- semver::Version::parse(version)
- .map_err(|error| Error::InvalidVersion(version.to_owned(), error))?;
- target_lexicon::Triple::from_str(platform)
- .map_err(|_error| Error::InvalidPlatform(platform.to_owned()))?;
+ validate_metadata(license, version, platform)?;
let odb = odb_at(repo)?;
let bin_tree = build_tree(&odb, bin_dir)?;
if bin_tree.entries.is_empty() {
return Err(Error::EmptyBin(bin_dir.to_owned()));
}
- let bin = RawTree::new(write_object(&odb, &bin_tree)?);
-
- let src = src_dir
- .map(|dir| -> Result<RawTree, Error> {
- let tree = build_tree(&odb, dir)?;
- Ok(RawTree::new(write_object(&odb, &tree)?))
- })
- .transpose()?;
+ let bin = Bin::Embedded(RawTree::new(write_object(&odb, &bin_tree)?));
+ let src = import_src(&odb, src_dir)?;
let toolchain = Toolchain {
bin,
@@ -163,8 +197,79 @@
version: version.to_owned(),
platform: platform.to_owned(),
};
- let oid = facet_git_tree::serialize_into(&toolchain, &odb)?;
+ store_toolchain(repo, name, toolchain, &odb)
+}
+/// Import a toolchain whose `bin` is a set of externally-hosted archives
+/// (see [`Bin::Downloaded`]) instead of a local directory: no tree is walked
+/// or written for `bin` itself, only the component list and the rest of the
+/// document. `src_dir`, if given, is still captured as a `RawTree` the usual
+/// way — provenance-only content with no natural external origin to point at
+/// instead.
+pub fn import_downloaded(
+ repo: &Path,
+ name: &str,
+ components: Vec<Component>,
+ src_dir: Option<&Path>,
+ license: &str,
+ version: &str,
+ platform: &str,
+) -> Result<ObjectId, Error> {
+ if !git_store::ref_segment_ok(name) {
+ return Err(Error::InvalidName(name.to_owned()));
+ }
+ if components.is_empty() {
+ return Err(Error::NoComponents);
+ }
+ validate_metadata(license, version, platform)?;
+ let odb = odb_at(repo)?;
+ let src = import_src(&odb, src_dir)?;
+
+ let toolchain = Toolchain {
+ bin: Bin::Downloaded(components),
+ src,
+ license: license.to_owned(),
+ version: version.to_owned(),
+ platform: platform.to_owned(),
+ };
+ store_toolchain(repo, name, toolchain, &odb)
+}
+
+/// `license` MUST be a valid SPDX license expression, `version` a valid
+/// semver version, and `platform` a valid target triple — shared by
+/// [`import`] and [`import_downloaded`].
+fn validate_metadata(license: &str, version: &str, platform: &str) -> Result<(), Error> {
+ spdx::Expression::parse(license)
+ .map_err(|error| Error::InvalidLicense(license.to_owned(), error))?;
+ semver::Version::parse(version)
+ .map_err(|error| Error::InvalidVersion(version.to_owned(), error))?;
+ target_lexicon::Triple::from_str(platform)
+ .map_err(|_error| Error::InvalidPlatform(platform.to_owned()))?;
+ Ok(())
+}
+
+/// Write `src_dir`, if given, as a `RawTree` — shared by [`import`] and
+/// [`import_downloaded`], since `src` is captured the same way regardless of
+/// how `bin` is provisioned.
+fn import_src(odb: &gix::odb::Handle, src_dir: Option<&Path>) -> Result<Option<RawTree>, Error> {
+ src_dir
+ .map(|dir| -> Result<RawTree, Error> {
+ let tree = build_tree(odb, dir)?;
+ Ok(RawTree::new(write_object(odb, &tree)?))
+ })
+ .transpose()
+}
+
+/// Serialize `toolchain` and fast-forward `refs/meta/toolchains/<name>` to a
+/// commit over it — the shared final step of [`import`] and
+/// [`import_downloaded`].
+fn store_toolchain(
+ repo: &Path,
+ name: &str,
+ toolchain: Toolchain,
+ odb: &gix::odb::Handle,
+) -> Result<ObjectId, Error> {
+ let oid = facet_git_tree::serialize_into(&toolchain, odb)?;
let store = Store::open(repo)?;
store.store_tree(
&toolchain_ref(name),
@@ -204,14 +309,26 @@
/// under `dest`, restoring the executable bit and symlinks. Refuses to write
/// into a `dest` that already has contents. Returns the resolved document,
/// so the caller can report the license alongside the exported files.
+///
+/// [`Bin::Embedded`] writes its (already self-contained: executables plus a
+/// sibling `lib/`) tree straight under `dest/bin`. [`Bin::Downloaded`]'s
+/// components already carry their own `bin/`/`lib/`/... top-level
+/// directories once their outer two path segments are stripped, so they are
+/// extracted directly under `dest` instead, landing at the same `dest/bin/…`
+/// shape by construction.
pub fn export(repo: &Path, name: &str, dest: &Path) -> Result<Toolchain, Error> {
let toolchain = resolve(repo, name)?;
let odb = odb_at(repo)?;
ensure_empty_dest(dest)?;
- let bin_dest = dest.join("bin");
- fs::create_dir_all(&bin_dest).map_err(|error| Error::Io(bin_dest.clone(), error))?;
- write_tree_to_disk(&odb, toolchain.bin.oid(), &bin_dest)?;
+ match &toolchain.bin {
+ Bin::Embedded(tree) => {
+ let bin_dest = dest.join("bin");
+ fs::create_dir_all(&bin_dest).map_err(|error| Error::Io(bin_dest.clone(), error))?;
+ write_tree_to_disk(&odb, tree.oid(), &bin_dest)?;
+ }
+ Bin::Downloaded(components) => download_components(components, dest)?,
+ }
if let Some(src) = &toolchain.src {
let src_dest = dest.join("src");
@@ -387,9 +504,115 @@
Ok(())
}
+/// Fetch, verify, and extract every component of a [`Bin::Downloaded`]
+/// toolchain into `dest`, in order — later components overlay earlier ones,
+/// matching how rustup itself layers `rustc`/`cargo`/`rust-std` onto one
+/// sysroot.
+fn download_components(components: &[Component], dest: &Path) -> Result<(), Error> {
+ for component in components {
+ let archive = fetch(&component.url)?;
+ let actual = sha256_hex(&archive)?;
+ if actual != component.sha256 {
+ return Err(Error::HashMismatch(
+ component.url.clone(),
+ component.sha256.clone(),
+ actual,
+ ));
+ }
+ extract_stripped(&archive, dest)?;
+ }
+ Ok(())
+}
+
+/// `GET url` via the system `curl`, returning the response body — shells out
+/// rather than adding an HTTP client dependency to this crate.
+fn fetch(url: &str) -> Result<Vec<u8>, Error> {
+ let output = Command::new("curl")
+ .args(["-fsSL", url])
+ .output()
+ .map_err(|error| Error::Fetch(url.to_owned(), error.to_string()))?;
+ if !output.status.success() {
+ return Err(Error::Fetch(
+ url.to_owned(),
+ String::from_utf8_lossy(&output.stderr).into_owned(),
+ ));
+ }
+ Ok(output.stdout)
+}
+
+/// Hex-encoded sha256 of `bytes`, via the system `shasum` (macOS) or
+/// `sha256sum` (Linux) — shells out rather than adding a hashing dependency
+/// to this crate.
+fn sha256_hex(bytes: &[u8]) -> Result<String, Error> {
+ let (program, args): (&str, &[&str]) = match std::env::consts::OS {
+ "macos" => ("shasum", &["-a", "256"]),
+ _ => ("sha256sum", &[]),
+ };
+ let mut child = Command::new(program)
+ .args(args)
+ .stdin(Stdio::piped())
+ .stdout(Stdio::piped())
+ .spawn()
+ .map_err(|error| Error::Fetch(program.to_owned(), error.to_string()))?;
+ child
+ .stdin
+ .take()
+ .ok_or_else(|| Error::Fetch(program.to_owned(), "no stdin".to_owned()))?
+ .write_all(bytes)
+ .map_err(|error| Error::Fetch(program.to_owned(), error.to_string()))?;
+ let output = child
+ .wait_with_output()
+ .map_err(|error| Error::Fetch(program.to_owned(), error.to_string()))?;
+ if !output.status.success() {
+ return Err(Error::Fetch(
+ program.to_owned(),
+ String::from_utf8_lossy(&output.stderr).into_owned(),
+ ));
+ }
+ let hex = String::from_utf8_lossy(&output.stdout);
+ hex.split_whitespace()
+ .next()
+ .map(str::to_owned)
+ .ok_or_else(|| Error::Fetch(program.to_owned(), "no hash in output".to_owned()))
+}
+
+/// Extract a gzipped tar `archive` into `dest`, stripping the outer two path
+/// segments every rust-lang dist archive (and most other distributors')
+/// wraps its payload in (`<package>-<version>-<target>/<component>/`).
+fn extract_stripped(archive: &[u8], dest: &Path) -> Result<(), Error> {
+ let mut child = Command::new("tar")
+ .args(["-xz", "--strip-components=2", "-C"])
+ .arg(dest)
+ .stdin(Stdio::piped())
+ .spawn()
+ .map_err(|error| Error::Fetch("tar".to_owned(), error.to_string()))?;
+ child
+ .stdin
+ .take()
+ .ok_or_else(|| Error::Fetch("tar".to_owned(), "no stdin".to_owned()))?
+ .write_all(archive)
+ .map_err(|error| Error::Fetch("tar".to_owned(), error.to_string()))?;
+ let status = child
+ .wait()
+ .map_err(|error| Error::Fetch("tar".to_owned(), error.to_string()))?;
+ if status.success() {
+ Ok(())
+ } else {
+ Err(Error::Fetch(
+ "tar".to_owned(),
+ "extraction failed".to_owned(),
+ ))
+ }
+}
+
#[cfg(test)]
mod tests {
- #![allow(clippy::unwrap_used, clippy::indexing_slicing, reason = "unit test")]
+ #![allow(
+ clippy::unwrap_used,
+ clippy::indexing_slicing,
+ clippy::unreachable,
+ reason = "unit test"
+ )]
use git_store::test_support::repo;
@@ -460,9 +683,12 @@
)
.unwrap();
let toolchain = resolve(repo_dir.path(), "gcc").unwrap();
+ let Bin::Embedded(bin) = &toolchain.bin else {
+ unreachable!("import always produces an embedded bin");
+ };
let odb = odb_at(repo_dir.path()).unwrap();
let mut buf = Vec::new();
- let tree = odb.find_tree(&toolchain.bin.oid(), &mut buf).unwrap();
+ let tree = odb.find_tree(&bin.oid(), &mut buf).unwrap();
assert!(tree.entries.iter().all(|entry| entry.filename != "empty"));
}
@@ -556,6 +782,99 @@
assert_eq!(fs::read(dest_path.join("bin/README")).unwrap(), b"hello\n");
}
+ /// Build a `file://` component archive matching real dist tarballs'
+ /// layout (`<pkg>-<version>-<target>/<component>/<payload>`), returning
+ /// its `file://` URL and sha256, ready to hand to [`Component`].
+ fn build_component(staging: &Path, payload: &[(&str, &[u8])]) -> Component {
+ let root = staging.join("pkg-1.0.0-target/component");
+ for (path, contents) in payload {
+ let full = root.join(path);
+ fs::create_dir_all(full.parent().unwrap()).unwrap();
+ fs::write(&full, contents).unwrap();
+ }
+ let archive = staging.join("component.tar.gz");
+ let status = Command::new("tar")
+ .args(["-czf"])
+ .arg(&archive)
+ .args(["-C"])
+ .arg(staging)
+ .arg("pkg-1.0.0-target/component")
+ .status()
+ .unwrap();
+ assert!(status.success());
+ let bytes = fs::read(&archive).unwrap();
+ Component {
+ url: format!("file://{}", archive.display()),
+ sha256: sha256_hex(&bytes).unwrap(),
+ }
+ }
+
+ #[test]
+ fn import_downloaded_then_export_extracts_and_strips_components() {
+ let repo_dir = repo();
+ let staging = tempfile::tempdir().unwrap();
+ let component = build_component(staging.path(), &[("bin/tool", b"#!/bin/sh\n")]);
+
+ import_downloaded(
+ repo_dir.path(),
+ "rustup-like",
+ vec![component],
+ None,
+ "MIT",
+ VERSION,
+ PLATFORM,
+ )
+ .unwrap();
+
+ let dest = tempfile::tempdir().unwrap();
+ let dest_path = dest.path().join("out");
+ let toolchain = export(repo_dir.path(), "rustup-like", &dest_path).unwrap();
+ assert!(matches!(toolchain.bin, Bin::Downloaded(_)));
+ assert_eq!(
+ fs::read(dest_path.join("bin/tool")).unwrap(),
+ b"#!/bin/sh\n"
+ );
+ }
+
+ #[test]
+ fn import_downloaded_rejects_an_empty_component_list() {
+ let repo_dir = repo();
+ let result = import_downloaded(
+ repo_dir.path(),
+ "rustup-like",
+ vec![],
+ None,
+ "MIT",
+ VERSION,
+ PLATFORM,
+ );
+ assert!(matches!(result, Err(Error::NoComponents)));
+ }
+
+ #[test]
+ fn export_rejects_a_component_whose_hash_does_not_match() {
+ let repo_dir = repo();
+ let staging = tempfile::tempdir().unwrap();
+ let mut component = build_component(staging.path(), &[("bin/tool", b"#!/bin/sh\n")]);
+ component.sha256 = "0".repeat(64);
+
+ import_downloaded(
+ repo_dir.path(),
+ "rustup-like",
+ vec![component],
+ None,
+ "MIT",
+ VERSION,
+ PLATFORM,
+ )
+ .unwrap();
+
+ let dest = tempfile::tempdir().unwrap();
+ let dest_path = dest.path().join("out");
+ let result = export(repo_dir.path(), "rustup-like", &dest_path);
+ assert!(matches!(result, Err(Error::HashMismatch(_, _, _))));
+ }
+
#[test]
fn export_refuses_a_non_empty_destination() {
let repo_dir = repo();