Uploading a TUF repo fails while unpacking the composite control plane tarball

[jgallagher]

I tried uploading a TUF repo on dublin, and it failed with this error:

oxide system update repo upload --path tuf-mupdate.zip
Uploading tuf-mupdate.zip...
[00:01:00] [██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 2.58 GiB/2.58 GiB (43.37 MiB/s, 0s
error
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "8fa40fc4-8712-46d1-9444-b7f9667fedec", "content-length": "171", "date": "Wed, 23 Jul 2025 19:55:11 GMT"}; value: Error { error_code: None, message: "error extracting tarball for control_plane from repository\nCaused by:\n  -> invalid gzip header", request_id: "8fa40fc4-8712-46d1-9444-b7f9667fedec" }

We're pretty sure this was introduced by #7906. The control plane tarball inside the TUF repo now contains all the zones it did before, but also a couple of measurement CBOR files:

./oxide.json
./measurements
./measurements/e0ffbd6e86b3ba9891ee21d7d1e2a98667ef598070eef941d8088bc724af526b.cbor
./measurements/c1e0ef5f799cb4df2cb41e91c4dd82978e5b4abe5a18fe83715b8f3df05f8958.cbor
./zones
./zones/crucible_pantry.tar.gz
./zones/internal_dns.tar.gz
./zones/clickhouse_keeper.tar.gz
./zones/crucible.tar.gz
./zones/nexus.tar.gz
./zones/clickhouse.tar.gz
./zones/probe.tar.gz
./zones/cockroachdb.tar.gz
./zones/external_dns.tar.gz
./zones/oximeter.tar.gz
./zones/ntp.tar.gz
./zones/clickhouse_server.tar.gz

As we're unpacking this composite artifact, we attempt to extract the artifact versions from each zone or manifest file. This assumes those files are themselves gzipped tar files containing an oxide.json:

omicron/update-common/src/artifacts/update_plan.rs

Lines 920 to 955 in de056f0

	ControlPlaneZoneImages::extract_into(reader, |_, kind, reader| {
	let known_kind = match kind {
	ControlPlaneEntry::Zone => KnownArtifactKind::Zone,
	ControlPlaneEntry::MeasurementCorpus => {
	KnownArtifactKind::MeasurementCorpus
	}
	};
	let mut out = self.extracted_artifacts.new_tempfile()?;
	io::copy(reader, &mut out)?;
	let data = self
	.extracted_artifacts
	.store_tempfile(known_kind.into(), out)?;
	// Read the zone name and version from the `oxide.json` at the root
	// of the zone.
	let data_clone = data.clone();
	let file = Handle::current().block_on(async move {
	std::io::Result::Ok(data_clone.file().await?.into_std().await)
	})?;
	let mut tar = tar::Archive::new(flate2::read::GzDecoder::new(file));
	let metadata =
	tufaceous_brand_metadata::Metadata::read_from_tar(&mut tar)?;
	let info = metadata.layer_info()?;
	let artifact_id = ArtifactId {
	name: info.pkg.clone(),
	version: ArtifactVersion::new(info.version.to_string())?,
	kind: known_kind.into(),
	};
	self.record_extracted_artifact(
	artifact_id,
	data,
	known_kind.into(),
	self.log,
	)?;
	Ok(())

These CBOR files are not gzipped tar files, so we get the error above.

For now we're going to revert #7906 to unblock update testing; I'll put some more notes in a comment below.

[jgallagher]

It's a little surprising we didn't catch this in CI; we have several tests that unpack a TUF repo that exercise this code path. However, all of those do so by assembling a fake TUF repo. tufaceous's support for fake measurements is incorrect: it's going through the same code path it does to create fake zones, which means it emits a gzipped tarball containing an oxide.json for fake measurements, which means the code linked in the original issue succeeds on these fakes, even though it fill fail on real CBOR files.

Some ideas for a hopefully-quick fix:

• Change tufaceous so it produces more realistic fake measurements. (https://github.com/oxidecomputer/tufaceous/tree/john/split-fake-zones-measurements is one way to approach that, although it produces garage measurement files and not real CBOR, which may be critically wrong given the next bullet. With this branch, we have omicron tests that fail with the same invalid gzip header seen on dublin.)
• Teach UpdatePlanBuilder::extract_control_plane_zones_impl() to extract the version from the measurement CBORs instead of trying to interpret them as zone tarballs. @labbott noted we can do this via https://github.com/oxidecomputer/rats-corim.

---

Medium term suggestion from @iliana:

i'm wondering if the releng CI job should try pushing the archive zip through the update-common zip-reading machinery when it's done

---

Longer term: Some context on why we're adding the measurements to the control plane zones tarball in the first place is in oxidecomputer/tufaceous#21. We know it's a little sketchy, but we need to make these available to installinator, which means either sneaking them into the control plane zones tarball as we've done or doing some nontrivial (and tricky to test across releases) rework to the wicket <-> installinator interface. If the hopefully-quick fix doesn't pan out, it may be worth investigating how difficult that would be? Some ideas on that front:

• We could separate the artifacts at the TUF level but have wicket repack them (@iliana noted this in Add measurement corpus as part of the control plane artifacts tufaceous#21 (review)).
• @andrewjstone and @sunshowers have previously both suggested that instead of fetching the control plane artifact only, installinator could fetch a manifest from wicket describing artifacts to fetch, then have it fetch those. This might have to be staged across two releases since installinator has to talk to old wicket, but IIRC wicket will happily serve artifacts it doesn't understand so there might be a way to do it backwards compatibly.

[labbott]

Longer term: Some context on why we're adding the measurements to the control plane zones tarball in the first place is oxidecomputer/tufaceous#21. We know it's a little sketchy, but we need to make these available to installinator, which means either sneaking them into the control plane zones tarball as we've done or doing some nontrivial (and tricky to test across releases) rework to the wicket <-> installinator interface. If the hopefully-quick fix doesn't pan out, it may be worth investigating how difficult that would be? Some ideas on that front:

* We could separate the artifacts at the TUF level but have wicket repack them ([@iliana](https://github.com/iliana) noted this in [Add measurement corpus as part of the control plane artifacts tufaceous#21 (review)](https://github.com/oxidecomputer/tufaceous/pull/21#pullrequestreview-2810615481)).

* [@andrewjstone](https://github.com/andrewjstone) and [@sunshowers](https://github.com/sunshowers) have previously both suggested that instead of fetching the control plane artifact only, installinator could fetch a manifest from wicket describing artifacts to fetch, then have it fetch those. This might have to be staged across two releases since installinator has to talk to old wicket, but IIRC wicket will happily serve artifacts it doesn't understand so there might be a way to do it backwards compatibly.

What's the chance we're going to need to add more artifact types to the TUF repository? "Never" is probably wrong but I don't have a good sense for what else might be added. Maybe as a concrete example, would anything we need to do for comso require this change?

[jgallagher]

Maybe as a concrete example, would anything we need to do for comso require this change?

We've already added cosmo.rom to the TUF repository, but (similar to this) it's packed inside the composite "host OS tarball", and currently none of our unpacking code (or wicket) knows to look for it. More in #8623 (including that we should switch from pulling out the rom file for gimlets to pulling out gimlet.rom more explicitly). I don't think there's anything else coming with cosmo, assuming the SP / RoT / bootloader images are "more of the same" in that we already support various gimlet board revisions anyway.

I know @sunshowers did some work in this area and was pretty uncomfortable in general with these composite artifacts that have to be unpacked; I think we hoped that any new artifact additions would not make this situation worse. But that seems to be pretty strongly in tension with general urgency; moving away from composite artifacts or adding new top-level artifacts is may require reworking some finicky things like the installinator <-> wicket interface, depending on how we go about it.