Make fields length and hashes mandatory on MetadataDescription

[heartsucker]

The spec says they can be left out at times, but we might as well include them and bolt them down to be required and match in all cases. This would add a small amount of additional security during checks, but it would add additional security to the internals of the lib by preventing misusing the struct to not include fields when it should.

I can't think of a reason not to do this (other than saving bytes in transit). @trishankkarthik, any thoughts?

[trishankkarthik]

Sorry, where does the spec say that they can be left out?

For the targets metadata, the "custom" attribute is optional, but the "target" attribute (containing the length and hashes) are not.

I am assuming you mean the snapshot metadata? For targets metadata files, we require only the version numbers. I think we additionally require the hashes and length of only the root metadata file in there. We don't need the length in there, because that can be fixed in the updater client code. @JustinCappos, do you think we need the hashes of the root metadata in there? What attacks can happen if they are not there?

Cc: @vladimir-v-diaz

[heartsucker]

https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt#L677-L681

The HASHES and LENGTH are the hashes and length of the file, both of which are only specified for the root file. VERSION is listed for the root file and all other roles available on the repository. LENGTH is an integer. HASHES is a dictionary that specifies one or more hashes, including the cryptographic hash function. For example: { "sha256": HASH, ... }.

This is metadata about other metadata, not targets.

[trishankkarthik]

Thanks for the clarification! Here's my TLDR summary:

1. You need to list only the version numbers of all targets metadata files, including the top-level targets role.

2. You do not need to list the hashes of the targets metadata files. This will increase bandwidth costs, and is useful only in the threat model where you are worried about malicious mirrors being able to substitute some targets metadata files (because they somehow have the private signing keys to these files), but not the snapshot metadata file.

More details will be published soon in the Mercury paper next week.

I'll let Justin and Vlad answer whether we need to list in the snapshot metadata file the hashes of the root metadata file.

[vladimir-v-diaz]

@heartsucker

We used to list the hashes of all delegated role files in Snapshot, but discovered that it added a large bandwidth cost to community repositories like PyPI. The Mercury paper shows that including the hashes of all metadata (and their version numbers) in Snapshot incurs an initial cost of 360% and a recurring cost of 24%. The cost is the overhead of adding TUF metadata to PyPI.

In contrast, only listing version numbers results in an initial cost of 48% and recurring cost of 24%.

[heartsucker]

Ok, so then this goes back to the edge case of the snapshot metadata needing to include the hashes / length of the root from some other issues / email / discussion somewhere. I'm not sure what edge case that solves then? Isn't that just the case where the snapshot is compromised and is intentionally falsely reporting the root? What attacker would do that? Wouldn't they try to cover their tracks by just reporting the root correctly?

[vladimir-v-diaz]

I think @JustinCappos recommended that we list the hash of Root in Snapshot to help recover from fast-forward attacks, but this was before any of the changes proposed in the TAPs (specifically, TAP 5?)

[vladimir-v-diaz]

Adding to my previous reply...

Adding Root's hash was also done before CoreOS's contribution to Root key rotations.

[heartsucker]

to help recover from fast-forward attacks

A fast forward on which role?

[trishankkarthik]

Fast forward attacks: where attackers arbitrarily increase the version numbers of targets metadata files compared to their actual values. This results in denial-of-service attacks, where TUF will reject the target metadata file because its actual version number != the alleged version number in the snapshot metadata.

To recover, administrators revoke and replace timestamp+snapshot keys using the root metadata. When TUF clients update, they will now always first download the root metadata. When they see that these keys have been replaced, they will delete and replace previous timestamp+snapshot metadata, hence removing the denial-of-service attacks.

[vladimir-v-diaz]

@heartsucker
Suppose an attacker launches a fast forward attack, where they list a maliciously high version number for Root (and no hash is listed for it). For instance, the actual version number for Root is 4, but the attacker says it is now 666.

The repository detects the attack and attempts to release a new version of Root to revoke Snapshot's keys. The new Root file has a version of 5. Unfortunately, a client would reject the Root file because version 5 < 666. If Root's hash were instead listed in Snapshot, the client could reset its cache of currently-trusted version numbers and use it to recover from the fast-forward attack.

At least, that's how it would have been done before we introduced Root key rotations and TAP 5...
(I hope my explanation is clear and correct. So many things have changed over time, including switching back-and-forth between listing and not listing Root hashes, that I might have missed a reason for a change. The TAP process actually stems from this...)

[heartsucker]

I knew what fastforward attacks were in general, but in a previous discussion, it was said to blow everything away after a root rotation and start over. New root implies new keys for the snapshot role which means the evil snapshot would be out of the question and the client would just grab the newest snapshot and start trusting that, no?