Implement a text-based lockfile format

[Jarred-Sumner]

This is a tracking issue for implementing a text-based lockfile format for bun install and making it the default format going forward. There will be a smooth migration path from bun.lockb → bun.lock.

Why?

When first working on the package manager, the flamegraph showed parsing JSON was the biggest number. So instead of a JSON lockfile, we designed two efficient binary formats: one for the registry manifest cache and one for the lockfile.

The binary lockfile format has served us well, but it isn't worth the cost in developer experience, particularly for larger teams working together. We suggest workarounds today for various things, but none of them are great.

• Merge conflicts are hard. Do you pick bun.lockb.1 or bun.lockb.2? What about both?
• How do you inspect the lockfile from a PR on GitHub/GitLab/etc?
• How do you diff the lockfile? You can print as a yarn.lock and configure git to diff that way, but that's a whole lot more complicated than it not being a problem in the first place
• When there are lockfile changes, it's not clear enough why. Sometimes, you run bun install and the lockfile has changes due to hashes changing, package.json scripts changing, or other reasons and this is hard to understand right now because it's hard to read in a text editor.
• People too frequently add bun.lockb to the .gitignore file

Will bun install still be fast?

Yes.

Will it make bun install slower?

Based on what we've seen, about 1-20 milliseconds.

What will the new format be?

Probably JSON with Trailing Commas, like tsconfig.json.

Bun already supports this schema format for the runtime (and package.json). TOML is another option and it's what we use for bunfig.toml, though I kind of think that was a mistake and it should've been JSON with Trailing Commas. TOML's editor tooling support is not as mature as JSON with Trailing Commas.

Why JSON with Trailing Commas instead of JSON?

Too many collective human lifetimes have been spent fixing merge conflicts from diffs caused by adding or removing trailing commas at the end of lists in JSON

-    }
+    },

Why JSON with Trailing Commas instead of YAML?

YAML is fine. We don't have a YAML parser in Bun yet, and the indentation gets really confusing sometimes. Also, YAML parsers tend to be slower than JSON parsers (and YAML parsers are also JSON parsers). My favorite YAML fact is that the two-digit country code for Norway in YAML is parsed as false, though this isn't relevant to a lockfile

Why JSON with Trailing Commas instead of JSON5?

We don't have a JSON5 parser in Bun, and would like to avoid formats that're slower to parse than JSON

When will bun.lock be released?

Q3.

What will the migration plan be?

bun install will support both bun.lockb and bun.lock for awhile, but once released, new features will only be supported via bun.lock.

Do people really edit lockfiles manually?

Yes, for small tweaks it can be important

Why not fix merge conflicts via bun install instead?

We will support that as well. But, it shouldn't be impossible to do manually.

[whyman]

Feels like json5 could also be a good choice, that has a real spec at least. I realise there is no parser currently but surely it's better than having jsons that are not valid with most parsers

[styfle]

Regarding json5 vs jsonc, there is a related issue with some discussion about tsconfig.json from a few years ago:

• [settings] Use JSON5 instead of JSONC for configuration files microsoft/vscode#100688

[anru]

Good design goals from yarn:

1. Must be easy to read by a human
2. Must be easy to diff
3. Must be fast to parse

To achieve 3 it is useful to have a dedicated format and parser.
Common formats, formats that support many options and functionality require additional logic, branches in the code, etc. which negatively affects the parsing speed.

Diffability at yarn.lock is simply top notch. Yes, commas in JSON fixes the situation from catastrophic to acceptable, but yarn.locks are always more compact and win in readability over JSON files.

The downside of a custom format is that if it is not compatible or is not a subset of some other, more common format, it will be harder to work with it at the user or infrastructure code level.

In any case, this undertaking cannot but rejoice.

[rhuanbarreto]

This would really unblock bun for being usable in storybooks as parseable outputs are important for it to adopt as a supported package manager. storybookjs/storybook#28164
JSON with trailing commas is good. But it's also important to have a --json parameter on the package manager commands so scripts can JSON.parse its output.

[GermanJablo]

I'm not as familiar with the technical implications, but as a user I have to say that YAML makes a lot of sense in a long, auto-generated file like the lock file.

[nurulhudaapon]

Wondering why just re-using package-lock.json is not an option?

[rhuanbarreto]

I think it's a good idea to keep a way to support a backwards compatible (package-lock.json) in addition to the current (binary) one. Then people could choose in the bunfig.toml if they are willing to pay the performance penalty in order to be more compatible with the node ecosystem.

[LitoMore]

Two same lockfile content bun.lockb files were generated from different versions of Bun may have different md5 bun.lockb results.

This is super annoying with Git.