Locking dependency versions in reqs.txt

[aiqc]

The package broke last week because it was incompatible with Werkzeug==2.1.0.

So I looked at requirements.txt expecting to see version rules, but it's all loosey goosey

# requirements.txt
dash
requests
flask
retrying
ipython
ipykernel
ansi2html
nest-asyncio

https://github.com/plotly/jupyter-dash/blob/master/requirements.txt

---

I get that you want it free floating with the latest dash, but shouldn't things be a bit more nailed down to prevent future breakage?

[alexcjohnson]

@flying-sheep's plotly/dash#1992 (comment) and the article it points to describe well the philosophy we're trying to follow. We took this breakage as evidence that we were dependent too much on the internals of werkzeug, and since it was an option here, the fix implemented in #82 was to remove direct dependence on werkzeug altogether.

[aiqc]

Thanks for sharing. The logic around the section "Version limits break code too" helps me gain perspective about how this can be a divisive issue:

If everyone pins versions then the whole ecosystem gets rigid [or reliable/ accountable depending on who you ask] e.g. if you want to use packages A and B, but their pinned versions are incompatible, then you are stuck.

Every single major release of every dependency you cap immediately requires you to make a new release or everyone using your library can no longer use the latest version of those libraries

I guess I've just come to expect reliability from the word release, and don't understand how can you can indefinitely include entire libraries in a given release because they aren't guaranteed to work.

[flying-sheep]

There’s a few ways ways to guarantee reliability:

1. Only lock down minimum versions. This works fine if you have a CI that regularly checks for breakages caused by other packages updating, and you commit to fixing those. Only when something else has an upper version boundary, you can run into unfixable problems.
2. Lock everything down only in the final applications that nothing will ever import things from. This works well, as long as you frequently update. Otherwise updates will be a nightmare since you’ll have to update everything and deal with all tiny breakages at once.
3. Lock everything down in a library. Your project now defines most versions that are installed. Since as we know even individual upper boundaries can cause problems, as soon as a second library in your dep tree does the same, people will be unable to install both at the same time (note that e.g. npm doesn‘t have this problem as it can have nested package libraries like node_modules/some-dep/node_modules/subdep)

I’m afraid your notion of reliability doesn’t match the practical reality of the Python ecosystem. There are many projects who at some point thought they’re important enough to dictate pinned versions to other projects, but they were all soon humbled by the outcry of dozens of configurations they just broke. I’m happy dash doesn’t make the same mistake.

[alexcjohnson]

There's a flavor of (2) that IMO doesn't get enough attention: working locally on a final application, start with only unpinned versions of your direct dependencies in something like requirements.in, adjust for any problems you find, then for production pip freeze the full dependency set to a second file requirements.txt. When you're ready to update, wipe your environment, go back to requirements.in, and repeat.

This is more or less what the JS ecosystem does with package.json / package-lock.json, in addition to the major difference of nested dependencies allowing multiple simultaneous versions. A variety of tools in Python try to get at this type of pattern, but none has really caught on as the standard and anyway it's relatively easy to do this just with pip, once you get in the habit.

Again, this applies only to final applications, and it works best when libraries do NOT pin their dependencies.

[flying-sheep]

Since PEP 621 exists, I’d do away with “requirements.in” and just use the project.dependencies table in pyproject.toml, but yes, that’s a good approach for applications. I didn’t go into it since dash and dash-jupyter are libraries and it’s therefore a bit off topic.

There’s also other cool ideas once could get into like cargo update -Z minimal-versions, which allows you check if you’re really telling the truth about your lower bounds and will lead you to specify them correctly. (Not a big problem since most people will run your code using new-ish versions of most things, but a good idea nontheless!). But since I’m not aware of that existing in pip, it’s probably also off topic.

[aiqc]

plotly/dash#2516

[alexcjohnson]

Thanks @aiqc - plotly/dash#2516 is certainly a departure for us, directly contradicting my previous statements. Perhaps you can sense my frustration at how many times this pattern has repeated 🙄