CLI functionality related to aggregator preventing yarn commands 1.19-1.22 - Reverted to 1.16

[DavideDaniel]

<!-- *Before creating an issue please make sure you are using the latest version of yarn.* -->
^ Problems started with upgrading to 1.19 and then snowballed into 5 hours of hell trying to figure out what went wrong.

Do you want to request a feature or report a bug?
error An unexpected error occurred: "expected workspace package to exist for \"rimraf\""
It started with some dev dependency in a workspace repo and after messing around with nohoist settings I started noticing that the same modules keep coming up. Rimraf, inquirer, react-dev-tools.. generally things that were listed many times over in transitive dependencies. Read through many different similar issues regarding symlinks but ended up being unable to add new modules through cli which was the problem I was trying to address for automation.

What is the current behavior?

These commands do not seem to work in version v1.19.2 through v1.22.0-20191201.1637:

yarn workspace @scope/A add uuid lodash.get --dev
cd packages/A && yarn add uuid lodash.get --dev
yarn workspace @scope/A add js-cookie@^2.2.1 -D
cd packages/A && yarn add lodash.get@^2.2.1 --dev

Tried yarn cache clear, removing node_modules, reinstalling yarn to no avail.

If the current behavior is a bug, please provide the steps to reproduce.

Sorry, work repos.

What is the expected behavior?
Workspaces command and yarn add should work from cli.

Please mention your node.js, yarn and operating system version.
Tested on the following:

Node:
v8.9.4
v8.15.1
v10.11.0
v10.16.0
v10.16.2

Yarn:
v1.19.2
v1.21.0
v1.22.0-20191201.1637

OS:
MacOS Mojave 10.14.6

[DavideDaniel]

For anyone that comes looking for a quick fix.. I was able to get back to a working state by installing v16.0 https://raw.githubusercontent.com/karlhorky/homebrew-core/f8bc263f93dcfd1cc50c98976a8503c8a7cb032f/Formula/yarn.rb via homebrew. I'm gonna try seeing if the update with 1.17.4 works and go as far up as I can without losing functionality.

I was able to go up to 1.17.4 but 1.19 and up the same issues come up.

1.17.4 continues to have lots of symlink issues which were addressed in 1.19 but is not usable currently for me. 1.16 seems more stable but is not secure... sigh. Please let me know how I can help. Need a break and will check in later.

[DavideDaniel]

A clue about symlink issues while running --verbose. The uuid module keeps missing even as far back as 1.16. The common pattern right now appears to be modules that are used in many other modules but are listed as a peerDependency in a workspace can run into issues with workspace aggregator during certain cli commands. This could be because a lot of commonly used modules are listed as peerDependencies vs dependencies, following advice from https://yarnpkg.com/blog/2018/04/18/dependencies-done-right/.

[DavideDaniel]

From #6953
I'm wondering if 2.0 will address this with the above.. but still unsure why this became a bigger problem past 1.16. Is it because changes to 1.16 are being made with the same lens applied for code going in for 2.0? If so, should we just consider pinning our version of yarn and wait for 2.0 if we are using workspaces?

[danoc]

I think we're running into this issue at @thumbtack. In particular, this affects us when trying to upgrade a package within a workspace.

This snippet should reproduce the issue:

git clone git@github.com:thumbtack/thumbprint.git && cd thumbprint && yarn && cd packages/thumbprint-react && yarn upgrade @thumbtack/thumbprint-tokens --latest

This is the error you'll see:

error An unexpected error occurred: "expected workspace package to exist for \"@babel/plugin-transform-dotall-regex\"".
info If you think this is a bug, please open a bug report with the information provided in "/Users/<redacted>/thumbprint/packages/thumbprint-react/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/upgrade for documentation about this command.

[danoc]

Confirmed that downgrading Yarn to 1.16.0 works. Thanks for digging into this, @DavideDaniel. Saved me lots of time!

Here are downgrade instructions for those that installed with Homebrew: thumbtack/thumbprint#592 (comment)

[troywweber7]

Downgrading worked for me. But I'd definitely like to see this fixed in the latest version of yarn so I'm not lagging behind by 5 minor versions... yikes!

[iplus26]

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

[roni-frantchi]

Downgrading to v1.19.0 worked for me. Looking forward to fix this~

Same here.

[amirburbea]

Any update on this?

In my lerna monorepo I have some packages (like husky and lint-staged) installed at the root

(My .yarnrc file has --ignore-workspace-root-check true)

When I try to upgrade packages (using yarn upgrade-interactive --latest) I always get Invariant Violation: expected workspace package to exist for "" where the quoted portion includes the name of the root package. Downgrading to yarn 1.17.0 has no such issue

[troywweber7]

I just want to hop in and clarify my situation, in case it helps anyone else.

Failing Command

The commands that were failing for me were of the yarn workspace ${workspace} add ${package} variety.

Working Yarn Version

The indicated command is FAILING for me when using yarn=1.19.2-1 is SUCCEEDING for me when using yarn=1.19.1-1.

Install Older Yarn

First, follow the directions here: https://yarnpkg.com/lang/en/docs/install/#debian-stable

However, when it comes to the installation command, sudo apt update && sudo apt install -y --no-install-recommends yarn, you'll want to change yarn to yarn=${version}. You can find appropriate versions by doing apt-cache policy yarn.

Once you install a yarn version, you may also want to consider doing a sudo apt-mark hold yarn until this is resolved in later versions, otherwise apt upgrade will update your yarn again. You can confirm it is held by doing apt-mark showhold.