Looking for brilliant yarn member who has first-hand knowledge of prior issues with symlinking modules

I was recently informed by @ thejameskyle that:

Before being open sourced, Yarn used to do symlinking. However it broke far too much of the ecosystem to be considered a valid option.

If you are a yarn member, who has first hand knowledge of what actually broke, and can technically explain why, I implore you to respond to this issue.

I believe I have found a solution, and my initial experiments indicate it is entirely viable. However, like thejameskyle, the only responses I seem to get as to why it wouldn't work at all, let alone on a broad scale, are entirely ancetodal.

Please, I'm desperate to find someone who knows what they are talking about, that has the technical acumen and understanding of node module resolution, as well as first hand knowledge of the issues yarn encountered, who can repudiate my solution by technical means of cause and effect, by first explaining what issues yarn encountered in its earlier attempts to exploit symlinking.

With my solution, one can have independent physical dependency tree renderings, were the specific module@versions used within a given tree remain specific to that particular physical tree, and can be locked, but where all modules across all physical trees on a given machine can all be symlinked to a single machine wide copy centrally stored, even when within each physical tree a common module@version is used in several trees, yet still resolves it's specific dependency versions based on the physical tree it is used in, and where those dependencies versions are slightly different between trees for whatever reason (but still within the semver range spec in package.json).

You're probably the right person if a) you understand exactly what I just wrote, and b) believe it's impossible.

Having this ability would mean modules no longer need to be copied all over the place, saving gigabytes of storage, and once centrally stored, all 'installs' could symlink all modules (but wouldn't have to), reducing install times from minutes to seconds.

@kittens Would you be a possible candidate to speak to this issue? Or know a more appropriate person?

[sebmck]

I've talked about this briefly in other places but haven't really given comprehensive reasoning as to the difficulties we faced with it. The truth is that the existing ecosystem does not cooperate very well when you start using symlinks.

Operating system differences

Symlinks are supported differently on various operating systems. In Windows they aren't allowed unless you're an administrator for example. You can however use NTFS junctions that operate in a similar way but with the following restrictions:

• Only work on NTFS partitions (as the name implies).
• They can only refer to folders on the same drive.
• They can only operate on folders (this is fine for Yarn since we only need junctions for folders).
• They don't operate across network shares.

If we want seamless Windows support then we'd need to impose restrictions on the development environment of Yarn users and when the existing alternatives don't have these same restrictions it's hard to justify.

Alternatively we could support both symlinks/junctions and the current flat version but one of the big motivators behind Yarn is determinism and having different ways of representing the files on disk that are distinguishable from one another goes against this. It'd also lead to an explosion on support since we'd be forking the workflow and internals to support symlink resolution. (In fact somewhere in the git history you'd find we once supported both of these installation methods)

Tooling not supporting file system cycles

Tooling such as Jest would run into weird recursion errors when crawling the file system since symlinks allow cycles to appear. ie. a nested directory referencing another included in it's heirarchy. Jest is a lot better now and it probably fixed but this is a common problem that existing tools don't take into consideration.

Poor support for file watching

File watching across operating system is already a massive issue with their being a lot of inconsistencies and problems with normal files and folders. This issue is even more exaseparted when you take symlinks into consideration. Tools such as watchman don't support symlinks for specific.

Tooling relying on node_modules hierarchy

Tools like ESLint rely on the hierarchy of node_modules to use plugins. For example eslint-plugin-foobar needs to be accessible in the hierarchy of eslint to be accessible and includable in your projects. Node resolves symlinks which means that their absolute path (unsymlinked) is used which breaks this assumption that ESLint makes. A lot of tools use this sort of resolution to work around particular issues with module loading and path issues.

---

To summarise the advantages of symlinks:

• Slightly faster package installation.
• Less disk usage

Are outweighed by the significant disadvantages of:

• Operating system differences
• Making Yarn less deterministic by adding multiple modes of install
• File watching incompatibility
• Poor existing tool compatibility
• Tool recursion errors due to cycles
• etc...

I'm going to leave this issue open if there's any rebuttals to the points I've made and for future discussion.

[Daniel15]

Thanks for the wonderful comment, @kittens! ❤️

I'm going to leave this issue open if there's any rebuttals to the points I've made

I'm just passing through and don't know anywhere near as much as @kittens does with regards to the topic of symlinks and their usage here, but thought I'd add some quick points.

In the end, I think a filesystem that supports copy on write will make everything better. With a file system that uses copy on write, copies are "lazy". This means that creating a copy of a file doesn't actually copy the data, it simply makes the new file a pointer to the old one, similar to what you'd get if you hardlinked the file. A true copy (allocating space on disk for the file, and actually copying the bytes across) is only performed when you modify the file. The end result is that you get all the benefits of symlinking (faster installations, less disk usage) with none of the disadvantages (modifying a copy doesn't modify the original, and tooling just sees everything as regular files, since they are regular files).

On Linux, you can use btrfs or zfs for this. Even with Yarn in its current state today, using btrfs should give you a nice performance boost due to the CoW semantics. However, neither Mac OS nor Windows have a good copy-on-write filesystem today. In #499, @dfreeman said:

In theory both Apple and Microsoft's next-gen filesystems will be copy-on-write, which would change the landscape here considerably. APFS is allegedly due out in 2017, but it's not clear when ReFS will be ready for general-purpose use, so ¯_(ツ)_/¯

Hardlinks solve some of the drawbacks that symlinks have, but they also have their own issues.

Positive (advantages of hardlinks over symlinks):

• Hardlinks don't have an issue with cycles, as you can only hardlink files
• Hardlinks still give you the same disk space savings you'd get with symlinks, as there's just a single copy of the files
• Tooling doesn't need to know anything about hardlinks, as a hardlink looks exactly the same as a regular file. No need to treat it specially

Negative (or negative-ish):

• Hardlinking a lot of files is slower than simply symlinking the directory (but it should still be faster than copying the files)
• Hardlinks still have one of the "problems" symlinks have: Editing a file in one project will actually modify it everywhere it's used. That can be pretty confusing. This can be solved by using a copy-on-write filesystem, neither Windows nor Mac OS support any COW filesystems so that's limited to Linux
• By default, hardlinks require admin permissions on Windows :(

We have a backlog item to investigate hardlinking (#499), nobody's actively working on that right now though.

Alternatively we could support both symlinks/junctions and the current flat version but one of the big motivators behind Yarn is determinism and having different ways of representing the files on disk that are distinguishable from one another goes against this.

This is a great point! The fewer possible combinations there are, the easier debugging becomes.

Sebastian, Thank you kindly for taking the time to explain your experience with symlinking. It's greatly appreciated.

OS File System Linking Differences

Regarding file system linking, I'm presuming yarn did something similar to how ied works, in that folders lower in the physical dependency tree were linked to top level folders under project/node_modules where the modules physically existed?

If true, and given that FAT32 filesystems are also prevented from storing links when mounted to 'nixs, the only difference in linking behavior between the OS's that node supports, that would have a material impact in how yarn was using links, is that in Windows, folder links can't reference network volumes (junctions can cross local volumes, but not to network shares; see here). Would it therefore be fair to state that in the context of how node resolves module loading, all the other enumerated differences in linking behavior, while accurate, would have no relevance and could be taken out of scope?

Assuming the singular difference in linking behavior between OS's relevant to yarns use of linking, in your experience how many developers were you aware of, either first or second hand, that were attempting to run yarn in such a way that the folder from which it was ran against, containing the project's package.json, was somehow expected to have links to module folders that existed on a network share? And how was yarn configured to know to deploy the physical tree in that way?

Given the one difference that would have had some impact, when you say:

If we want seamless Windows support then we'd need to impose restrictions on the development environment of Yarn users and when the existing alternatives don't have these same restrictions it's hard to justify.

..is my understanding correct then that the only imposed restriction would have been, on non Windows systems, yarn will not create local links to network shares to modules, ensuring that is behaved identically across all OS's?

If that were something never required; i.e. a developer having local links to network shares for modules within a project being developed, would it be fair to say that, practically speaking, there really wouldn't be any issues that needed justification of restrictions to account for the difference in OS linking behavior in order to seamlessly support Windows?

I ask only because the way in which I'm proposing to use symlinks, while not at all like how ied or yarn is able to, might be impacted if lots of developers were needing, for some reason, to have the their project/node_modules folders contain local links to network module folders. I would be shocked if that were ever the case, but you've clearly had a great deal more experience and exposure to the ecosystem. If true, could you elaborate on the purpose these developers where needing that for?

This was under Operating System Differences, but I couldn't understand how it was related to OSs?

Alternatively we could support both symlinks/junctions and the current flat version but one of the big motivators behind Yarn is determinism and having different ways of representing the files on disk that are distinguishable from one another goes against this. It'd also lead to an explosion on support since we'd be forking the workflow and internals to support symlink resolution. (In fact somewhere in the git history you'd find we once supported both of these installation methods)

When you say having different ways of representing the files on disk that are distinguishable in the context of determinism, are you implying that when yarn used symlinks, it used a different folder organization than when no symlinks where used, or rather that the folder structure's were identical but the physical location of the files in the folders was some how different between the two? Or more specifically, where is your physical boundary separating what must be physically deterministic and what is irrelevant?

I ask because I can't see a practical difference with respect to how node would deterministically walk the folder structure that organizationally didn't change, regardless if the folder was a physical copy of a module folder vs. a symlink to a module folder; the contents and path would be identical from node's perspective (assuming launched with --preserve-symlinks)

The way in which I'm intending to use symlinks would not bind the folder structure in any way to any given folder's physical representation. I.e. regardless if any particular module folder was symlinked or a copy, and even if that characteristic of the folder varied from time to time, node would always deterministically walk the structure in exactly the same way, and see exactly the same content on every walk (to the degree node itself was looking for modules deterministically). If that were the case, would that qualify as deterministic enough to meet yarns guarantees of determinism?

Tooling not supporting file system cycles

If it could be guaranteed that it was physically impossible for cycles to exist in the organization of the dependency tree, as represented by the physical folder structure, even though every single module folder was a symlink, would the issue of tooling not supporting file system cycles be irrelevant and could be taken out of scope?

Poor support for file watching

While watching for changes of a project's source files is a common place thing, like how yarn itself is shipped to have gulp watch for changes to .js to then compile with babel, can you elaborate on the use cases you encountered that required watching files that lived somewhere within the project/node_folders hierarchy? I would be shocked if this ever was required, and even more so if it was a common need, but again you have a great deal more experience and exposure to the ecosystem, and your specific experiences with this use case would be immensely valuable.

Tooling relying on node_modules hierarchy

Node has a --preserve-symlinks switch to address such things, although it does not preserve the symlink of the entry.js arg passed on its command line. I have been told by nodejs members they would prefer the default be that all symlinks always be preserved, but there are currently some edge cases that have given them pause to change the default behaviour of resolving symlinks to their realpath.

My intended use of symlinks would require they be preserved in all cases. The only issue that's been brought to my attention regarding this behavior (and in no way am I implying it's the only one), is that if different symlinks pointed to the same physical addon.node file, node would crash as a consequence of how it loads binary dynamic link libraries and associates them to node module instances. First, the way in which I intend to use symlinks would never allow for such a state of the dependency tree to occur, but if in the remote chance there was a reason such a state would be required, the package manager would know and simply make enough copies of the addon module, linking to each copy individually, so that the OS would have no idea they were logically identical.

If symlinks were always preserved, would the issue of tooling relying on node_modules hieararchy be effectively mitigated and for practical purposes could be taken out of scope?

Advantages of a different way of symlinking

I'm going to make a very bold claim:

I know of a way to reduce the total overall module storage on a given developer's machine by at least 10x's (50GB to 5GB) or more, and install times by at least 50x's (50 sec to 1 sec) or more

This is fundamentally achieved by only ever storing a single physical copy of any given module to a central machine wide location, then symlinking to it from where ever it's used anywhere on the machine. Edge cases effected by this change may be less than one half of one percent of the ecosystem, although this still needs to be empirically proven.

---

To summarize, assuming best case answers to above questions

Potential Advantages:

• 10x's or more reduction of total space required to store modules on a given machine
• 50x's or more reduction in install times (after first install)

Potential Disadvantages:

• Operating system differences
• Difference in support of links to network shares
  • (It's atypical a project/node_modules folder would require this)
• Making Yarn less deterministic by adding multiple modes of install
  • (The solution would still allow yarn to render deterministically regardless if using symlinks or not)
• Tooling not supporting file system cycles
  • (Solution would guarantee this would be physically impossible)
• Poor support for file watching
  • (It's atypical files within a project/node_modules folder would require watching during development)
• Tooling relying on node_modules hiearachy
  • (Solution requires symlinks be preserved in all cases, and addresses known issues)
• There may be breaking edge cases, but potentially within a minimal acceptible spectrum, and the proposed solution would be opt-in on a case by case basis to effectively mitigate while offering a path to migration

For the moment, if the above were in any way possible, would you at least say continued work on developing a POC to empirically demonstrate would be worth perusing?

I know it sounds almost too good to be true, but I want to find out, and I'm already working on the POC of this. I have done some initial experiments with tweaks to node (yes node, about 10 lines of relevant code) and npm to at least show me it's feasible, and that success is spurring me on, but I prefer to continue the package manager side of the POC with yarn.

Are you interested in the solution? In the end, it's unbelievably simple, and comes down to using a ., in addition to a / at just the right time and place!!

[Daniel15]

@phestermcs - What do you think of the idea of using a copy-on-write filesystem to achieve the disk space reduction? That's something I've been thinking about a bit as well (as per my comment above)

@Daniel15 What do I think? I think any solution that fixes this !@$# problem needs to be implemented IMMEDIATELY!!! hahaha.

One of the great things of node is how it normalizes the OS to irrelevancy. If an OS effectively supports some COW solution, that is unknown to node, and if you can use it you should.

I have machines with all three major OS's (and others), and I've done lots of both OS agnostic and OS specific development on all of them. But I prefer Windows. I could of course use a VM to run linux within, but I like speed... like, alot! (hence this issue).

@Daniel15 I've spent a little time pondering you comments.

I think using hard links could be a decent option. I was so locked on to linking the whole folder (but never it's node_modules subfolder) as the absolute fastest way, I missed the hardlink approach entirely (blush)! In fact, I'm going to explore that a little more, as you're correct that it should still save just about the same amount space (by there being only one physical copy on the machine), and should still be faster than copies (my first gating test will be to take that measure). Hardlinks also have only one realpath so there would be no issue with node having to "preserve" the link as with symlinks, which is good. It also has similar constraints to linking a folder (around lifecycle events), that I had to address while researching that approach, so I think I have a tad head start on it. And my cross off's of the disadvantages @kittens raised would still be crossed off.

My approach requires a tiny change to node Modules, which is locked, and I already know faces resistance, so trying with hardlinks is more likely to sooner get into the hands of those who want the benefit (assuming it's significant enough and works with out breaking too much). Hardlinks do have the constraint they can only link to files on the same volume, but at least that's the same across OS's (speaking to @kittens concern of seamless Windows support). Assuming the benefit is significant, I'd take the trade.

Windows 8+ no longer requires a user have administrative rights to make hard links. So while that story is a little better, it's still not the best on Windows NT < 8. However, its the more common case developers are administrators of their machines, even in many enterprise IT departments, so still a pretty good story. And while creating hard links does require administrative rights on Win <8, it does not require running with elevated permissions on any Win version, contrary to creating symbolic links. So if a user has admin rights, yarn could link without having to run elevated, which means it's as user friendly as it ever was.

The way I was thinking of using symlinks, would have come with the constraint that module folders where always read only (I see them logically like *.so's or *.dll's that shouldn't be altered regardless of where used), and I would still apply that same constraint when using hardlinks.

Which leads to one of the changes that could create issues with linking to a global location, and the application of the read-only constraint. That is that the preinstall, install, and postinstall lifecycle events would only run once, when a module was first "installed" to the global location, while the build event would still need to run on every link-install for a given project (which all worked in my proto, because 'build' puts things to local folders). When I prototyped using npm, just to see if it would work at all (by making copies of projects, tweaking each slightly to have different version specs, then installing to ensure each still had their specific dependency tree), I did not find modules that altered their local install from one place to the next, based on their surroundings or specific versions of their dependencies that were installed locally with them. However, I was not at all looking super hard for that, and this is definitely something that could be a problem.

If that issue does show itself to be a problem, only if even marginally, it would obviously need to be addressed. The first way is that you just don't use linking on a particular project; that fallback will always be available (although some might use the issue as justification to just not even provide a linking feature to begin with, as they'd fear it would confuse noobs (which frankly are already usually confused by things, so what's a little more confusion? giggle)). Another approach is that packages could flag themselves as linkable, and then yarn could ignore or honor the flag based on a switch (and I can hear that same someone complain about more switches, to whom I'd say "maybe you should get into a different line of work, because this line is based on billions of switches all working together in imperfect harmony!" j/k).

Regarding COW, I'm just starting to dig into yarn code, but at first glance it appears to first unpack a module into a global cache if not already there, then copy that into a given install location. So COW might work without having to change anything in yarn. (This would be different with npm which always unpacks a tar into the install location, and that would have to change to leverage COW). But like you pointed out it's not available on all OS's, so not much to move forward with using that approach.

I'm going to run some tests to measure performance of hardlinking thousands of files, to see just how much faster it is than copying. If it's at least 4 or 5 times, I'm going to wrench yarn enough to actually use hardlinks so can start POC'ing actually using with projects (but not going to worry about things like if yarn running concurrently, and things like that; later if the POC shows further promise). yarn should still be able to guarantee determinism in the same way it currently does, if the guarantee is really about just the folder structure and hierarchy (which I would opinion is all that's actually required to prove node would be deterministic in its walk of that structure when loading modules, without actually running node)

I will say it's a quite a bit nicer working in yarn as it's better organized and using async/await; npm was really starting to show it's age, as I would at times start looking around for a jar of Ragu sauce. So thanks @kittens!

Finally took some initial measurements using react repo ~ 21k files. Ran the tests 3 times each, so here's their ballparks.

SSD "Install" (482MBs seq, 30k iops rand)

yarn lnkdep	hardlink	adj-nm & symlink
70 sec	28 sec	2 sec

SSD Delete node_modules

yarn lnkdep	hardlink	adj-nm & symlink
35 sec	35 sec	< 1 sec

7200RPM HD "Install"

yarn lnkdep	hardlink	adj-nm & symlink
118 sec	30 sec	5 sec

So hardlinking can be twice as fast on SSD's, and up to 4x's as fast on HD. It's interesting it takes about the same time regardless if SSD or HD.

But clearly, --adjacent-node-modules with symlinking is way faster

[Pauan]

@phestermcs Thank you for pushing for this. In my opinion, slow installation is the current biggest problem with yarn (though it's still faster than npm).

It makes sense to me that hardlinks would be the same performance on both SSD and HDD, because they both do the same amount of work: they simply increment the ref count, but don't do any actual copying.

Symlinks are faster than hardlinks because you only need to create a single symlink, rather than creating multiple directories and multiple hardlinks.

Although I am in favor of hard/soft links, there is one downside: if somebody modifies a file in node_modules, it will affect other (completely unrelated) packages.

Personally, I think people should not be doing that, instead they should use local packages if they want to make file modifications. So I see it as more of an education/documentation problem.

[ljharb]

@Pauan yet, the ability to do that is a long-standing important part of node, and since node's require only cares about what's on disk, i should always be able to edit a file on disk and have the changes show up appropriately.