[Feature Request] Symlink support

[JohnyDays]

Is there any performance / complexity reason for not supporting symlinks?

With the release of react-native, and the use of watchman in its packager, it's troubling that npm's (react-native's chosen package manager) endorsed method of developing local interdependent libraries with npm link is not supported, and having to commit to a remote repo and download on every change is obviously not ideal.

[wez]

Watchman recursively watches files that are contained within a filesystem tree.
Symlinks can point to an arbitrary location on the filesystem.
Changes to symlink targets that are outside the tree are not observable.

Why not simply resolve the symlink and watch its target? Because it isn't simple. Here are a handful of reasons that make this a difficult prospect. This is not an exhaustive list, it's just a few reasons I can rattle off the top of my head:

• The target of the symlink may not exist at the time we want to establish the watch, and neither may any of the parent directories in the symlink path. The operating system will not allow us to watch a path that does not exist
• Any of the directory components in the target symlink path may themselves be a symlink and may change at an arbitrary time in their lifecycle in a way that is not detectable unless we watch every single directory component in the symlink target path
• The target of the symlink, or any of the directory components of the symlink target, may be a remote filesystem that doesn't support change notifications
• Each additional watched directory consumes more filesystem watch resources. There are finite limits on these, and not every system allows raising these limits.
• Because the hypothetical watches that we'd establish for each symlink are distinct from the watch used for the root of the real tree, we lose certain ordering guarantees when processing change notifications from the kernel. We wouldn't be able to tell whether a change to the symlink target happened logically before or after changes that we observe in the main tree. This important property is used to ensure that filesystem changes have been observed up to the logical point in time of a query. More details on this can be found at https://facebook.github.io/watchman/docs/cookies.html
• Filesystem changes are fraught with TOCTOU (time of check, time of use) issues already. Adding user-space resolution of symlinks increases the chances of more TOCTOU bugs dramatically; any component of the symlink path may change while we're processing information about them.

Correctly handling all of the above would be tremendously complex and still be error prone. As a result, it is unlikely that we'll ever add support for resolving and tracking symlink targets.

[wez]

Taking a step back from this, can you describe what you're trying to do and why symlinks are important? I'm a casual and occasional node and npm user and am not familiar with npm link

[JohnyDays]

I understand why this would be difficult, and I'll close the ticket and resign to using the above solution for developing local modules.

I'll attempt to explain the process behind npm link, with my use case as an example.

I am developing a module called react-native-waterfall, it implements a generic waterfall view.
I am at the same time developing, in a sister folder, a module called react-native-social-waterfall, and this module depends on the above module.
I am at the same time developing utility modules, which both of the above modules depend on.

npm requires modules from either:

1. a globally qualified name e.g react-native-waterfall
2. a path e.g ../react-native-waterfall
3. an url e.g https://github.com/facebook/watchman/

The second solution has 2 problems:

1. Every time you change any file in a dependent module, you have to run npm install ../react-native-waterfall to reinstall the module.
2. You are checking in code that doesn't mirror the final product, where you will be using globally qualified namespaces hosted on npm.

The third solution has 2 problems as well(both are slow)

1. Every time you change any file in a dependent module, you have to commit it and push it to origin.
2. You must then go back to your dependent module and do npm update to grab the latest commit

npm link works the following way:

1. You go into the depended upon module and write npm link, this will register react-native-waterfall as a globally qualified name
2. You go into the dependent module and write npm link react-native-waterfall, this will register a symlink to the globally qualified name, using the folder where you wrote npm link.

This allows the following benefits:

1. Your checked in code mirrors the final product.
2. You do not have to do anything to update the local dependency files, meaning you can quickly test code without committing / running commands.
3. You can even edit the depended upon module in your dependent folder, and the changes will be applied to the original module.

[wez]

Thanks. How does watchman fit into this?

[JohnyDays]

I am using watchman with the react-native packager, which automatically parses dependencies, transforms code and reloads the application when you change a file. However I have to continuously restart the packager because it doesn't detect the symlinked dependencies's changes.

[JohnyDays]

Idea:
I understand why the added complexity of implicitly following symlinks would cause too many troubles, both in implementation and in maintenance. So would it be possible to instead implement an explicit/manual symlink helper in the form of:

/dir1/
/dir2/

watch dir1
watch dir2 --link-to dir1

Any change in a file at dir2 would trigger an event for an analogous file in dir1
This could easily be automated into a npm link workflow, along with some kind of unlink function

[wez]

Sorry to disappoint; it's too complex to build something to handle this that will work according to expectations.
It sounds like this is something that the react package manager should be handling, since it is the component that has special knowledge of the situation.