Publish ES5 modules to NPM

[scurker]

It seems a common pattern for modules in npm these days is to publish their code as ES5, while keeping ES6 modules in the original source. The idea being is that it doesn't require any consumers to have any knowledge of how external modules are written, it just works out of the box.

Assuming most modules in npm are already compiled, most webpack configs you see tend to ignore node_modules for faster compilation time:

module: {
  rules: [{
    test: /\.js$/,
    exclude: /node_modules/,
    loader: 'babel-loader'
  }]
}

This can cause issues with bundlers if your code isn't ES5 because if you have this configuration, it chokes on ES6 specific code. You can work around this for autotrack by excluding these specific packages:

exclude: /node_modules\/(?!(autotrack|dom-utils))/

However, that can get pretty long if you have a lot of external packages that aren't precompiled. It's also not as straightforward for someone who is starting out and may not be familiar with the inner workings of webpack.

In either case, either this should be documented for bundle users or have published ES5 modules in npm so that it's not an issue. I'd be happy to create a pull request to help.

[philipwalton]

Thanks for the suggestion. I actually spent a lot of time thinking about this issue before publishing, and ultimately I chose to publish the source modules rather than the built modules.

Here was my initial thinking (I'm happy to reconsider if it becomes a burden for a lot of people):

• Publishing the source to npm forces users into the transpiler of my choosing, not their choosing. Autotrack source code is written to work with either babel or closure compiler (which produces much smaller output than babel), but if I published babel versions, closure compiler users wouldn't be able to get those gains.

• Publishing transpiled source files means those transpiled source files all have to include every helper function (e.g. _createClass(), etc.) with them. This means you can't optimize your build to only include those once.

• I feel pretty strongly that autotrack code should never affect load performance, so I think it's an antipattern to import it in your main site bundle (unless you're using dynamic import() and code splitting features). That's why I included the build tool with this release, so folks could build it separately and load it via <script async>.

• excluding /node_modules/ is a performance win on the initial build, but in my experience rebuilds are the main area where performance matters, and that won't be an issue since you're probably not changing the contents of node_modules during development.

I'd consider publishing both the source and built files, but I'd want to make sure there was a real demand for it before doing the work to support it.

Thoughts?

[scurker]

Thanks for your thoughts, it's great to see some solid thinking behind this.

However, since it does seem this does deviate from the norm for NPM published packages it would be nice if the loading via npm section mentioned you will need to be sure you transpile down to ES5 since it will ultimately be used in the browser.

[matthewlein]

Thanks @scurker !, I just spent 5-6 hours troubleshooting babel, webpack, grunt, the upgrade to webpack 2, downgrading to webpack 1, bouncing around commits, changes. It was because babel was not processing node_modules, as you said.

@philipwalton I think this is misleading:
If you use npm and a module loader that understands ES2015 imports (e.g. Webpack, Rollup, or SystemJS), you can include autotrack in your build by importing it as you would any other npm module
The imports worked fine, yes, but in my case the default function params (and maybe other es6 features) did not without babel processing. You definitely need a blurb in there and the upgrading guide.

exclude: /node_modules/, is the defacto standard everywhere I've seen, so I'd probably put a section on babel and adding exclude: /node_modules\/(?!autotrack)/,

The end fix is very simple, but understanding the problem these days is a quagmire of interconnected tools, not so clear error messages, and mountains of outdated config recommendations...

Edit: also worth noting that I did try removing the exclude on node_modules completely early on, but ran into another module that had some babel settings and a not so clear error message.

[scurker]

@matthewlein FYI you also need dom-utils in your exclusion list since it's also published as ES6.

I wish there was a better community standard around this, because I think Philip has perfectly valid reasons to not transpile code but there's not an official way to make those intentions clear.

[matthewlein]

Huh, haven't hit any errors yet without it. I will keep that in mind if I see anything.

[matthewlein]

Derp. There it is, dom-util required as well.

[mattecapu]

This is the reason I'm still sticking with v1. Compiling a module is non-standard practice so a lot of the tools I use are not really good at this. For me, it was easier to just not update autotrack.

I also think publishing ES2015 is great, but the ecosystem is not ready yet, mainly because there is still no agreement on how to do this.
I would have waited for the ES Modules proposal to land in Node.js. This will set a standard that any tool will eventually support.

May I propose to publish an ES5 version on the v1 branch? This would give users an opton.

[cycomachead]

I also just spent a couple hours on this. I 👍 the recommendations about improving the docs as well, and had the exact intuition about "ES2015 compatibility". Perhaps we could just link to this issue?

To be honest, adapting our webpack exclusions isn't tricky as long as I know that's what needs to be adapted.

[ggregoire]

Hi, just to add my two cents (sorry I'm late):

Create-React-App's build process doesn't transform the libs in node_modules.

From Dan (React team & author of CRA):

No, this is not supported because builds will get much slower. We don't recommend using libraries that don't precompile their code to ES5 but are supposed to be usable in ES5.

[bryce-larson]

My two cents:

After a couple of hours, was able to figure out the root issue and fix my build. Finding this thread helped, but it took a while to understand why this was the issue and further complicated since I am using Typescript. Here is the error I was getting:

ERROR in bundle.js from UglifyJs
Unexpected token operator «=», expected punc «,» [bundle.js:13291,12]

IMO, this is not the default format that npm modules are published. This is a hard / bigger question but until community figures the way to move forward from ES5 it would be nice if this repo followed the current defacto standard.

I opt'ed to add babel to just to get this module to work, here are the steps if anyone else is interested.

npm install babel-loader babel-core babel-preset-env babel-preset-es2015 --save-dev

webpack v2

  module: {
    rules: [
      {
        test: /\.tsx?$/,
        use: { loader: 'ts-loader' },
      },
      {
        test: /\.js?$/,
        use: { loader: 'babel-loader' },
        exclude: /node_modules\/(?!(autotrack|dom-utils))/,
      },
    ]
  },

[philipwalton]

I've spent a bit of time looking into what it would take to publish the plugin source files as ES5 so they could be more easily consumed via webpack with most users' exclude: /node_modules/ configuration, and I don't think there's a good solution.

The problem is, since autotrack depends on my dom-utils package, transpiling all the autotrack source files via babel into a /dist folder isn't sufficient, I'd also have to release a version of dom-utils with its files transpiled. But the problem there is even if I release both autotrack and dom-utils with transpiled /dist folders, there's no way to dynamically tells the import paths that were requested with a dist in their name to continue to use dist (rather than lib) in their sub-imports.

That means my options are to either (1) publish only babel-transpiled ES5 versions of all the plugins (which I don't want to do since closure compiler-transpiled versions are ~40% smaller), or (2) I can continue to publish ES2015 versions and ensure this issue is as clear as possible in the documentation.

At this point I'd much rather do option (2), since it's ultimately the most flexible and future-proof, and it still works with all existing tools (just not with all existing configurations of those tools).

On other hand, if I do option (1), I'd be preventing anyone who wants to use a transpiler other than babel from doing so, and I'd be forcing all consumers of autotrack to download far more code than they need.

I also want to point out that I already am providing an ES5 version of autotrack that users can include in their builds, it's just not referenced in package.json, so it doesn't get important when using the node module resolution algorithm.

I people want, I'd be willing to update the main key in package.json to point to the ES5 version. That would solve this problem for folks using just import 'autotrack'. Unfortunately, it still wouldn't solve the problem for folks who want to import individual plugins (though maybe that's a less common use-case).

[philipwalton]

I also want to reiterate that, even if you're using webpack to build your app, you don't have to bundle autotrack with the rest of your code. It's actually much better to create a custom build and load it separately via <script async> (as describe in the installation and usage section).

And if you really want to use webpack to bundle autotrack, then you should definitely be using code splitting and you can still transpile autotrack via babel-loader with the configuration @surker described.