Modules: Coordinating ESM implementation in Core

[jasnell]

@nodejs/tc39 @nodejs/v8 @nodejs/node-chakracore ... following the TC-39 meeting this week, there is a proposed path forward for getting modules implemented in core. While the node-eps documents need to be updated, it would be good to begin coordinating the various threads that would need to be pulled together for working out an actual implementation.

@nodejs/v8 and @nodejs/node-chakracore teams... can you please respond to this thread with details or links to each VM's plan and timeline regarding ECMAScript Module implementation. Pointers to existing documentation is perfectly fine, I just want to use this issue as a coordination point.

[targos]

This is the V8 bug tracking Module implementation: https://bugs.chromium.org/p/v8/issues/detail?id=1569

[ofrobots]

/cc @ajklein for V8 plans

[Fishrock123]

ChakraCore Appears to have 100% implemented but documentation doesn't seem to be available yet.
I have yet to be able to make more progress attempting to shim v8module.

@chrisdickinson has a branch that attempts to use the current V8 API on master, which is only somewhat functional on the V8 end but completely functional on ours; there is a full implementation of a html-spec compliant JS loader included as well: https://github.com/chrisdickinson/node-1/tree/implement-esm

[mikeal]

FYI, at the TC39 meeting we were told the html-spec for the loader is "in flux" and is likely to change quite dramatically. In fact, we expect it to change quite a bit based on our feedback.

[Fishrock123]

Ok, we'd might as well still work backwards from an impl that has anything that may not change already in place though, imo. Idk, we'll have to see what those changes are like but I think the spec as it exists today will be a good enough starting point to prototype with.

[ajklein]

V8 is quite close to a complete implementation of what's in ES2015 (note that it's only implemented in our interpreter, so passing --ignition is required to make it work at the moment). Our API probably isn't sufficient for Node's use yet: it assumes all modules will be created as "Source Text Module Records" (in spec-speak), since that's what we expect to need for an implementation of <script type="module"> in Blink. It would be useful to understand Node's API requirements.

Regarding the design explorations at TC39, we're still early enough that nothing from that conversation is likely to land in V8 as part of our initial modules implementation. So it would be great if we could find a sufficient API surface that experimentation in that area could happen outside of V8.

[domenic]

FYI, at the TC39 meeting we were told the html-spec for the loader is "in flux" and is likely to change quite dramatically. In fact, we expect it to change quite a bit based on our feedback.

I think something was lost in translation there. What's in flux is https://whatwg.github.io/loader/ (a "collection of interesting ideas" about loading in various environments), not https://html.spec.whatwg.org/multipage/webappapis.html#fetching-scripts and related sections (the HTML spec loader). The browser loader is finalized, pending implementation feedback (although implementations have progressed quite far so I don't anticipate much more).

I would definitely not bother implementing the collection of interesting ideas, as most of it is going to be removed and changed in various ways.

[bmeck]

@ajklein I think the solutions we described that VMs will need at the meeting are as follows:

1. Late linking validation : Need a new Module Record type that's shape is not validated until dependencies are run through ModuleEvaluation.
   • This requires a spec change to loosen ModuleDeclarationInstanciation on Source Text Module Records and implement a guard in ModuleEvaluation.
2. Property Delegation : New Module Record should be able to delegate to a value's properties rather than a scope's bindings. It was pointed out that this has similar effects as global variables wrt side-effects.
   • Does not require a spec change to existing structures, though requires some addition of a concrete Module Record type somewhere (outside of Ecma262?).
3. Re-linking : Currently linking is permanent, however until ModuleEvaluation occurs, some modules like meow may end up rewriting their parent's behavior to delegate to a new dynamically created module instead of the existing dep. Having linking become permanent after evaluation.
   • This requires a spec change to loosen ModuleDeclarationInstanciation on Source Text Module Records.

We have not discussed the API design for these or some other things discussed like meta-data to declare the shape of a late linked module up front. Would it make sense to give me time to write up a spec for the new type of ModuleRecord / gather the tweaks to spec prior to jumping into a C++ API design discussion?

[ajklein]

@bmeck When I talk about "API requirements", I mean something higher-level than actual C++ code, say, "ability to create a Module Record from outside the API and pass it into V8 to be linked".

But it does seem like we may still be at too early a stage for that: your point 2, for example, sounds like big extension to what we discussed at the meeting.

[bmeck]

@ajklein if the values go out of sync for "plucked" variables, it would make more sense just to not support named imports from CJS IMO (though others differ on this); Not having live delegation/named imports changes many parts of this discussion.

[joshgav]

/cc @digitalinfinity @boingoing @kunalspathak re Chakra

[boingoing]

@joshgav Chakra has a compliant implementation of ECMAScript modules now (barring any bugs). We don't have support for the Loader or BrowserLoader objects though we made some progress on the whatwg spec (chakra-core/ChakraCore#1472) which has not been pulled-in to the ChakraCore repo, yet. Looks like we will need to refactor this after the dust settles on a converged design.

Since the only way to test ESM is with help from the host (ie: static modules via <script>), testing with ChakraCore is possible via ch.exe with "-es6module" test flag. The feature can also be enabled in Edge on Windows Insider builds via the "Enable experimental JavaScript features" toggle under about:flags. I believe the current Insider flight (14936) has a mostly-complete modules implementation (missing module namespace object and has some bugs IIRC) but the next Windows Insider build should have our complete ESM implementation.