Extract runtime to a separate package and preprocess it with webpack for embedding

[MOZGIII]

Currently the runtime is not pre-processed, and this causes a lot of troubles when trying to use it with electron in a sandboxed environment.

[pmmmwh]

Is there any examples you can provide? All code in this repo is written as ES5-compliant as possible (and also in CommonJS), so I don't see it causing trouble in any environments unless you're expecting ES Modules. (I don't know a lot about electron)

[MOZGIII]

To be more specific, there are numerous issues.

1. ErrorOverlayEntry.js imports react-dev-utils/formatWebpackMessages, and that, in turn, imports chalks. chalks then tries to access process.env.*, that is not accessible in electron render process unless the process "has permissions". I have no idea why does ErrorOverlayEntry.js ends up being imported in the browser context - might only be required in CLI context, but nonetheless - it causes a runtime error at the browser (render process) end.

2. Similarly, something requires url in the browser context, which is a node module. This should be required, and I suspect something, that shouldn't be, ends up being included in the webpack bundle, and this is why all this occurs.

I hope this is enough context - if not - I can create a sample repo.

[pmmmwh]

I hope this is enough context - if not - I can create a sample repo.

This is great - I will work on fixing the issues this week.

[pmmmwh]

Issue 1 fixed by #58

[MOZGIII]

#58 is more of a workaround, however fundamentally the issue is not solved. This can happen again in the future, and there's nothing that would've prevented this.
While I'm grateful for the fix, I have some concerns about the reliability of the solution.

I can help with implementing the bundle split if needed.

[MOZGIII]

Same goes for #61.

@pmmmwh while the particular pain points I stumbled upon are fixed (kudos for that btw), I didn't hear your feedback on the subject of this issue - splitting the runtime (browser end) code and webpack (node end) code into separate packages.
I appreciate the time you spent on fixes, but I don't feel like I got a clear response.

Is your point that the code should be portable across the environments, and therefore the split is not required? Well, this might be true, but then I'll end up loading the node-related code in the browser environment and vice versa. This consumes compute cycles unnecessarily, and on its own, it's not a big deal, but coupled with other packages that do the same it becomes a huge problem. Please don't ignore this factor.

Anyhow, please reopen this issue, as it has not been addressed entirely. There are other issues, and I'd like to avoid going over them one-by-one.

[MOZGIII]

Maybe I'm not clear.

This project contains the code that runs as part of the webpack build process and the code that runs as part of the compiled resulting bundle. The code that runs in the webpack injects the code that runs at "runtime" into the resulting bundle.
My proposal is to process the runtime part that's injected differently than the code that's being run as part of the webpack execution at the package level. The easiest way to do it is to split it into a separate package.

[pmmmwh]

I didn't hear your feedback on the subject of this issue - splitting the runtime (browser end) code and webpack (node end) code into separate packages.

The easiest way to do it is to split it into a separate package.

At this stage I am against splitting the repo into separate packages, I don't think it is any better than the current directory split. Differential compilation on directories is also a way to achieve what you want.

Is your point that the code should be portable across the environments, and therefore the split is not required?

This is not true - code is currently already split into directories.

I'll end up loading the node-related code in the browser environment and vice versa.

I don't think this is true either, Node-related files (everything other than runtime) is never required in the browser context, and it is the same for Node's context too.

This consumes compute cycles unnecessarily, and on its own, it's not a big deal, but coupled with other packages that do the same it becomes a huge problem.

Unless we bundle everything into one file (e.g. Rollup/Webpack), require cycles will still happen, and will only happen with files that are specific to that context. However, most of the runtime code cannot be grouped because of their varying nature, and a lot of the code will not work until it is being injected by webpack.ProvidePlugin.

My proposal is to process the runtime part that's injected differently than the code that's being run as part of the webpack execution at the package level.

The point is, I don't see any need to pre-process/transpile any of the runtime code. No code in the runtime folder is non-compatible with browsers/non-native renderers. The nature of the issues you were facing are related to how Webpack 4 aggressively polyfills (all Node APIs if used) patches stuff for the browser (like defaulting process.env) but not for electron. That is not something pre-processing can help - especially when we know that this behaviour is going away in Webpack 5, which means by then the code will be obviously broken. We could do the same aggressive polyfilling, but it brings in bundle penalties for everybody.

Unless we're switching to ES Modules/ES6+ syntax, I don't feel that processing the current plain CommonJS, ES5 compliant code (other than use of the URL API which breaks IE11) is a right move. Writing tests (for various output targets) seems like a much simpler and beneficial approach - which I will need help.

[MOZGIII]

The point is, I don't see any need to pre-process/transpile any of the runtime code. No code in the runtime folder is non-compatible with browsers/non-native renderers. The nature of the issues you were facing are related to how Webpack 4 aggressively polyfills (all Node APIs if used) patches stuff for the browser (like defaulting process.env) but not for electron. That is not something pre-processing can help - especially when we know that this behaviour is going away in Webpack 5, which means by then the code will be obviously broken. We could do the same aggressive polyfilling, but it brings in bundle penalties for everybody.

I'm not following.

The issue, in particular, is things like this:

react-refresh-webpack-plugin/lib/utils/injectRefreshEntry.js

Line 1 in 9276e4d

const querystring = require('querystring');

This has nothing to do with polyfilling, but about cutting what gets evaluated in the renderer webpack context or now. You are assuming that only the content of the runtime folder gets included into the webpack bundle that's executing at runtime - however, in my case, things from lib ended up there too. This is the root cause of the problem, and what I'm proposing to solve.

I don't know why it only affects electron, possibly because there we have all the various execution contexts, while normally the situation would be way simpler.

We have:

• node context - for cli tooling, build process, webpack-dev-server and etc; runs at dev time, executed natively without webpack preprocessing.
• electron main context - for code that runs as part of the app (at runtime) in the main electron process; very similar to node context, webpack produces a bundle with all context, except for a few select require targets, like nodejs APIs or native modules.
• electron preload context - for code that runs at the render process before the real page is loaded; the code is processed into a bundle, but it still has access to node APIs (like require fn), but also to the browser API; this is where the issue occurs - the lib code is included into a bundle. This, in turn, will unconditionally trigger a non-webpack require at load (see the listing with code from lib above) effectively doing a privileged nodejs call, which triggers a security error and prevents further execution. This lib isn't intended for loading at runtime, but it's what's happening currently.
• electron render process - webpack processed code for the user-visible UI rendered in the browser; doesn't have access to nodejs require fn, but the lib code is also still loaded there - and we get a "no such fn" error upon loading.

So, again, if everything worked as you assume - with only runtime actually being loaded in runtime - that would be completely fine. Polyfills are different story, and I'm not worried about those at all, since in electron we're in control of the browser-side APIs and know exactly what version we have (URL Web API object is available in our case, hence no polyfill is required). The problem is with rouge files getting into the bundle, and the code intended to even be run at runtime issuing require-s and process.envs.

[MOZGIII]

Steps to reproduce the core issue:

git clone https://github.com/pmmmwh/react-refresh-webpack-plugin
cd react-refresh-webpack-plugin/examples/typescript-without-babel
yarn
yarn webpack --mode development --hot
grep '__webpack_require__.*querystring' dist/main.js

The last grep shows that querystring is indeed imported, yet it's supposed to not have been if, as you say, only runtime folder was included.

Again the querystring require comes from

react-refresh-webpack-plugin/lib/utils/injectRefreshEntry.js

Line 1 in 9276e4d

const querystring = require('querystring');