http,https: add built-in proxy support in http/https.request and Agent

[joyeecheung]

This patch implements proxy support for HTTP and HTTPS clients and
agents in the http and https built-ins`. When NODE_USE_ENV_PROXY
is set to 1, the default global agent would parse the
HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy
settings from the environment variables, and proxy the requests
sent through the built-in http/https client accordingly.

To support this, http.Agent and https.Agent now accept a few new
options:

• proxyEnv: when it's an object, the agent would read and parse
  the HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy
  properties from it, and apply them based on the protocol it uses
  to send requests. This option allows custom agents to
  reuse built-in proxy support by composing options. Global agents
  set this to process.env when NODE_USE_ENV_PROXY is 1.
• defaultPort and protocol: these allow setting of the default port
  and protocol of the agents. We also need these when configuring
  proxy settings and deciding whether a request should be proxied.

Example

Starting a Node.js process with proxy support enabled for all requests sent
through the default global agent:

NODE_USE_ENV_PROXY=1 HTTP_PROXY=http://proxy.example.com:8080 HTTPS_PROXY=http://proxy.example.com:8080 NO_PROXY=localhost,127.0.0.1 node client.js

To create a custom agent with built-in proxy support:

const http = require('node:http');
// Creating a custom agent with custom proxy support.
const agent = new http.Agent({ proxyEnv: { HTTP_PROXY: 'http://proxy.example.com:8080' } });
http.request({
  hostname: 'www.example.com',
  port: 80,
  path: '/',
  agent,
}, (res) => {
  // This request will be proxied through proxy.example.com:8080 using the HTTP protocol.
  console.log(`STATUS: ${res.statusCode}`);
});

Alternatively, the following also works:

const http = require('node:http');
// Use lower-cased option name.
const agent1 = new http.Agent({ proxyEnv: { http_proxy: 'http://proxy.example.com:8080' } });
// Use values inherited from the environment variables, if the process is started with
// HTTP_PROXY=http://proxy.example.com:8080 this will use the proxy server specified
// in process.env.HTTP_PROXY.
const agent2 = new http.Agent({ proxyEnv: process.env });

Implementation

Implementation-wise, this adds a ProxyConfig internal class to handle
parsing and application of proxy configurations. The configuration
is parsed during agent construction. When requests are made,
the createConnection() methods on the agents would check whether
the request should be proxied. If yes, they either connect to the
proxy server (in the case of HTTP reqeusts) or establish a tunnel
(in the case of HTTPS requests) through either a TCP socket (if the
proxy uses HTTP) or a TLS socket (if the proxy uses HTTPS).

When proxying HTTPS requests through a tunnel, the connection listener
is invoked after the tunnel is established. Tunnel establishment uses
the timeout of the request options, if there is one. Otherwise it uses
the timeout of the agent.

If an error is encountered during tunnel establishment, an
ERR_PROXY_TUNNEL would be emitted on the returned socket. If the proxy
server sends a errored status code, the error would contain an
statusCode property. If the error is caused by timeout, the error
would contain a proxyTunnelTimeout property.

This implementation honors the built-in socket pool and socket limits.
Pooled sockets are still keyed by request endpoints, they are just
connected to the proxy server instead, and the persistence of the
connection can be maintained as long as the proxy server respects
connection/proxy-connection or persist by default (HTTP/1.1)

Testing

Most of the diff of this patch are tests for various cases that a proxied client can run into. I also tested it with a production proxy server behind a firewall.

To check how transparent the sockets behave when they are going through a proxy, I also ran the existing http/https tests over a minimal testing proxy server:

NODE_USE_ENV_PROXY=1 HTTPS_PROXY=http://127.0.0.1:8000 HTTP_PROXY=http://127.0.0.1:8000 tools/test.py --timeout=20 "test/parallel/test-http*" "test/sequential/test-http*"

[03:36|% 100|+ 635|-  80]: Done

So >88% of the existing HTTP/HTTPS use cases work transparently when they go through a proxy. Among the failures most of them are caused by testing proxy server not being transparent since I didn't put a lot of thought into it, or that the tests are expecting specific things (e.g. events, errors) that have to come from a server in the same process. I think this is good enough as an initial implementation and we can continue iterating to make the proxied behavior as transparent as possible.

Some TODOs:

• Check that it allows other popular user-land agents (for proxy or not) to operate with/without NODE_USE_ENV_PROXY
• Check that it works transparently with popular npm packages that uses http/https.request

The first commit comes from #58950 - I split it off since it just moved the existing tests for fetch into a new client-proxy directory and made the testing proxy server a bit more versatile, not strictly tied to what this patch tries to implement.

Refs: #57872
Refs: #8381
Refs: #15620

[nodejs-github-bot]

Review requested:

• @nodejs/crypto
• @nodejs/http
• @nodejs/net

[codecov]

Codecov Report

Attention: Patch coverage is 93.95425% with 37 lines in your changes missing coverage. Please review.

Project coverage is 90.09%. Comparing base (b16a0e7) to head (036b1fd).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/http.js	94.27%	8 Missing and 3 partials ⚠️
lib/https.js	96.21%	8 Missing and 2 partials ⚠️
lib/_http_agent.js	91.57%	8 Missing ⚠️
lib/_http_client.js	85.71%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #58980      +/-   ##
==========================================
+ Coverage   90.05%   90.09%   +0.03%     
==========================================
  Files         645      645              
  Lines      189153   189730     +577     
  Branches    37091    37217     +126     
==========================================
+ Hits       170348   170943     +595     
+ Misses      11528    11506      -22     
- Partials     7277     7281       +4     

Files with missing lines	Coverage Δ
lib/internal/errors.js	97.50% <100.00%> (+<0.01%)	⬆️
lib/internal/process/pre_execution.js	90.21% <100.00%> (-0.54%)	⬇️
lib/_http_agent.js	97.28% <91.57%> (-0.89%)	⬇️
lib/_http_client.js	97.33% <85.71%> (-0.66%)	⬇️
lib/https.js	98.07% <96.21%> (-1.23%)	⬇️
lib/internal/http.js	95.66% <94.27%> (-4.34%)	⬇️

... and 43 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @joyeecheung.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[mcollina]

lgtm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/67953/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/67967/

[silverwind]

Could we in addition or alternatively to NODE_USE_ENV_PROXY have a --use-env-proxy or similar? Reason I'm asking is that such flag options can be set in a script's hashbang without having to mess with process.env:

#!/usr/bin/env -S node --use-env-proxy
fetch()

Such an option could also be passed via NODE_OPTIONS=--use-env-proxy for example, which integrates better than a bare env variable.

[joyeecheung]

Could we in addition or alternatively to NODE_USE_ENV_PROXY have a --use-env-proxy or similar? Reason I'm asking is that such flag options can be set in a script's hashbang without having to mess with process.env:

The idea SGTM though I think it should be done in a separate PR - this PR just adds support to http/https builtins, support for NODE_USE_ENV_PROXY in fetch already happened in #57165, so it will need another PR to support the flag in both. Can you open a feature request issue so I can add it to the tracking issue?

[silverwind]

Ok, opened #59100.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/67993/

[joyeecheung]

Last CI was orange, the CI status of node-test-commit-osx can't be updated by the bot because the job is currently disabled (nodejs/build#4105). Manually landed in b16a0e7...036b1fd.