HTTP/RPC Unbounded Resource Usage, Susceptible to Unauthenticated DoS

[schmichael]

Vulnerability ID: CVE-2020-7218
Versions: Previous versions of Nomad and Nomad Enterprise; fixed in 0.10.3.

A vulnerability was identified in Nomad such that unbounded resource usage, triggered by the establishment of many unauthenticated HTTP or RPC connections, may generate excessive load and/or crash the server. This vulnerability affects all previous releases of Nomad, and is fixed in the 0.10.3 release.

Background

Nomad utilizes HTTP for networked communication between Nomad cluster members and control/administration of Nomad servers.

Nomad clients are configured to communicate with their regional servers and communicate using remote procedure calls (RPC) to register themselves, send heartbeats for liveness, wait for new allocations, and update the status of allocations. These RPC requests/responses traverse the network between Nomad nodes using TCP.

Nomad’s HTTP and RPC both support using end-to-end TLS with optional client authentication.

Vulnerability Details

An internal security review identified a vulnerability such that it was possible to consume excessive Nomad server resources via HTTP and RPC services.

A Nomad server was exposed to denial of service attack by any party with network-level connectivity to that server. Authentication via mutual TLS is not required in order to launch an attack.

While successful exploitation of this vulnerability would not affect confidentiality or data integrity within a Nomad deployment, availability would be affected.

Remediation

Users should upgrade to Nomad 0.10.3, or newer. Please refer to Upgrading Nomad for general guidance and version-specific upgrade notes.

Several new configuration options are introduced (rpc_max_conns_per_client, rpc_handshake_timeout, http_max_conns_per_client, https_handshake_timeout) with default values which restrict resource usage and reduce exposure to attack.

If upgrade is not possible, customers may consider other mitigation options:

• Restrict connectivity to Nomad servers to trusted sources/networks only. TCP ports used by Nomad for HTTP and RPC default to 4646 and 4647.
• Restrict resource usage on Nomad servers by enforcing network connection limits. For example:

iptables -A INPUT -p tcp --syn --dport 4646 -m connlimit --connlimit-above 100 -j REJECT --reject-with tcp-reset

Enabling mutual TLS as a defense-in-depth mechanism for Nomad clusters, per documentation, remains strongly recommended.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.