Akka.Discovery.Dns #3365

anpin · 2025-07-03T19:50:46Z

Discovery via DNS

A/AAAA records resolved via default inet-resolver
SRV records are resolved via async-dns (name inherited from JVM Akka)

Fixes #3364

Changes

New IDnsProvider for async-dns resolver to issue direct SRV queries as underlying dotnet implementation resolves only A/AAAA
Hosting extensions to configure
Examples for docker-compose
Fix for IPv6 formatting in BootstrapCordinator

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

This change follows the Akka.NET API Compatibility Guidelines.
This change follows the Akka.NET Wire Compatibility Guidelines.
I have reviewed my own pull request.
Design discussion issue Port akka.io.dns and akka.discovery.dns akka.net#5136 Discovery via DNS #3364
Changes in public API reviewed, if any.
I have added website documentation for this feature.

Latest `dev` Benchmarks

Include data from the relevant benchmark prior to this change here.

This PR's Benchmarks

Include data from after this change here.

anpin · 2025-07-08T19:15:36Z

I managed to get the DNS cluster example working, for both A and SRV records. As mentioned in previous issues SRV required to implement custom resolve handler.

Aaronontheweb · 2025-07-09T03:44:21Z

@anpin VERY nice! is this ready for review or are you still working on it?

anpin · 2025-07-09T11:51:40Z

Thanks. Still working on it. Main concerns for now:

new SRV resolver doesn't have unit tests yet
Not sure how to properly instantiate new DNS provider. In pekko DnsProvider marked as obsolete. Akka.Net relies on DnsExt, but it doesn't respect IDnsProvider.ManagerClass and always uses hard-coded SimpleDnsManager. I think this should be fixed in main Akka.Net repo.

Aaronontheweb · 2025-07-09T14:53:31Z

think this should be fixed in main Akka.Net repo.

100% - honestly, I'm not even sure how much Akka.IO's DNS code is even used. I just spent a bunch of time in April / May purging tons of poorly designed older code from the TCP stack in Akka.IO. Wouldn't surprise me if the DNS stack was full of rot too.

anpin · 2025-07-09T15:14:12Z

Well it seems that TcpOutgoingConnection is using dotnet dns client these days.
https://github.com/akkadotnet/akka.net/blob/c10cfc16d25879a9db3ce5f9de1a3e1074e0401f/src/core/Akka/IO/TcpOutgoingConnection.cs#L108

However as pointed out in the previous discussion, dotnet dns client is not capable of resolving SRV records, unlike JVM counterpart

Aaronontheweb · 2025-07-11T20:30:34Z

@anpin merged your PR on the main Akka.NET project - are you blocked from working on this until we do a new release of that or are you good to go for now?

anpin · 2025-07-11T21:22:47Z

no pressure on my end for a new release. thanks @Aaronontheweb

anpin · 2025-07-14T21:23:11Z

Fixed examples and added tests to resolve some publicly known records. I think it is close to be considered for review.

anpin · 2025-07-14T21:27:44Z

Couldn't find a way to unit test TCP fallback for SRV requests, maybe akka/pekko tests have some clues

anpin · 2025-07-14T21:37:59Z

Above mentioned PR akkadotnet/akka.net#7727 not present on nuget feed yet, so the tests in CI are still trying to use SimpleDnsManager

anpin · 2025-07-15T14:14:28Z

looked at the jvm hocon again and noticed that a few major features were overlooked, e.g. multiple nameservers

anpin · 2025-07-15T17:04:35Z

Trying to implement caching now, existing SimpleDnsCache can't be reused here as SimpleDnsCache.Put method and IPeriodicCacheCleanup are internal. Any idea how to move forward @Aaronontheweb?

Edit: Actually existing SimpleDnsCache cache is not a great fit for SRV records as cached instances use IPAddress class, without a port. For SRV records cache using EndPoint would be more appropriate.

only A/AAAA records are supported at this moment

see akkadotnet/akka.net#7727

…ependent expirience

anpin · 2025-07-18T11:03:47Z

seems like some tests in per-existing code are unreliable on linux

Aaronontheweb · 2025-07-18T14:23:32Z

seems like some tests in per-existing code are unreliable on linux

yeah some of the K8s specs are flaky - we'll get to work on reviewing this. thanks for all of your hard work @anpin

Arkatufus · 2025-07-21T18:46:32Z

Question, there's a public static object in AsyncDnsClient called TcpDropped, it is being sent to the actor parent when there is a problem with the TCP connection. Where is this message being handled?

Arkatufus · 2025-07-21T19:08:51Z

Comparing to the scala code, this is supposed to be handled by the DnsClient to cleanup its inflight TCP request list

Arkatufus

Found some problems with the current implementation.

src/discovery/dns/Akka.Discovery.Dns/Internal/AsyncDnsClient.cs

anpin · 2025-07-22T12:34:47Z

Switched query ID from short to int and added lookup to verify ID doesn't exists in the collection.

Refactored TcpDropped message and added a handle.

I'm on a very unreliable network today, so got to test those TCP failure cases while correcting the code. Made the TCP failures reply back with error on connection error and fail discovery before timeout occurs, for untruncated UDP requests the client will still fail on timeout.

@Arkatufus please let me know if I can improve anyhow.

Arkatufus

Reviewed parts so far:

DNS UDP/TCP packet binary serializer/deserializer, everything looks good.
General DNS client actor flow, looks good.

Would love to have more people to look over this, but it looks good to me.

Aaronontheweb · 2025-07-28T18:37:12Z

Thanks @Arkatufus - I'll take a stab at this in the next day or two

Aaronontheweb · 2025-07-28T18:37:42Z

Gonna ask CoPilot to review it too - why not?

Copilot

Pull Request Overview

This PR introduces DNS-based service discovery support to Akka.NET Management, enabling cluster bootstrap through DNS A/AAAA records and SRV records. This new capability allows Akka clusters to discover nodes using DNS infrastructure without requiring specialized service discovery systems.

Implements a complete DNS discovery provider with async DNS client supporting both standard DNS (A/AAAA) and SRV record resolution
Adds IPv6 support enhancement to BootstrapCoordinator for proper URI formatting
Provides comprehensive examples and documentation for various DNS record types

Reviewed Changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 9 comments.

File	Description
src/management/Akka.Management/Cluster/Bootstrap/Internal/BootstrapCoordinator.cs	Fixes IPv6 address formatting in URI construction
src/discovery/dns/Akka.Discovery.Dns/*.cs	Complete DNS service discovery implementation including async DNS client, protocol handling, and caching
src/discovery/dns/Akka.Discovery.Dns.Tests/*.cs	Unit tests for DNS discovery functionality
src/cluster.bootstrap/examples/discovery/dns/*	Docker-compose examples demonstrating A, AAAA, and SRV record-based discovery

Comments suppressed due to low confidence (1)

src/discovery/dns/Akka.Discovery.Dns/Internal/DnsProtocol.cs

src/discovery/dns/Akka.Discovery.Dns/Internal/AsyncDnsClient.cs

src/discovery/dns/Akka.Discovery.Dns/DnsServiceDiscovery.cs

src/discovery/dns/Akka.Discovery.Dns/Internal/TcpDnsClient.cs

Aaronontheweb

So I tried running the demo locally and got this:

[+] Running 6/6
 ✔ node1                    Built                                                                                                                                                                                                        0.0s  ✔ Network src_akkanet      Created                                                                                                                                                                                                      0.1s  ✔ Container src-coredns-1  Created                                                                                                                                                                                                      0.2s  ✔ Container src-node1-1    Created                                                                                                                                                                                                      0.2s  ✔ Container src-node3-1    Created                                                                                                                                                                                                      0.2s  ✔ Container src-node2-1    Created                                                                                                                                                                                                      0.3s Attaching to coredns-1, node1-1, node2-1, node3-1
coredns-1  | .:1053
coredns-1  | [INFO] plugin/reload: Running configuration SHA512 = ff9cb9f62c926ec7a3b0d3ee6a542fbe41139b95a7f80cd26dd277e714af89d68c326d83caf0302723cfd0a3ffb7faf4d55ee9220a9e6ac89018003ded3abfef
coredns-1  | CoreDNS-1.10.1
coredns-1  | linux/amd64, go1.20, 055b2c3
node1-1    | ==== AKKA DNS CLUSTER NODE STARTING ====
node1-1    | Hostname: node1.akkacluster
node1-1    | IP addresses: 172.28.0.10
node1-1    | Environment variables:
node1-1    |   CLUSTER__PORT: 4053
node1-1    |   CLUSTER__IP: 0.0.0.0
node1-1    |   MANAGEMENT__PORT: 18558
node1-1    |   ACTORSYSTEM: DnsCluster
node1-1    |   SERVICENAME: akkacluster.dns.oci
node1-1    |   PORTNAME: management
node2-1    | ==== AKKA DNS CLUSTER NODE STARTING ====
node3-1    | ==== AKKA DNS CLUSTER NODE STARTING ====
node1-1    |   DNS_PORT: 1053
node2-1    | Hostname: node2.akkacluster
node1-1 exited with code 9
node3-1    | Hostname: node3.akkacluster
node1-1    |   DNS_NAMESERVER: 172.28.0.2
node2-1    | IP addresses: 172.28.0.20
node3-1    | IP addresses: 172.28.0.30
node1-1    | ===================================
node2-1    | Environment variables:
node3-1    | Environment variables:
node2-1 exited with code 9
node1-1    | \nPerforming DNS resolution test for 'akkacluster.dns.oci'...
node2-1    |   CLUSTER__PORT: 4053
node3-1    |   CLUSTER__PORT: 4053
node1-1    | ;; communications error to 127.0.0.11#1053: connection refused
node2-1    |   CLUSTER__IP: 0.0.0.0
node3-1    |   CLUSTER__IP: 0.0.0.0
node1-1    | ;; communications error to 127.0.0.11#1053: connection refused
node3-1 exited with code 9
node2-1    |   MANAGEMENT__PORT: 28558
node3-1    |   MANAGEMENT__PORT: 38558
node1-1    | ;; communications error to 127.0.0.11#1053: connection refused
node2-1    |   ACTORSYSTEM: DnsCluster
node3-1    |   ACTORSYSTEM: DnsCluster
node1-1    |
node2-1    |   SERVICENAME: akkacluster.dns.oci
node3-1    |   SERVICENAME: akkacluster.dns.oci
node1-1    | ; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> -p 1053 akkacluster.dns.oci
node2-1    |   PORTNAME: management
node3-1    |   PORTNAME: management
node1-1    | ;; global options: +cmd
node2-1    |   DNS_PORT: 1053
node3-1    |   DNS_PORT: 1053
node1-1    | ;; no servers could be reached
node2-1    |   DNS_NAMESERVER: 172.28.0.2
node3-1    |   DNS_NAMESERVER: 172.28.0.2
node2-1    | ===================================
node3-1    | ===================================
node2-1    | \nPerforming DNS resolution test for 'akkacluster.dns.oci'...
node3-1    | \nPerforming DNS resolution test for 'akkacluster.dns.oci'...
node2-1    | ;; communications error to 127.0.0.11#1053: connection refused
node3-1    | ;; communications error to 127.0.0.11#1053: connection refused
node2-1    | ;; communications error to 127.0.0.11#1053: connection refused
node3-1    | ;; communications error to 127.0.0.11#1053: connection refused
node2-1    | ;; communications error to 127.0.0.11#1053: connection refused
node3-1    | ;; communications error to 127.0.0.11#1053: connection refused
node2-1    |
node3-1    |
node2-1    | ; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> -p 1053 akkacluster.dns.oci
node3-1    | ; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> -p 1053 akkacluster.dns.oci
node2-1    | ;; global options: +cmd
node3-1    | ;; global options: +cmd
node2-1    | ;; no servers could be reached
node3-1    | ;; no servers could be reached

It looks like all of the nodes killed themselves after a DNS failure:

I'm running on Windows, but the same thing happened to me when I ran this on a Linux box too. What's supposed to happen when we run the sample?

Aaronontheweb · 2025-07-29T20:21:47Z

src/discovery/dns/Akka.Discovery.Dns/README.md

+- **A/AAAA records**: Standard DNS address records that return IP addresses
+- **SRV records**: Service records that provide both IP addresses and port information
+
+## Enabling DNS Discovery Using Akka.Hosting


Stupid question from yours, truly: so in order for this to work, do we need a DNS server that is writeable and accessible somewhere inside the network? I'm totally unfamiliar with how to do this outside of using tools like Pulumi to modify DNS records on specific cloud providers - so if I wanted to run this on a bare metal setup, would I need to stand up something like CoreDNS or PiHole?

anpin · 2025-07-30T00:11:09Z

All three examples are running fine on my linux machine, but I will test it in another environment. Attaching logs produced as such:

cd ~/projects/Akka.Management/src/cluster.bootstrap/examples/discovery/dns/
./build.ps1 a > a.log
./build.ps1 aaaa > aaaa.log
./build.ps1 srv > srv.log

srv.log
a.log
aaaa.log

anpin · 2025-07-30T14:36:02Z

I had a rough time running docker on my windows VM and I don't have a physical windows machine I can test it on right now

anpin force-pushed the feature/dns branch 4 times, most recently from 8f61cdd to 9bf29c4 Compare July 8, 2025 19:13

anpin mentioned this pull request Jul 9, 2025

DnsExt doesn't respect IDnsProvider.ManagerClass and always uses hard-coded SimpleDnsManager akkadotnet/akka.net#7726

Closed

anpin marked this pull request as ready for review July 17, 2025 12:35

anpin added 12 commits July 17, 2025 09:42

init Akka.Discovery.Dns

13f5341

only A/AAAA records are supported at this moment

init dns cluster example

94f10eb

SRV example cluster is starting

a970620

cleanup commented code and typos

398cecb

Custom override for DnsExt is no longer required

20e50b2

see akkadotnet/akka.net#7727

fix IPv6 formating

2230205

fix logging of parsed IP records

f7eccd6

separate A and AAAA examples

533fe19

use dns.oci root domain instead of default dns.podman for runtime ind…

8740cff

…ependent expirience

use docker-compose to run examples as it is more common

fbec5dc

fix reference HOCON

e26b4df

cleanup hosting extensions

3d8fc58

reduce line noise

13e8c07

Merge branch 'dev' into feature/dns

01da986

Fix/add documentation

17c1896

Arkatufus requested changes Jul 21, 2025

View reviewed changes

src/discovery/dns/Akka.Discovery.Dns/Internal/AsyncDnsClient.cs Outdated Show resolved Hide resolved

src/discovery/dns/Akka.Discovery.Dns/Internal/AsyncDnsClient.cs Outdated Show resolved Hide resolved

anpin added 4 commits July 22, 2025 09:23

make constructors for immutable messages private

e3c5320

Make TcpDropped message immutable and handle it

29f5883

Remove unreferenced DropRequest message and handlers

287fb0b

Reply with error on TCP failure

b61745b

Refactor query ID generation

1b2deab

anpin force-pushed the feature/dns branch from b1e5c74 to 1b2deab Compare July 24, 2025 11:46

anpin requested a review from Arkatufus July 24, 2025 11:56

Arkatufus approved these changes Jul 28, 2025

View reviewed changes

Aaronontheweb requested review from Aaronontheweb and Copilot July 28, 2025 18:37

Copilot AI reviewed Jul 28, 2025

View reviewed changes

anpin added 2 commits July 29, 2025 08:33

applied copilot suggestions

5351c9c

add better comment

04917ac

Aaronontheweb mentioned this pull request Jul 29, 2025

Added missing .gitattributes file #3377

Merged

Merge branch 'dev' into feature/dns

2cf78c9

Aaronontheweb requested changes Jul 29, 2025

View reviewed changes

Aaronontheweb reviewed Jul 29, 2025

View reviewed changes

Akka.Discovery.Dns #3365

Are you sure you want to change the base?

Akka.Discovery.Dns #3365

Uh oh!

Conversation

anpin commented Jul 3, 2025 • edited by Aaronontheweb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Checklist

Latest dev Benchmarks

This PR's Benchmarks

Uh oh!

anpin commented Jul 8, 2025

Uh oh!

Aaronontheweb commented Jul 9, 2025

Uh oh!

anpin commented Jul 9, 2025

Uh oh!

Aaronontheweb commented Jul 9, 2025

Uh oh!

anpin commented Jul 9, 2025

Uh oh!

Aaronontheweb commented Jul 11, 2025

Uh oh!

anpin commented Jul 11, 2025

Uh oh!

anpin commented Jul 14, 2025

Uh oh!

anpin commented Jul 14, 2025

Uh oh!

anpin commented Jul 14, 2025

Uh oh!

anpin commented Jul 15, 2025

Uh oh!

anpin commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anpin commented Jul 18, 2025

Uh oh!

Aaronontheweb commented Jul 18, 2025

Uh oh!

Arkatufus commented Jul 21, 2025

Uh oh!

Arkatufus commented Jul 21, 2025

Uh oh!

Arkatufus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anpin commented Jul 22, 2025

Uh oh!

Arkatufus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb commented Jul 28, 2025

Uh oh!

Aaronontheweb commented Jul 28, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

anpin commented Jul 3, 2025 •

edited by Aaronontheweb

Loading

Latest `dev` Benchmarks

anpin commented Jul 15, 2025 •

edited

Loading

Arkatufus left a comment •

edited

Loading