Configurable Broker tier selection for Realtime servers#19062
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Currently the BrokerServerView shares the same strategy configured by druid.broker.select.tier for both historicals and realtime servers. This patch allows operators to optionally overide that behavior based on how realtime servers are setup via druid.broker.select.realtime.tier property. If this property isn't specified, the realtime servers' strategy will continue to share the historical's strategy (backwards compatble).
abhishekrb19
force-pushed
the
configurable_broker_realtime_tier_strategy
branch
from
5e5a322 to
441f48d
Compare
abhishekrb19
force-pushed
the
configurable_broker_realtime_tier_strategy
branch
from
734a358 to
fd405ba
Compare
| public String toString() | ||
| { | ||
| return "PreferredTierSelectorStrategy{" + | ||
| ", config=" + config + |
There was a problem hiding this comment.
| ", config=" + config + | |
| "config=" + config + |
| final FilteredServerInventoryView baseView, | ||
| final TierSelectorStrategy tierSelectorStrategy, | ||
| final TierSelectorStrategy historicalTierSelectorStrategy, | ||
| @Named(REALTIME_SELECTOR) final TierSelectorStrategy realtimeTierSelectorStrategy, |
There was a problem hiding this comment.
Can we also have @Named for historicalTierSelectorStrategy? This makes things more explicit in the test code
There was a problem hiding this comment.
Hmm agreed, that would make it clearer for the historical one as well. However, it would require introducing a separate provider to bind the @Named variant and move the existing bindings from CliBroker, since the same class cannot be bound multiple times in Guice. I punted on that for now in favor of a brief comment here indicating where these bindings are injected from. Hopefully that clarifies it?
There was a problem hiding this comment.
Ahh I see, thanks for clarifying. Makes sense, and the comments help!
| instanceof LowestPriorityTierSelectorStrategy | ||
| ); | ||
| } | ||
|
|
There was a problem hiding this comment.
Could we have one additional test case for the Preferred strategy just to validate the bindings? Thanks!
There was a problem hiding this comment.
Thanks for calling that out! Added tests for PreferredTierSelectorStrategy as well.
3 participants
Currently, the Broker uses the same tier selection and balancer strategies configured by
druid.broker.select.tieranddruid.broker.balancer.typefor both Historicals and realtime servers.This patch allows operators to optionally override that behavior for realtime servers via the
druid.broker.realtime.select.tieranddruid.broker.realtime.balancer.typeproperties. This is useful in environments where Historicals and Realtime servers are configured differently.druid.broker.realtime.select.tieranddruid.broker.realtime.balancer.typeare not configured (default behavior), realtime servers will continue to use the default strategy or the one configured viadruid.broker.select.tieranddruid.broker.select.balancer.type, for backward compatibility.Approach:
BrokerRealtimeSelectorModulethat's added toCliBrokerTierSelectorStrategyandServerSelectorStrategydruid.broker.balancer.typeanddruid.broker.select.tierare applied for the realtime servers (backward compatible behavior)While at it, I also added
toString()and related helpers forTierSelectorStrategyimplementations to improve debuggability and test verification.I can document this configuration separately, since the relevant sections in the existing docs for
druid.broker.select.tiercould also use an update.Release note:
Added
druid.broker.realtime.select.tieranddruid.broker.realtime.balancer.typeon the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are unset (by default), realtime servers continue to use the existingdruid.broker.selectanddruid.broker.balancerconfigurations that applies to both historical and realtime servers.This PR has: