Skip to content

Add a new custom router scheduler plugin : presto-plan-checker-router-plugin#25035

Merged
aditi-pandit merged 3 commits into
prestodb:masterfrom
pdabre12:router-bolt
Jun 10, 2025
Merged

Add a new custom router scheduler plugin : presto-plan-checker-router-plugin#25035
aditi-pandit merged 3 commits into
prestodb:masterfrom
pdabre12:router-bolt

Conversation

@pdabre12

@pdabre12 pdabre12 commented May 2, 2025

Copy link
Copy Markdown
Contributor

Description

Add a new custom router scheduler plugin : presto-plan-checker-router-plugin.

Motivation and Context

This change introduces a custom router plugin designed to facilitate the seamless migration from Java-based clusters to native C++ clusters in production environments. The plugin acts as an intermediary between two routers - one pointing to a group of native (C++) clusters and the other to a group of Java clusters.

The plugin uses a StatementClient to send each incoming request to a sidecar-enabled plan checker cluster. Based on the response from the sidecar, the plugin determines the routing part:

  • If the sidecar enabled cluster does not return an error, the query is routed to the native C++ router.
  • If the sidecar indicates that the query is native incompatible, the plugin routes the incoming query to the Java router.
  • If at any stage, we run into any syntax/user errors, the query is returned back to the original client with the QueryError field populated to indicate the failure.

Impact

This dynamic routing mechanism allows users to gradually migrate workloads from Java to native C++ with minimal risk and without manual intervention. Additionally , it allows users to retain full control over scheduling in both the native and the Java routers.

Test Plan

Test cases included, more e2e tests need to be added.

Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Router Changes
* Add a new custom router scheduler plugin, the `Presto Plan Checker Router Scheduler Plugin <https://github.com/prestodb/presto/tree/master/presto-plan-checker-router-plugin/README.md>`_.

@prestodb-ci prestodb-ci added the from:IBM PR from IBM label May 2, 2025
@pdabre12 pdabre12 force-pushed the router-bolt branch 5 times, most recently from be00ae3 to a67d074 Compare May 13, 2025 23:30
@pdabre12 pdabre12 force-pushed the router-bolt branch 10 times, most recently from 1be19aa to 88029b1 Compare May 26, 2025 19:38
@pdabre12 pdabre12 force-pushed the router-bolt branch 3 times, most recently from f8261fc to 0037958 Compare May 26, 2025 20:02
@pdabre12 pdabre12 changed the title [Do not review] Add custom scheduler in presto-router Add a new router plugin : presto-plan-checker-router-plugin May 26, 2025
@pdabre12 pdabre12 changed the title Add a new router plugin : presto-plan-checker-router-plugin Add a new router plugin : presto-plan-checker-router-plugin May 26, 2025
@pdabre12 pdabre12 force-pushed the router-bolt branch 8 times, most recently from 30c7707 to da3c6c3 Compare May 29, 2025 20:25
@pdabre12 pdabre12 changed the title Add a new router plugin : presto-plan-checker-router-plugin Add a new custom router scheduler plugin : presto-plan-checker-router-plugin May 29, 2025
@pdabre12

pdabre12 commented Jun 9, 2025

Copy link
Copy Markdown
Contributor Author

Thanks @steveburnett , made the changes, can you PTAL?

@pdabre12 pdabre12 requested a review from aaneja June 9, 2025 20:31
@steveburnett

Copy link
Copy Markdown
Contributor

Thanks @steveburnett , made the changes, can you PTAL?

Thanks for updating the link! Could you change the section heading General Changes to Router Changes?

aaneja
aaneja previously approved these changes Jun 10, 2025
@pdabre12

pdabre12 commented Jun 10, 2025

Copy link
Copy Markdown
Contributor Author

Thanks @steveburnett , made the changes, can you PTAL?

Thanks for updating the link! Could you change the section heading General Changes to Router Changes?

@steveburnett Done, thanks !

aaneja
aaneja previously approved these changes Jun 10, 2025
@steveburnett

Copy link
Copy Markdown
Contributor

@steveburnett Done, thanks !

I'm sorry if I was unclear! We've added a new section heading Router Changes that this note should go into, instead of the General Changes section heading. Apologies for this nit taking so much of your time.

== RELEASE NOTES ==

Router Changes
* Add a new custom router scheduler plugin, the `Presto Plan Checker Router Scheduler Plugin <https://github.com/prestodb/presto/tree/master/presto-plan-checker-router-plugin/README.md>`_.

@pdabre12

pdabre12 commented Jun 10, 2025

Copy link
Copy Markdown
Contributor Author

@steveburnett Done, thanks !

I'm sorry if I was unclear! We've added a new section heading Router Changes that this note should go into, instead of the General Changes section heading. Apologies for this nit taking so much of your time.

== RELEASE NOTES ==

Router Changes
* Add a new custom router scheduler plugin, the `Presto Plan Checker Router Scheduler Plugin <https://github.com/prestodb/presto/tree/master/presto-plan-checker-router-plugin/README.md>`_.

@steveburnett My bad, I misread your suggestion , I have moved it to Router Changes now.

@aditi-pandit aditi-pandit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pdabre12

@aditi-pandit aditi-pandit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pdabre12.

We need to investigate these failures the e2e test failures but I don't think they are related to this work https://github.com/prestodb/presto/actions/runs/15566344717/job/43839187753?pr=25035

@pdabre12

Copy link
Copy Markdown
Contributor Author

Thanks @pdabre12.

We need to investigate these failures the e2e test failures but I don't think they are related to this work https://github.com/prestodb/presto/actions/runs/15566344717/job/43839187753?pr=25035

Yes, these failures were seen earlier too: https://github.com/prestodb/presto/actions/runs/15566813960/job/43833560154

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

from:IBM PR from IBM

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants