Add config to specify metastore catalog name by AnuragKDwivedi · Pull Request #24235 · prestodb/presto

AnuragKDwivedi · 2024-12-10T16:02:16Z

Description

This PR introduces a new configuration that can be applied to Hive, Hudi, Delta, and Iceberg catalog properties. The configuration enables the catalog name to be passed to the metastore, significantly enhancing the metastore's capabilities for managing and organizing schemas and tables based on the catalog name.

By passing the catalog name, the metastore can now support unique schema creation under different catalogs, as it already recognizes the combination of catalog and schema as unique. Additionally, this change allows the metastore to filter schemas at the metastore layer itself, making schema management more efficient.

Motivation and Context

Previously, due to the absence of the catalog name in metastore interactions, all schemas were created under the default "hive" catalog. This limitation made it impossible for users to filter or retrieve schemas associated with a specific catalog. The metastore lacked the ability to distinguish between schemas created under different catalogs.

With this update:

Schemas can now be managed and organized by catalog, leveraging the metastore's existing support for catalog-schema uniqueness.
Users can create schemas with the same name under different catalogs, enabling better schema organization and reducing naming conflicts.
Schema filtering at the metastore layer becomes possible, providing more accurate and efficient responses to schema queries.
This change addresses a long-standing limitation and significantly improves schema management in environments using Hive, Hudi, Delta, and Iceberg catalogs.

Example:
A user needs to connect to two different metastores that both have the same catalog name (foo) already registered. This can be done by creating two separate properties files in Presto. In each file, set hive.metastore.catalog.name to foo and specify different hive.metastore.uri values for each metastore. In this case, the user can create two properties files with names like foo-a-metastore.properties and foo-b-metastore.properties, and set the catalog name to foo within both files.

Fixes: #22895

Impact

NA

Test Plan

CI passed

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

== RELEASE NOTES ==

General Changes
* Add configuration property ``hive.metastore.catalog.name`` to pass catalog names to the metastore, enabling catalog-based schema management and filtering.

prestodb-ci · 2024-12-13T15:18:12Z

Saved that user @AnuragKDwivedi is from IBM

steveburnett · 2025-01-29T14:38:08Z

Consider adding documentation for this configuration property. Perhaps in https://github.com/prestodb/presto/blob/master/presto-docs/src/main/sphinx/admin/properties.rst.

steveburnett · 2025-02-03T15:04:18Z

New release note guidelines as of last week: PR #24354 automatically adds links to this PR to the release notes. Please remove the manual PR link in the following format from the release note entries for this PR.

:pr:`12345`

I have updated the Release Notes Guidelines to remove the examples of manually adding the PR link.

ZacBlanco

Just a few things on my first pass. Please take a look a all your usages of Optional and make sure there are no Optional.of[Nullable](Optional::get) occurrences. I found two while skimming

steveburnett · 2025-02-11T15:12:31Z

Thanks for the release note! To improve the release note entry, please include the name of the configuration property.

steveburnett

LGTM! (doc)

Pull branch, local doc build, looks good. Thanks for the doc!

ZacBlanco · 2025-02-28T18:50:23Z

    private boolean readNullMaskedParquetEncryptedValueEnabled;
    private boolean useParquetColumnNames;
    private boolean zstdJniDecompressionEnabled;
+    private String catalogName;


I feel like a sane default would be the name of the presto catalog. Would it be possible to set that here?

If we set a default value here, Presto will start sending the catalog name even when the configuration is not explicitly defined in the properties. This could cause issues with different metastores that do not support catalog names as input. The intention was to keep this configurable and only pass it when the metastore can handle it.

Could you share the advantages you see in adding a default value here?

The advantage is that users would not have to specify it in the first place when using HMS. Consider two hive catalog configured through hive1.properties and hive2.properties. If they were configured to use the same HMS, then inside the config files then in the catalog configuration you would need to set this property as hive.metastore.catalog.name=hive1, etc. If presto already has the catalog name available, it would seem to make sense to default the catalog name to the configured presto catalog name

I do however understand the desire for passing the catalog name to be optional for the case where a metastore doesn't support the "catalog" feature. I'm not sure what the best course of action is other than adding a boolean flag in addition to this one to support both having a sane default and verifying that the feature is enabled

Won't using a boolean hive.metastore.send-catalog-name=true in the core.config suffice?

linux-foundation-easycla · 2025-04-21T07:23:54Z

The committers listed above are authorized under a signed CLA.

✅ login: AnuragKDwivedi / name: Anurag Kumar Dwivedi (23ee6bc)

majetideepak

Isn't the catalog name derived from the catalog filename? Why do we need another config?

majetideepak · 2025-04-23T13:00:06Z

    private boolean readNullMaskedParquetEncryptedValueEnabled;
    private boolean useParquetColumnNames;
    private boolean zstdJniDecompressionEnabled;
+    private String catalogName;


Won't using a boolean hive.metastore.send-catalog-name=true in the core.config suffice?

majetideepak

Thanks for the clarification offline. I included the example you shared in the description.
Please update the commit title and description as well. That is what gets landed.

### Description: This commit introduces a new configuration to allow users to specify a catalog name for the metastore in Presto. By using the `hive.metastore.catalog.name` property, users can manage and connect to multiple metastores with the same catalog name. This enhancement enables a more flexible and configurable approach to handling catalogs and schemas across different metastores and storage backends, especially in complex, multi-region, and multi-tenant environments.

majetideepak · 2025-04-23T14:41:08Z

@ZacBlanco can you re-approve?

ZacBlanco · 2025-04-23T17:41:44Z

prestocpp-macos-build / prestocpp-macos-build-engine (pull_request)
prestocpp-macos-build / prestocpp-macos-build-engine (pull_request) has known issues. Going to merge this change is highly unlikely to have caused more failures in the native build

mohsaka · 2025-05-05T22:07:45Z

@AnuragKDwivedi This PR requires a regeneration of the presto protocol. Opened an issue here
#25049

FYI @aditi-pandit @ZacBlanco @majetideepak

AnuragKDwivedi force-pushed the catalog-name-to-metastore branch from 00027a7 to e72d7ff Compare December 12, 2024 05:28

tdcmeehan added the from:IBM PR from IBM label Dec 13, 2024

prestodb-ci requested review from a team, Dilli-Babu-Godari and ShahimSharafudeen and removed request for a team December 13, 2024 15:18

AnuragKDwivedi mentioned this pull request Dec 27, 2024

Pass catalog name in create schema request #22896

Closed

6 tasks

AnuragKDwivedi force-pushed the catalog-name-to-metastore branch from 9b9b2d9 to c97e5bf Compare February 3, 2025 10:41

AnuragKDwivedi marked this pull request as ready for review February 3, 2025 12:29

AnuragKDwivedi requested review from a team, 7c00, ZacBlanco, elharo, hantangwangd, steveburnett and vinothchandar as code owners February 3, 2025 12:29

AnuragKDwivedi requested a review from presto-oss February 3, 2025 12:29

AnuragKDwivedi marked this pull request as draft February 3, 2025 12:30

AnuragKDwivedi marked this pull request as ready for review February 4, 2025 05:43

ZacBlanco reviewed Feb 11, 2025

View reviewed changes

AnuragKDwivedi requested a review from ZacBlanco February 26, 2025 07:50

AnuragKDwivedi force-pushed the catalog-name-to-metastore branch from cb29758 to ab9a9b9 Compare February 26, 2025 11:05

steveburnett previously approved these changes Feb 26, 2025

View reviewed changes

ZacBlanco reviewed Mar 3, 2025

View reviewed changes

AnuragKDwivedi dismissed steveburnett’s stale review via b27555d March 6, 2025 13:34

AnuragKDwivedi requested a review from vivek-bharathan as a code owner April 21, 2025 07:23

AnuragKDwivedi force-pushed the catalog-name-to-metastore branch 7 times, most recently from 79d55bd to 739f218 Compare April 22, 2025 10:08

ZacBlanco previously approved these changes Apr 22, 2025

View reviewed changes

majetideepak reviewed Apr 23, 2025

View reviewed changes

majetideepak changed the title ~~Added new configuration to pass catalog name to metastore when creati…~~ Add config to specify metastore catalog name Apr 23, 2025

majetideepak requested changes Apr 23, 2025

View reviewed changes

AnuragKDwivedi dismissed ZacBlanco’s stale review via 531645b April 23, 2025 14:28

AnuragKDwivedi force-pushed the catalog-name-to-metastore branch from 739f218 to 531645b Compare April 23, 2025 14:28

AnuragKDwivedi requested a review from majetideepak April 23, 2025 14:29

majetideepak previously approved these changes Apr 23, 2025

View reviewed changes

AnuragKDwivedi dismissed majetideepak’s stale review via 23ee6bc April 23, 2025 14:39

majetideepak approved these changes Apr 23, 2025

View reviewed changes

ZacBlanco approved these changes Apr 23, 2025

View reviewed changes

This was referenced May 2, 2025

[refactor]Extract WindowNode.Specification as a separate class #25033

Merged

[native]Presto Protocol needs to be re-generated after addition of metastore catalog name #25049

Open

ZacBlanco mentioned this pull request May 29, 2025

Add release notes for 0.293 #25222

Merged

21 tasks

This was referenced May 30, 2025

Add release notes for 0.293 unix280/presto#31

Closed

Add release notes for 0.293 unix280/presto#32

Closed

Conversation

AnuragKDwivedi commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Uh oh!

prestodb-ci commented Dec 13, 2024

Uh oh!

steveburnett commented Jan 29, 2025

Uh oh!

steveburnett commented Feb 3, 2025

Uh oh!

ZacBlanco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveburnett commented Feb 11, 2025

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

ZacBlanco Feb 28, 2025

Choose a reason for hiding this comment

Uh oh!

AnuragKDwivedi Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

ZacBlanco Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

majetideepak Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

linux-foundation-easycla Bot commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

majetideepak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

majetideepak Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

majetideepak left a comment

Choose a reason for hiding this comment

Uh oh!

majetideepak commented Apr 23, 2025

Uh oh!

ZacBlanco commented Apr 23, 2025

Uh oh!

mohsaka commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

AnuragKDwivedi commented Dec 10, 2024 •

edited

Loading

linux-foundation-easycla Bot commented Apr 21, 2025 •

edited

Loading