-
Notifications
You must be signed in to change notification settings - Fork 88
Description
Describe the bug
Projects using s3tables are not able to use incremental materialization, due to the file_format argument not being passed into the adapter.get_relation in the base is_incremental macro.
Steps To Reproduce
Run an incremental model while using s3tables as the specified file_format. The model config could look something like
{{ config(
materialized='incremental',
incremental_strategy='append',
file_format='s3tables'
) }}
Expected behavior
is_incremental() should return True under the right conditions. Currently, it will always return False as calls to adapter.get_relation will always return False when using s3tables as a file format.
System information
The output of dbt --version:
1.10.11
The operating system you're using: MacOS
The output of python --version: Python 3.13.5
Additional context
If I override the is_incremental macro with the following code, I am able to get the correct behavior:
{% macro is_incremental() %}
{#-- do not run introspective queries in parsing #}
{% if not execute %}
{{ return(False) }}
{% else %}
{% set relation = adapter.get_relation(this.database, this.schema, this.table, config.get('file_format')) %}
{{ return(relation is not none
and relation.type == 'table'
and model.config.materialized == 'incremental'
and not should_full_refresh()) }}
{% endif %}
{% endmacro %}
the only line that was changed was:
{% set relation = adapter.get_relation(this.database, this.schema, this.table, config.get('file_format')) %}
to pass in the file format argument.
If the file_format is not passed in as "s3tables", then the correct table retrieval is never called.
dbt-glue/dbt/adapters/glue/impl.py
Lines 218 to 245 in 79ba9c4
| def get_relation(self, database, schema, identifier, file_format=None): | |
| session, client = self.get_connection() | |
| if not identifier: | |
| logger.debug(f"get_relation returns None for schema : {schema} as identifier is not set") | |
| return None | |
| if file_format == 's3tables': | |
| # Use S3 Tables catalog ID for get_table call | |
| import os | |
| s3_tables_bucket = os.getenv('DBT_S3_TABLES_BUCKET') | |
| if s3_tables_bucket: | |
| schema_stripped = self._strip_catalog_from_schema(schema) | |
| try: | |
| response = client.get_table( | |
| CatalogId=s3_tables_bucket, | |
| DatabaseName=schema_stripped, | |
| Name=identifier | |
| ) | |
| # Create relation for S3 Tables | |
| computed_schema = self.__compute_schema_based_on_type(schema=schema_stripped, identifier=identifier) | |
| return self.Relation.create( | |
| database=computed_schema, | |
| schema=computed_schema, | |
| identifier=identifier, | |
| type='table' | |
| ) | |
| except Exception as e: | |
| return None |