Skip to content

get_relation always returns None with s3tables #620

@dbeachnau-allcloud

Description

@dbeachnau-allcloud

Describe the bug

Projects using s3tables are not able to use incremental materialization, due to the file_format argument not being passed into the adapter.get_relation in the base is_incremental macro.

Steps To Reproduce

Run an incremental model while using s3tables as the specified file_format. The model config could look something like

{{ config(
    materialized='incremental',
    incremental_strategy='append',
    file_format='s3tables'
) }}

Expected behavior

is_incremental() should return True under the right conditions. Currently, it will always return False as calls to adapter.get_relation will always return False when using s3tables as a file format.

System information

The output of dbt --version:

1.10.11

The operating system you're using: MacOS

The output of python --version: Python 3.13.5

Additional context

If I override the is_incremental macro with the following code, I am able to get the correct behavior:

{% macro is_incremental() %}
    {#-- do not run introspective queries in parsing #}
    {% if not execute %}
        {{ return(False) }}
    {% else %}
        {% set relation = adapter.get_relation(this.database, this.schema, this.table, config.get('file_format')) %}
        {{ return(relation is not none
                  and relation.type == 'table'
                  and model.config.materialized == 'incremental'
                  and not should_full_refresh()) }}
    {% endif %}
{% endmacro %}

the only line that was changed was:

{% set relation = adapter.get_relation(this.database, this.schema, this.table, config.get('file_format')) %}

to pass in the file format argument.

If the file_format is not passed in as "s3tables", then the correct table retrieval is never called.

def get_relation(self, database, schema, identifier, file_format=None):
session, client = self.get_connection()
if not identifier:
logger.debug(f"get_relation returns None for schema : {schema} as identifier is not set")
return None
if file_format == 's3tables':
# Use S3 Tables catalog ID for get_table call
import os
s3_tables_bucket = os.getenv('DBT_S3_TABLES_BUCKET')
if s3_tables_bucket:
schema_stripped = self._strip_catalog_from_schema(schema)
try:
response = client.get_table(
CatalogId=s3_tables_bucket,
DatabaseName=schema_stripped,
Name=identifier
)
# Create relation for S3 Tables
computed_schema = self.__compute_schema_based_on_type(schema=schema_stripped, identifier=identifier)
return self.Relation.create(
database=computed_schema,
schema=computed_schema,
identifier=identifier,
type='table'
)
except Exception as e:
return None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions