Releases: aws-samples/dbt-glue
Releases · aws-samples/dbt-glue
1.10.15
- Added ability to have specific models use the same glue session separate from the project level session by specifying a "group_session_id" in the model's meta.
- Ensured that when "group_session_id" is set, "enable_session_per_model" is not enabled for that model so that session usage behavior acts appropriately.
- Added testing to capture proper behavior of "group_session_id".
- Upgrade dependencies: dbt-core 1.10.15, moto 5.1.17, black 25.11.0.
1.10.13
- Added ability to use Spark temporary views when using iceberg targets instead of physical tables to store intermediate results, by setting model config use_iceberg_temp_views to 'True'
- Added ability to purge iceberg and s3tables underlying data s3 object during drop table when config purge_dropped_iceberg_data is set to 'True' in model config
- Improved unit testing of drop_relation by separating out glue__get_drop_sql and unit testing the output for each condition
- Improved coverage of unit testing for glue__make_target_relation and glue__create_temporary_view
- Separated glue__make_target_relation into utils/make_target_relation.sql so that we could mock this often called macro, allowing us to better unit test macros that call this one
- Fixed several bugs on S3 tables support
1.10.11
- Fix duplicate table macro issue in published package (GitHub #608)
- Add test to prevent duplicate macro definitions in future releases
- Improve build process to ensure clean package builds
- Fix issue where including unrelated values in
metasection would create new sessions whenglue_session_idis set - Fix NoneType error in meta configuration handling
- Upgrade dependencies: dbt-core 1.10.11, dbt-spark 1.9.3, moto 5.1.12, mypy 1.18.1.
1.10.9
- Allow temporary tables to be created in different schema/database than target model using profile temp-schema variable
- Add experimental Python model support with Iceberg file format (requires AWS Glue 4.0+)
- Add experimental Amazon S3 Tables support with
file_format='s3tables' - Upgrade dependencies: dbt-core 1.10.9, dbt-tests-adapter 1.18.0, moto 5.1.10, freezegun 1.5.5.
- Fix merge strategy with schema changes for Iceberg tables
1.9.4
- Reduce debug logging
- Fix invalid Spark relation type: iceberg_table
- Fix Iceberg snapshot issue of incorrect DROP VIEW calls
- Upgrade dependencies: dbt-core 1.9.4, dbt-spark 1.9.2, moto 5.1.3.
1.9.2
- Correctly handle EntityNotFound when trying to determine session state, setting state to does not exist instead of STOPPED.
- Allow spawning new isolated sessions for the models that require different session configuration.
- Correctly handle EntityNotFound when listing relations.
- Add configuration property to allow spark casting of seed column types.
- Fix the get_columns_in_relation function error when on_schema_change is specified.
- Upgrade dependencies: dbt-core 1.9.2, dbt-spark 1.9.1, dbt-tests-adapter 1.11.0, mypy 1.15.0, moto 5.0.28, and black 25.1.0.
- Fix the temporary view creation issue when using schema_on_change
- Fix error handling and logging
- Fix missing catalog identifier related issues for Iceberg
- Add update_iceberg_ts column with an add_iceberg_timestamp option.
- Add refresh table when missing columns are detected
- Fix ALTER TABLE ADD COLUMN returning None
1.9.0
- Allow to load big seed files
- Migrates the PySpark code for the Iceberg file format at a macro level, making the impl.py file more readable.
- Fixes the get_columns_in_relation function to work for both Iceberg and non-Iceberg tables without hard-coding the catalog name.
- Fixes the get_location table function to work for both Iceberg and non-Iceberg tables on macOS and Windows.
- Adds a helper function to retrieve the Iceberg catalog namespace from the profile.yaml file.
- Adds merge_exclude_columns and incremental_predicates features.
- Drop Python 3.8 support
- Upgrade default Glue version to 5.0
- Upgrade dependencies: dbt-core 1.9.0 and dbt-spark 1.9.0
https://github.com/aws-samples/dbt-glue/blob/main/CHANGELOG.md
1.8.6
- Fix session provisioning timeout and delay handling
- Add write options for Delta format
- Add on_schema_change possibility
- Fix table materialization for Delta models
- Change GlueColumn parent from base Column to SparkColumn
- Fix unraised DbtDatabaseError
- Fix get_columns_in_relation function to stop returning additional partition columns
- Fix null values handling in seeds
- Fix exceptions import for FailedToConnectError and ExecutableError
- Fix the case-sensitive comparison on the seed name
https://github.com/aws-samples/dbt-glue/blob/main/CHANGELOG.md