Skip to content

Support Restatements on SCD Type 2 and Use Them When Creating Preview Clone #3642

@eakmanrq

Description

@eakmanrq

Currently SCD Type 2 supports only restating the model from the beginning of time. One could also restate a model at any point in history too but also end with the latest date. This how how it would work:

Lets say I am restating Jan 10th

  • Delete from the table where all valid_from >= Jan 10th
  • Update the table where all valid_to >= Jan 10th

This would reset the table to look like it was last run before the given start date.

Ideally, once this is implemented, when we create a preview clone of a model we would then configure the clone to restate this latest model with the new logic.

Suggestion for resolving that is to update the SCD Type 2 logic to always go through this delete/update logic on each insert. That way in most cases where we are just processing the newest interval it would be no-op (does require a scan though) and when we have to restate it would delete/update as expected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ImprovementImproves existing functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions