Skip to content

Create new employee history start date default#29

Merged
fivetran-avinash merged 9 commits into
schema-change/deprecate-support-for-old-tablesfrom
bugfix/employee-history-new-start-date-default
Apr 7, 2026
Merged

Create new employee history start date default#29
fivetran-avinash merged 9 commits into
schema-change/deprecate-support-for-old-tablesfrom
bugfix/employee-history-new-start-date-default

Conversation

@fivetran-avinash

@fivetran-avinash fivetran-avinash commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

PR Overview

Package version introduced in this PR:

  • v0.9.0

This PR addresses the following Issue/Feature(s):

  • GA-1020281

Summary of changes:

  • Add workday history start date filter

Submission Checklist

  • Alignment meeting with the reviewer (if needed)
    • Timeline and validation requirements discussed
  • Provide validation details:
    • Validation Steps: Check for unintentional effects (e.g., add/run consistency & integrity tests)
    • Testing Instructions: Confirm the change addresses the issue(s)
    • Focus Areas: Complex logic or queries that need extra attention
  • Merge any relevant open PRs into this PR

Changelog

  • Draft changelog for PR
  • Final changelog for release review

@fivetran-avinash fivetran-avinash self-assigned this Mar 26, 2026
@fivetran-avinash fivetran-avinash marked this pull request as ready for review March 26, 2026 02:11
@fivetran-avinash fivetran-avinash added the docs:ready Triggers the docs generator workflow. label Mar 27, 2026
Comment thread CHANGELOG.md Outdated
| `workday__employee_daily_history` <br> `workday__worker_position_org_daily_history` | Data change | Default `employee_history_start_date` = `2005-03-01` | Default `employee_history_start_date` = `2025-03-01` | Aligns the dbt Core default with the Quickstart default. Users who do not explicitly set `employee_history_start_date` will now receive just over one year of history instead of all available history by default. See how to configure the `employee_history_start_date` to another date in the README. |

## Bug Fixes
- Fixed an issue where `workday__employee_daily_history` and `workday__worker_position_org_daily_history` would fail on Snowflake with a `Date 'None' is not recognized` error. The models now handle null results gracefully by falling back to the `employee_history_start_date` variable and today's date respectively.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Fixed an issue where `workday__employee_daily_history` and `workday__worker_position_org_daily_history` would fail on Snowflake with a `Date 'None' is not recognized` error. The models now handle null results gracefully by falling back to the `employee_history_start_date` variable and today's date respectively.
- Fixes an issue where `workday__employee_daily_history` and `workday__worker_position_org_daily_history` would fail on Snowflake with a `Date 'None' is not recognized` error. The models now handle null values by falling back to the `employee_history_start_date` variable and the current date respectively.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two end models should need the flags.which..., otherwise the section meant for compilation will never kick in.

@fivetran-avinash fivetran-avinash Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh good catch. Added.

{{ dbt_utils.date_spine(
datepart="day",
start_date = "greatest(cast('" ~ start_date[0:10] ~ "' as date), cast('" ~ var('employee_history_start_date','2005-03-01') ~ "' as date))",
start_date = "greatest(cast('" ~ start_date[0:10] ~ "' as date), cast('" ~ var('employee_history_start_date','2025-03-01') ~ "' as date))",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is the only place the '2025-03-01' default is applied. If we introduce the default earlier, we could likely avoid some of these downstream cleaning steps. Or is there a reason we only want to bring it in at this stage?

Also, is it that we are expecting None during compilation, or are there cases where min and max dates truly wouldn’t exist? flags.which should address the compilation scenario.

@fivetran-avinash fivetran-avinash Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Re: '2025-03-01'
    Because it's a history model, we want to maintain all records at the staging layer to avoid data loss. We apply it here since we are only looking to bound the spine, since the daily history is most at risk of row explosion.

We don't apply this upstream because there's a risk of records being cut off. For example, if there's a record with a _fivetran_start on 2025-01-01 and a _fivetran_end after 2025-06-01, that record wouldn't be captured in a staging filter, and the employee history would be empty from March to June.

  1. Re: None
    I believe this was to avoid empty source table errors when both the min/max calls would return null and thus None in Jinja.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as workday__employee_daily_history.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

@fivetran-avinash fivetran-avinash added docs:ready Triggers the docs generator workflow. and removed docs:ready Triggers the docs generator workflow. labels Mar 30, 2026
@fivetran-avinash fivetran-avinash added docs:ready Triggers the docs generator workflow. and removed docs:ready Triggers the docs generator workflow. labels Mar 31, 2026
@fivetran-avinash fivetran-avinash added docs:ready Triggers the docs generator workflow. and removed docs:ready Triggers the docs generator workflow. labels Mar 31, 2026

@fivetran-catfritz fivetran-catfritz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the responses. This is good for pre-release!

@fivetran-avinash fivetran-avinash added the pre-release Triggers the auto-releaser workflow. label Mar 31, 2026
@fivetran-avinash fivetran-avinash changed the base branch from main to schema-change/deprecate-support-for-old-tables April 7, 2026 19:57
…gfix/employee-history-new-start-date-default
@fivetran-avinash fivetran-avinash merged commit c8aca9d into schema-change/deprecate-support-for-old-tables Apr 7, 2026
9 checks passed
fivetran-avinash added a commit that referenced this pull request Apr 14, 2026
* Schema Change: Deprecate support for oldtables

* Create new employee history start date default (#29)

* Create new employee history start date default

* Identifier fix for employee history start date

* Generate dbt docs via GitHub Actions

* PR review

* spine refactor

* Generate dbt docs via GitHub Actions

* Generate dbt docs via GitHub Actions

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Final checks

* schema & changelog

* Final touchups

* Generate dbt docs via GitHub Actions

* data type

* schema

* CHANGELOG reversion/PR review

* Generate dbt docs via GitHub Actions

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs:ready Triggers the docs generator workflow. pre-release Triggers the auto-releaser workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants