Create new employee history start date default#29
Conversation
| | `workday__employee_daily_history` <br> `workday__worker_position_org_daily_history` | Data change | Default `employee_history_start_date` = `2005-03-01` | Default `employee_history_start_date` = `2025-03-01` | Aligns the dbt Core default with the Quickstart default. Users who do not explicitly set `employee_history_start_date` will now receive just over one year of history instead of all available history by default. See how to configure the `employee_history_start_date` to another date in the README. | | ||
|
|
||
| ## Bug Fixes | ||
| - Fixed an issue where `workday__employee_daily_history` and `workday__worker_position_org_daily_history` would fail on Snowflake with a `Date 'None' is not recognized` error. The models now handle null results gracefully by falling back to the `employee_history_start_date` variable and today's date respectively. |
There was a problem hiding this comment.
| - Fixed an issue where `workday__employee_daily_history` and `workday__worker_position_org_daily_history` would fail on Snowflake with a `Date 'None' is not recognized` error. The models now handle null results gracefully by falling back to the `employee_history_start_date` variable and today's date respectively. | |
| - Fixes an issue where `workday__employee_daily_history` and `workday__worker_position_org_daily_history` would fail on Snowflake with a `Date 'None' is not recognized` error. The models now handle null values by falling back to the `employee_history_start_date` variable and the current date respectively. |
There was a problem hiding this comment.
These two end models should need the flags.which..., otherwise the section meant for compilation will never kick in.
There was a problem hiding this comment.
Ahhh good catch. Added.
| {{ dbt_utils.date_spine( | ||
| datepart="day", | ||
| start_date = "greatest(cast('" ~ start_date[0:10] ~ "' as date), cast('" ~ var('employee_history_start_date','2005-03-01') ~ "' as date))", | ||
| start_date = "greatest(cast('" ~ start_date[0:10] ~ "' as date), cast('" ~ var('employee_history_start_date','2025-03-01') ~ "' as date))", |
There was a problem hiding this comment.
It looks like this is the only place the '2025-03-01' default is applied. If we introduce the default earlier, we could likely avoid some of these downstream cleaning steps. Or is there a reason we only want to bring it in at this stage?
Also, is it that we are expecting None during compilation, or are there cases where min and max dates truly wouldn’t exist? flags.which should address the compilation scenario.
There was a problem hiding this comment.
- Re: '2025-03-01'
Because it's a history model, we want to maintain all records at the staging layer to avoid data loss. We apply it here since we are only looking to bound the spine, since the daily history is most at risk of row explosion.
We don't apply this upstream because there's a risk of records being cut off. For example, if there's a record with a _fivetran_start on 2025-01-01 and a _fivetran_end after 2025-06-01, that record wouldn't be captured in a staging filter, and the employee history would be empty from March to June.
- Re: None
I believe this was to avoid empty source table errors when both the min/max calls would return null and thus None in Jinja.
There was a problem hiding this comment.
same comments as workday__employee_daily_history.
…s://github.com/fivetran/dbt_workday into bugfix/employee-history-new-start-date-default
fivetran-catfritz
left a comment
There was a problem hiding this comment.
Thanks for the responses. This is good for pre-release!
…gfix/employee-history-new-start-date-default
c8aca9d
into
schema-change/deprecate-support-for-old-tables
* Schema Change: Deprecate support for oldtables * Create new employee history start date default (#29) * Create new employee history start date default * Identifier fix for employee history start date * Generate dbt docs via GitHub Actions * PR review * spine refactor * Generate dbt docs via GitHub Actions * Generate dbt docs via GitHub Actions --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Final checks * schema & changelog * Final touchups * Generate dbt docs via GitHub Actions * data type * schema * CHANGELOG reversion/PR review * Generate dbt docs via GitHub Actions --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
PR Overview
Package version introduced in this PR:
This PR addresses the following Issue/Feature(s):
Summary of changes:
Submission Checklist
Changelog