Skip to content

[BUG]: feature_samples uses a "target/" folder for schemas which is ignored by DABs #45

@Ben-Dowling

Description

@Ben-Dowling

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

In samples/bronze_sample/src/dataflows/feature_samples/schemas schemas are split into source and target folders.

Due to the behaviour of DABs, folders named "target" are not deployed to the workspace, resulting in missing schemas on sample pipeline runs.

Expected Behavior

All schema files are deployed to the workspace and are available at runtime for the sample pipeline to run successfully.

Steps To Reproduce

  1. Clone or fork the repo.
  2. Run the samples/deploy.sh script into a clean workspace.
  3. Check the deployed workspace files — the Workspace/.../.bundle/bronze_sample/files/src/dataflows/feature_samples/schemas contains only a "source" folder no "target" despite both being in the source code.
  4. Run Lakeflow Framework - Run 1 - Load and Schema Initialization feature_samples tasks fail.

Channel

CURRENT

Relevant log output

# Example: on feature_samples_pipeline_data_quality run (triggered by Lakeflow Framework - Run 1 - Load and Schema Initialization)
PYTHON.VALUE_ERROR
Path does not exist: /Workspace/.../.bundle/bronze_sample/dev/files/src/dataflows/feature_samples/schemas/target/customer_address_schema.json

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions