[BUG]: (YAML) legacy utility function (get_json_from_file) is being used in dlt_pipeline_builder

[Ben-Dowling]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Attempting to use a pipeline_configs/global.yaml file results in an error.

Preferring YAML over JSON we attempted to override "pipeline_bundle_spec_format" using a pipeline_configs/global.yaml.

It resulted in a ValueError following a caught json.JSONDecodeError in load_config_file.

The root cause appears to be the use of utility.get_json_from_file() in src/dlt_pipeline_builder.py. This function is marked as depreciated as 'utility.py'.

Using utility.load_config_file_auto() instead of utility.get_json_from_file() resolves the issue.

Simply using a global.json instead works well, so it's a very minor issue, but since all the hard work to support .yaml seems in place, I thought it worth raising.

P.S. I really really like this framework, apologies for raising a few bugs, they are all minor but I've raised them as I have hit them, hope it helps.
I have not raised a PR for the fix, but I did for a couple of issues in the samples. I noticed that the docs mention a develop branch, but that current PRs are going in against main, so I did the same, hope this was the right place.

Expected Behavior

global.yaml should be valid for use in pipeline_bundle/src/pipeline_configs in order to override global values.

Steps To Reproduce

1. In a pipeline bundle create a pipeline_bundle/src/pipeline_configs/global.yaml

# example global.yaml content
pipeline_bundle_spec_format:
   format: yaml

2. deploy bundle and run pipeline

Channel

CURRENT

Relevant log output

utility.py:142

ValueError: Error loading JSON file '/Workspace/.../.bundle/.../dev/files/src/./pipeline_configs/./global.yaml': Expecting value at line 1, column 1

[haillew]

Thanks for raising this Ben - Glad you're liking the framework! and please keep any feedback coming as we're looking to plug any holes we can find. Yes that's correct on the PRs, we've have moved to trunk based git strategy from git flow so will fix the outdated doco.