Add Met Office Global Deterministic height level dataset

[mo-jeff]

Description

Adding content for 8 collections. Things to note:

• Description.md needs adding a separate file and linking from template.json
• No dataset.yml files included
• item_assets is a static list based on the collection themes

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

n/a

Checklist:

Please delete options that are not relevant.

• I have performed a self-review
• Changelog has been updated
• Documentation has been updated
• Unit tests pass locally (./scripts/test)
• Code is linted and styled (./scripts/format)

[ghidalgo3]

Thanks @mo-jeff! Some thoughts:

• It is not strictly necessary to add a long description.md, but remember that the contents of description.md are rendered to HTML in the Planetary Computer explorer. Here you have a chance to fully describe your dataset to your users in rich detail, including links (see the red box):

• Which values? I'll eventually end up documenting them, just curious...
• A collection's temporal extent can be null on one side, which indicates an open internal. You can reference other continuously updated collections like MODIS, Sentinel2, HLS2, or GOES for an example.
• item_assets at the collection level is an indication to the users of this collection what the expected asset keys are of all STAC items assets in this collection. It should be a relatively small set of asset keys.
• You might want to consider adding the datacube STAC extension too, and documenting the dimensions of all the assets in this collection.

Let us handle the dataset.yml file, that's a workflow specification that you don't have to write.

[mo-jeff]

@ghidalgo3 - updated this PR. To your comments:

• I need to add richer descriptions once we've agreed them internally.
• Asset keys have been updated, with detail similar to ECMWF datasets. More data documentation will be linked to in the description to ensure consistency across platforms.
• Open to using the datacube extension, focused on main content of the collection at the moment.

@gadomski - most of the technical detail is here now I think. Descriptions to follow soon.

[mo-jeff]

@mo-jeff please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Met Office"

[gadomski]

Gave it a quick once-over, no major issues. We'll want to populate the item_assets with more information once that's prepared ... I'm working on that now.

[gadomski]

I'm still not convinced on the value or correctness of putting 1H_timesteps in cube:dimensions. Any reason not to just keep it in the text descriptions?

[mo-jeff]

I'm still not convinced on the value or correctness of putting 1H_timesteps in cube:dimensions. Any reason not to just keep it in the text descriptions?

I thought Temporal Dimensions might help us with this but maybe not.

Happy to keep the Timestep header and content in the collection description and call out in the description for any parameters that don't follow that pattern? For example, there is a precip accumulation that is hourly from T+0 - T+51 (rather than T+54)

[gadomski]

Happy to keep the Timestep header and content in the collection description and call out in the description for any parameters that don't follow that pattern? For example, there is a precip accumulation that is hourly from T+0 - T+51 (rather than T+54)

Yeah, the datacube extension is intended to describe a single file. Since your forecast horizons are split into their own standalone files, it's not a good fit for the forecast horizons IMO. Still a good fit for the other dimensions.

I think a good text description will be super-helpful for users, so 👍🏼

[gadomski]

@mo-jeff just a heads up that I'm going to be adding a few commits on top of your branch to add ingestion code, etc. I'll try my best to not stomp on any of your changes, but letting you know so you aren't surprised if new commits appear and you need to sync your branch before comitting.

[gadomski]

@mo-jeff I've renamed each folder to match the collection id (removed the -level) so take care with any new changes so you don't end up in merge hell.

[gadomski]

@mo-jeff updated the template.json w/ collection-level datacube extension and a few other tweaks. scripts/cleanup.py is the automated process, so if we need to do it again or change anything, should be straightforward.