Skip to content

MNSP_PEROFFER cannot be read for periods after ~April 2021 #68

@nick-gorman

Description

@nick-gorman

Summary

dynamic_data_compiler(table_name="MNSP_PEROFFER", ...) fails to return
usable data for any start_time from approximately April 2021 onward,
because AEMO has changed the underlying source twice and NEMOSIS's
defaults.names["MNSP_PEROFFER"] still points at the original (now stale)
archive name.

Symptoms

  • April 2021 → July 2024: the request succeeds at the network layer
    (the PUBLIC_DVD_MNSP_PEROFFER_YYYYMM010000.zip archive still exists)
    but the file's contents are now MNSP_BIDOFFERPERIOD data with
    different column names (TRADINGDATE, OFFERDATETIME instead of
    SETTLEMENTDATE, OFFERDATE). NEMOSIS errors with
    KeyError: 'SETTLEMENTDATE' inside filters.construct_interval_datetime_from_period_id.
  • August 2024 onward: the archive stub is renamed to
    PUBLIC_ARCHIVE#MNSP_BIDOFFERPERIOD#FILE01#YYYYMM010000.zip, so the
    URL NEMOSIS builds (PUBLIC_ARCHIVE#MNSP_PEROFFER#FILE01#...) returns
    HTTP 404.

Reproduction

```python
from nemosis import dynamic_data_compiler
dynamic_data_compiler(
start_time="2021/05/01 00:00:00",
end_time="2021/05/01 01:00:00",
table_name="MNSP_PEROFFER",
raw_data_location="/tmp/nemosis_cache",
)

-> KeyError: 'SETTLEMENTDATE'

dynamic_data_compiler(
start_time="2024/09/01 00:00:00",
end_time="2024/09/01 01:00:00",
table_name="MNSP_PEROFFER",
raw_data_location="/tmp/nemosis_cache",
)

-> HTTPError: 404

```

MNSP_DAYOFFER is unaffected: the day-offer archive kept its
SETTLEMENTDATE / OFFERDATE column names across both transitions.

Suggested fix

Either:

  1. Add a per-era filename + column-name mapping to defaults.py so
    requests after April 2021 hit the renamed file and use the new
    column names; or
  2. Register MNSP_BIDOFFERPERIOD as a separate table and document
    MNSP_PEROFFER as legacy / pre-2021 only.

Notes

Surfaced while building end-to-end fixtures for the test suite refactor;
test coverage is currently capped at the 2018-05 and 2021-02 eras for
this table.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions