Summary
dynamic_data_compiler(table_name="MNSP_PEROFFER", ...) fails to return
usable data for any start_time from approximately April 2021 onward,
because AEMO has changed the underlying source twice and NEMOSIS's
defaults.names["MNSP_PEROFFER"] still points at the original (now stale)
archive name.
Symptoms
- April 2021 → July 2024: the request succeeds at the network layer
(the PUBLIC_DVD_MNSP_PEROFFER_YYYYMM010000.zip archive still exists)
but the file's contents are now MNSP_BIDOFFERPERIOD data with
different column names (TRADINGDATE, OFFERDATETIME instead of
SETTLEMENTDATE, OFFERDATE). NEMOSIS errors with
KeyError: 'SETTLEMENTDATE' inside filters.construct_interval_datetime_from_period_id.
- August 2024 onward: the archive stub is renamed to
PUBLIC_ARCHIVE#MNSP_BIDOFFERPERIOD#FILE01#YYYYMM010000.zip, so the
URL NEMOSIS builds (PUBLIC_ARCHIVE#MNSP_PEROFFER#FILE01#...) returns
HTTP 404.
Reproduction
```python
from nemosis import dynamic_data_compiler
dynamic_data_compiler(
start_time="2021/05/01 00:00:00",
end_time="2021/05/01 01:00:00",
table_name="MNSP_PEROFFER",
raw_data_location="/tmp/nemosis_cache",
)
-> KeyError: 'SETTLEMENTDATE'
dynamic_data_compiler(
start_time="2024/09/01 00:00:00",
end_time="2024/09/01 01:00:00",
table_name="MNSP_PEROFFER",
raw_data_location="/tmp/nemosis_cache",
)
-> HTTPError: 404
```
MNSP_DAYOFFER is unaffected: the day-offer archive kept its
SETTLEMENTDATE / OFFERDATE column names across both transitions.
Suggested fix
Either:
- Add a per-era filename + column-name mapping to
defaults.py so
requests after April 2021 hit the renamed file and use the new
column names; or
- Register
MNSP_BIDOFFERPERIOD as a separate table and document
MNSP_PEROFFER as legacy / pre-2021 only.
Notes
Surfaced while building end-to-end fixtures for the test suite refactor;
test coverage is currently capped at the 2018-05 and 2021-02 eras for
this table.
Summary
dynamic_data_compiler(table_name="MNSP_PEROFFER", ...)fails to returnusable data for any
start_timefrom approximately April 2021 onward,because AEMO has changed the underlying source twice and NEMOSIS's
defaults.names["MNSP_PEROFFER"]still points at the original (now stale)archive name.
Symptoms
(the
PUBLIC_DVD_MNSP_PEROFFER_YYYYMM010000.ziparchive still exists)but the file's contents are now
MNSP_BIDOFFERPERIODdata withdifferent column names (
TRADINGDATE,OFFERDATETIMEinstead ofSETTLEMENTDATE,OFFERDATE). NEMOSIS errors withKeyError: 'SETTLEMENTDATE'insidefilters.construct_interval_datetime_from_period_id.PUBLIC_ARCHIVE#MNSP_BIDOFFERPERIOD#FILE01#YYYYMM010000.zip, so theURL NEMOSIS builds (
PUBLIC_ARCHIVE#MNSP_PEROFFER#FILE01#...) returnsHTTP 404.
Reproduction
```python
from nemosis import dynamic_data_compiler
dynamic_data_compiler(
start_time="2021/05/01 00:00:00",
end_time="2021/05/01 01:00:00",
table_name="MNSP_PEROFFER",
raw_data_location="/tmp/nemosis_cache",
)
-> KeyError: 'SETTLEMENTDATE'
dynamic_data_compiler(
start_time="2024/09/01 00:00:00",
end_time="2024/09/01 01:00:00",
table_name="MNSP_PEROFFER",
raw_data_location="/tmp/nemosis_cache",
)
-> HTTPError: 404
```
MNSP_DAYOFFERis unaffected: the day-offer archive kept itsSETTLEMENTDATE/OFFERDATEcolumn names across both transitions.Suggested fix
Either:
defaults.pysorequests after April 2021 hit the renamed file and use the new
column names; or
MNSP_BIDOFFERPERIODas a separate table and documentMNSP_PEROFFERas legacy / pre-2021 only.Notes
Surfaced while building end-to-end fixtures for the test suite refactor;
test coverage is currently capped at the 2018-05 and 2021-02 eras for
this table.