Summary
bid_table_gen in src/nemosis/date_generators.py:79-91 fails to fetch
the previous month's archive when a user queries BIDPEROFFER_D for a
window near the AEMO trading-day start (04:05) on day 1 of a month,
if that month is pre-2021-04 (MMS monthly format).
Result: NEMOSIS silently returns an empty frame when the data exists in
the prior month's archive.
Cause
AEMO partitions BIDPEROFFER_D by trading day, not calendar day. The
May 2018 archive (PUBLIC_DVD_BIDPEROFFER_D_201805010000.zip) contains
INTERVAL_DATETIME from 2018/05/01 04:05:00 to 2018/06/01 04:00:00.
The calendar May 1 00:00–04:00 window lives in the April 2018 archive.
bid_table_gen's pre-2021-04 buffer-back branch only fires when
start_time.day == 1 AND hour == 0 AND minute == 0 (line 85). Any other
start_time on day 1 of a pre-2021-04 month skips the prev-month fetch
entirely.
Reproduction
```python
from nemosis import dynamic_data_compiler
data = dynamic_data_compiler(
start_time="2018/05/01 03:00:00",
end_time="2018/05/01 04:00:00",
table_name="BIDPEROFFER_D",
raw_data_location="/tmp/nemosis_cache",
)
assert data.empty # but the data exists in the April 2018 archive
```
Expected behaviour
For any start_time whose trading-day partition points at the previous
calendar month's archive, bid_table_gen should buffer back to that month.
Scope
- Pre-2021-04 (MMS monthly bidding) only. The post-2021-04 branch
(line 82) unconditionally subtracts a day and is unaffected.
- Affects BIDPEROFFER_D (the only table routed through
bid_table_gen
for monthly-format queries).
Summary
bid_table_geninsrc/nemosis/date_generators.py:79-91fails to fetchthe previous month's archive when a user queries BIDPEROFFER_D for a
window near the AEMO trading-day start (04:05) on day 1 of a month,
if that month is pre-2021-04 (MMS monthly format).
Result: NEMOSIS silently returns an empty frame when the data exists in
the prior month's archive.
Cause
AEMO partitions BIDPEROFFER_D by trading day, not calendar day. The
May 2018 archive (
PUBLIC_DVD_BIDPEROFFER_D_201805010000.zip) containsINTERVAL_DATETIMEfrom2018/05/01 04:05:00to2018/06/01 04:00:00.The calendar May 1 00:00–04:00 window lives in the April 2018 archive.
bid_table_gen's pre-2021-04 buffer-back branch only fires whenstart_time.day == 1 AND hour == 0 AND minute == 0(line 85). Any otherstart_timeon day 1 of a pre-2021-04 month skips the prev-month fetchentirely.
Reproduction
```python
from nemosis import dynamic_data_compiler
data = dynamic_data_compiler(
start_time="2018/05/01 03:00:00",
end_time="2018/05/01 04:00:00",
table_name="BIDPEROFFER_D",
raw_data_location="/tmp/nemosis_cache",
)
assert data.empty # but the data exists in the April 2018 archive
```
Expected behaviour
For any
start_timewhose trading-day partition points at the previouscalendar month's archive,
bid_table_genshould buffer back to that month.Scope
(line 82) unconditionally subtracts a day and is unaffected.
bid_table_genfor monthly-format queries).