Skip to content

bid_table_gen returns empty frame for pre-2021-04 queries near trading-day start #70

@nick-gorman

Description

@nick-gorman

Summary

bid_table_gen in src/nemosis/date_generators.py:79-91 fails to fetch
the previous month's archive when a user queries BIDPEROFFER_D for a
window near the AEMO trading-day start (04:05) on day 1 of a month,
if that month is pre-2021-04 (MMS monthly format).

Result: NEMOSIS silently returns an empty frame when the data exists in
the prior month's archive.

Cause

AEMO partitions BIDPEROFFER_D by trading day, not calendar day. The
May 2018 archive (PUBLIC_DVD_BIDPEROFFER_D_201805010000.zip) contains
INTERVAL_DATETIME from 2018/05/01 04:05:00 to 2018/06/01 04:00:00.
The calendar May 1 00:00–04:00 window lives in the April 2018 archive.

bid_table_gen's pre-2021-04 buffer-back branch only fires when
start_time.day == 1 AND hour == 0 AND minute == 0 (line 85). Any other
start_time on day 1 of a pre-2021-04 month skips the prev-month fetch
entirely.

Reproduction

```python
from nemosis import dynamic_data_compiler
data = dynamic_data_compiler(
start_time="2018/05/01 03:00:00",
end_time="2018/05/01 04:00:00",
table_name="BIDPEROFFER_D",
raw_data_location="/tmp/nemosis_cache",
)
assert data.empty # but the data exists in the April 2018 archive
```

Expected behaviour

For any start_time whose trading-day partition points at the previous
calendar month's archive, bid_table_gen should buffer back to that month.

Scope

  • Pre-2021-04 (MMS monthly bidding) only. The post-2021-04 branch
    (line 82) unconditionally subtracts a day and is unaffected.
  • Affects BIDPEROFFER_D (the only table routed through bid_table_gen
    for monthly-format queries).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions