Monthly MMS dynamic table fetches fail from 2024-08 onward (`PUBLIC_ARCHIVE#...`) while older `PUBLIC_DVD_*` months still work

[mrvictoru]

Summary

In our environment, nemosis.dynamic_data_compiler() can still fetch monthly MMS tables such as DISPATCHPRICE and DISPATCHLOAD for older months (for example 2020-01), but it fails from 2024-08 onward with NoDataToReturn.

The failure affects at least:

• DISPATCHPRICE
• DISPATCHLOAD

What works

These succeed in a fresh empty cache:

dynamic_data_compiler(
    start_time="2020/01/15 00:00:00",
    end_time="2020/01/16 00:00:00",
    table_name="DISPATCHPRICE",
    raw_data_location="/tmp/test-cache",
    select_columns=["SETTLEMENTDATE"],
)

dynamic_data_compiler(
    start_time="2020/01/15 00:00:00",
    end_time="2020/01/16 00:00:00",
    table_name="DISPATCHLOAD",
    raw_data_location="/tmp/test-cache",
    select_columns=["SETTLEMENTDATE"],
)

What fails

These fail with NoDataToReturn:

dynamic_data_compiler(
    start_time="2024/08/15 00:00:00",
    end_time="2024/08/16 00:00:00",
    table_name="DISPATCHPRICE",
    raw_data_location="/tmp/test-cache",
    select_columns=["SETTLEMENTDATE"],
)

dynamic_data_compiler(
    start_time="2024/08/15 00:00:00",
    end_time="2024/08/16 00:00:00",
    table_name="DISPATCHLOAD",
    raw_data_location="/tmp/test-cache",
    select_columns=["SETTLEMENTDATE"],
)

Recent months such as 2026-04 and 2026-05 also fail the same way.

Observed behavior

The post-2024-07 filename logic produces monthly stubs like:

• PUBLIC_ARCHIVE#DISPATCHPRICE#FILE01#202408010000
• PUBLIC_ARCHIVE#DISPATCHLOAD#FILE01#202408010000

nemosis then attempts URLs of the form:

• https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/2024/MMSDM_2024_08/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202408010000.zip

In our tests, representative older PUBLIC_DVD_* URLs return HTTP 200, while representative post-2024-07 PUBLIC_ARCHIVE#... URLs return HTTP 400.

Expected behavior

Either:

1. the post-2024-07 monthly filename / URL contract should be updated so these months download correctly, or
2. the docs should clarify the new monthly archive location / naming convention if AEMO changed it.

Additional data point from downstream testing:

We verified that the manual local-cache workaround is viable even for the post-2024-07 monthly naming era.

Using synthetic cache files named exactly as NEMOSIS expects, e.g.

• PUBLIC_ARCHIVE#DISPATCHPRICE#FILE01#202408010000.CSV
• PUBLIC_ARCHIVE#DISPATCHLOAD#FILE01#202408010000.CSV

plain nemosis.dynamic_data_compiler() successfully returned rows from the local cache for a 2024-08 window.

Observed results:

• DISPATCHPRICE: 1440 rows
• DISPATCHLOAD: 143712 rows

So FILE01 local monthly cache ingestion works.

One nuance: in default mode, NEMOSIS still probes FILE02 after reading FILE01 for these PUBLIC_ARCHIVE#... monthly stubs. That extra probe is not needed for the local-cache recovery path, but it does not prevent FILE01 data from being returned when present.

This suggests:

1. the post-2024-07 monthly archive fetch path is broken upstream for at least some MMS tables, but
2. manual monthly cache files with the expected PUBLIC_ARCHIVE#...#FILE01#YYYYMM010000.CSV naming still work as a recovery path.

Why this matters

This breaks automated historical cache population for any workflows that rely on post-2024-07 monthly MMS tables, including AEMO replay / preprocessing pipelines built on DISPATCHPRICE and DISPATCHLOAD.

Environment

• Python: 3.12 host repro, also validated in project runtime
• nemosis: installed from pip in our environment
• Failing tables: DISPATCHPRICE, DISPATCHLOAD
• Boundary observed: works for 2020-01, fails from 2024-08 onward

[nick-gorman]

@mrvictoru which NEMOSIS version did you get this bug on?

[nick-gorman]

Confirmed reproducing on current master (post-PR #83). Surfaced this while running smoke tests for an unrelated downloader refactor (#85). Sharing the root-cause investigation in case it helps whoever picks up the fix.

Root cause

The post-2024-07 archives on nemweb are stored on the server's disk with literal %23 characters in the filename — not literal #. You can see this in the directory listing for any post-cutover month:

$ curl -s https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/2024/MMSDM_2024_12/MMSDM_Historical_Data_SQLLoader/DATA/ | grep -o 'PUBLIC[^\"]*DISPATCHPRICE[^\"]*\.zip' | head -1
PUBLIC_ARCHIVE%2523DISPATCHPRICE%2523FILE01%2523202412010000.zip

Reading that back: the link's href contains %2523. %25 is the URL-encoded form of %, so %2523 represents the literal three-character string %23 — which is the actual filename on disk. (Not a #. Not even a URL-decoded %23 representing #. A literal %, 2, 3.)

Meanwhile NEMOSIS's download_unzip_csv does:

def download_unzip_csv(url, down_load_to):
    url = url.replace('#', '%23')
    r = requests.get(url, ...)

So a constructed URL with literal # characters becomes …%23DISPATCHPRICE%23…. nemweb URL-decodes %23 → # and looks for a file with literal # in its name. No such file exists (the file is named with literal %23). 400 Bad Request.

To match the real file, the URL needs to contain %2523 (so that nemweb decodes it to %23 and finds the actual filename). Verified directly against the real server:

import requests
ua = {'User-Agent': '…Chrome/130…'}

# Single %23 encoding (what NEMOSIS sends today)
r = requests.get('…/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202412010000.zip', headers=ua)
# → 400 Bad Request

# Double %2523 encoding (what the file actually needs)
r = requests.get('…/PUBLIC_ARCHIVE%2523DISPATCHPRICE%2523FILE01%2523202412010000.zip', headers=ua)
# → 200 OK, 2,140,354 bytes

Why offline tests don't catch this

The test fixture filenames in tests/fixtures/data/.../MMSDM_2024_*/…/DATA/ are stored on disk with literal # characters:

PUBLIC_ARCHIVE#DISPATCHPRICE#FILE01#202412010000.zip

Python's http.server URL-decodes %23 back to # when looking up local files, so NEMOSIS's …replace('#', '%23') → %23 in URL → server decodes to # → finds the local file → 200 OK. The fixture's name disagrees with how real nemweb stores its files, so the offline path can't surface this.

Fix direction

Whichever fix is picked, the offline fixture filenames and the URL-encoding step need to agree:

• Option A (cleanest): double-encode # → %2523 in download_unzip_csv, and rename the post-2024-07 fixture files from …#…zip to …%23…zip so the mock server hands them out under the same name the real server uses. The local extracted/cached files keep their existing #-in-name convention.
• Option B: construct the filename_stub with %23 already in it for PUBLIC_ARCHIVE format. Means the local feather/parquet files end up named PUBLIC_ARCHIVE%23DISPATCHPRICE%23…feather which is uglier on disk.

I'd lean A.

Where this fits in the queue

I've got an open downloader rework (#85, awaiting CI). This fix is a sibling to it, not part of it — happy to take it on after #85 + the rest of #67's downloader work merges if nobody else picks it up first. Adding to my queue as a follow-up.

[mrvictoru]

looking forward to the fix coming to release! i am using the latest version so 3.8.1