[codex] Add structured Fama/French data provider

README.md

@@ -10,6 +10,7 @@ Agent Market Data Terminal (aMDT) is a lightweight LLM skill built to give both
 - BLS public v1
 - World Bank
 - OECD
+- Kenneth French Data Library via direct Fama/French CSV zips
 - FRED
 - VIXCentral
 - CBOE
@@ -20,7 +21,7 @@ Agent Market Data Terminal (aMDT) is a lightweight LLM skill built to give both
 - Multpl
 - Yahoo Finance via direct `yfinance`
 
-The skill is native-first. It prefers direct machine-readable provider sources such as BEA public files, Treasury FiscalData JSON, Treasury rate feeds, BLS v1 JSON, World Bank indicator JSON, OECD SDMX-JSON, FRED CSV, Macrotrends JSON, Westmetall chart XML, linked EIA XLS workbooks for history pages, VIXCentral XHR-style routes, CBOE CSV endpoints, `vix_utils`' CBOE-backed history loaders, and Multpl's inline chart payload.
+The skill is native-first. It prefers direct machine-readable provider sources such as BEA public files, Treasury FiscalData JSON, Treasury rate feeds, BLS v1 JSON, World Bank indicator JSON, OECD SDMX-JSON, Kenneth French `_CSV.zip` feeds, FRED CSV, Macrotrends JSON, Westmetall chart XML, linked EIA XLS workbooks for history pages, VIXCentral XHR-style routes, CBOE CSV endpoints, `vix_utils`' CBOE-backed history loaders, and Multpl's inline chart payload.
 
 ## Install
 
@@ -45,6 +46,7 @@ After installation, Codex can use the skill as `$agent-market-data-terminal`.
 - VIXCentral no longer depends on Playwright. The helper uses direct requests first, then a homepage-primed `requests.Session()`, then optional `curl_cffi` impersonation if needed.
 - BEA is implemented as a no-auth file/catalog source. It uses direct public files, BEA release pages, and `https://apps.bea.gov/Data.json` instead of the key-based API.
 - Treasury yield-curve and bill-rate data come from Treasury's official TextView-linked CSV/XML feeds, not FiscalData.
+- Fama/French support is structured-feed only. The helper accepts direct Kenneth French `_CSV.zip` feeds and direct archive zip URLs, and rejects HTML page scraping.
 - Westmetall chart parsing is XML-first. Raw chart fetches remain XML-only, while parsed chart mode can fall back to the daily HTML table if the XML endpoint is temporarily unavailable.
 - `vix_utils` is exposed as a separate historical provider and keeps its cache inside a workspace-local directory when used through the bundled helper.
 - The Yahoo path uses direct `yfinance` inside this repo and keeps only the lightweight `history` and `current_snapshot` datasets.

SKILL.md

@@ -1,11 +1,11 @@
 ---
 name: "agent-market-data-terminal"
-description: "Use when tasks require a lightweight, no-auth market-data skill for BEA, Treasury FiscalData, Treasury rates feeds, BLS, World Bank, OECD, FRED, VIXCentral, Macrotrends, Yahoo Finance, Westmetall, EIA DNAV pages, Multpl, CBOE, or dougransom/vix_utils. Prefer BEA public files and `Data.json`, Treasury FiscalData JSON, Treasury TextView-linked XML/CSV feeds, BLS public v1, World Bank indicator JSON, OECD SDMX-JSON, direct FRED CSV URLs, Macrotrends `/economic-data/{page_id}/{chart_frequency}` JSON, Westmetall chart XML from `/api/marketdata/{lang}/{field}/`, EIA linked XLS/history pages, Multpl inline `pi` chart payloads and table views, VIXCentral term-structure routes with lightweight session-based retries, direct and archive-discovered CBOE CSV endpoints, direct `yfinance` for Yahoo history and current snapshots, and `vix_utils` for historical VIX term structure."
+description: "Use when tasks require a lightweight, no-auth market-data skill for BEA, Treasury FiscalData, Treasury rates feeds, BLS, World Bank, OECD, Kenneth French Data Library, FRED, VIXCentral, Macrotrends, Yahoo Finance, Westmetall, EIA DNAV pages, Multpl, CBOE, or dougransom/vix_utils. Prefer BEA public files and `Data.json`, Treasury FiscalData JSON, Treasury TextView-linked XML/CSV feeds, BLS public v1, World Bank indicator JSON, OECD SDMX-JSON, direct Kenneth French `_CSV.zip` feeds, direct FRED CSV URLs, Macrotrends `/economic-data/{page_id}/{chart_frequency}` JSON, Westmetall chart XML from `/api/marketdata/{lang}/{field}/`, EIA linked XLS/history pages, Multpl inline `pi` chart payloads and table views, VIXCentral term-structure routes with lightweight session-based retries, direct and archive-discovered CBOE CSV endpoints, direct `yfinance` for Yahoo history and current snapshots, and `vix_utils` for historical VIX term structure."
 ---
 
 # Agent Market Data Terminal (aMDT)
 
-Use this skill when a task needs lightweight, no-auth raw downloads, parsed tables, or provider-aware analysis from BEA, Treasury FiscalData, Treasury rates feeds, BLS, World Bank, OECD, FRED, VIXCentral, Macrotrends, Yahoo Finance, Westmetall, EIA DNAV pages, Multpl, CBOE, or `vix_utils`.
+Use this skill when a task needs lightweight, no-auth raw downloads, parsed tables, or provider-aware analysis from BEA, Treasury FiscalData, Treasury rates feeds, BLS, World Bank, OECD, Kenneth French Data Library, FRED, VIXCentral, Macrotrends, Yahoo Finance, Westmetall, EIA DNAV pages, Multpl, CBOE, or `vix_utils`.
 
 ## Workflow
 
@@ -35,6 +35,9 @@ Use this skill when a task needs lightweight, no-auth raw downloads, parsed tabl
 - OECD
   Open `references/oecd.md`, then use `scripts/fetch_oecd.py`.
   Use this for SDMX-JSON datasets and keep series/observation dimensions provider-native.
+- Fama/French
+  Open `references/famafrench.md`, then use `scripts/fetch_famafrench.py`.
+  Use this for Kenneth French Data Library factors and portfolio datasets available as direct `_CSV.zip` feeds.
 - FRED
   Open `references/fred.md`, then use `scripts/fetch_fred.py`.
 - Macrotrends
@@ -72,6 +75,7 @@ python .\scripts\fetch_treasury_rates.py --dataset daily_treasury_yield_curve --
 python .\scripts\fetch_bls.py --series-id CUSR0000SA0 --series-id LNS14000000 --start-year 2020 --end-year 2024 --format parsed --output .\outputs\bls_2020_2024.json
 python .\scripts\fetch_worldbank.py --country US --indicator NY.GDP.MKTP.CD --date 2000:2024 --format parsed --output .\outputs\worldbank_us_gdp.json
 python .\scripts\fetch_oecd.py --dataset MEI_CLI --series-key LOLITONO.USA.M --start-time 2000-01 --end-time 2001-12 --format parsed --output .\outputs\oecd_mei_cli.json
+python .\scripts\fetch_famafrench.py --feed-id F-F_Research_Data_Factors_CSV --format parsed --output .\outputs\famafrench_ff3.json
 python .\scripts\fetch_fred.py --series-id GS10 --start 2020-01-01 --end 2026-04-01 --output .\outputs\gs10.csv
 python .\scripts\fetch_macrotrends.py --page-url "https://www.macrotrends.net/datasets/1476/copper-prices-historical-chart-data" --chart-frequency D --output .\outputs\macrotrends_copper_daily.json
 python .\scripts\fetch_vixcentral.py --mode historical --date 2026-04-02 --output .\outputs\vixcentral_2026-04-02.json
@@ -93,6 +97,7 @@ python .\scripts\fetch_yfinance.py --dataset current_snapshot --ticker ^VIX --ti
 - Open `references/bls.md` for BLS public v1 request shapes.
 - Open `references/worldbank.md` for indicator endpoint behavior and pagination notes.
 - Open `references/oecd.md` for SDMX-JSON route structure and flattening expectations.
+- Open `references/famafrench.md` for direct Kenneth French `_CSV.zip` feed ids, archive-url handling, and section parsing expectations.
 - Open `references/fred.md` for valid FRED parameters and the clean CSV contract.
 - Open `references/macrotrends.md` for page-id extraction and valid `chart_frequency` codes.
 - Open `references/vixcentral.md` for route behavior, array shapes, and the lightweight retry order.
@@ -111,6 +116,7 @@ python .\scripts\fetch_yfinance.py --dataset current_snapshot --ticker ^VIX --ti
 - Use BLS public v1 only; v2 registration features are out of scope here.
 - World Bank raw mode should keep the native two-element JSON page response. Parsed mode may paginate to complete the requested panel.
 - For OECD, flatten SDMX dimensions and attributes into rows without inventing a cross-source schema.
+- For Kenneth French Data Library, use only direct `_CSV.zip` feeds or direct archive `_CSV.zip` URLs. Reject `_TXT.zip` files and HTML pages.
 - Do not scrape visible HTML tables from Macrotrends when the JSON endpoint is available.
 - Prefer Westmetall `/api/marketdata/{lang}/{field}/` chart XML over HTML tables when the task is a chart-series request.
 - For EIA DNAV pages, prefer linked XLS files for raw downloads and for history parsing. Use HTML only for summary parsing, metadata extraction, or explicit history fallback.

agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Agent Market Data Terminal (aMDT)"
   short_description: "Lightweight no-auth market and macro data for agents and cloud chatbots"
-  default_prompt: "Use $agent-market-data-terminal to fetch BEA, Treasury FiscalData, Treasury rates, BLS, World Bank, OECD, FRED, VIXCentral, CBOE, vix_utils, Macrotrends, direct Yahoo Finance via yfinance, Westmetall, EIA DNAV, or Multpl data into this workspace."
+  default_prompt: "Use $agent-market-data-terminal to fetch BEA, Treasury FiscalData, Treasury rates, BLS, World Bank, OECD, Kenneth French Data Library, FRED, VIXCentral, CBOE, vix_utils, Macrotrends, direct Yahoo Finance via yfinance, Westmetall, EIA DNAV, or Multpl data into this workspace."

references/famafrench.md

@@ -0,0 +1,53 @@
+# Fama/French via Kenneth French Data Library
+
+Use only direct structured Kenneth French CSV zip feeds.
+
+## Current-feed pattern
+
+- Base pattern:
+  - `https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/{feed_id}.zip`
+- `feed_id` must be the exact CSV zip stem ending in `_CSV`, for example:
+  - `F-F_Research_Data_Factors_CSV`
+  - `F-F_Research_Data_Factors_daily_CSV`
+  - `F-F_Research_Data_5_Factors_2x3_CSV`
+  - `6_Portfolios_2x3_CSV`
+  - `Developed_3_Factors_CSV`
+
+## Archive support
+
+- Archive access is allowed only through a direct archive `_CSV.zip` URL, for example:
+  - `https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/Data_Library/Historical_Archives/08%202025%20Update/ftp/F-F_Research_Data_Factors_CSV.zip`
+- Do not use archive HTML pages at runtime.
+
+## Native payload
+
+- The provider returns a zip file containing a CSV member.
+- The embedded CSV commonly contains:
+  - preamble lines describing the data source and methodology
+  - one or more blank-line-delimited table sections
+  - section titles such as `Average Value Weighted Returns -- Monthly`
+  - blank-first-column headers like `,Mkt-RF,SMB,HML,RF`
+
+## Parsed output contract
+
+- `provider`
+- `source_url`
+- `feed_id`
+- `zip_members`
+- `csv_member`
+- `preamble`
+- `sections`
+
+Each section keeps:
+
+- `title`
+- `columns`
+- `rows`
+
+## Guidance
+
+- Raw mode should preserve the native zip payload unchanged.
+- Parsed mode should read the embedded CSV and preserve the original period token as a string in `period`.
+- Convert provider missing sentinels like `-99.99` and `-999` to `null`.
+- Reject `_TXT.zip` feeds and reject HTML page URLs.
+- Do not scrape `data_library.html` or any other Kenneth French HTML page during runtime.

references/normalization.md

@@ -16,7 +16,7 @@ Only use this schema when the user explicitly asks to combine providers.
 
 ## Notes
 
-- `source` should be one of `fred`, `macrotrends`, `vixcentral`, `vix_utils`, `cboe`, `yahoo`, `westmetall`, `eia`, `multpl`, `bea`, `fiscaldata`, `treasury_rates`, `bls`, `worldbank`, or `oecd`.
+- `source` should be one of `fred`, `macrotrends`, `vixcentral`, `vix_utils`, `cboe`, `yahoo`, `westmetall`, `eia`, `multpl`, `bea`, `fiscaldata`, `treasury_rates`, `bls`, `worldbank`, `oecd`, or `famafrench`.
 - `dataset` should preserve the provider-specific dataset or route identity.
 - `series_id` is the provider-native id when available, such as a FRED series code or Macrotrends page id.
 - `symbol` is optional and should only be filled when the source exposes a natural ticker or contract symbol.
@@ -48,3 +48,5 @@ Only use this schema when the user explicitly asks to combine providers.
   - `source=worldbank`, `dataset=indicator`, `series_id=NY.GDP.MKTP.CD`
 - OECD
   - `source=oecd`, `dataset=MEI_CLI`, `series_id=LOLITONO.USA.M`
+- Fama/French
+  - `source=famafrench`, `dataset=F-F_Research_Data_Factors_CSV`, `series_id=Mkt-RF`