Agenda
- RNG updates (RSG)
- Transport for New South Wales Agency Update (TfNSW)
- Transport for New South Wales Agency Sharrow Implementation (TfNSW)
Notes
Agenda Adjustments
The original agenda included questions for Joe Flood on the Application and Analysis Guide. Given time constraints, Joe Castiglione proposed deferring those questions to a smaller follow-up meeting with interested parties. The revised agenda covered two topics: (1) an EET random number generation update from RSG, and (2) a TfNSW auto-calibration tool presentation.
EET Random Number Generation Update (RSG)
David Hensle and Sijia Wang co-presented the latest findings on random number generation (RNG) for ActivitySim's Explicit Error Terms (EET).
RSG Presentation
WSP Presentation
Background
EET requires drawing a random number per alternative (not just one draw per chooser as in Monte Carlo), making the choice of RNG approach a potentially meaningful performance consideration.
RNG Strategy Benchmarking
Three RNG strategies were assessed:
- Legacy (simple channel): NumPy default random state; slow due to reseeding overhead with large alternative sets.
- Faster channel: Stateless hash-based approach storing raw bit generator states per row, generating draws in bulk. Testing on SANDAG's disaggregate accessibility model showed approximately 35% runtime reduction vs. legacy EET, though still roughly 2x slower than a no-EET baseline.
- Random number caching (I/O approach): Writes generated random numbers to disk and loads them on subsequent runs. Effective when many scenario runs share the same chooser population. Demonstrated significant load-time savings (e.g., ~800 seconds to generate vs. ~49 seconds to load for a Poisson-sampling run). Key trade-off: cache files for a production-scale run (500K workers × 6,000 TAZs) reach approximately 23 GB — and real-world problem sizes may be substantially larger.
Poisson Sampling Results
With Poisson sampling enabled, the runtime gap between EET and non-EET runs becomes minimal. David reported that, across most model components, no meaningful runtime difference was observed between no-EET, EET with Poisson, and faster EET with Poisson configurations.
Additional Optimization
David reported that a skim lookup optimization (replacing slow dictionary mapping with bulk NumPy operations) yielded approximately 45% reduction in skim lookup time — a benefit independent of the RNG work.
Discussion and Direction
The group discussed whether to continue RNG optimization or move forward with Poisson sampling as the default EET sampling method. Joel Freedman raised approximate log calculations as a possible further speedup. David noted that with Poisson sampling, RNG is no longer a dominant share of runtime, reducing the marginal value of further RNG optimization.
Decision: Adopt Poisson sampling as the EET default and deprioritize further RNG optimization for now. RNG work to be documented for future reference. Joe Castiglione noted interest in a broader and detailed performance profiling effort across all ActivitySim components.
Action: Joe Castiglione to follow up with Joe Flood and others to schedule a smaller meeting on Application and Analysis Guide questions.
TfNSW Auto-Calibration Tool: FineTune
Amarin (TfNSW) presented FineTune, an off-model auto-calibration package for ActivitySim developed at Transport for NSW.
Overview
FineTune is an external wrapper that runs ActivitySim iteratively, adjusting model coefficients to minimize the difference between simulated outputs and calibration targets. Key design choices:
- Configuration via a single YAML file — specifies ActivitySim settings, model sequence, output tables, parameter bounds, targets, and optimizer settings.
- Off-model architecture — no modifications to ActivitySim source code; runs ActivitySim as a subprocess.
- Optimizers supported: Log ratio (sequential, model-by-model) and SPSA (simultaneous multi-model calibration). SPSA is more expensive and slower to converge but allows holistic optimization across model components.
- Real-time dashboard showing convergence plots, residual trends, and parameter trajectories by iteration.
Performance Example
A work-from-home model calibration using 100,000 households converged in 5 iterations (~6 minutes total). TfNSW is currently applying it jointly to work-from-home and telecommute models using the log ratio optimizer.
Relationship to RSG Auto-Calibration Work
Amarin and David noted significant design overlap between FineTune and the auto-calibration approach RSG is developing for the consortium. Key differences: FineTune uses a unified YAML vs. RSG's proposed spec files; FineTune is external while RSG's approach would be built into ActivitySim. The TfNSW team shared their design learnings with RSG, including reporting functionality that RSG may incorporate.
Next Steps
Amarin indicated additional material to share. Joe Castiglione committed to scheduling a future meeting with adequate time for a fuller TfNSW presentation.
Action: Joe Castiglione to coordinate with Amir and Amarin to schedule a dedicated session for the full TfNSW update.
Action Items Summary
| Action |
Owner |
| Schedule follow-up meeting on Application and Analysis Guide questions |
Joe Castiglione |
| Adopt Poisson sampling as EET default; document RNG work for future reference |
RSG (David/CJ) |
| Scope broader performance profiling task across ActivitySim components |
Joe Castiglione / RSG |
| Schedule dedicated TfNSW presentation slot (FineTune + remaining items) |
Joe Castiglione |
Agenda
Notes
Agenda Adjustments
The original agenda included questions for Joe Flood on the Application and Analysis Guide. Given time constraints, Joe Castiglione proposed deferring those questions to a smaller follow-up meeting with interested parties. The revised agenda covered two topics: (1) an EET random number generation update from RSG, and (2) a TfNSW auto-calibration tool presentation.
EET Random Number Generation Update (RSG)
David Hensle and Sijia Wang co-presented the latest findings on random number generation (RNG) for ActivitySim's Explicit Error Terms (EET).
RSG Presentation
WSP Presentation
Background
EET requires drawing a random number per alternative (not just one draw per chooser as in Monte Carlo), making the choice of RNG approach a potentially meaningful performance consideration.
RNG Strategy Benchmarking
Three RNG strategies were assessed:
Poisson Sampling Results
With Poisson sampling enabled, the runtime gap between EET and non-EET runs becomes minimal. David reported that, across most model components, no meaningful runtime difference was observed between no-EET, EET with Poisson, and faster EET with Poisson configurations.
Additional Optimization
David reported that a skim lookup optimization (replacing slow dictionary mapping with bulk NumPy operations) yielded approximately 45% reduction in skim lookup time — a benefit independent of the RNG work.
Discussion and Direction
The group discussed whether to continue RNG optimization or move forward with Poisson sampling as the default EET sampling method. Joel Freedman raised approximate log calculations as a possible further speedup. David noted that with Poisson sampling, RNG is no longer a dominant share of runtime, reducing the marginal value of further RNG optimization.
Decision: Adopt Poisson sampling as the EET default and deprioritize further RNG optimization for now. RNG work to be documented for future reference. Joe Castiglione noted interest in a broader and detailed performance profiling effort across all ActivitySim components.
Action: Joe Castiglione to follow up with Joe Flood and others to schedule a smaller meeting on Application and Analysis Guide questions.
TfNSW Auto-Calibration Tool: FineTune
Amarin (TfNSW) presented FineTune, an off-model auto-calibration package for ActivitySim developed at Transport for NSW.
Overview
FineTune is an external wrapper that runs ActivitySim iteratively, adjusting model coefficients to minimize the difference between simulated outputs and calibration targets. Key design choices:
Performance Example
A work-from-home model calibration using 100,000 households converged in 5 iterations (~6 minutes total). TfNSW is currently applying it jointly to work-from-home and telecommute models using the log ratio optimizer.
Relationship to RSG Auto-Calibration Work
Amarin and David noted significant design overlap between FineTune and the auto-calibration approach RSG is developing for the consortium. Key differences: FineTune uses a unified YAML vs. RSG's proposed spec files; FineTune is external while RSG's approach would be built into ActivitySim. The TfNSW team shared their design learnings with RSG, including reporting functionality that RSG may incorporate.
Next Steps
Amarin indicated additional material to share. Joe Castiglione committed to scheduling a future meeting with adequate time for a fuller TfNSW presentation.
Action: Joe Castiglione to coordinate with Amir and Amarin to schedule a dedicated session for the full TfNSW update.
Action Items Summary