Python tools for procedurally generating Graph of Events in Space and Time (GEST) specifications and executing them deterministically with the GEST-Engine (mta-sim) — either locally on a single machine or in parallel across many VMware Workstation Pro worker VMs.
This repository contains:
simple_gest_random_generator.py— a procedural random GEST generator that reads the engine's capability registry and produces executable GESTs by construction.main.py— single-story / maintenance CLI for one-off operations (export the engine's capability registry, simulate a stored story, configuration utilities). Also exposes the LLM-based concept / casting / detail generation pipeline (see note below).batch_generate.py— local batch runner that drives the GEST-Engine on the host machine. Useful for small batches, smoke tests, debugging, or any setup without VMware.vmware_orchestrator.py— VMware Workstation Pro orchestrator that clones worker VMs from a master snapshot, distributes generation jobs, monitors worker health with auto-restart, merges artifacts, and uploads results to Google Drive. Used for corpus-level production.
Two generator backends. The
simple_randomgenerator produces GESTs that are executable by construction: every action is sampled from the engine's capability registry, every chain follows a valid POI sequence, and every story succeeds in the engine (modulo the engine's own bugs). Thellmgenerator produces GESTs that are narratively richer but not simulatable out of the box — it operates at a higher semantic level, ignoring the engine's exact action vocabulary and constraints. The released ICLR checkpoint and the GTASA-01 corpus were produced exclusively withsimple_random. Usellmonly when you want semantically structured graphs for downstream non-simulation purposes.
The GEST line of work, from the formalism to the engine, the agentic authoring layer, and the corpus. Checkpoints of this repository used for a paper are noted.
- M. Masala, N. Cudlenco, T. Rebedea, M. Leordeanu. Explaining Vision and Language through Graphs of Events in Space and Time. ICCV 2023 Workshops (CLVL), pp. 2818–2823. CVF Open Access · DOI · arXiv:2309.08612
- M. Masala, N. Cudlenco, T. Rebedea, M. Leordeanu. GEST: the Graph of Events in Space and Time as a Common Representation between Vision and Language. Preprint, 2023. arXiv:2305.12940
- N. Cudlenco, M. Masala, M. Leordeanu. [Tiny Paper] GEST-Engine: Controllable Multi-Actor Video Synthesis with Perfect Spatiotemporal Annotations. ICLR 2026, 2nd Workshop on World Models. OpenReview — checkpoint
v1.0-iclr2026 - N. Cudlenco, M. Masala, M. Leordeanu. The GEST-Engine: From Event Graphs to Synthetic Video. A Full Technical Report. 2026. arXiv:2607.12231 — documents this system together with the GEST-Engine
- N. Cudlenco, M. Masala, M. Leordeanu. Authoring for Living Worlds: Tool-Constrained LLM Agents for Executable Multi-Actor Scenarios. ECCV 2026 Workshop on Agent in World: Living Worlds with Interactive Agents (AIW). OpenReview · arXiv:2604.10383 (earlier versions titled Agentic Video Generation: From Text to Executable Event Graphs via Tool-Constrained LLM Planning) — checkpoint
v1.1-eccv2026 - N. Cudlenco, M. Masala, M. Leordeanu. GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models. ACCV 2026. arXiv:2604.10385 — corpus generated with checkpoint
v1.0-iclr2026
The corpus produced with this orchestrator, GTASA-01 (1,008 procedurally generated multi-actor stories, including the 938-scenario set reported in the GTASA paper), is publicly available on HuggingFace: nnc-001/gtasa-01.
| Tag | Date | Reference |
|---|---|---|
v1.0-iclr2026 |
March 2026 | ICLR 2026 Workshop Tiny Paper — state used to generate the GTASA-01 sample corpus |
v1.1-eccv2026 |
August 2026 | ECCV 2026 AIW Workshop paper — adds the agentic authoring layer: a Director agent and a Scene Builder subagent that build executable GESTs exclusively through validated tool calls |
Future checkpoints will be listed here as the system evolves.
- Windows 10 / 11
- Python 3.10+
- GTA San Andreas PC v1.0 + MTA 1.6 + the
sv2lresource, installed and configured following the mta-sim installation guide. - Google Drive API credentials — optional, only required if you pass
--google-drive-folder/--output-g-driveto upload results.
- VMware Workstation Pro 25H2 — free for personal, educational, and commercial use as of November 11, 2024. Release notes and download: VMware Workstation Pro 25H2.
- A pre-built master VM image containing Windows 10/11, GTA San Andreas, MTA 1.6, and the
sv2lresource. This image is not distributed with the repository because it requires a legitimate Windows license and a legitimate GTA San Andreas license; each user must build their own.
VMware Workstation Pro cannot run VMs on a machine where Hyper-V is active. If WSL (Windows Subsystem for Linux) or any other Hyper-V–based feature is enabled on the host, you must disable the hypervisor before running the VM orchestrator:
# Run as Administrator
bcdedit /set hypervisorlaunchtype off
# Reboot the machineTo re-enable WSL afterwards:
# Run as Administrator
bcdedit /set hypervisorlaunchtype auto
# Reboot the machineThis is a Windows/VMware compatibility issue, not a limitation of this system.
git clone https://github.com/ncudlenco/multiagent_story_system.git
cd multiagent_story_system
pip install -r requirements.txtFollow the mta-sim installation instructions to install GTA San Andreas, MTA 1.6, and the sv2l resource. This step is required for both local and VM modes.
For local single-machine execution, no further setup is needed beyond step 2. Use main.py and batch_generate.py directly on the host.
Install VMware Workstation Pro 25H2. If Hyper-V / WSL is enabled on the host, disable it first (see the limitation note above).
Create a Windows 10/11 master VM containing a working installation of GTA San Andreas, MTA 1.6, and the sv2l resource (i.e. complete step 2 inside the guest). The orchestrator on the host invokes scripts from this repository inside the guest via vmrun runProgramInGuest to drive the MTA server + client for each input graph.
Recommended VM hardware (these match the configuration used to produce GTASA-01):
| Resource | Value |
|---|---|
| Memory | 3 GB |
| Processors | 2 |
| Cores per processor | 1 |
| Hard disk | 200 GB (NVMe) |
| Network adapter | NAT |
| USB controller | Present |
| Sound card | Auto detect |
| Display — number of monitors | 1 |
| Display — maximum resolution | 1280 × 720 |
| Display — 3D graphics acceleration | Enabled (required) |
| Display — graphics memory | 8 GB (recommended by VMware) |
| Virtualization engine | Virtualize Intel VT-x/EPT or AMD-V/RVI ✓ |
GTA San Andreas requires Direct3D 9 acceleration, so the VM must have Display → Accelerate 3D graphics enabled. The display must also be set to a single monitor — multi-monitor configurations confuse the screenshot capture pipeline and prevent reliable simulation.
Auto-start the worker on logon. The orchestrator on the host signals each worker by writing a configuration file into a shared folder; inside the guest, vm_auto_runner.py watches that file and drives the per-story simulation loop. For this to work, vm_auto_runner.py must be running automatically when the guest user logs in — the orchestrator only powers the VM on, it does not log in or open a terminal.
Set this up by creating a Task Scheduler entry in the guest with the following definition. Save it as VMAutoRunner.xml and import it via Task Scheduler → Action → Import Task…, or via PowerShell with Register-ScheduledTask -Xml (Get-Content VMAutoRunner.xml | Out-String) -TaskName "VMAutoRunner".
<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.2" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
<RegistrationInfo>
<Description>Reads the configuration for the worker from the shared folder on the host and starts simulation and collection.</Description>
<URI>\VMAutoRunner</URI>
</RegistrationInfo>
<Triggers>
<LogonTrigger>
<Enabled>true</Enabled>
</LogonTrigger>
</Triggers>
<Principals>
<Principal id="Author">
<LogonType>InteractiveToken</LogonType>
<RunLevel>HighestAvailable</RunLevel>
</Principal>
</Principals>
<Settings>
<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
<DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
<StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
<AllowHardTerminate>true</AllowHardTerminate>
<StartWhenAvailable>false</StartWhenAvailable>
<RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
<IdleSettings>
<StopOnIdleEnd>true</StopOnIdleEnd>
<RestartOnIdle>false</RestartOnIdle>
</IdleSettings>
<AllowStartOnDemand>true</AllowStartOnDemand>
<Enabled>true</Enabled>
<Hidden>false</Hidden>
<RunOnlyIfIdle>false</RunOnlyIfIdle>
<WakeToRun>false</WakeToRun>
<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>
<Priority>7</Priority>
</Settings>
<Actions Context="Author">
<Exec>
<Command>powershell.exe</Command>
<Arguments>-NoExit -Command "python 'C:\mta1.6\server\mods\deathmatch\resources\multiagent_story_system\vm_auto_runner.py'"</Arguments>
</Exec>
</Actions>
</Task>Adjust the <Arguments> path if you cloned the repository to a different location inside the guest. Make sure the task is set to run with highest privileges so it can launch MTA, and that the trigger is At log on — LogonTrigger is what makes the runner come up automatically every time the orchestrator powers the worker VM on.
The default Windows account in the example task is user, matching guest_os.username in vmware_config.yaml. If you use a different guest username, edit the <UserId> field (or remove it and re-import; Task Scheduler will fill it in based on whoever imports the XML).
Once the VM is configured, the auto-runner starts on logon, and a manual end-to-end run succeeds, take a snapshot. The default snapshot name in vmware_config.yaml is Ready.
Edit vmware_config.yaml to match your VMware layout and guest credentials:
vmware:
master_vm_path: "W:\\VMs\\2\\Windows 10 and later x64.vmx"
master_snapshot: "Ready"
workers_dir: "W:\\VMs\\workers"
vmrun_paths:
- "C:\\Program Files (x86)\\VMware\\VMware Workstation\\vmrun.exe"
- "C:\\Program Files\\VMware\\VMware Workstation\\vmrun.exe"
guest_os:
username: "user"
password: "user"
work_dir: "C:\\mta1.6\\server\\mods\\deathmatch\\resources\\multiagent_story_system"The repository ships a pre-generated data/simulation_environment_capabilities.json that has been manually curated and is the version used by all released checkpoints. Use it as-is unless you have a specific reason to regenerate it — the hand edits fix inconsistencies that a fresh export does not.
A fresh capability registry can be produced by running the engine with "EXPORT_MODE": true in its config.json (see the mta-sim README) or by running python main.py --export-capabilities. A raw export will likely not be fully compatible without additional manual fixups.
There are three command-line entry points: main.py for one-off operations and maintenance, batch_generate.py for local batches, and vmware_orchestrator.py for VM batches.
Replace <YOUR_GDRIVE_FOLDER_ID> in any example with the ID of a Google Drive folder you own, or omit the flag entirely to keep results local.
python .\vmware_orchestrator.py `
--num-vms 25 `
--stories-per-vm 20 `
--random-chains-per-actor 2 `
--random-max-actors-per-region 5 `
--random-max-regions 2 `
--generate-description prompt `
--generator-type simple_random `
--simulation-retries 0 `
--google-drive-folder <YOUR_GDRIVE_FOLDER_ID> `
--ensure-target `
--no-monitorpython .\batch_generate.py `
--output-folder local_batch_out\ `
--story-number 20 `
--generator-type simple_random `
--random-chains-per-actor 2 `
--random-max-actors-per-region 5 `
--random-max-regions 2 `
--generate-description prompt `
--collect-simulation-artifacts `
--simulation-retries 0 `
--ensure-targetpython .\main.py --export-capabilitiespython .\main.py --simulate 22597965 --collect-artifacts| Flag | Type | Default | Description |
|---|---|---|---|
--export-capabilities |
flag | — | Export the engine's capability registry from MTA. |
--preprocess-capabilities, --preprocess |
flag | — | Preprocess the exported capabilities into optimized cache files using an LLM. |
--invalidate-capabilities-cache |
flag | — | Invalidate any existing capabilities cache files before preprocessing. |
--skip-episodes |
flag | — | Skip episode summarization (faster, optional data). |
--generate |
flag | — | Generate a story using the (LLM-based) Concept + Casting agents. |
--max-num-protagonists |
int | 2 |
Maximum number of protagonist actors. Use -1 to let the LLM decide. |
--max-num-extras |
int | 0 |
Maximum number of background / extra actors. Use -1 to let the LLM decide. |
--num-actions |
int | 5 |
Number of distinct action types to use. |
--seeds |
str... | [] |
Narrative seed sentences (space-separated, in quotes). |
--from-text-file FILE |
path | — | Read narrative seeds from a text file (one sentence per line). Overrides --seeds. |
--stop-phase |
int | — | Number of generation phases to run (default: all). |
--scene-number |
int | 0 |
How many scenes to generate. |
--resume STORY_ID |
str | — | Resume an existing story from checkpoint (8-character UUID). |
--from-phase {1,2,3} |
int | — | Phase to resume from when using --resume (1=concept, 2=casting, 3=detail). |
--resume-from-stage {1..5} |
int | — | Resume the reactive detail workflow from stage N (1=grounding, 2=segmentation, 3=setup, 4=screenplay, 5=translation). Requires --use-react. |
--use-cached-detail |
flag | — | Use cached detail expansions if available (only for the detail phase). |
--use-react |
flag | — | Use the reactive (tool-based) detail workflow instead of the standard one. |
--simulate STORY_ID |
str | — | Simulate an existing story in MTA (8-character UUID). |
--scene SCENE_ID |
str | — | Specific scene to simulate. Requires --simulate. |
--timeout SECONDS |
int | 3600 |
Simulation timeout in seconds. |
--collect-artifacts |
flag | false |
Enable artifact collection during simulation (videos, logs, spatial relations, segmentation, …). |
--capture-segmentations / --no-capture-segmentations |
flag | enabled | Toggle per-frame instance segmentation capture. |
--save-prompts |
flag | — | Save all LLM prompts and responses during story generation. |
--save-raw-responses |
flag | — | Include raw LLM responses in the prompt logs (verbose; requires --save-prompts). |
--config FILE |
path | config.yaml |
Path to the configuration file. |
--verbose |
flag | — | Enable verbose (DEBUG) logging. |
Mode selection (mutually exclusive — pick one):
| Flag | Type | Description |
|---|---|---|
--story-number N |
int | Generate N stories. |
--from-existing-stories PATH |
path | Path to a folder of existing stories to (re-)simulate. |
--from-text-files JSON_FILE |
path | Path to a JSON file listing text files to convert into stories. |
--resume-batch BATCH_ID |
str | Resume an interrupted batch (e.g. batch_20231103_143022). |
--reset-failed BATCH_ID |
str | Reset all failed stories in a batch and clear simulation artifacts. |
--reset-success BATCH_ID |
str | Reset all successful stories in a batch and clear simulation artifacts. |
--reset-simulations BATCH_ID |
str | Reset all simulations (success + failed) and clear all simulation artifacts. |
General arguments:
| Flag | Type | Default | Description |
|---|---|---|---|
--output-folder PATH |
path | — | Output folder for batch results. Required unless using a reset / resume mode. |
--retry-story STORY_ID |
str | — | Retry simulations for a specific story. Requires --resume-batch. |
--take TAKE_NUMBER |
int | — | Specific take to retry. Requires --retry-story. |
--config FILE |
path | config.yaml |
Path to the configuration file. |
--verbose |
flag | — | Enable verbose (DEBUG) logging. |
Story generation parameters:
| Flag | Type | Default | Description |
|---|---|---|---|
--num-actors |
int | 2 |
Number of protagonist actors. |
--num-extras |
int | 1 |
Number of extra / background actors. |
--num-actions |
int | 5 |
Number of distinct actions. |
--scene-number |
int | 4 |
Number of scenes per story. |
--seeds |
str... | [] |
Narrative seed sentences. |
--generator-type {llm,simple_random} |
str | llm |
Story generator. Use simple_random for procedural execution-by-construction generation. |
--random-chains-per-actor |
int | 3 |
Number of action chains per actor (simple_random only). |
--random-max-actors-per-region |
int | unlimited | Maximum actors per region (simple_random only). |
--random-max-regions |
int | unlimited | Maximum number of regions visited (simple_random only). |
--random-seed |
int | — | Random seed for reproducibility (simple_random only). |
--episode-type {classroom,gym,garden,house} |
str | random | Constrain to a single episode type (simple_random only). |
--ensure-target |
flag | — | Keep generating until the target number of successful stories is reached. |
--parallel-workers N |
int | 1 |
Number of parallel workers (only used in text-file mode). |
--skip-simulation |
flag | — | Skip the MTA simulation phase (generation only, for text-file mode). |
Variation parameters:
| Flag | Type | Default | Description |
|---|---|---|---|
--same-story-generation-variations |
int | 1 |
Number of detail variations per story. |
--same-story-simulation-variations |
int | 1 |
Number of simulation runs per detail variation. |
Retry / timeout parameters:
| Flag | Type | Default | Description |
|---|---|---|---|
--generation-retries |
int | 3 |
Maximum generation retry attempts. |
--simulation-retries |
int | 3 |
Maximum simulation retry attempts. |
--simulation-timeout |
int | 3600 |
Simulation timeout in seconds. |
Artifact / output parameters:
| Flag | Type | Default | Description |
|---|---|---|---|
--collect-simulation-artifacts |
flag | false |
Enable artifact collection during simulations (videos, logs, spatial relations, segmentation, …). |
--capture-segmentations / --no-capture-segmentations |
flag | enabled | Toggle per-frame instance segmentation capture. |
--generate-description {prompt,full} |
str | — | Write the GPT-4o prompt only (prompt) or also call the OpenAI API in-line (full). |
--output-g-drive FOLDER_ID |
str | — | Google Drive folder ID for upload. |
--keep-local |
flag | — | Keep the local copy after Google Drive upload. |
--force |
flag | — | Force overwrite if the output folder already exists. |
Mode selection (mutually exclusive — pick one):
| Flag | Description |
|---|---|
--num-vms N |
Spawn N worker VMs and run a batch (requires --stories-per-vm). |
--update-master |
Update the master VM by pulling the latest code from GitHub repositories and refreshing the Ready snapshot. |
--purge-vms |
Delete all previous worker VM batch folders from disk. |
--merge-gdrive-results FOLDER_ID |
Merge per-worker subfolders inside a Google Drive folder into a flat layout (standalone, no VMs). |
--merge-flat-folders DEST_ID SRC_ID... |
Merge several flat Google Drive folders into one destination (first ID is the destination). |
Common arguments:
| Flag | Type | Default | Description |
|---|---|---|---|
--stories-per-vm N |
int | — | Target number of successful stories per worker VM. Required with --num-vms. |
--config FILE |
path | vmware_config.yaml |
Path to the orchestrator config. |
Code-sync arguments (used with --update-master):
| Flag | Description |
|---|---|
--github-token TOKEN |
GitHub personal access token (or set GITHUB_TOKEN env var). |
--purge |
Completely remove repository directories before cloning. |
--skip-snapshot |
Don't update the Ready snapshot after sync. |
--refresh-google-token |
Delete the cached Google token and re-authenticate. |
Story generation parameters (override defaults from config):
| Flag | Type | Default | Description |
|---|---|---|---|
--num-actors |
int | — | Number of protagonist actors. |
--num-extras |
int | — | Number of extra / background actors. |
--num-actions |
int | — | Number of distinct actions. |
--scene-number |
int | — | Number of scenes per story. |
--same-story-generation-variations |
int | — | Detail variations per story. |
--same-story-simulation-variations |
int | — | Independent engine simulations per generated GEST. |
--ensure-target |
flag | — | Keep generating stories until the target number of successful stories is reached per worker. |
--simulation-retries |
int | 3 |
Number of retries per failed simulation. |
Generator selection:
| Flag | Type | Default | Description |
|---|---|---|---|
--generator-type {llm,simple_random} |
str | llm |
Story generator. Use simple_random for procedural execution-by-construction generation. |
--random-chains-per-actor |
int | — | Action chains per actor (simple_random only). |
--random-max-actors-per-region |
int | — | Max actors per region (simple_random only). |
--random-max-regions |
int | — | Max regions to visit (simple_random only). |
--random-seed |
int | — | Random seed (simple_random only). |
--episode-type {classroom,gym,garden,house} |
str | — | Constrain to a single episode type (simple_random only). |
Description / capture / statistics:
| Flag | Type | Default | Description |
|---|---|---|---|
--generate-description {prompt,full} |
str | — | Write the GPT-4o prompt only (prompt) or also call the OpenAI API in-line (full). |
--capture-segmentations / --no-capture-segmentations |
flag | enabled | Toggle per-frame instance segmentation capture. |
--no-count-segmentations |
flag | — | Skip counting segmentation frames in post-run statistics. |
--no-count-spatial |
flag | — | Skip counting spatial relations in post-run statistics. |
Google Drive:
| Flag | Description |
|---|---|
--google-drive-folder FOLDER_ID |
Google Drive parent folder ID for uploads. |
--keep-local |
Keep local copies after Google Drive upload. |
--merge-gdrive |
Merge all per-worker Google Drive folders into a single batch folder after completion. |
Worker control / debugging:
| Flag | Description |
|---|---|
--no-restart |
Don't restart crashed/hung workers — show errors and fail fast. |
--no-monitor |
Fire-and-forget: launch workers and exit without monitoring. |
--keep-vms |
Keep worker VMs after completion (for debugging). |
main.py— single-story / maintenance entry point.batch_generate.py— local batch runner (no VMs).vmware_orchestrator.py— VMware batch orchestrator (parallel VMs).vm_auto_runner.py— script that runs inside each worker VM; drives the MTA server + client and thesv2lresource for each input graph.vm_monitor.py— heartbeat-based worker health monitoring and auto-restart.simple_gest_random_generator.py— procedural GEST generator implementation.gdrive_manager.py— Google Drive upload, folder management, and merging.gdrive_statistics.py/download_gdrive_folders.py— post-run statistics and retrieval helpers.vmware_config.yaml— VMware paths, snapshot name, worker limits, guest credentials.config.yaml— common configuration (paths, logging, optional OpenAI settings).data/simulation_environment_capabilities.json— pre-generated, manually curated capability registry consumed by the generator.
If you use this orchestrator in your research, please cite the paper matching the part you use: the ICLR 2026 Tiny Paper for procedural generation and the engine pipeline, the ECCV 2026 AIW paper for the agentic authoring layer, and the ACCV 2026 GTASA paper for the corpus and its evaluation.
@inproceedings{cudlenco2026tiny,
title={[Tiny Paper] {GEST}-Engine: Controllable Multi-Actor Video Synthesis with Perfect Spatiotemporal Annotations},
author={Nicolae Cudlenco and Mihai Masala and Marius Leordeanu},
booktitle={ICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling},
year={2026},
url={https://openreview.net/forum?id=uUofPYVMZH}
}
@inproceedings{cudlenco2026authoring,
title={Authoring for Living Worlds: Tool-Constrained {LLM} Agents for Executable Multi-Actor Scenarios},
author={Nicolae Cudlenco and Mihai Masala and Marius Leordeanu},
booktitle={ECCV 2026 Workshop on Agent in World: Living Worlds with Interactive Agents (AIW)},
year={2026},
url={https://openreview.net/forum?id=kFK4sVgN9b}
}
@inproceedings{cudlenco2026gtasa,
title={{GTASA}: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models},
author={Nicolae Cudlenco and Mihai Masala and Marius Leordeanu},
booktitle={Asian Conference on Computer Vision (ACCV)},
year={2026},
eprint={2604.10385},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2604.10385}
}See LICENSE. Use of the system requires a licensed copy of Windows and a licensed copy of GTA San Andreas; Rockstar Games / Take-Two Interactive retain all rights to in-game assets. Research data derived from this system is released for non-commercial academic research only.
- Nicolae Cudlenco, corresponding author and maintainer:
nicolae.cudlenco@gmail.com - Mihai Masala:
mihaimasala@gmail.com - Marius Leordeanu:
leordeanu@gmail.com
For questions, bug reports, or collaboration inquiries, open an issue or email the corresponding author.