Hi there,
Thanks for sharing this repository. I am interested in your work on EnactToM benchmark and am looking forward to evaluate on it.
I noticed your page mentions "Dataset(coming soon)", but it appears the last commit of the code was about 2 months ago. Thus, I tried to generate tasks as the instruction ./enacttom/run.sh generate --num-tasks 1 --difficulty standard with the default model but failed. Could you release the dataset or tell me the reason why it always couldn't generate tasks the meet the standards. I found that "failure_explanations": ["vase_0 is not is_on_top table_32", "glass_4 is not is_on_top table_32"] even if {"agent": "agent_0","action": "Place[glass_4, on, table_32, None, None]", "success": true,"observation": "Successful execution!\n\nagent_0_VisualSummary:\n- No notable new visual observations"} and {"agent": "agent_2","action": "Place[vase_0, on, table_32, None, None]", "success": true,"observation": "Successful execution!\n\nagent_0_VisualSummary:\n- No notable new visual observations"}
Thank you for your time and contribution!
Hi there,
Thanks for sharing this repository. I am interested in your work on EnactToM benchmark and am looking forward to evaluate on it.
I noticed your page mentions "Dataset(coming soon)", but it appears the last commit of the code was about 2 months ago. Thus, I tried to generate tasks as the instruction
./enacttom/run.sh generate --num-tasks 1 --difficulty standardwith the default model but failed. Could you release the dataset or tell me the reason why it always couldn't generate tasks the meet the standards. I found that"failure_explanations": ["vase_0 is not is_on_top table_32", "glass_4 is not is_on_top table_32"]even if{"agent": "agent_0","action": "Place[glass_4, on, table_32, None, None]", "success": true,"observation": "Successful execution!\n\nagent_0_VisualSummary:\n- No notable new visual observations"}and{"agent": "agent_2","action": "Place[vase_0, on, table_32, None, None]", "success": true,"observation": "Successful execution!\n\nagent_0_VisualSummary:\n- No notable new visual observations"}Thank you for your time and contribution!