reproduction help!

[kw9-21]

hi authors!
I’m really hoping to reproduce the results of your paper in the ScienceWorld environment. I’ve already integrated ScienceWorld and got it running, but I haven’t come up with a good prompt yet, so I was wondering if you could share some of the prompts you used in your experiments? Specifically, prompts for generating insights from successful and failed trajectories, as well as the format specifications for those insights.

[bingreeky]

We have already uploaded SciWorld! Could you plz take a look and see whether it is helpful? Thx!

[kw9-21]

May I ask if I can confirm that Sciworld can now be run directly? I seem to have only found envs/sciworld_env.py, and when I run bash run_mas.sh, I get the following error. Also, I would really like to know if there should be a file named sciworld_prompt.py in the directory tasks/prompts.
I would greatly appreciate it if you could provide some necessary files, such as data/sciworld/test.jsonl and tasks/prompts/sciworld_prompt.py!
P.S.: When I tried to integrate sciworld myself earlier, I could only load the dataset from the SciWorld environment in code ; I couldn’t find any open-source versions available for direct download online.

Traceback (most recent call last):
File "/home/GMemory/tasks/run.py", line 21, in
from envs import BaseEnv, BaseRecorder, get_env, get_recorder, get_task
File "/home/GMemory/tasks/envs/init.py", line 31, in
with open(TASKS_PATH['sciworld'], 'r+', encoding='utf-8') as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'data/sciworld/test.jsonl'

[bingreeky]

data/sciworld/test.jsonl added! Sorry for the miss.

[kw9-21]

Thanks for your added data! But I still have two questions:

1. why data/sciworld/test.jsonl only has 90 samples? The statistics I obtained using the code below are Train: 3592 (49.8%)| Dev: 1796 (24.9%)| Test: 1819 (25.2%) . How do you get those 90?
   train_vars = env.get_variations_train()
   dev_vars = env.get_variations_dev()
   test_vars = env.get_variations_test()
2. Could you please provide tasks/prompts/sciworld_prompt.py？I'm afraid I cannot reproduce the result in paper without corresponding prompts.

[kw9-21]

To add to the second question: using your provided test.jsonl and my prompt, got result 35.6%(qwen14b), while the table 1 in paper shows 67.4%(gpt-4o-mini). Even though the models are different, surely there wouldn't be such a big gap?

[bingreeky]

Sure, we have updated the config and prompts. The test data is from the AgentSquare codebase; you may refer to it if interested. Many previous issues also encountered reproducibility difficulties, but they were eventually able to identify the causes and reproduce the results, e.g., #19. The 35.6% is surely not expected, and you may start from using the update codebase for reproduction. There is no need to rush — we can continue to discuss and work through this step by step : ) Thank you for the patience!

[kw9-21]

Sorry for getting a little worked up, I just really want to get a good result as soon as possible, haha.
I download your updated codebase and bash run_mas.sh directly, finally the result is ave reward: 0.5638227513227513, ave done: 0.2111111111111111, which is also strange. Could you plz help me?

python3 tasks/run.py
--task sciworld
--reasoning io
--mas_memory g-memory
--max_trials 30
--mas_type autogen
--model Qwen2.5-14B-Instruct \

[bingreeky]

Thank you for your message. The score reported for G-Memory on AutoGen + SciWorld is 60% (Table 3 of the paper), whereas the score reported in our paper is the progress rate on SciWorld (as noted in Appendix A.1). In that sense, the current 56% result is not far off.

One simple possibility is randomness from the LLM and memory system themselves. Another is variation in the quality of the LLM API, if an API-based model was used. Would it be possible for you to run the experiment two more times to help determine whether this gap is due to randomness or to an actual setup difference? Thanks!

[bingreeky]

Hi! Just for a quick update; we are wondering if the results have been aligned successfully or still warrant further discussion? We are more than glad to help.

[kw9-21]

Sure! I tried it twice, and the results were:
ave reward: 0.4383465608465608, ave done: 0.2222222222222222
ave reward: 0.47244708994708984, ave done: 0.18888888888888888
The QWEN 2.5-14B model I used was deployed via VLLM. Perhaps there is indeed randomness in other aspects, such as the model itself, because when I reproduced the ALFWorld results, the three runs yielded the following averages:
ave done: 0.7611940298507462
ave done: 0.6716417910447762
ave done: 0.7164179104477612
while the result in Table 3 is 85.82

In addition, while reviewing the logs, I noticed that in some cases won=false was reported even though score=100 (which is the JVM score for the environment). I later discovered that this was caused by overly rigid specifications for sub-goals in test.jsonl, resulting in some sub-goals being marked as won=false despite having actually completed, simply because they failed to match the criteria. After fixing this issue, the success rate improved.

I would like to inquire whether ScienceWorld includes a training task set, and if so, whether subgoal annotations are provided for computing progress rate.
I was unable to locate this in the AgentSquare codebase, and I would greatly appreciate any guidance.

[bingreeky]

Thank you! Please let us re-run the codebase from our devices. This might take a few hours and we will get back shortly.

[kw9-21]

hi author~ wondering that how things are going over there.

[kw9-21]

@bingreeky very much looking forward to receiving your response regarding the reproduction, and particularly whether ScienceWorld includes a training task set, and if so, whether subgoal annotations are provided for computing progress rate.