Hello, while testing the models on the benchmark, I encountered a couple of minor bugs, so I thought of mentioning them here to get them fixed.
- In the
evaluation/evaluate_ecgbench.py script, item dictionary does not contain key No; I think it should be id(according to the test split from ECGBench(Testing) benchmark on HuggingFace).
def eval_mmmu(dir):
print("====mmmu====")
with open("/path/to/PULSE/data/ECGBench/mmmu-ecg.json", "r", encoding='utf-8') as f:
data = json.load(f)
answer_dict = {item["No"]: item["conversations"][1]["value"] for item in data}
score_dict = {}
for root, dirs, files in os.walk(dir):
for file in files:
if file.endswith(".jsonl"):
predict_list = []
golden_list = []
file_path = os.path.join(root, file)
if "mmmu-ecg" not in file_path:
continue
- In the
LLaVA/llava/eval/model_ecg_resume.py script, the line "text": line["conversations"][0]["value"].replace("<image>\n",""), causes AttributeError due to being a list; looks like it should be a string "text": "".join(line["conversations"][0]["value"]).replace("<image>\n",""), to fix the issue.
def eval_model(args):
# Model
disable_torch_init()
model_path = args.model_path
model_name = get_model_name_from_path(model_path)
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name)
questions = []
with open(args.question_file, "r") as f:
json_data = json.load(f)
for line in json_data:
print(line)
questions.append({"question_id": line["id"],
"image": line["image"],
"text": line["conversations"][0]["value"].replace("<image>\n",""),
"ans": line["conversations"][1]["value"]})
Great work on the paper. Thank you!
Hello, while testing the models on the benchmark, I encountered a couple of minor bugs, so I thought of mentioning them here to get them fixed.
evaluation/evaluate_ecgbench.pyscript,itemdictionary does not contain keyNo; I think it should beid(according to the test split from ECGBench(Testing) benchmark on HuggingFace).LLaVA/llava/eval/model_ecg_resume.pyscript, the line"text": line["conversations"][0]["value"].replace("<image>\n",""),causesAttributeErrordue to being a list; looks like it should be a string"text": "".join(line["conversations"][0]["value"]).replace("<image>\n",""),to fix the issue.Great work on the paper. Thank you!