Questions about Event-OCR and absolute distance evaluation in EventBench

[HYLZ-2019]

Hi, thank you for releasing EventBench and EventGPT-Plus-2B.

I am trying to reproduce the EventGPT+(B) results from Table 3 in the paper. I downloaded the released EventBench dataset and EventGPT-Plus-2B checkpoint, and wrote an evaluation script based on the released inference_eventgpt_plus.py pipeline. The script is attached as evaluate_eventbench_eventgpt_plus.py.

evaluate_eventbench_eventgpt_plus.py

Most of my reproduced metrics are close to Table 3, but I am unsure about the official evaluation protocols for Event-OCR and absolute distance:

Metric	Paper Table 3	My script
DU	78.3	78.26
CR	78.2	80.00
AR	79.8	80.27
GR	55.5	54.47
E-OCR	53.6	11.11 exact match / 79.29 CER
SR	50.2	49.83
AD	51.9	MAE 0.8319 / RMSE 1.1844 / 45.34% at 0.5m tolerance / 69.47% at 1.0m tolerance
OC	62.8	63.04

Could you please clarify the following two points?

1. Event-OCR metric

   • What metric is used for the E-OCR score in Table 3?
   • Is it exact match, character-level accuracy, edit distance/CER, token-level F1, or another metric?
   • What text normalization is applied before scoring, e.g. whitespace removal, case folding, Unicode/accent normalization, punctuation removal, or partial matching?

2. Absolute distance metric

   • How is the numeric distance prediction converted into the AD score in Table 3?
   • Is a fixed tolerance threshold used to decide whether a prediction is correct?
   • If so, what threshold is used? For example, 0.25m, 0.5m, 1.0m, relative error, or another rule?
   • Are predictions rounded before comparison?

If there is an official evaluation script or answer-normalization code for these two metrics, it would be very helpful.

Thank you!

[HYLZ-2019]

@XduSyL 🧐

[XduSyL]

Thank you for your detailed questions and great effort!

We are pleasantly surprised and thrilled to see that you have successfully reproduced the vast majority of the metrics in our main table (Table 3)! This truly affirms the objective and rigorous evaluation standards of EventBench.

Regarding your questions about the specific evaluation metrics:

• Absolute Distance (AD):
  In our preliminary evaluations, we calculated the percentage of correct predictions within multiple error thresholds (10%, 20%, and 30%). To simplify the final presentation in Table 3, we unified the metric by adopting a 30% error threshold. That is, a prediction is considered correct if the relative error is within 30%.
• Event-OCR:
  For the E-OCR score, we generally rely on standard exact match accuracy after applying basic text normalization (such as lowercasing and basic whitespace/punctuation removal). Admittedly, we are aware that there are currently a few minor issues with the OCR evaluation pipeline, which we plan to refine and update in subsequent versions.

The detailed answer-normalization code and the official evaluation scripts for all metrics will be fully released alongside our upcoming ECCV challenge (You can check the challenge here: EventBench Challenge).

Thanks again for your outstanding work and support!

[HYLZ-2019]

Thank you very much for the detailed clarification and for the kind words!

It is great to know that the AD score in Table 3 uses a 30% relative-error threshold. This resolves my main uncertainty about the absolute distance metric. Thank you also for clarifying the current Event-OCR evaluation protocol and the planned refinement of the OCR pipeline.

I appreciate your plan to release the official evaluation scripts and answer-normalization code together with the upcoming ECCV challenge. That will be very helpful for making future comparisons fully consistent and reproducible.

Thanks again for releasing EventBench and for your careful response!

[Channingss]

@HYLZ-2019 I used the script you shared and ran inference with the default parameters, but the OCR score is only 0.1. Should I adjust the parameters, or are there other things I should pay attention to?

[HYLZ-2019]

@Channingss My reproduced OCR score is also only 0.11. According to the above discussion, the number seems to be right.

[Channingss]

{
"action_recognition": 0.8,
"casual_reasoning": 0.4222222222222222,
"detailed_understanding": 0.5445134575569358,
"gesture_recognition": 0.5436241610738255,
"event_ocr": 0.1111111111111111,
"avg_score": 0.5446257197696737
}
@HYLZ-2019 这是我用你的脚本跑出来的结果，除了OCR有问题以外，casual_reasoning和detailed_understanding的评估指标得分也过低。
@

[HYLZ-2019]

@Channingss 这是不是codabench评测机算出来的结果？
我前两天和@XduSyL 邮件联系过，他表示现在确实存在codabench评测机和huggingface数据集题目版本不一致的问题，说他周末会修。

[XduSyL]

{ "action_recognition": 0.8, "casual_reasoning": 0.4222222222222222, "detailed_understanding": 0.5445134575569358, "gesture_recognition": 0.5436241610738255, "event_ocr": 0.1111111111111111, "avg_score": 0.5446257197696737 } @HYLZ-2019 这是我用你的脚本跑出来的结果，除了OCR有问题以外，casual_reasoning和detailed_understanding的评估指标得分也过低。 @

你采用的评测数据的query来自于哪里？是codabench上还是hf上的，这两个版本是不一致的

[Channingss]

我用来推理的数据来自于https://huggingface.co/datasets/XduSyL/EventBench @HYLZ-2019

[Channingss]

@HYLZ-2019 是的，codabench跑出来的结果。请问您也在打比赛吗？

[HYLZ-2019]

@Channingss 是的，坐等数据集更新中^_^

[XduSyL]

{ "action_recognition": 0.8, "casual_reasoning": 0.4222222222222222, "detailed_understanding": 0.5445134575569358, "gesture_recognition": 0.5436241610738255, "event_ocr": 0.1111111111111111, "avg_score": 0.5446257197696737 } @HYLZ-2019 这是我用你的脚本跑出来的结果，除了OCR有问题以外，casual_reasoning和detailed_understanding的评估指标得分也过低。 @

在codabench上的Get Started中File里的Event Understanding Uniform Option Input data中获取test_question文件进行评测，目前DU和CR是进行选项均匀后的结果，因此版本不一致，在HF上的数据 @HYLZ-2019 第一个表格的数据是置信的

[Channingss]

@HYLZ-2019 可否交换一下联系方式，哈哈

[HYLZ-2019]

@Channingss 好啊，可以把微信二维码发我邮箱：hylz@pku.edu.cn