Evaluating Problems

[ikerrrrrrrrrrr]

Hi,

I did close-loop eval but I found that I can only hit approx. 50 points of driving score with your checkpoint provided with no modifications.
With the Crazy Traffic Light mode on, I notice that some agents will get stuck in crossroads, causing extremely low average speed of ego car, and failures simultaenously.
On my device, Infraction Score is super low, while road completion and collision rates are similar to your paper.
Is the default config of CARLA right? How can i get same driving score as your paper?

Thanks a lot!!

[ikerrrrrrrrrrr]

My scores are listed below:
Avg. driving score : 57.130278
Avg. route completion : 93.035833
Avg. infraction penalty : 0.608259
Collisions with pedestrians : 0.050574
Collisions with vehicles : 0.101963
Collisions with layout : 0.127259
Red lights infractions : 0.000000
Stop sign infractions : 0.010444
Off-road infractions : 0.011185
Route deviations : 0.000000
Route timeouts : 0.011981
Agent blocked : 0.039926
Average Speed (m/s) : 1.005835

[Rookie-From-East]

Hello, I've recently encountered a similar issue. I'm unsure whether the author provided a script file within the code repository to compute the final evaluation metrics. I wrote my own statistical script following the evaluation method described in the paper, but the results still failed to meet the metrics outlined in the paper. My overall findings are quite similar to yours. I was wondering if you've managed to resolve this issue?

My scores are listed below:我的成绩如下： Avg. driving score : 57.130278平均驾驶分数：57.130278 Avg. route completion : 93.035833平均路线完成：93.035833 Avg. infraction penalty : 0.608259平均违规罚款：0.608259 Collisions with pedestrians : 0.050574与行人的碰撞：0.050574 Collisions with vehicles : 0.101963与车辆碰撞：0.101963 Collisions with layout : 0.127259布局冲突：0.127259 Red lights infractions : 0.000000红灯违规：0.000000 Stop sign infractions : 0.010444停车标志违规：0.010444 Off-road infractions : 0.011185越野违章：0.011185 Route deviations : 0.000000路由偏差：0.000000 Route timeouts : 0.011981路由超时：0.011981 Agent blocked : 0.039926座席阻塞：0.039926 Average Speed (m/s) : 1.005835平均速度（m/s）： 1.005835