Submission Evaluation

Each time you submit an agent, it is run for one episode to be evaluated. Every submission will generate a score, used for leaderboard positioning, and will unlock achievements.

The score is a function of both the total cumulative reward and the submission difficulty you selected at submission time, which can be either “Easy”, “Medium” or “Hard”. Every game has a different difficulty level scale, so a specific mapping is applied and is represented by the following table:

The relation that links score with total cumulative reward and difficulty is shown in the picture below. When “Easy” is selected, the score is exactly equal to the total cumulative reward. When “Medium” (or “Hard”) is selected, the score is obtained multiplying the total cumulative reward by a weighting value that varies linearly with the total cumulative reward obtained. It is equal to 1 if you obtain the lowest possible total cumulative reward (i.e. same score as if “Easy” was selected), and it is equal to the ratio between the game difficulty level for “Medium” (or “Hard”) and the game difficulty level for “Easy” if you obtain the highest possible total cumulative reward.

So, for example, for Dead or Alive ++, the weighting values for “Medium” and “Hard” vary linearly between

$$ \begin{equation} \begin{gathered} k_M = \left[1.0, \frac{3}{2} \right] = \left[1.0, 1.5 \right] \\ k_H = \left[1.0, \frac{4}{2} \right] = \left[1.0, 2.0 \right] \end{gathered} \end{equation} $$