Submission Evaluation

Each time you submit an agent, it is run for one episode to be evaluated. Every submission will generate a score, used for leaderboard positioning, and will unlock achievements.

The score is a function of both the total cumulative reward and the submission difficulty you selected at submission time, which can be either “Easy”, “Medium” or “Hard”. Every game has a different difficulty level scale, so a specific mapping is applied and is represented by the following table:

GameEasyMediumHard
Dead Or Alive ++234
Street Fighter III468
Tekken Tag Tournament579
Ultimate Mortal Kombat 3345
Samurai Showdown 5468
King of Fighters ‘98468
Marvel VS Capcom468
X-Men VS Street Fighter468
Soul Calibur345

The relation that links score with total cumulative reward and difficulty is shown in the picture below. When “Easy” is selected, the score is exactly equal to the total cumulative reward. When “Medium” (or “Hard”) is selected, the score is obtained multiplying the total cumulative reward by a weighting value that varies linearly with the total cumulative reward obtained. It is equal to 1 if you obtain the lowest possible total cumulative reward (i.e. same score as if “Easy” was selected), and it is equal to the ratio between the game difficulty level for “Medium” (or “Hard”) and the game difficulty level for “Easy” if you obtain the highest possible total cumulative reward.

So, for example, for Dead or Alive ++, the weighting values for “Medium” and “Hard” vary linearly between

$$ \begin{equation} \begin{gathered} k_M = \left[1.0, \frac{3}{2} \right] = \left[1.0, 1.5 \right] \\ k_H = \left[1.0, \frac{4}{2} \right] = \left[1.0, 2.0 \right] \end{gathered} \end{equation} $$

Scoring as a function of Total Cumulative Reward and Submission Difficulty