Styjun

I'm working on a rating algorithm. I have a set of exercises. They are all categorized in levels (1 = easiest, 5 = hardest).

Users get shown two exercises and should decide which one is harder or if both are equal. Based on user ratings, the levels should get adjusted.

What I've done:
I experimented with the Elo rating.

My Questions:
Are there any better algorithms for doing this use case? (found nothing so far)

Thanks in advance and cheers.
Toby

I would try to solve the problem in a simple yet (I hope) effective way.

First, you only update an exercise rating when the vote is different that what the system actually expects. From now on, I will only considers the cases where the user output differs from what the system actually expects.

Second, I would give more weight to the votes where the two levels have a big difference. A wrong expectation on two esercises with rating 2 and 3 should have less impact than a wrong expectation on two exercises with rating 1 and 5.

That said, my algorithm would be along the lines of:

1- A constant percentage is set, let's call it increment. It establishes the percentage of impact that a vote has, and can be modified along the way based on the number of users.

2- For an "unexpected" vote, I would calculate the difference between the original levels (minimum of 1).

diff = max(1, abs(ex1.level - ex2.level))

3- I would update each exercise rating by a percentage, based on the multiplication of increment and diff.

if (ex1 level expected bigger)
 ex1.rating = ex1.rating + diff*increment;
else
 ex1.rating = ex1.rating - diff*increment;

Rating would be a float, and level would be the rounding of rating:

ex1.level = round(ex1.rating)

Example:

let's set increment = 0.1. exA, with a rating of 2.0 and level 2 is compared with exB, rating of 3.0 and level 3.

The first user selects exB as the hardest. Nothing changes, because it is the result expected by the system.

The second user selects exA. It is not the expected result. The difference between the two exercises is 1, so the rating is modified by a factor 1*0.1 = 0.1,
resulting in a exA.rating = 2.1 for exB.rating = 2.9