Developing a ranking system - looking for feedback

[El Dude]

Hi folks,
So you're asking, what's El Dude up to now? Well as many/most of you know, I like dabbling with different systems to rank players with (obviously ). While I realize that no system will ever be perfect, the quest for the "perfect system" goes on - or at least the best system I can come up with. I've come up with simple and more complex systems, and right now I'm working on a revision of a more complex approach that is adaptable enough to be simple.

One of my main concerns is being able to compare players across generations. This becomes more and more problematic the further back you go, as information becomes sparser. For instance, we only have computer rankings from 1973 onward - and even after that, there's a weird glitch in the early 80s where the rankings are all messed up for a few years. Or as another example, if you look at John McEnroe's titles on Wikipedia it says that 23 are "high category" tournaments (equivalent to today's ATP 500) and 1 are "low category" (ATP 250), yet in the list that follows it doesn't differentiate the two. On the other hand, on Ivan Lendl's list of titles it does differentiate them. So if I am to come up with a formula that gives more points for high vs. low category tournaments on a year-to-year basis, this will benefit Lendl over McEnroe because the info is more complete. What I'd like to be able to do is select which categories and criteria I can compare with. When comparing two players one should only compare with information that exists for both players.

Anyhow, what I'm hoping to get out of this thread is feedback. I'll ask various questions and hope that some of you erudite folks will weigh in. I may or may not heed your advice, but I will consider everything.

What I'll do is ask specific questions which I will put in bold. I do this only to differentiate my questions from other conversation that may arise (the more the merrier). I will continue to post other "official" questions in bold as the thread develops.

Thanks for any participation you might offer....now onto the questions, with a few to start.

I'm looking at different areas to include within my formula, in both "broad" categories and "narrow" sub-categories. The broad categories are: Slams, Other Tournaments, and Rankings, each of which has numerous sub-categories that I won't go into now (I'll get there later). My first question is this: How would you weight the importance of these three broad categories relative to each other?

Secondarily, is there a difference in how these categories are weighed in terms of historical greatness versus current/contextual ability?

One more, of a different nature. In terms of historical greatness, how much do you weigh peak level vs. career longevity/career results? Obviously both are important, but how do you weigh them? This could be called the "Borg vs. Connors Problem."

I look forward to your responses!

 

[mrzz]

Slams > Rankings > Other tournaments, I guess most will agree with that.

IMO Slams > 1.5 * Rankings > 1.5 * Other tournaments.

Random toughts:

As a suggestion about how could you weigh in tournaments back in the day (the AO problem), using a hint given by yourself, you could calculate the average ranking of the seeds. If it falls bellow a certain threshold value, it would not count as a "true" slam. Easier said than done...

But anyway you cannot use this the evaluate the "value" of the tournament directly. Nowadays, for example, the field is basically the same for a MS1000 and for a slam. But as most of them as percieved as important tournaments, no one is missing them.

But maybe the MS from the current era can give some insight, as there seems to be some hierarchy between them. I mean, clearly Miami, Indian Wells and Rome > Monte Carlo, Madri, Cincy and Rogers Cup > Shangai and Bercy.

But if you find, for example, AO was really not a "trully" slam before 1987 (anyway, what about the 60's?), so players in that era has less slams per year, putting them in disadvantage with the current crop. Maybe their slams should count more...

 

[El Dude]

Great thoughts, mrzz, thanks. Good point about the field being the same for Slams and Masters. Not sure how that impacts weighing the pre-1987 AO, though. As I said in another thread, the field before 1987 was about as deep as current ATP 500s - there were anymore from 1-3 elite players, but other than that everyone was outside the top 10.

Anyhow, my impulse is to treat the AO before 1987 similarly as Amateur and Pro Slams - in a similar category as Masters and year-end finals. So we'd have something like this:

Tier 1: Open Era Slams except AO before 1987
Tier 2: Pro Slams, Amateur Slams, Open Era AO 1969-85, Masters (and equivalent), Year-end finals (various including WTF, Year-end Championships, Grand Slam Cup, etc)
Tier 3: ATP 500 and equivalent, maybe Olympics
Tier 4: ATP 250, maybe Davis Cup

Or something like that. There might be more specific point differences, but those are the general groups as far as I see it. I really can't justify equating amateur or pro Slams as akin to Open Era Slams, but I'd probably rank them higher than Masters.

 

[mrzz]

Every time I see your posts about this kind of subject, and thus think of it, I see how difficult the problem is. I agree with your stratification, I would only add that, in order to the cross eras comparison to be fair, one constraint you must have is that the total points avaialable for the year is roughly the same (basically what I meant about the "old" slams, that is, if you have only three "real" slams per year, they have to have more value).

I know that the number of tournaments fluctuate, but having some kind of grand total would allow you to compare different seasons with a bit more fairness. In a nutshell, a perfect year in 2015 must be equivalent to a perfect year in 1984, whatever tournaments are played back then.

 

[GameSetAndMath]

Great thoughts, mrzz, thanks. Good point about the field being the same for Slams and Masters. Not sure how that impacts weighing the pre-1987 AO, though. As I said in another thread, the field before 1987 was about as deep as current ATP 500s - there were anymore from 1-3 elite players, but other than that everyone was outside the top 10.

Anyhow, my impulse is to treat the AO before 1987 similarly as Amateur and Pro Slams - in a similar category as Masters and year-end finals. So we'd have something like this:

Tier 1: Open Era Slams except AO before 1987
Tier 2: Pro Slams, Amateur Slams, Open Era AO 1969-85, Masters (and equivalent), Year-end finals (various including WTF, Year-end Championships, Grand Slam Cup, etc)
Tier 3: ATP 500 and equivalent, maybe Olympics
Tier 4: ATP 250, maybe Davis Cup

Or something like that. There might be more specific point differences, but those are the general groups as far as I see it. I really can't justify equating amateur or pro Slams as akin to Open Era Slams, but I'd probably rank them higher than Masters.

I would IPTL in between Tier 1 and Tier 2 :snicker

 

[El Dude]

Every time I see your posts about this kind of subject, and thus think of it, I see how difficult the problem is. I agree with your stratification, I would only add that, in order to the cross eras comparison to be fair, one constraint you must have is that the total points avaialable for the year is roughly the same (basically what I meant about the "old" slams, that is, if you have only three "real" slams per year, they have to have more value).

I know that the number of tournaments fluctuate, but having some kind of grand total would allow you to compare different seasons with a bit more fairness. In a nutshell, a perfect year in 2015 must be equivalent to a perfect year in 1984, whatever tournaments are played back then.

Yeah, its a pain in the ass - I don't know why I do this to myself. :lolz: I guess I get some kind of sick pleasure out of it!

I also like your view about equal points across different years - I've thought of this but not sure how to approach it, other than in a very tedious way! But I think if I'm going to take away from the AO before 1987, I should add to the other three Slams.

So for instance, let's say current Slams have a value of 15 points for a win each, so 60 total. If we go back to 1980, let's say I'm reducing the AO to 9 points - which is probably about right, maybe even too generous. I could then distribute those 6 points to the other three Slams, so they are 17 points each. Of course a Slam in 1980 is not necessarily more competitive than one now, and in fact it could be less overall as tennis has a much deeper international field now than it did then (like little "inconsequential" nations like Serbia), but it gives recognition to someone like Borg, for whom winning two Slams in 1980 is probably more meaningful than winning two Slams is now.

This is also why I devised "dominance shares" in that other thread about the Big Four's dominance. It translates the 14 big tournaments into 100 total points, with Slams worth 14 points, WTF worth 8, and the nine Masters 4 each. Those aren't perfect numbers, but they equal 100! I couldn't figure out another way that worked as well. But this sort of system is different than what I'm trying to do here. It looks only at titles, not performance in non-wins - which are important too.

 

[El Dude]

I would IPTL in between Tier 1 and Tier 2 :snicker

LOL. Well, on a serious note, I was thinking about how to value the Grand Slam Cup. At the time (the 90s) it wasn't part of the ATP with any points or title value, although later it would be added in to the title count. That said, it had the highest pay-out - so I imagine the competition was quite fierce. I actually decided to count it as equivalent to the other year-end championships, although that may be considerably over-valuing it.

 

[Great Hands]

The problem of tournaments changing in value is not limited to the AO. Like the AO, winning Olympic Gold used to be not very prestigious, but now all the top players play in it and really want to win it, and value it second only to slams I would imagine. I certainly see Olympic Singles Gold as the fifth most prestigious title in tennis now, but it didn't used to be. I would certainly place it much higher now than an ATP 500, as you have placed it above.