Placement is the most controversial of these: it confers the least information about "difficulty" of a bracket path, it gives higher seeds a much easier path to the same placing, it is the most reliant on luck, and it disproportionately punishes players who end up in the path of a top seed who loses early. The data-minded panelists hate placement; it's seen as a lazy shorthand for a player's performance which skirts the issue of determining the relative value of all of their wins. And yet, there is still something appealing about it. If you look at just a single player, it is strictly better to place higher compared to placing lower. Your opponent in any given part of bracket is unimportant, because the expected reward will go up if you win and it will go down if you lose. In a sense, it is a proper scoring rule, where you have no real incentive to "hedge" your prediction, if you are just trying to predict placement.

Placement, ultimately, is something which I consider to be a generally bad metric. It is, however, not a useless metric, and I think a useful comparison point comes from something I consider to be a generally good metric which has some weird edge cases.

Head-to-head, by comparison, is a generally uncontroversial metric. The logic for head-to-heads is very simple: wins against strong players are very valuable, losses against weaker players are quite bad. If your bracket path involved you beating a bunch of very strong players, then it was good even if your placement wasn't as impressive. If you imagine a relatively stable seeding list and vaguely try to simulate the outcomes of 100 tournaments in your head, you end up having to answer a lot of questions like "how often will Mango beat Hungrybox?" in order to determine how well everybody would do on average. With plentiful data, this seems obviously like the metric to use, and it is therefore usually the metric of choice for people who spend every ranking period looking at spreadsheets.

This is all well and good, but there is unfortunately a slight issue with head-to-heads as a metric: it isn't actually what any of the competitors are at the tournament to do. One very easy example of this from 2022 was Hungrybox, whose records you can see in the below crosstable:

Hungrybox has what I consider to be the clear "worst" head-to-heads this year, with just one positive record and one even record, the rest losing. However, you will notice that Hungrybox is an absolutely mind-boggling 11-1 against Jmook this year, which played a relatively large role in the fact that Hungrybox's placements this year were quite competitive within the 2022 season.

Even putting aside tournament victories (which I will get to in a moment), Hungrybox is a near-perfect illustration of why head-to-heads do not tell the complete picture, and about how in extreme cases they can be misleading. In a world where head-to-head is the only thing people care about, Hungrybox is actually highly incentivized to throw sets against Jmook in order to get more chances to correct his poor records against other players. Absolutely nobody looking at head-to-head would notice any difference between 11-1 and 10-2, but the chance to turn that "aMSa" box from red to yellow is very valuable. If he loses this set, no harm done, the box was already red anyways, and the green box is still green. If he wins the set, his head-to-heads look great now.

This would, of course, be very stupid. Nobody is showing up to tournaments in order to correct their tournament records against specific players. They are there to defeat whichever opponent is placed in the chair next to them, at any given moment. Head-to-heads provide the most accurate picture of how well players do against each other, but it is important as a panelist to juggle that with the fact that players are not there to improve their set records - nay, they largely can't even control which opponents they get to play in the first place, other than by throwing on purpose.

Most people I talk to have an opinion that Hungrybox is the clear fifth this year, in 2022. As head-to-head oriented as "nerd" panelists typically are, it is clear that he is not fifth because he has the fifth best set records. There is something else going on here.

I would be remiss if I did not talk about arguably the most important metric of all - winning the actual tournament. This is something which in my experience usually gets more visibly reflected in the end-of-year list compared to the summer list, but it is one which is important enough such that players often point to this as the biggest source of "incorrectness" about the list³ - specifically that this is not considered important enough by the nerds.

In some sense this is the most "proper scoring rule" of them all - there's no way to game the metric since "getting first place" is a completely binary variable, unlike placement or set counts. Notably, this is mostly irrelevant for the vast majority of the list – most top 100 players will not win any majors, so you're strong-armed into using one of the earlier two metrics. However, it is an important additional factor when it comes to evaluating the very top of the list, so I'll talk a bit about it here.

There are actually two elements to this which I want to briefly outline, since it actually is a bit more nuanced than it sounds. The two elements are roughly what I will refer to as "accumulated prestige" and "ability to win", which are correlated but distinct things which are both captured by this umbrella of "tournament victory".