The Wason 2-4-6 Task is a famous reasoning test used to demonstrate confirmation bias. The goal of this task is to infer the hidden rule, which will take three ordered numbers and return either True or False. They are told that the test case (2,4,6) is True, and that they can submit any number of test cases in order to test the output of the rule. Participants often seek to submit tests which confirm their initial suspicions about the rule (e.g. (4,6,8) -> (10,12,14) -> The rule is "numbers ascending separated by two"), despite the true rule being "any three numbers in ascending order".

To be good at the Wason 2-4-6 Task, you need to be able to form hypotheses and then specifically target the edges of that hypothesis, in order to falsify your own suspicions. As a result, it's a nice shorthand for testing inductive logic – I find that this test is an effective way to teach younger people about what kind of mindset is useful for research work¹. I have been wanting to design a super-simple reasoning metric for LLMs which is resistant to memorizing lots of solutions, and landed upon this as a potential solution. Code for this project can be found on github here.

It is not possible to "confirm" the rule in the Wason 2-4-6 Task. This is the most important property of the task.

At nearly any stage in the process of the task, there always remains an arbitrarily higher resolution which could be disproven. Take the canonical "any three values ascending" rule used in the original task. Even in the ideal situation where you quickly catch on to the general rule, there are many variants of the simple rule which could disprove your hypothesis. For example:

• Any three natural numbers, ascending ((-1, 0, 1) fails)
• Any three integers, ascending ((1, 2, 3.5) fails)
• Any three rational numbers, ascending ((1, 2, pi) fails)
• Any three numbers, ascending, where none of the values are 0 ((0, 3, 5) fails)
• Any three numbers, ascending, where the first number does not equal 30
• Any three numbers, ascending, where the first number does not equal 31
• …

What is nice about this task is that you have to optimize your final response for how well it explains your observations vs how reasonable the rule is. You are probabilistically arriving at a conclusion which describes the function, rather than somehow being able to completely uncover it using only black-box observations. To arrive at a probability of 1, you would need an infinite amount of evidence, but to simply arrive at a very high probability, you need to seek out high-information test examples.

Here we introduce the Wason Inductive Logic Test (WILT) which is a procedurally generated reasoning benchmark modeled after the Wason 2-4-6 task. For each example, the model has up to 30 attempts to maximize the number of correct answers, ideally using the fewest number of test cases necessary. We will include a set of 2 versions: a "canon" version with 100 pre-selected functions, and a way to generate simple functions to create a "procedural" version robust to memorization effects.

The value of this benchmark is easy to understand: to get a good score the model must successfully model what it does and does not know, it must be able to generate plausible hypotheses, it must be able to pursue refuting those hypotheses, and is must make a judgement call on when its degree of support for a hypothesis is sufficient to guess the rule.

Scoring the benchmark is straightforward. We report three values: Accuracy, Average Test Count, and Total Points. Points are determined by awarding 1000 points for each correct answer, and \(100 * (1 - \frac{X}{M})\) bonus points are awarded for each question, where \(X\) is the number of guesses taken on that question, and \(M\) is the number of allotted guesses (default 30). This way the model is rewarded for correct answers, but two models with similar accuracy can be differentiated in the event of one having much better test case selection.

Starting this out we will just show the results of several models on 10 very easy tests, and update this with a scaled up longer run on more tests later. Here's the very easy mini version of this benchmark:

TESTS = {
    '1': lambda x, y, z: x > y > z,
    '2': lambda x, y, z: x < y < z,
    '3': lambda x, y, z: x >= y >= z,
    '4': lambda x, y, z: x <= y <= z,
    '5': lambda x, y, z: x == y and y == z,
    '6': lambda x, y, z: x != y and y != z, and x != z,
    '7': lambda x, y, z: x < 0 and y < 0 and z < 0,
    '8': lambda x, y, z: x + y == z,
    '9': lambda x, y, z: x * y == z,
    '10': lambda x, y, z: x < y and y > z
}

Here we test some models on this task. Llama 3, Mixtral, and Gemma were done via the groq api. OpenAI models are called via the OpenAI api. DeepSeek is called via the DeepSeek API using the openai client. Anthropic is called via the anthropic API.

In general, almost all of the failure points tracks with one of three things:

1. Never encountering any evidence (e.g. no Trues or Falses, usually from selecting poor tests)
2. Overcomplicating the Rule and not eliminating hypotheses well
3. Tunnelling down test cases for things it already knows are true or false (confirmation bias)

A lot of the individual "character" of each of the LLMs comes out when you test them this way. The behaviors that pop out are really interesting.

A very funny thing I'm noticing about all of these models is that a common point of failure for all of them is Occam's Razor, and that I think a lot of them struggle to identify what "the simplest hypothesis that fits the data" means in the context of the observed data. I think there's a lot of noteworthy things here:

First, I think that cracking this is going to be a really important capability if these models are supposed to be able to actually perform meaningful scientific tasks. I think doing productive work in the sciences is often about this sort of hypothesis rejection, and identifying this as a capability models specifically struggle with is a valuable barometer for how "intelligent" the models are.

Second, I think that vibe evals are underrated in general. This was roughly 20-30 API calls each across 10 tests that produced an almost exactly expected full ordering of all the models I tried. You could argue this list could be shifted around based on better prompt engineering, more test examples, harder tests, easier tests, etc. But it's important to remember that this is a weekend project's worth of evaluation and not a full academic paper².

Overall I am hopeful that these sorts of models will be someday able to identify highly plausible hypotheses and select effective tests for evaluating them. I think if we get there, it's not that much further away for models to generate "good ideas". I think that would be a sight to see.