When I was in high school I remember reading Harry Lorayne's The Memory Book, and using it to learn to memorize a deck of cards in about 35 minutes of memorization time. It was nice to learn about the basic memory techniques, and they were helpful for my studies (especially when I needed to learn ordered lists). I also read Josh Foer's Moonwalking with Einstein, which was a nice glimpse into the world of competitive memory sports.

However, this all always seemed like a lot of work just to remember things. I eventually forgot how to do everything in these books, never having really made a huge effort to practice them to begin with.

But in pursuit of hacking away at the idea of LLM-powered knowledge management software that could "make you remember things", I cracked open a few of these books again, and made an effort to learn them in earnest to see how viable this would be. I'm having a lot of fun in this rabbit hole, and I'll probably continue to update this document as I read a few more of the books I have ordered about this topic. I think I've learned a lot of very useful things about how my own head works, and I've done some fun experiments using these techniques which I think is potentially worth sharing.

I pretty much immediately dove into creating a PAO system, since I could envision a lot of different use cases for spots where I would want to use three numbers. The way PAO works is that for each number from 00 to 99, you come up with a person, an action, and an object. This way, you can memorize groups of 6 digits by taking the person from the first pair, the action from the second pair, and the object from the third pair. To memorize cards, you repurpose 52 of these as representing playing cards, and you place them in a specific order using a memory palace. This was the system prominently featured in Moonwalking with Einstein, so I understood how it worked and didn't need that much help to get off the ground.

My PAO pegs can be found here. Most of these were made on a long road trip with my wife; it was a fun word game to kill time ("Who is an anime character whose name starts with 'M'? What can we give Michael Jordan for Object, given that Kuroko has a basketball already? etc"). Since this involves creation 300 unique images, this took a very long time, and it still takes me a moment to really remember what the associated words are both in encoding and decoding. But, it's surprisingly straightforward once you get going!

Making the sheet took a couple days of on-off idle thoughts / asking chatGPT for ideas when I gave up. I practiced memorizing cards for about two hours before I could do it while looking at the sheet. After four or five attempts, I was able to use this to get a deck of cards memorized in just over 8 minutes, with no errors.

The big "reason" I went down this rabbit hole again was to explore the question of if it would be possible to somehow write software which would "make me remember things". To reiterate, this is slightly different from software which would "let me practice remembering things", which is what the multitude of very good flashcard applications are useful for. What I was specifically after was exploring whether or not it was possible to use these techniques in order to directly put something into my memory, to reduce the friction involved in encoding things after reading them.

I, of course, do not think that this sort of thing should fly in memory competitions, in the same way using a chess engine is not allowed in a chess tournament. However, remembering things is a useful thing people supposedly need in their day-to-day life. If I could demonstrate the possibility of a computer program making something easy for me to remember extremely quickly, that is cause for further investigation.

Here's a super simple example: a python program which will randomly shuffle a deck of cards, print them to screen, pull from my PAO pegs and encode every necessary group, and print those also. With no cost of encoding, I can just memorize the already-encoded inputs, stop the timer, and then decode them as normal.

There's a very funny analogy to be made here with diffusion models. In the early days of those models, they would slowly add random noise to images, and then train a model to remove that random noise step by step until they arrived back at their original image. To make things more efficient, stable diffusion would encode the image into latent space with a variational autoencoder (VAE), and then perform the diffusion process with the noise directly upon the latents. They found that they could do this and it was pretty much the same as doing it directly on the pixels, but required a lot less processing power. What I'm doing here is sort of similar, in a way – I'm hoping to skip the step where I have to do all this "learning of information by encoding and decoding" by directly learning encoded versions of information. The gamble here is that I can learn well with a decoder-only method, if the way I do this encoding is well-constructed enough.

And here I think we get a pretty convincing validated hypothesis. I make 0 errors and complete the memorization phase just 14 seconds shy of the International Master of Memory qualifying time, as a complete beginner. The decoding still takes kind of a lot of time, since I am so new at it, but it seems pretty plausible with this self-test that we can offload some of the encoding costs for learning new information to some sort of software responsible for using some a priori known encoding scheme².

It is also validating to run this experiment and to compare it to the world record time, which was about 20 seconds the last time I checked. This method I used essentially skips the encoding cost, and the format does not time the decoding cost. If there was no difficulty in "placing" the information in memory, I should be theoretically able to read the generated images and then instantaneously stop the timer. The fact that I cannot do that, that I have to try to see the image and then put it down in mental space somewhere, is a good indicator that this middle step is what is really being tested here.

Creating PAO pegs is a pretty high up-front cost, so I don't know if this sort of thing is ever going to have huge market capture. But all sorts of useful things can be encoded as images like this: 2D (Person-Action) and 3D (Person-Action-Object) 100x100(x100) grid locations, numbers, letters, playing cards, ordered lists, etc. LLMs know what these systems are (they have been around for thousands of years after all, and vector databases can be used to show it anything it doesn't already know), so something which can immediately spit out something using your selected pegs + encodings could be a super useful study paradigm given the right angle.

A belief I have is that the most human thing in the world is to do something nobody else cares about.

Something I see often surrounding the topics of memory training and mnemonics is that it is useless. And after engaging with it somewhat, I can say that becoming skilled at memorizing arbitrary objects in random orders is something which sounds much more useful than it actually is. You can argue about all of the things I've written above, that my insights about encoding costs or creation of systems or whatever are all things I could have learned without wasting 10 hours staring at playing cards, or arguing with my spouse about whether "Phelps" is more memorable than "Federer". I think all of this is probably true. In a world hyper-optimized for useful insights per unit time, I think there are likely better ways to spend your time.

But you could say this about a lot of things. You could very easily say this about competing video games, which is something I've done for years, which has led me to meet my spouse, let me befriend the best friends I've ever made, made me try harder than I've ever tried at something before, and got me a great job in a field I'm passionate about. To call something pointless usually ignores the broader context, which is that we can assign meaning to any arbitrary task we can imagine. It's possible to go through your entire life only doing things which are beholden to market forces, only retaining interest in things which the broader population finds equal value in compared to yourself.

That is one way to do things, I suppose.

But I do think sometimes it's nice to stand in the sun, and to feel the warmth and the breeze. Sometimes it's nice to simply feel yourself being alive, capturing those fleeting sensations that one day you will lose access to for all eternity. Sometimes it's nice to feel like a child again, and to lie to yourself about how swinging cool fallen tree branches around will be of enormous practical help to you if you're ever attacked by bad guys. In the end, it's not really about the practical utility, it's about the act of swinging.

If you disconnect from things the appropriate distance, I think you'll find that there's no real reason to do things that you do not like. And, likewise, that there's no need to make excuses to do those things you do like. Thinking something is cool is reason enough.