Notes on SRS
Table of Contents
- Notes on Spaced Repetition
Notes on Spaced Repetition
Draft: v1.0 | Posted: 6/8/2018 | Updated: 6/8/2018 | confidence of success: 85% | estimated time to completion: 3/5/2018 | importance: Medium
What is Spaced Repetition?
Spaced Reptition is a method of efficiently reviewing flashcards by scheduling them based upon how easily you remember something. The gist of the idea is that it's much easier to remember something in your long-term memory than your short-term memory, so if you are trying to memorize a large body of information you should prioritize things you don't know very well and things you are just about to forget. Below is a good video explaining the concept and the relevant research behind it.
The cool thing about spaced repetition is that it allows you to review and manage an extremely large body of information with relatively little daily review1. I use Anki to review flashcards, but other people use Mnemosyne which works just as well. I use this little application a lot to remember pretty much anything I think is worth remembering, so I figured it would be useful to write up some of my notes from using it.
Spaced Repetition as a Deep Reading Tool
I am a chronic skimmer - I often read books very quickly, consume a lot of books in a short window, and then forget everything in them after a short while. My first solution to this was very detailed reviews on goodreads. It didn't help me much in committing my reading to memory, but at least I was able to go back and reread my notes as a refresher whenever I needed to. I find that I have relatively weak retention when I read2, so spaced repetition (more specifically, the use of a single, uncategorized deck separated by tags rather than decks) was a gamechanger for me in this respect - I started reading things with my phone open next to me (I shelled out the money for Anki's ridiculously overpriced app, and have since recouped the value spent on it by using it for literally hundreds of hours), and making a short flashcard for any detail or quote I felt like I wanted to remember. This, in turn, let me do three things:
A: It allowed me to absorb as much as I wanted from my reading, down to the level of direct quotes
B: It allowed me to realistically understand how much I could read in a day / a single sitting
C: It allowed me to better engage with the material I was reading, since I'd actually grasp details instead of just the concepts
The second point is very important, because it's one that I struggled with a great deal for a very long time. I'd often sit down with a theory-dense book and just start reading it, as if it was a novel, more or less straight through. I'd inevitably find myself not really grasping things, and it'd eventually become a huge pain to read it. Making flashcards of important details in a book as I went allowed me to roughly see how much information I was actually absorbing, via my daily reviews. Somehow I was surprised to realize that my absorption rate for more difficult books was worse, and that I ought to be reading those slower3 (for example, a chapter of Tae Kim's Guide to Japanese Grammar wasn't so bad, but a chapter of Introductory Statistics with R was very difficult and those cards frequently gave me a hard time). This ended up being a reasonably useful tool I could use to adjust my studying habits - if I was going through a book and it felt too easy, I could try studying more each day; if I felt like it was hard, I could go slower and give myself more time to think about the ideas.
The third point was pretty useful because actively reading almost anything while actively stopping to make flashcards makes it rather difficult to zone out while reading, which is a bad habit I had for a long time. I'd read a page, look back up, and realize that I was so distracted that I didn't really understand the literal last page I had just read. Actively making flashcards as I read made it so that I had to actually be reading when I was reading, which did wonders for keeping me on-task during study time.
Spaced Repetition as Removal of "XP Waste"
Old School Runescape Players have a wonderful concept that they refer to as "XP waste". Playing Runescape is mostly about maximizing your experienced gained per unit time (the game doesn't exactly involve much else), so players will often calculate how much experience they ought to gain performing a certain task (e.g. "if you cut this type of tree for an hour, you should gain 10,000 experience per hour"). However, there are many reasons why you might not actually hit that amount of experience in one hour - trips to the in-game bank, going to the bathroom in real life, deciding to take a nap, getting bored, etc. Therefore, the difference between experience gained and theoretical experience possible is referred to as "XP waste" (i.e. "you could have earned this much more experience but you wasted your time and lost out on it"). Granted, people have differing levels of commitment to eliminating XP waste (e.g using better training spots vs optimizing for global timer vs not sleeping), but the overall idea I think is pretty interesting.
The nice thing, I find, about spaced repetition, is the low effective time cost to actually doing it. Even studying a good amount each day (50+ new cards, plus reviews) will only swell your review time up to ~45 minutes per day, and that sounds much worse than it actually is, compared to something like Exercise or Meditation. Doing 45 minutes of exercise feels like a good chunk of time, since it needs to be a solid 45 minute block of your day in order to be useful. But Anki can be done at any time, for any amount of time. Waiting for your eggs to cook? Do a few flashcards. Walking to class? Do a few flashcards. On the bus home after work? Do a few flashcards. You can't exercise in 8 randomly spaced 150 second chunks throughout the day, but you can do flashcards like that4. Spaced repetition requires some time, but it can come from any time in your day, including dead time. This is time you normally would have spent doing literally nothing, and it ends up making "XP waste" moments in your life into ones that actually have a purpose. Doing flashcards is just free experience, and making your default nervous phone habit "open Anki" instead of "open twitter" will essentially make this a zero-impact habit on your day-to-day routine.
Useful Habits for Using Spaced Repetition
I think being good at using anki is a weirdly not trivial thing, and it's something I have been continually working on as I've dumped hundreds of hours into this program. Here are some useful tidbits.
Making Anki a default part of your learning
Interrupting confusion
- Every time you look something reasonably useful up (e.g. "wait, what does this word mean?" Or "didn't they do a study on football causing CTE?") make a flashcard about it (usually takes 20 seconds)5
- Associate feeling of being bored and waiting for something with the act of doing flashcards
Reading
Given that anki lets you remember things, it's popular among people who do a lot of reading and want to remember things they read.
Michael Neilsen has a great article about using anki to augment long term memory, in which he gives the following strategy for reading something:
- Do a very quick read / skim of it, and make a handful of flashcards for the key points you pick up
- Do the above 5-6 times, each time going a bit deeper
- Attempt a full thorough read, and make flashcards for the important details
I find this works quite well for research papers.
Likewise the supermemo people have an idea they call "Incremental Reading" which is a feature which lets them dump thousands of articles into supermemo, lets them read through articles a few sentences at a time, and make cloze deletion cards of sentences they deem important. The claim here is that you can read "thousands of articles at once".
Incremental Reading is definitely a bit too manic for me to actually do with an anki addon, but there are a few good points to this idea which I think are worthy of implementing into your own reading practice:
- Keep a big list of things to read, read all over the place, don't tunnel vision too hard on any one thing
- If you are getting frustrated reading something, it's perfectly fine to read something else for a little bit
- Collecting information on the same topic from multiple sources helps resolve contradictions, juxtaposing them randomly next to each other helps resolve these sorts of conflicts (this is just a general point in favor of spaced repetition)
- You don't have to read everything all the way through to get the most important benefits out of them, it's not "giving up" if you've already gotten most of the juice out of it.
- One-sentence extract cloze deletions are a super easy way to make cards of something you are reading.
Card Creation advice
Card Atomization
Minimum Information Principle
Another point about SRS which has been around since the 90s is the minimum information principle which is also called "atomization" in some anki circles. (Neilsen, med students).
Basically, it's much easier to do five cards on a single topic rather than remembering five things in one card. Splitting cards into their smallest atomic units makes reviewing more pleasant, faster, easier, sexier, etc. Longer flashcards are frequent leeches for me, and usually end up deleted or refactored sooner or later.
Critically, it's okay if you have multiple cards on the same thing. Neilsen gives the example of memorizing the unix command "ln -s filename linkname" by instead making two cards, one for "ln -s" and one for the order of "filename linkname". Beginners (and myself, early on) often make cards which really should be ten cards rather than one. From supermemo comes the following example:
Q: What are the characteristics of the Dead Sea?
A: Salt lake located on the border between Israel and Jordan. Its shoreline is the lowest point on the Earth's surface, averaging 396 m below sea level. It is 74 km long. It is seven times as salty (30% by volume) as the ocean. Its density keeps swimmers afloat. Only simple organisms can live in its saline waters
compare to the following
Q: Where is the Dead Sea located?
A: on the border between Israel and Jordan
Q: What is the lowest point on the Earth's surface?
A: The Dead Sea shoreline
Q: How long is the Dead Sea?
A: 70 km
etc.
Sets and Enumerations are almost always bad
unless you specifically want to remember a small list of things, "what are the six topics described in X book" is an annoying card to get while reviewing. It makes you stop going through cards as you mentally go through all of them, and you have to make sure you aren't double counting anything in your head, and you end up getting it wrong even if you know the answers for this sort of reason.
"Visualize"
- Use images often, and make drawings when applicable - they can be terrible ms paint drawings, if you don't plan on sharing them then nobody will judge you for your art and it will help you remember.
- Just doing "visualize diagram explaining X from book Y" is usually good enough for the card to be useful later, making the card memorable and useful is all you really need so don't go crazy making detailed contexts for all the image cards
Generally Avoiding Card Islands
I have noticed when I am doing flashcards sleep deprived that I sometimes glaze over a card and do not really absorb the meaning of certain questions, but think something like "oh it's this card, and the answer to this card is Krebs Cycle". Remember that you are trying to learn facts about the world, not the answer to your flashcards. Remembering the answers based on some meta-reasoning about your deckbuilding itself is a shortcut and in cognitive science goes by the name "unintended cue learning".
This usually is a symptom of that card being the only thing in your deck which is even remotely like that thing, which means you're not getting an actual representation of the knowledge "around" that item. Likewise, once you notice this, you can't really stop it. You can't try to "honestly remember it" if you are already aware of the answer.
Another example: Q: "This south asian dynasty was responsible for building the taj mahal"; A: "Mughal Empire". Perhaps a useful thing to know, but you'll get this card every single time unless you also have some other south asian dynasties in the mix there somewhere.
I think the solution to these cards is adding, rather than subtracting. In a sort of adversarial training, something you can do if you find a certain topic too lacking in your deck is go out and find some other things around it that might make remembering it more difficult.
Deckbuilding / actively reviewing
Mixed Practice and The Single Deck
Use one big deck, and separate things by tags instead.
There's a number of things that make this better than using separate decks, in my view, one of the big ones just being research6. People don't typically do this by default because, well, it sucks at first. In a study about massed practice vs subject-interleaved practice, students typically self-reported that they felt less successful during their studying while doing interleaved practice, as opposed to massed practice. The punchline here is that the mixed practice students outperformed the massed practice students, despite feeling like they got worse practice7.
There's a certain kind of mental flexibility that mixed practice requires which seems to help generalization, and that by itself would be a good reason to do it. But generally speaking I think it's also both more fun and more likely to actually stick with it.
With regards to it being more fun, it's hilarious to get a card about Grace Bedell immediately after being asked to translate a Japanese sentence. It keeps you on your toes and it makes you feel like you know lots of things, and it makes it feel more like you're on a weirdly specific quiz show rather than studying for an exam or something.
With regards to productivity, there's explicit value in being unable to tell ahead of time what the content of your reviews will be. When I used multiple decks I would regularly not feel like doing certain decks and then the reviews for that deck would pile up, which would make it worse (I'm fine as long as I study all these other subjects, right?). Having only a vague number of cards you need to study makes the pain of starting comparatively much lower.
Reviewing Practices
- If you encounter a card that sucks, or a card which needs to be refactored into multiple smaller cards, just do it then and then delete the card.
- Don't add things you don't care about, or you will not want to actually do your reviews. I have made it a point to delete anything I come across during reviews that I decide is a badly made card or a card on something I don't care about.
Some Musings on Finding Things to Learn
Quizbowl as a Content Aggregator
Something I think is fun and heavily applicable to spaced repetition is Quizbowl, or more broadly "trivia"8
I was "decent" at quizbowl in high school, where my geographic location played a large part of the lack of serious competition, but after being admitted to Yale (at the time arguably the best/second best collegiate team, period, with Matt Jackson spearheading the team) I played in a few collegiate novice tournaments my freshman year of college, found out I was terrible, and promptly quit the team to pursue competitive Super Smash Brothers Melee instead.
Fast forwards to my realization that it is possible to remember things you read for more than a week after you read them, and suddenly I felt extremely motivated to hit up quizbowlpackets.com. The nice thing about spaced repetition is that you can memorize tons of things, but the weird thing about it is that the question "what do you want to memorize?" is surprisingly hard to answer. Robb Seaton puts it well:
"Or here’s a common hangup people have, and that I had, when starting with spaced repetition. It’s the question, 'What ought I memorize?' and people think, well, maybe the presidents or something, because that’s what they’ve associated memorization with. It’s the wrong question. Ask 'What’s interesting?' and start ankifying that." 9
What's really exciting about quizbowl is that it's quite literally structured to provide you with useful information. You're rewarded on a question-by-question basis for knowing more obscure things about a topic, and you're rewarded on a round-by-round basis by knowing more answers than your opponent. I'm not eligible for collegiate tournaments and I'm not skilled enough to solo open tournaments, so you might ask "why care about quizbowl?" The answer here lies in utilizing quizbowl packets as cool shit aggregators. By its nature, a good quizbowl question contains a range of clues about a topic, and if you don't know about something then a quizbowl question can serve as a unit of 3-6 things about it (quizbowl players typically advocate reading packets as a method of improving at the game). Knowing about random things, even on a somewhat shallow level, often leads me to some of my most interesting projects, which in turn makes my knowledge of those things less shallow. Viewing it from this lens, playing quizbowl is like reading a manic RSS feed of snippets cut out of textbooks, lowering the chance that you spend lots of time on something boring, and making it easy to find new things worth reading about. And its pretty fun!10
edit May 2020: I don't do this much anymore, but that's mostly just because my learning is a lot more directed these days – I still am fond of this comparison and I think it's a great way to look for cool things to learn.
Closing
I plan to periodically add to this page as I come up with useful thoughts on flashcarding, but overall I think that using flashcards is one of those "Real-Life Cheat Codes" that I wish I found out earlier. It's one of those things like Counting Calories or Index Funds that is actually capable of producing large changes with minimal effort, and I think things like that deserve some good documentation even if it isn't entirely complete.
Footnotes:
Further reading http://www.gwern.net/Spaced-repetition
I generally consider myself more slanted towards fluid intelligence, rather than crystallized intelligence, which makes SRS a really useful tool for me.
This point is rarely, if ever brought up in SRS circles, but there's actually some adaptive learning literature on modeling the forgetting curves of words in spaced repetition systems, with more difficult words having harsher forgetting curves. Some cards are genuinely harder than others, although generally speaking in time it peters out.
Speaking of exercise, there's some fun papers on exercise + study dual task settings, usually involving studying on an exercise bike. It seems to not harm performance on tests despite feeling harder (sounds similar to massed vs mixed practice) and also can make exercise feel easier since people typically pedal harder if they are distracted by some other task during stationary biking. Food for thought.
my rule of thumb is thinking of gwern's estimation of a card roughly equaling 5 minutes of time and thinking "am I willing to stare at this for five minutes if it means memorizing it"
On top of that, since I always have emacs open these days I have a (very very basic) capture template that creates an anki card which makes the process pretty straightforward - no matter what I'm looking at I can do Win-3 C-c c a and be in the process of making a card.
("a" "anki basic" entry (file+headline "~/Dropbox/org/logs/added_anki.org" "Basic")
"* know :deck: \n** Item :note: \n\t:PROPERTIES:\n\t:ANKI_NOTE_TYPE: Basic\n\t:ANKI_TAGS: \n\t:END:\n*** Front\n%?\n*** Back\n")
I view this as a fun case of The Illusion of Explanatory Depth, but specifically cultivated. Jumping from subject to subject makes it more likely that you will feel a shallow explanation or vague answer would be correct, followed by immediate confirmation that you were wrong. It's very jarring, and feels bad, but it's specifically what you are trying to avoid. I don't know if this is actually grounded, it's just how I internalize the feeling of doing mixed cards.
this word is sort of frowned upon in quizbowl circles, since quizbowl tries to put a strong emphasis on, for lack of a better term, "useful academic knowledge", rather than "arbitrary memorized facts". "Trivia" typically refers to information that offers no reason why anybody would know it (e.g. "Nikola Tesla was born on July 10, 1856"), whereas pop culture questions are affectionately referred to as "Trash"; and typically get regaled to side tournaments or the odd 1-2 questions per game. Further reading.
Robb Seaton uses anki for, as far as I can tell, literally everything - he puts little factoids about people he likes in his deck, so that he can memorize stuff like "My coworker's favorite color" or "someone's favorite type of cheese" which strikes me as a little much, but knowing people's birthdays certainly seems like a great example of a nontraditional use of anki.
for more information see Nana Maru San Batsu