Math Is an API Spec

01 Opening

MAYASam, I'm going to show you one line from a paper, and you tell me what the code does. A big sigma, a little i equals one underneath, an n on top, and then $x_i w_i$ .

SAMHonestly? It looks like a ransom note. A Greek capital letter, some subscripts, no types, no declarations. I'd send it back in code review.

MAYAHere is the same thing in Python. Picture two lists of numbers with the same length, called x and w. Start a running total at zero. Walk down both lists together, multiply each pair of matching items, and add that to the total. In code: total equals zero; for i in range n, total plus equals x of i times w of i.

SAMOh, so it's just: multiply two equal-length lists item by item, then add it all up. A loop with an accumulator. If x is one, two and w is three, four, that's one times three plus two times four: eleven.

MAYAExactly. Math calls that a dot product, and you'll meet it everywhere. It's even the core calculation inside a neural network, which we'll build later in the course. And that's the thesis of this module: mathematical notation is a programming language. It's very compressed, very old, and it has no compiler.

SAMSo when I read a paper, I'm the compiler.

MAYAYes. Today we install that compiler in your head for the four constructs on almost every page of every machine learning paper: sets, functions, sums and products.

02 Quick recall

SAMWhat do I need to know already?

MAYAOnly things you use every day: types, function signatures, loops with an accumulator, and comprehensions. Everything in this episode is a new name for one of those.

SAMSo I'm learning syntax, not concepts.

MAYASyntax, plus one idea about culture. Jeremy Kun, in A Programmer's Introduction to Mathematics, argues that mathematics has a culture the way programming does, with idioms, legacy conventions and in-jokes. Once you know the culture, the notation stops looking hostile.

03 The phenomenonLanguage

MAYALet's start with language. Any language system needs a vocabulary: every word, or piece of a word, that it knows. Each of those units is called a token, so the vocabulary is all the tokens the system knows. Mathematically, that's a set.

SAMLike a Python set. No duplicates, no order.

MAYAExactly. A tokenizer is a function from strings to sequences of token IDs: text goes in, a list of integers comes out. Each integer points to one entry in the vocabulary, the fixed set of every token the model knows, numbered so that each token has its own ID. That set is the V on screen, and the vertical bars around V mean its size, here fifty thousand tokens. So Seq of V just means "a list whose items come from V". The arrows on screen read "goes from this set to that set"; we'll take that notation apart properly in a few minutes. A language model is a function from a context, meaning the list of token IDs so far, to a probability for every possible next token: one number per vocabulary entry, all adding up to one. That's Dist of V, short for a distribution over V. In Python terms: the tokenizer takes a str and returns a list of ints; the language model takes that list of ints and returns a dict from each token ID to its probability.

SAMSo the tokenizer's output is literally the model's input. The context is just the token IDs so far.

MAYARight. And this whole course is a story about increasingly clever functions between sets.

SAMWhere does it get hard?

MAYATake the word bank. Try writing a function from words to meanings. "Bank" has to go to the side of a river and also to the place that holds your money.

SAMOne input, two outputs. That's not a function, that's a bug. My dictionary would have to store a list.

MAYARight, and that observation matters. A function must give exactly one output per input, so meaning as a function of the word alone fails. How would you fix it without breaking the rule?

SAMChange the input type. Meaning is a function of the word and its context. "Bank" next to "river" goes one way, next to "loan" the other.

MAYAThat's the fix the field converged on. On screen, "words cross contexts" means the input is now a pair: a word together with its context. We'll meet that cross symbol properly in a minute. Older systems give each word one fixed list of numbers, the same list wherever the word appears; those are called static word vectors. Modern systems compute a fresh list of numbers for each word from the sentence around it; those are called contextual representations, and they're what a transformer produces, the kind of model behind today's chatbots. That change of signature, from word alone to word plus context, is roughly the step from the first kind to the second. You'll build both later in the course, in the embeddings module and the transformers module.

SAMSo it's like overloading in C++ or Java? One name, say print, and which version runs depends on the arguments you pass. Mention the name alone, without its arguments, and the compiler can't tell which one you mean. "Bank" without its neighbours is that ambiguous name.

MAYAYes, and virtual dispatch is even closer: the same method call does different things depending on the object it lands on at run time. Or think of a void pointer in C: the raw address means nothing until a cast tells you how to read it. A token ID is like that. The tokenizer looks only at spelling, never at meaning, so "bank" gets the same ID whichever bank you mean; the dependence on context comes later, from the model, which we'll build later in the course. And here's where the analogy breaks. Overloading picks one of a few versions that someone wrote by hand, chosen by type. A modern model computes a fresh list of numbers for each occurrence of "bank" by blending in information from the other words in its context. So there aren't two banks. There are endlessly many shades of "bank": a river bank, a bank on a loan form, a blood bank.

SAMSo the type signature isn't decoration. Choosing it is the modelling decision.

MAYAVery often, choosing the domain is the research idea.

04 Building the mathMath & data structures

MAYAOne piece at a time. A set is a collection of distinct things, written in curly braces, like the set containing cat, dog and fish. "x is in A" is written $x \in A$ , with a curvy E.

SAMThat's x in A . Python even uses the same word. And the slashed version on screen, "cow is not an element of A", is just that check coming back False.

MAYANext is set-builder notation: $\{x^2 : x \in \{0,\dots,9\},\ x \text{ even}\}$ . The colon reads "such that".

SAMLet me try. That's a set comprehension. Curly brace, x times x, for x in range ten, if x mod two is zero.

MAYAPerfect, and in both the expression comes first. One difference to remember: a set forgets duplicates. The set of x squared, for x from minus two to two, has only three elements, zero, one and four. Minus two and two both land on four.

SAMSo it's always a set comprehension, never a list comprehension.

MAYATwo more set facts. Vertical bars, $\lvert V \rvert$ , mean the size of the set. In a paper, that's usually the vocabulary size.

SAMLen of V.

MAYAAnd the Cartesian product, $A \times B$ , is every ordered pair, one element from A and one from B.

SAMA nested loop yielding tuples.

MAYAYes. The real numbers, written with a double-struck R, are every point on the number line, decimals included; floats are our approximation of them. The plane is $\mathbb{R} \times \mathbb{R}$ , written $\mathbb{R}^2$ , all pairs of real numbers. A vector, a word you'll hear constantly, is for now just a fixed-length list of numbers, so a two-dimensional vector is just a pair.

MAYANow functions. We write $f : A \to B$ , "f goes from A to B". A is the domain, the input type. B is the codomain, the output type.

SAMdef f(x: A) -> B . I'd have guessed that.

MAYAHere's what you wouldn't guess. Formally, a function is a set of pairs, a lookup table. Clive Newstead's book names three conditions that make a table a legitimate function. Totality: every input in the domain has an output. Existence: that output really lives in the codomain. Uniqueness: no input gets two different outputs.

SAMTranslation. Totality failing is a key error on a valid input. Existence failing is returning a string when the signature says int. Uniqueness failing is nondeterminism, the same call giving different answers.

MAYAThat's better than most textbooks. A definition that passes all three is called "well-defined", and when you write definitions in a paper, you're expected to check them.

MAYANow sums. The capital Greek sigma means "add these up". Underneath is where the index starts, i equals one. On top is where it stops, n. To the right is the summand, the thing being added.

SAMAccumulator starts at zero, loop from one to n, add the summand each time.

MAYAFrom one to n inclusive. Math ranges include both ends.

SAMAnd Python's range one to n stops before n. So it's range one to n plus one. I can already feel the off-by-one bugs.

MAYAIn my experience, they're the most common bug in research code that translates formulas. The index i exists only inside the sum, like the loop variable of a comprehension, which doesn't exist outside it, so you can rename it to k and nothing changes. Mathematicians call that a bound variable.

SAMWhat if the range is empty, i from one to zero?

MAYAYou tell me.

SAMThe loop body never runs, so I return the starting value. Zero.

MAYAThe empty sum is zero. Capital pi means "multiply these". What's its starting value?

SAMOne, because multiplying by one changes nothing. So the empty product is one.

MAYAAnd that is the whole reason zero factorial equals one. n factorial is the product one times two times three, all the way up to n, so four factorial is twenty-four. Zero factorial is the product of nothing at all: the empty product. It isn't a special rule. It's the starting value of the loop.

SAMThat's genuinely satisfying. The empty loop returns the identity of its operation.

MAYASubscripts. $x_i$ is $\mathtt{x}\lbrack i \rbrack$ . Two subscripts on a grid of numbers called M, $M_{ij}$ , means row i, column j. Papers usually index from one, so the M sub two three on screen is M of one, of two in Python, which counts from zero. Conditions can also sit under the sigma, like "k less than a hundred, k odd".

SAMA generator with an if clause.

MAYALast piece is argmax. Max returns the biggest output. Argmax returns the input that produced it. Ties go to the first one, like NumPy.

SAMSo argmax is the key of the dict entry with the largest value, not the value.

SAMIf a model scores the, cat and sat at zero point two, zero point seven and zero point one, max gives zero point seven, and argmax gives "cat".

MAYAAnd plenty of models in this course end with exactly that argmax. A model that picks a label, like spam or not spam, scores every label and returns the argmax; that kind of model is called a classifier. And the simplest way to generate text is to score every possible next token, take the argmax, append it, and repeat; that's called greedy decoding.

05 The data structureMath & data structures

SAMWhere do these live in memory? A math set has no hash function, no mutability, no lookup cost. It just works.

MAYAKun makes exactly that point. When we implement one, we choose a structure. A vocabulary is a hash set of strings, with constant-time membership on average. But it really needs two maps, token to ID and ID back to token.

SAMA bidirectional dict.

MAYAMathematically, what property does the token-to-ID map need for decoding to work?

SAMNo two tokens can share an ID, or I can't go back. And every ID should be used, or I'd decode garbage from a hole.

MAYAThose are called injective and surjective, and having both is a bijection. A bijection is exactly a function with an inverse: a second function that undoes it, the way ID-to-token undoes token-to-ID. Deeper point: when the domain is finite, any function can be stored as a table. Give each of fifty thousand tokens its own fixed list of d numbers, where d is however many numbers per token we choose. Store those lists as the rows of one grid, fifty thousand rows by d columns, as on screen. That table is called the embedding matrix, and each token's row of numbers is its embedding.

SAMAnd when the domain is infinite, like all strings?

MAYAThen you can't store the table, so you store an algorithm. Code is how we represent functions whose table is too big to write down.

06 Code walkthroughCode

MAYAThe lab's reference code. Sam, narrate the first one.

SAMSum to n. Total starts at zero, i runs from one through n inclusive, we add i and return the total. So sum to four is one plus two plus three plus four, which is ten. Product to n is the same with one and times: start at one and multiply instead, so product to four is one times two times three times four, twenty-four. Nothing clever.

MAYANothing clever is the point. The notation was hiding a loop you've written a thousand times.

MAYAThe interesting one is "is function". It takes a list of input-output pairs, a domain and a codomain, meaning the allowed inputs and the allowed outputs. It scans the pairs while building a dictionary from input to output, and each new pair either fits the dictionary built so far or is rejected on the spot, with a reason. One failure, totality, can't be caught pair by pair, so after the scan it checks that every domain element got an output.

SAMSo an input outside the domain is rejected. An output outside the codomain means existence failed. A repeated input with a different output means uniqueness failed. And at the end, any domain element missing from the dictionary means totality failed.

MAYAAnd it returns a reason string, not just false. A test that only says "false" teaches nothing. Last is the dot product, the multiply-matching-pairs-and-add loop from the start of the episode. Before the loop, we check that the two lists have the same length, and raise a value error if not.

SAMWhy bother? Zip stops at the shorter one.

MAYAThat's exactly why. Zip silently truncates: it pairs items up only until the shorter list runs out. On screen, one, two against one, two, three: zip pairs one with one and two with two, quietly drops the three, and returns five. So a length-two times length-three dot product quietly returns a plausible number. The notation assumes both have length n. That's part of the type, so we enforce it.

07 Look insideCode

MAYARun the file. What do you see first?

SAMSum to a hundred prints five thousand and fifty, matching the closed form, a formula that gives the answer directly with no loop: $\frac{n(n+1)}{2}$ . Product to five prints a hundred and twenty. Then the empty cases: product to zero prints one, sum to zero prints zero.

MAYANext, the sum of odd squares below a hundred, a hundred and sixty-six thousand, six hundred and fifty. It's computed twice: once as a filtered generator, once with an Iverson bracket, a one-or-zero switch: a condition in square brackets that equals one when the condition is true and zero when it's false, so multiplying a term by it either keeps the term or drops it. The two must agree. Then "is function" on one-to-a, two-to-b, one-to-c prints false, with "uniqueness fails: input 1 maps to both a and c".

SAMAnd in the notebook?

MAYAYou plot the running total as a staircase of bars beside that closed-form curve, and they coincide. The notebook's corpus is a toy we wrote, "the cat sat on the mat. the dog sat on the log.", and you call describe() from our inspection library on its word lengths, and it prints a one-line summary: the shape, which for a plain list is just how many items it holds, fourteen here, because the two full stops count as tokens too; the data type, here whole numbers; the mean, meaning the average; std, a measure of spread you'll meet in the probability episode; then the min, the max and the first values.

SAMAnd then a real sentence?

MAYAThe opening line of A Tale of Two Cities: its vocabulary as a set, its size, and an argmax over word counts. Real text has the same types as the toy.

08 Origin story

SAMWho invented the sigma?

MAYAAccording to Graham, Knuth and Patashnik's Concrete Mathematics, Joseph Fourier introduced the delimited form, index underneath and limit on top, in 1820, and it took the mathematical world by storm. The notation f of x is credited to Leonhard Euler in the eighteenth century. The equals sign is older: Robert Recorde introduced it in 1557, with two parallel lines because no two things can be more equal.

SAMAnd sets?

MAYASets as the foundation of mathematics are young, late nineteenth century, with Georg Cantor. There was an early crash: allowing any collection you can describe to count as a set leads to contradictions. The famous one is Russell's paradox: take the set of all sets that don't contain themselves. If it contains itself, it doesn't, and if it doesn't, it does, so it would have to both contain itself and not contain itself. So mathematicians had to restrict what counts as a set. The nugget is that Iverson bracket. Kenneth Iverson invented it for his programming language APL, and Knuth's group adopted it in Concrete Mathematics. That's notation travelling from a programming language into mathematics, the opposite of the usual direction.

SAMAnd young Gauss adding one to a hundred instantly?

MAYAA well-loved anecdote whose details vary with each telling, so treat it as folklore. The trick is real, though. Can you reconstruct it?

SAMPair the ends. One plus a hundred, two plus ninety-nine, all the way in. Fifty pairs, each worth a hundred and one, so five thousand and fifty. That's the closed form, and it turns a loop into one multiplication.

09 Where it breaks

SAMMy "why not just": why not descriptive names? "Sum over indices of input times weight" would be readable.

MAYAKun's answer is that math is written on blackboards in real time and rewritten over and over during a derivation. With long names, the structure disappears after the third rewrite. One letter plus a convention is faster: n counts, i indexes, f is a function. The price is that you must learn the conventions, and letters are reused across fields.

SAMOverloading, basically.

MAYAWith a rule attached. Kun notes that you only reuse plus for something that behaves like addition. Now the failures. You list the ones you'd hit.

SAMOne, translating sigma with range and forgetting the top is inclusive, so the loss, the single number that scores how wrong a model is, comes out one term short and still looks plausible. Two, a set-builder translated as a list keeps duplicates, so the vocabulary size is wrong. Three, the zip truncation we just saw.

MAYAAnd a fourth, which is reading equals as assignment. In math, x equals y is a claim that both sides are the same, more like an assert than a store.

SAMAnd the empty mean?

MAYAThe mean, the plain average, is $\frac{1}{n}\sum_i x_i$ . At n equals zero, that's undefined, so the empty sequence is outside the domain. Returning zero would be a plausible lie, so raise an error.

10 Real world and research

MAYAEvery paper you'll reproduce, from Vaswani's attention to DeepSeek's latent attention, defines its model as a composition of typed functions and writes its loss as a sum over tokens. Attention is the step that lets each token look at other tokens in the text, and you'll build both of those versions later in the course. On screen is that loss, the error score, for a language model. Script L is the loss. The sigma loops over positions t, from one up to capital T, the length of the text. $x_t$ is the token at position t, and $x_{<t}$ means all the tokens before it. So $p(x_t \mid x_{<t})$ is the probability the model gave to the token that really came next, given everything before it. The "given" bar gets its full story in the probability episode, and the log and the minus sign in the log-space episode. For now, read it as: one penalty per token, added up. Read sigma, subscripts and arrows fluently, and you can read the method section.

SAMAnd for research?

MAYATwo threads. Choosing domain and codomain is often the idea itself, like word versus word-in-context. And formal mathematics: proof assistants, programs that check a mathematical proof the way a compiler checks code, such as Lean, check every definition for exactly this well-definedness: totality, existence and uniqueness.

SAMAnd getting language models to translate informal math into that checkable form is an active research area.

MAYAIn both, notation is the API, and you're learning to read the spec.

11 Recap

MAYATo recap. A set has no duplicates, and set-builder notation is a set comprehension. A function from A to B is a typed signature backed by a table that is total, lands in the codomain, and is unique. Sigma is an accumulator loop starting at zero, pi starts at one, and the empty versions return those identities. Subscripts are indexing, shifted by one. Three questions to answer out loud. What three conditions make a table of pairs a function, and which bug matches each? Why is an empty sum zero and an empty product one? Why does "bank" break meaning-as-a-function, and what fixes it?

12 Your turn: the lab

SAMAnd the lab?

MAYAThe first exercise: translate ten formulas into Python, covering sums, products, filtered sums, set-builder, Cartesian products, the mean, the dot product, a double sum, the function checker and argmax.

SAMWhat do I predict before running anything?

MAYAThree things we did not compute today. What should the minimum of an empty set return, if it has to behave like the empty sum and the empty product? How many elements are in the image of x squared over minus three to three, meaning the set of all the outputs it actually produces? And for one-to-a, two-to-a, with domain one, two, three, which condition fails? Then do the break-it: replace the length check in dot with plain zip, and predict the number it returns for three-four against five-six-seven.

SAMA plausible number, without a word of complaint. That's the scary part.

MAYANext time, vectors are arrays and matrices are functions, and the embedding table gets its shape.