Python closures and the late-binding trap

Closures and the cell mechanism · late binding · what a closure keeps alive

RECALL that in ten minutes, then go deeper)*

The sentence to own: a closure is a function plus the cells of the variables it uses from an enclosing scope — and a cell is a live, shared, heap-allocated box, not a snapshot.

Day 030 showed you the bug. Today is the mechanism, the memory consequences, and the three patterns you will actually use — because tomorrow's decorators are nothing but closures with syntax.

---

Part 1 — Cells, seen directly

def counter(start=0):
    n = start
    def inc():
        nonlocal n
        n += 1
        return n
    def peek():
        return n              #    SAME cell — inc and peek SHARE the variable
    return inc, peek

inc, peek = counter()
inc(); inc(); peek()          #   2

inc.__closure__               #   (<cell at 0x...: int object at 0x...>,)
inc.__closure__[0].cell_contents          #    2 — watch it change as you call inc()
inc.__code__.co_freevars      #   ('n',)  — names this function takes from an enclosing scope
counter.__code__.co_cellvars  #   ('n',)  — names this function PROVIDES to inner functions
inc.__closure__[0] is peek.__closure__[0]     #    True — one cell, two functions

   WHAT THE COMPILER DOES — and this is a COMPILE-TIME decision, like Day 030's locals:
    1. it sees an inner function referencing `n`
    2. ⇒ `n` becomes a CELLVAR in the outer function instead of an ordinary local
    3. ⇒ the value lives in a heap-allocated CELL object, not in the frame's fast-locals array
    4. ⇒ the inner function object carries a reference to that cell in __closure__
    ⇒    SO the variable OUTLIVES the call, and every closure over it sees the SAME box.

"Sees the same box" is the whole of late binding. The loop-variable trap (Day 030) is not a timing accident — there is literally one cell, and every lambda holds a reference to it.

---

Part 2 — What a closure keeps alive

def make_handler():
    huge = load_500mb_dataset()
    def handler(x):
        return x * 2                 #    never touches `huge`
    return handler
#   FINE: `huge` is not a freevar, so no cell is created and it is freed when make_handler returns.

def make_handler_bad():
    huge = load_500mb_dataset()
    def handler(x):
        return x * 2 if huge else 0  # ⚠️⚠️    now `huge` IS captured
    return handler
# ⇒    the 500 MB stays alive for as long as ANY reference to `handler` exists.

   THE MEMORY RULE: a closure keeps its captured cells alive, and a cell keeps its object alive.
    ⇒   so a long-lived callback that captures a request object, a database session, a
      response body, or a big frame keeps ALL of it alive — for the life of the callback.
    ⇒    In practice this shows up as: registered event handlers, cached decorated functions,
         and asyncio tasks that were created and never awaited (Day 071).

⚠ The capture is per-name, not per-object, and it is decided by the compiler — so you cannot "capture just the field I need" by accident. If you only need huge.count, bind it to a local first (count = huge.count) and let the closure capture the small thing:

def make_handler_fixed():
    huge = load_500mb_dataset()
    count = len(huge)                #    capture the int, not the dataset
    def handler(x): return x * count
    return handler                   #   `huge` is freed when this returns

Cells and cycles (Day 027): a closure that refers to the object holding it — a method stored as an attribute on the instance it closes over — is a reference cycle, collectable only by the cyclic GC. Common in callback registries and observer patterns.

⚠ Threads: a cell is shared mutable state, so two threads calling inc() above race on n += 1 exactly as they would on a global (Day 067). A closure is not a thread-safety mechanism.

---

Part 3 — The three patterns worth knowing

#    1. THE FUNCTION FACTORY — parameterised behaviour without a class
def between(lo, hi):
    return lambda x: lo <= x <= hi
in_range = between(10, 20)

def retry(times, delay):                      #   tomorrow this becomes a decorator
    def run(fn, *a, **kw):
        for attempt in range(times):
            try: return fn(*a, **kw)
            except Exception:
                if attempt == times - 1: raise
                time.sleep(delay * 2 ** attempt)
    return run

#   2. PRIVATE STATE WITHOUT A CLASS
def make_account(balance=0):
    def deposit(n):
        nonlocal balance; balance += n; return balance
    def get(): return balance
    return deposit, get
# ⇒    `balance` is genuinely inaccessible from outside — more private than a leading underscore.

#    3. LATE BINDING, USED DELIBERATELY — the config that can change
def make_logger():
    def log(msg):
        if CONFIG["debug"]:      #    read at CALL time, not at definition time
            print(msg)
    return log
# ⇒   late binding is a FEATURE here: changing CONFIG changes behaviour with no re-registration.

Late binding is not a bug — it is a default that is wrong in loops and right almost everywhere else. Being able to say "I want the current value, so I capture it with a default argument; I want the live value, so I let the closure read it" is the mark of understanding rather than pattern-matching.

Closure vs class vs partial

	Closure	Class	partial
State	in cells	in self	pre-bound arguments
Inspectable	awkward (__closure__)	yes	yes (.func, .args)
Picklable	no	yes	yes
Several operations	clumsy	natural	no
Best for	one function + a little config	a thing with behaviour	fixing arguments

⚠ "Closures are not picklable" is a real constraint, not trivia. It is why multiprocessing (Day 068) and Celery cannot send a lambda or a closure to a worker — the AttributeError: Can't pickle local object you will meet is exactly this. functools.partial over a module-level function is picklable, which is why it is the recommended shape there.

---

Common mistakes

Mistake	Correction
Expecting a closure to snapshot the value	It captures the cell. Use x=x to snapshot.
Thinking late binding is always a bug	It is the right default outside loops.
Capturing a large object accidentally	Bind the small thing you need to a local first.
Forgetting nonlocal when mutating	Rebinding creates a new local (Day 030).
Treating a closure as thread-safe	A cell is shared mutable state.
Sending a closure to a process pool	Not picklable. partial over a module-level function.
Using a closure for several related operations	That is a class.
Not realising callback closures create cycles	Day 027 — the cyclic GC handles it, eventually.

---

Interview questions

Q: What is a closure, precisely?

A function object that carries references to variables from an enclosing scope. The compiler detects that an inner function reads an outer local, promotes that local from the frame's fast storage into a heap-allocated cell, and stores a reference to the cell on the inner function in __closure__. So the variable outlives the call that created it, and every closure over that name shares one cell — which is exactly why late binding works the way it does.

Q: Why does the loop-variable trap happen, mechanically?

Because there is one cell, not one per iteration. Every lambda created in the loop holds a reference to that same cell, and reading it happens when the lambda is called — by which point the loop has finished and the cell holds its final value. Snapshotting with a default argument works because defaults are evaluated at definition time and stored on the function object rather than read from a cell.

Q: Can a closure cause a memory problem?

Yes, and it's easy to do accidentally. A closure keeps its cells alive, and a cell keeps its object alive, so if the inner function references a large object anywhere in its body — even in a branch that rarely runs — that object lives as long as the closure does. Long-lived event handlers, registered callbacks and cached decorated functions are where it shows up. The fix is to bind the small piece you actually need to a local before defining the inner function, so the closure captures the integer rather than the dataset.

Q: Closure, class, or partial?

A closure for one function with a little configuration — a predicate, a formatter, a retry wrapper. A class when there are several related operations, or the state needs a name and wants to be inspected. partial when I'm purely pre-binding arguments to an existing function. The deciding factor is often picklability: closures and lambdas can't be pickled, so anything that needs to cross a process boundary — multiprocessing, a Celery task — has to be a module-level function, optionally wrapped in partial. That Can't pickle local object error is one of the more confusing ones the first time you see it.

Q: Is late binding a design flaw?

No — it's the right default in most places, and only wrong in a loop. Reading the current value at call time is what lets a closure see updated configuration without being re-registered. The problem is specifically that a loop creates many closures over one variable, which is a case where people expect per-iteration capture. Knowing which behaviour I want, and having a way to get each — a default argument to snapshot, plain capture for live reads — is more useful than calling either one a bug.

---

Mini task

1. Build counter() returning inc and peek. Confirm they share one cell with is on __closure__[0].
2. Print cell_contents after each inc() and watch the box change.
3. Print co_freevars and co_cellvars for both the inner and outer functions.
4. Write make_handler_bad, keep the returned function, and measure the retained memory with tracemalloc. Then fix it by capturing only the small value.
5. Remove nonlocal from inc and read the UnboundLocalError.
6. Try to send a closure to a ProcessPoolExecutor. Read the pickling error. Then do it with partial over a module-level function.
7. Build the "live config" logger and change CONFIG after creating it. Late binding as a feature.
8. Write the private-account closure and confirm the balance is unreachable from outside.

---

Exit questions

Answer aloud, no notes.

1. What is a closure, in terms of cells?
2. When is the decision to create a cell made?
3. Where do you look to see the captured values?
4. Explain the loop trap mechanically — why "one cell" is the whole answer.
5. Why does a default argument snapshot the value?
6. What does a closure keep alive, and where does that bite?
7. How do you avoid capturing a large object?
8. Why is a closure not thread-safe?
9. Why can't a closure be pickled, and where does that matter?
10. Closure vs class vs partial — one line each.
11. Give a case where late binding is exactly what you want.

Articulation drill

Record two minutes: "Explain closures, and give one place they cause a real problem."

Mechanism first, briefly: the compiler notices an inner function using an outer local, moves that local into a heap cell, and hands the inner function a reference to it. So the variable outlives the call, and all closures over that name share one box.

Then pick the memory problem rather than the loop trap, because everyone says the loop trap: "the one that has actually cost me time is retention. A closure keeps its cells alive and a cell keeps its object alive — so a long-lived callback that mentions a large object anywhere in its body, even in a branch that never runs, keeps that object alive for as long as the callback is registered. It looks like a leak and there is no obvious owner. The fix is to bind the small value you need to a local before defining the inner function, so the closure captures an integer instead of a dataset."

Close on the practical constraint: closures cannot be pickled, so anything crossing a process boundary needs a module-level function — which is the real reason partial exists alongside lambdas.

---

Previous: Day 042 · Tomorrow: Day 044 — decorators: the pattern, functools.wraps, and decorators with arguments