PythonPython internals13 min read

Python generators and lazy evaluation

Generators, yield and yield from: how lazy evaluation keeps memory constant over a file you could never fit in RAM.

Day 47 of the 488-day pathway. Published in full — nothing held back.

Generators — lazy evaluation, constant memory, yield from

The sentence to own: a generator function does not run when you call it. It returns a generator object, and the body executes one yield at a time, resuming where it left off.

This is the highest-value day in the functional block. "Process a 50 GB file in a service with 512 MB of RAM" is a normal backend requirement, and generators are the entire answer — plus they are the mechanism async is built on (Day 048).

Part 1 — What `yield` does

def count_to(n):
    print("starting")
    for i in range(n):
        yield i                 #    SUSPEND here, hand out i, remember everything
    print("done")

g = count_to(3)                 #    NOTHING PRINTS. No code has run.
next(g)                         #   "starting", then 0
next(g)                         #   1     — resumes INSIDE the for loop
next(g); next(g)                #   2, then "done" and StopIteration

   THE MECHANISM: the presence of `yield` anywhere in the body makes it a GENERATOR FUNCTION.
    calling it creates a generator object holding a SUSPENDED FRAME —
    local variables, the instruction pointer, the exception state — all preserved on the heap.
    ⇒    next() resumes the frame; yield suspends it again.
    ⇒   a generator IS an iterator (Day 046): it has __next__ and __iter__ returning self.
    ⇒    SO it is single-pass, and everything from Day 046 about exhaustion applies.

"A suspended frame on the heap" is the answer that shows you know what is happening. A normal function's frame dies on return; a generator's survives between yields, which is why local state persists with no object and no attributes.

g.gi_frame.f_locals      #   the live locals of the suspended frame
g.gi_running             #   is it currently executing
inspect.getgeneratorstate(g)     #   GEN_CREATED / GEN_SUSPENDED / GEN_RUNNING / GEN_CLOSED

Part 2 — The memory argument

# ⚠️    EAGER — everything in RAM, twice:
def load(path):
    return [transform(line) for line in open(path).readlines()]

#    LAZY — one line at a time, constant memory, regardless of file size:
def load(path):
    with open(path, encoding="utf-8") as f:      #   Day 037
        for line in f:
            yield transform(line)

   THE NUMBERS: a 50 GB log file
    readlines()  ⇒    50 GB+ of str objects ⇒ OOM-killed
    generator    ⇒    ONE line at a time ⇒ a few KB, forever
  ⇒   and the FIRST RESULT arrives immediately, rather than after the whole file is read.

#    PIPELINES — each stage is lazy, and the whole chain uses constant memory:
lines    = (l for l in open(path, encoding="utf-8"))
parsed   = (parse(l) for l in lines)
errors   = (p for p in parsed if p.level == "ERROR")
recent   = (e for e in errors if e.ts > cutoff)
for e in recent:                    #    NOTHING has been read until this line runs
    alert(e)

The composition is the point: four transformations, no intermediate lists, and each record travels the whole pipeline before the next one is read. This is the Unix pipe model, in Python — and it is also why the pipeline can process an infinite stream.

⚠ The trade you must be able to name: generators give constant memory and early first results, but no len(), no indexing, no re-iteration, and they are slightly slower per element (a resume is more expensive than a list index). And debugging is harder, because the code runs interleaved with the consumer rather than in one place.

Part 3 — `yield from`, and returning a value

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)      #    delegate — not `for x in flatten(item): yield x`
        else:
            yield item

def chain(*iterables):
    for it in iterables:
        yield from it

   yield from IS NOT JUST A LOOP SHORTHAND. It also forwards:
    · send()   — values pushed IN (Day 048)
    · throw()  — exceptions injected
    · close()  — shutdown
    ·    and it CAPTURES the subgenerator's RETURN VALUE:

    value = yield from sub()      ⇒    `value` is whatever sub() RETURNED

  ⇒    THAT is why `await` could be built on it (PEP 380 → PEP 492). The delegation
       machinery IS the coroutine machinery.

def counter():
    total = 0
    for x in data:
        total += x
        yield x
    return total                 #    becomes StopIteration.value — NOT yielded

#   You only see it via `yield from`, or by catching StopIteration:
try:
    next(g)
except StopIteration as e:
    e.value                      #   the returned value

A bare return in a generator just stops it (raises StopIteration). ⚠ Returning a value is invisible to a normal for loop, which is why it is rarely useful outside delegation.

Part 4 — Lifetime, cleanup, and the resource trap

def read_rows(path):
    f = open(path)                    # ⚠️    who closes this?
    for line in f:
        yield parse(line)
    f.close()                         #    NEVER REACHED if the consumer stops early

for row in read_rows(p):
    if row.bad: break                 #    the generator is abandoned, suspended, file open

   WHAT HAPPENS WHEN A GENERATOR IS ABANDONED:
    when it is garbage collected, Python throws   GeneratorExit into it at the yield point.
    ⇒   so a `finally` or a `with` DOES run —    but only WHEN it is collected (Day 026),
         which under a reference cycle or on PyPy may be much later, or at shutdown.
    ⇒    SO: put the resource in a `with` INSIDE the generator, and never rely on
         the code after the loop being reached.

def read_rows(path):
    with open(path, encoding="utf-8") as f:    #    closed on GeneratorExit, break, or exception
        for line in f:
            yield parse(line)

⚠ This is Day 008's CLOSE_WAIT and Day 026's __del__ lesson arriving a third time, and it is the most common real bug in generator code: a generator that opens a file, a socket or a database cursor and is not fully consumed. g.close() throws GeneratorExit explicitly if you need deterministic shutdown.

#   Generator expressions — the same thing, inline (Day 050 goes deeper):
sum(x*x for x in range(10**7))       #    constant memory
sum([x*x for x in range(10**7)])     # ⚠️ builds a 10-million-element list first
max((score(x) for x in items), default=0)

When NOT to use a generator: you need len(), random access, or several passes; the data is small and clarity wins; or you are in a hot numeric loop where NumPy's vectorisation beats any Python-level iteration (Day 028).

Common mistakes

Mistake	Correction
Expecting the body to run on call	It runs on the first `next()`.
Iterating a generator twice	Single pass, silently empty (Day 046).
`len(gen)` or `gen[0]`	Neither exists. Materialise if you need them.
Cleanup after the loop in a generator	Not reached on early `break`. Use `with` inside.
Assuming `GeneratorExit` runs promptly	Only at collection time.
`for x in sub(): yield x`	`yield from` — and it forwards send/throw/close.
Expecting a `return` value in a `for`	It becomes `StopIteration.value`.
`sum([...])` where `sum(...)` works	The brackets build the whole list.
Generators for small data	Clarity wins; a list is fine.
Generators in a hot numeric loop	NumPy — the loop should not be in Python.

Interview questions

Q: What happens when you call a generator function?

Nothing runs. It returns a generator object holding a suspended frame — the locals, the instruction pointer and the exception state, allocated on the heap rather than the stack. The body starts executing on the first next(), runs to the first yield, hands out the value and suspends again with all its state intact. That's why local variables persist between yields without any object or attributes involved.

Q: Why use a generator?

Constant memory and early results. Reading a 50 GB file with readlines() needs 50 GB of RAM; yielding line by line needs a few kilobytes regardless of file size, and the first result is available immediately rather than after the whole file is read. And they compose — a chain of generator expressions is a pipeline where each record travels the whole chain before the next is read, with no intermediate lists. It's the Unix pipe model, and it's why the same pipeline works on an infinite stream.

Q: What are the costs?

No len, no indexing, and one pass only — so anything that needs a second look has to materialise it, which defeats the purpose. They're slightly slower per element, because resuming a frame costs more than indexing a list. And they're harder to debug, since the code runs interleaved with the consumer rather than top to bottom. For small data I'd just build the list.

Q: What's the resource trap?

Cleanup written after the loop inside a generator never runs if the consumer stops early. If I break out of a for over a generator that opened a file, the generator is left suspended with the file open; the close() after the loop is simply never reached. Python does throw GeneratorExit in at the yield point — but only when the object is collected, which under a reference cycle could be much later. The fix is to put the resource in a with inside the generator, so it's released on early exit, on exception, and on abandonment.

Q: What does yield from do beyond looping?

It delegates fully. It forwards send, throw and close to the subgenerator, and it evaluates to the subgenerator's return value — result = yield from sub(). That's substantially more than for x in sub(): yield x, and it's the reason it exists: the delegation machinery introduced in PEP 380 is what await was built on top of in PEP 492. async def is that mechanism with different syntax and a scheduler attached.

Mini task

Write a generator with a print before the first yield. Call it. Confirm nothing prints. Then next() it.
Print inspect.getgeneratorstate(g) at each stage of its life.
Read a large file with readlines() and with a generator. Compare peak RSS.
Build a four-stage generator pipeline over a file and confirm nothing is read until the final loop starts.
Write flatten with yield from and with an explicit loop. Test on deeply nested lists.
Write a generator with a return value and retrieve it two ways — StopIteration.value and yield from.
Write read_rows with f.close() after the loop, break out of it early, and prove with lsof or f.closed that the file is still open. Then fix it with with.
Call g.close() explicitly and catch GeneratorExit inside the generator.
Time and measure sum(x*x for x in range(10**7)) vs the list-comprehension version.

Exit questions

Answer aloud, no notes.

What does calling a generator function return, and what has run?
What exactly is preserved between yields, and where does it live?
Give the memory argument with numbers.
What is a generator pipeline, and how much memory does a four-stage one use?
Name four costs of generators.
Why is cleanup after the loop unsafe? What runs instead, and when?
What is the correct pattern for a generator that opens a resource?
Give three things yield from does that a loop does not.
What happens to a return value in a generator?
Why is sum(genexp) better than sum([listcomp])?
Name three cases where a generator is the wrong choice.

Articulation drill

Record two minutes: "Process a 50 GB log file in a service with 512 MB of RAM."

Lead with the shape: a generator pipeline. Read line by line — a file object is already an iterator — then parse, filter and transform as chained generator expressions, so each record travels the whole chain before the next is read and total memory is one record, not one file. Note the second benefit: the first result is available immediately, which matters if anything downstream is streaming.

Then raise the trap yourself, because it is the part that bites in production: "the thing I'd be careful about is resource cleanup. If any consumer breaks out early — a limit, an error, a timeout — the generator is left suspended and any close() written after the loop is never reached. So the file goes in a with inside the generator, which releases it on early exit and on abandonment. That's the same lesson as leaked sockets and __del__: with is the only cleanup that runs on every path."

Close on the honest limitation: one pass, no len, no indexing — so anything needing a second look either re-opens the source or is a different design.

Previous: Day 046 · Tomorrow: Day 048 — generators as coroutines: send, throw, close, and the direct ancestor of async/await