Chapter 35: Pin and Why Async Is Hard

Step 1 - The Problem

Some values are fine to move around in memory. Others become invalid if moved after internal references have been created.

This is the self-referential problem. A simple version in many languages looks like “store a pointer to one of your own fields.” If the struct later moves, that pointer becomes stale.

Async Rust encounters this problem because the compiler-generated future for an async fn may contain references into its own internal state across suspension points.

Without a rule here, polling a future, moving it, and polling again could produce a dangling reference inside safe code. That is unacceptable.

Step 2 - Rust’s Design Decision

Rust introduced Pin<P> and Unpin.

• Pin<P> says the pointee will not be moved through this pinned access path
• Unpin says moving the value even after pinning is still harmless for this type

Rust accepted:

• a harder mental model
• explicit pinning APIs
• more advanced error messages when custom futures or streams are involved

Rust refused:

• hidden runtime object relocation rules
• GC-based fixing of internal references
• making all async values heap-allocated by default just to avoid movement concerns

Step 3 - The Mental Model

Plain English rule: pinning means “this value must stay at a stable memory location while code relies on that stability.”

Important refinement: pinning is about the value, not about the pointer variable that refers to it.

If a type is Unpin, pinning is mostly a formality. If a type is !Unpin, moving it after pinning would break its invariants.

Step 4 - Minimal Code Example

#![allow(unused)]
fn main() {
use std::future::Future;
use std::pin::Pin;

fn make_future() -> Pin<Box<dyn Future<Output = u32>>> {
    Box::pin(async { 42 })
}
}

This is not the whole theory of pinning, but it is the most common practical encounter: a future is heap-allocated and pinned so it can be polled safely from a stable location.

Step 5 - Line-by-Line Compiler Walkthrough

1. async { 42 } creates an anonymous future type.
2. Box::pin(...) allocates that future and returns Pin<Box<...>>.
3. The heap allocation gives the future a stable storage location.
4. The Pin wrapper expresses that the pointee must not move out of that location through safe access.

Why this matters for async:

polling a future may cause it to store references between its internal states. The next poll assumes those references still point to the same memory. Pinning is the mechanism that makes that assumption legal.

Step 6 - Three-Level Explanation

Why Async Rust Feels Harder Than JavaScript or Go

This is not accidental. Rust exposes complexity that those languages hide behind different runtime tradeoffs.

JavaScript hides many lifetime and movement issues behind GC and a single-threaded event-loop model.

Go hides much of the scheduling and stack management behind goroutines and a runtime that can grow and move stacks.

Rust refuses both tradeoffs. So you must reason about:

• which tasks may move between threads
• which futures are Send
• when cancellation drops in-progress state
• when pinning is required
• when holding a lock across .await can stall other work

That is harder. It is also why well-written async Rust can be both predictable and efficient.

tokio::pin! and pin-project

Pinned stack storage often looks like this:

use tokio::time::{sleep, Duration};

#[tokio::main]
async fn main() {
    let task = sleep(Duration::from_millis(10));
    tokio::pin!(task);

    (&mut task).await;
}

And pinned field projection in libraries often uses a helper macro crate:

• pin-project
• pin-project-lite

Those crates exist because manually projecting pinned fields is easy to get wrong in unsafe code.

Step 7 - Common Misconceptions

Wrong model 1: “Pin means the pointer itself cannot move.”

Correction: pinning is about the pointee’s location and the promise not to move that value through the pinned access path.

Wrong model 2: “All futures are self-referential.”

Correction: not all futures need pinning for the same reasons, and many are Unpin. The abstraction exists because some futures are not.

Wrong model 3: “Pin is only for heap allocation.”

Correction: stack pinning exists too, for example with tokio::pin!.

Wrong model 4: “If I use Box::pin, I understand pinning.”

Correction: you may understand the common application pattern without yet understanding the deeper contract. Those are different levels of mastery.

Step 8 - Real-World Pattern

You will encounter pinning in:

• manual future combinators
• stream processing
• select! over reused futures
• library internals using projection macros
• executor and channel implementations

Tokio and related ecosystem crates use projection helpers specifically because Pin is not ornamental. It is part of the soundness boundary of async abstractions.

Step 9 - Practice Block

Code Exercise

Create a function that returns Pin<Box<dyn Future<Output = String> + Send>>, then use it inside a Tokio task and explain why pinning was convenient.

Code Reading Drill

Explain what is pinned here and why:

#![allow(unused)]
fn main() {
let sleep = tokio::time::sleep(Duration::from_secs(1));
tokio::pin!(sleep);
}

Spot the Bug

Why is a self-referential struct like this dangerous without special handling?

#![allow(unused)]
fn main() {
struct Bad<'a> {
    data: String,
    slice: &'a str,
}
}

Refactoring Drill

Take code that recreates a timer future each loop iteration and redesign it so the same pinned future is reused where appropriate.

Compiler Error Interpretation

If the compiler says a future cannot be unpinned or must be pinned before polling, translate that as: “this value’s correctness depends on staying at a stable address while it is being driven.”

Step 10 - Contribution Connection

After this chapter, you can read:

• async combinator code
• custom stream and future implementations
• library code using pin-project-lite
• select! loops that pin a future once and poll it repeatedly

Approachable PRs include:

• replacing ad hoc pinning with clearer helper macros
• documenting why a type is !Unpin
• simplifying APIs that unnecessarily expose pinning to callers

In Plain English

Some values can be moved around safely. Others break if they move after work has already started. Pin is Rust’s way of saying “this must stay put now.” That matters to systems engineers because async code is really a collection of paused state machines, and paused state machines still need their memory layout to make sense when resumed.

What Invariant Is Rust Protecting Here?

A pinned !Unpin value must not be moved in a way that invalidates self-references or other address-sensitive internal state.

If You Remember Only 3 Things

• Pin exists because some futures become address-sensitive across suspension points.
• Box::pin and tokio::pin! are the common practical tools; pin-project exists for safe field projection.
• Async Rust is harder partly because Rust refuses to hide movement, lifetime, and scheduling costs behind a GC or mandatory runtime.

Memory Hook

Think of a future as wet concrete poured into a mold. Before it sets, you can move the mold. After internal supports are in place, moving it cracks the structure. Pinning says: leave it where it is.

Flashcard Deck

Chapter Cheat Sheet