What repeats
A shared computational core.
Independent technical reference
Recurrent Depth,
Explained.
A visual guide to shared-block recurrence, latent computation, and test-time depth.
Recurrent depth is an architecture technique in which a model repeatedly applies a shared computational block to an internal hidden representation. Each pass adds another sequential computation step while reusing the block’s parameters.
Last reviewed: September 3, 2026
On the left, an abstract state moves once through four distinct blocks. On the right, an abstract numerical state returns to the same shared core four times, changing after each pass. The diagram does not represent hidden states as readable language.
Distinct blocks · one route
Updated state · 1 of 4 passes
Read the diagram transcript
Fixed stack: input moves through Block A, B, C, and D, which may each hold different learned parameters. Recurrent core: a prelude prepares the state, the same core parameter set is applied four times to updated numerical representations, and a coda prepares the output.
00 / The essential idea
Recurrent Depth in 30 Seconds
A standard Transformer commonly processes information through a fixed stack of distinct blocks. A recurrent-depth system can apply a shared block repeatedly to its hidden state. The repeated passes add executed computation without creating a fresh parameter set for every pass.
What changes
The hidden representation passed into the next iteration.
What rises
Executed serial computation and usually inference latency.
What can stay fixed
The stored parameters of the repeated core.
Recurrent depth separates the number of stored parameter sets from the number of block applications executed during inference.
Repeated execution can increase inference computation and latency. Additional passes do not guarantee a better answer.
A hidden state is a numerical representation, not a readable sentence. A recurrent loop should never be illustrated as though it contains a verified secret transcript of the model’s reasoning.
Cite This Guide
This is an educational webpage, not a peer-reviewed publication.
01 / Architecture
One route.
Two kinds of depth.
A fixed stack usually moves through distinct blocks. A depth-recurrent system can move through a prelude, revisit one shared core, then exit through a coda. The counters below are conceptual, not FLOP estimates.
Each block may have its own learned parameters. The executed route follows the fixed stack defined by the architecture.
The same core parameter set can be applied repeatedly to an updated hidden state. Each pass adds an executed block application. Implementations differ.
Text description: the left route executes four distinct blocks once. The right route enters a prelude, applies one shared core 4 times to changing numerical states, then exits through a coda.
Illustrative
1Distinct core parameter sets
4Recurrent passes
4Executed core applications
4×Relative serial core work
Reuse terms available upon request.
02 / Recurrence lab
Try Recurrent Depth
Illustrative
Try Recurrent Depth
Adjust one architectural axis and watch executed work change while the shared parameter set stays fixed.
Recurrent passes4
Core applications4
Distinct core parameter sets1
Relative serial core work4×
Conceptual latencyModerate
h₀h1h2h3h4y
In this illustration, increasing the pass count increases the number of times the shared core is executed. It does not create a new stored copy of the core for each pass.
Real systems may use fixed, randomized, learned, or adaptive recurrence schedules.
This simulator explains architecture behavior. It does not predict the quality, cost, latency, or accuracy of a specific production model.
03 / Four levels
Choose your depth.
The same architecture can be explained without collapsing it into a slogan. Every level remains present in semantic HTML.
The shortest useful definition
A model applies a shared block repeatedly to its hidden state, adding computation through repeated passes.
04 / Technical anatomy
Anatomy of a Recurrent-Depth Model
A compact teaching equation captures the recurring pattern without pretending every paper uses the same design.
h₀ = Prelude(x)hᵣ₊₁ = Core(hᵣ; θ)y = Coda(hᴿ)- x
- The model input or an internal representation derived from it.
- Prelude
- Computation that prepares the initial hidden representation.
- hᵣ
- The hidden representation after recurrent pass r.
- Core
- The shared computational block.
- θ
- The learned parameters reused by the core.
- R
- The total number of recurrent passes.
- Coda
- Computation that converts the final state toward an output.
This equation is a conceptual teaching aid. Published systems may use different residual paths, normalization schemes, embeddings, recurrence schedules, stopping rules, losses, or output heads.
05 / Vocabulary
What does “depth” mean?
These quantities interact, but none is a synonym for another.
- Stored or parameterized depth
- The number of distinct learned block parameter sets.
- Executed depth
- The number of sequential block applications performed during a run.
- Token length
- The number of tokens generated or processed.
- Wall-clock latency
- The elapsed time required to complete computation.
- Total compute
- The computational work performed.
Each executed once
Executed four times
Both examples execute four block applications in this simplified illustration, but they do not have identical parameters, training behavior, representations, latency, or capability.
06 / Where computation appears
Visible Tokens vs Hidden-State Recurrence
Token-based reasoning externalizes intermediate computation through generated tokens. Hidden-state recurrence performs repeated computation within internal numerical representations. These approaches can coexist.
Visible token-based reasoning
QuestionIntermediate output tokensFinal answer
Hidden-state recurrence
QuestionHidden representation ↻Hidden representation ↻Output
Hidden-state recurrence should not automatically be described as a complete, interpretable, or human-readable chain of thought. A 2025 Huginn probing study found inconsistent evidence for clean token-like trajectories; that study is limited to one model, selected arithmetic tasks, and its probes.
Read the cautionary primary study07 / Compare
Five mechanisms, separated.
Simple yes-or-no cells are avoided where the answer depends on implementation.
| Question | Fixed-stack Transformer | Depth-recurrent Transformer | Token chain of thought | Sequence RNN | External self-refinement |
|---|---|---|---|---|---|
| Main recurrence axis | None across depth | Architectural depth | Generated-token sequence | Sequence positions | Separate calls or turns |
| Where extra work occurs | More distinct blocks | Repeated hidden-state passes | Additional output tokens | State transition per position | New generations or tools |
| Shared parameters | Implementation-dependent | Commonly in the core | No inherent requirement | Usually recurrent weights | May reuse the same model |
| Extra output tokens | No inherent requirement | No inherent requirement | Usually yes | No inherent requirement | Usually yes |
| Readable intermediate text | No | Not inherently | Usually | No | Usually |
| Context-window impact | Ordinary | Can stay small | Grows with trace length | Sequence-dependent | Can grow across turns |
| Inference latency | Fixed by stack | Rises with passes | Rises with tokens | Rises with sequence length | Rises with iterations |
| Training requirements | Standard stack training | Must support tied recurrence | Often prompting or post-training | Sequence-model training | Can work without retraining |
| Adaptive stopping | Not applicable | Implementation-dependent | Stop-token or budget | Sequence ends | Controller decides |
| Main caveat | More parameters per depth | Serial work and stability | Token cost and faithfulness | Different recurrence axis | External-loop overhead |
08 / Boundaries
What recurrent depth is not
Precision starts by refusing convenient equivalences.
Recurrent depth is not automatically the same as a standard RNN.
Sequence RNNs recur as positions advance; depth recurrence repeats computation along architectural depth.
Recurrent depth is not chain-of-thought prompting.
It updates hidden numerical states instead of requiring intermediate output tokens.
Recurrent depth is not proof of hidden human-like reasoning.
A numerical representation is not a readable transcript.
Recurrent depth is not free computation.
Repeated passes require additional inference work.
Recurrent depth is not guaranteed to improve every task.
Performance can plateau, vary, or decline.
Recurrent depth is not one standardized architecture.
Loop scope, residual paths, supervision, and stopping rules differ.
Recurrent depth is not proof that parameter scaling is obsolete.
It adds another axis for allocating computation.
Recurrent depth is not external self-refinement.
Repeated prompting runs separate generation or tool loops outside one recurrent core.
09 / Research interest
Why researchers care
Recurrent depth opens a compute axis, but the strongest reasons to study it are specific.
Parameter reuse
Repeated applications can reuse the same learned core rather than storing a fresh block set for every pass.
Test-time computation
Some architectures can vary pass count at inference, within the limits created by training and stability.
Generalization research
Controlled studies report systematic and depth-extrapolation gains; those task settings are narrow and explicit.
Interpretability & monitorability
Hidden-state computation raises observation, diagnosis, and control questions without making recurrence inherently unsafe.
10 / Evidence and limits
What we know—and what we do not.
Architecture facts, experiment results, and open questions belong in separate bins.
Established by the architecture
- A shared block can be executed repeatedly.
- The same parameter set can be reused.
- The hidden representation can change after each pass.
- Executed block applications rise with pass count.
- Additional passes require computation.
Observed in specific experiments
- Selected reasoning-benchmark gains in Huginn.
- Depth extrapolation in controlled multi-hop tasks.
- Adaptive recurrence on selected puzzle benchmarks.
- Parameter-efficient iteration in selected architectures.
Open or contested
- Broad transfer across production tasks.
- Reliable stopping at commercial scale.
- Calibration and safety-evaluation effects.
- How much hidden recurrence is interpretable.
- Which current commercial systems use which implementation.
11 / Claims ledger
Claims, checked in public.
Each row states what the evidence supports, its main caveat, and the source used for review.
| Claim | Status | Evidence | Caveat | Last reviewed | Source |
|---|---|---|---|---|---|
| A recurrent core can be executed repeatedly without adding a fresh parameter set for each pass. | SUPPORTED | This follows from weight-tied recurrent-core designs. | Surrounding prelude, coda, embeddings, and heads may still use distinct parameters. | 2026-09-03 | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach |
| Additional recurrent passes increase executed computation. | SUPPORTED | Every extra pass executes the shared core again. | Actual latency and cost also depend on hardware, batching, caches, and implementation. | 2026-09-03 | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach |
| Additional recurrent passes improve every task. | UNSUPPORTED | Controlled studies report plateaus and overthinking failures. | The useful pass count is task-, model-, and training-dependent. | 2026-09-03 | Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers |
| Recurrent depth automatically produces interpretable latent chain-of-thought. | UNSUPPORTED | Probing studies do not recover one consistent, readable trajectory from recurrence alone. | Other architectures and explicitly supervised latents may behave differently. | 2026-09-03 | Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer |
| Recurrent depth and standard chain-of-thought prompting are the same mechanism. | UNSUPPORTED | One repeats computation in numerical hidden states; the other emits intermediate tokens. | They can coexist, and theoretical work relates their computational roles. | 2026-09-03 | Reasoning with Latent Thoughts: On the Power of Looped Transformers |
| A depth-recurrent model can use a variable number of passes at inference time. | SUPPORTED WITH LIMITS | Huginn and adaptive-halting research explicitly vary inference recurrence. | Useful extrapolation beyond training depths is not guaranteed. | 2026-09-03 | Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers |
| Weight sharing always reduces total production cost. | UNSUPPORTED | Fewer stored parameters can coexist with more serial work per token. | Memory, throughput, cache behavior, energy, and serving economics vary. | 2026-09-03 | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach |
| A named commercial AI system uses recurrent depth. | REPORTED, UNCONFIRMED | Reporting links Astra to a limited recurrent-depth technique. | No official OpenAI technical source located by the review cutoff confirms that architecture. | 2026-09-03 | Safety overview: GPT-6 Astra |
| Recurrent depth is a new term created in 2026. | UNSUPPORTED | Peer-reviewed RNN research used the phrase in 2016. | That earlier formal definition differs from current Transformer usage. | 2026-09-03 | Architectural Complexity Measures of Recurrent Neural Networks |
| Looped Transformers and depth-recurrent Transformers always refer to identical implementations. | TERMINOLOGY-DEPENDENT | The labels overlap across several weight-sharing and recurrence patterns. | Residual paths, loop scope, supervision, halting, and state updates differ by paper. | 2026-09-03 | Universal Transformers |
12 / Current attention
Why Recurrent Depth Is Getting Attention
The September 2026 news cycle combined established architecture research with reporting about one commercial model. These are not the same evidentiary category.
Peer-reviewed recurrent-depth research exists.
Huginn’s depth-recurrent architecture was published in the NeurIPS 2025 main conference. Looped Transformer work predates it.
NeurIPS proceedingsAstra was linked to recurrent depth.
The Information reported on September 1, 2026 that Astra uses a limited recurrent-depth technique; TechCrunch covered that report on September 2.
The Information, 2026-09-01 TechCrunch, 2026-09-02Astra and added CoT monitoring are official.
OpenAI’s September 1 and September 3 pages confirm Astra, its release context, and monitorability work. The reviewed official pages do not name recurrent depth.
OpenAI safety overviewNo public architecture specification was located.
As of September 3, 2026, no official OpenAI technical source located in this review confirmed that Astra uses recurrent depth or specified its implementation. Reporting should not be converted into technical confirmation.
13 / Research timeline
A lineage, not a sudden invention.
Selected entries are included for direct relevance—not merely because they contain the word “recurrent.”
- PEER-REVIEWED
Architectural Complexity Measures of Recurrent Neural Networks
Defines recurrent depth as an over-time nonlinear-complexity measure for RNN architectures, alongside feedforward depth and recurrent skip coefficient. It establishes that the phrase predates the current Transformer discussion and had a distinct formal meaning.
Limitation: Its graph-theoretic RNN definition is not interchangeable with every present-day depth-recurrent Transformer design.
Direct source - PEER-REVIEWED
Universal Transformers
Applies a recurrent self-attention transition across depth and introduces a per-position adaptive halting mechanism. It is an important shared-computation and adaptive-depth precursor to modern looped architectures.
Limitation: Universal Transformers have their own recurrence and halting details; they are related to, not identical with, all later recurrent-depth language models.
Direct source - PEER-REVIEWED
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Studies cross-layer parameter sharing as one of two parameter-reduction techniques for BERT-style encoders. It shows a major Transformer lineage in which layer parameters can be reused across depth.
Limitation: Cross-layer sharing alone is not proof of the inference-time recurrence, dynamic stopping, or latent-reasoning claims studied in later systems.
Direct source - PEER-REVIEWED
Looped Transformers as Programmable Computers
Constructs constant-depth looped Transformers that execute iterative programs encoded in their inputs. It gives a formal and constructive account of repeated Transformer computation as programmable iteration.
Limitation: Programmed-weight constructions do not directly establish capabilities of large pretrained language models.
Direct source - PEER-REVIEWED
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Introduces the 3.5-billion-parameter Huginn proof-of-concept, trained from scratch on 800 billion tokens, and studies scaling inference work through a recurrent core. It is the central empirical reference for the current prelude–recurrent-core–coda design and variable test-time recurrence.
Limitation: Results are model- and benchmark-specific; equivalent-compute comparisons are not evidence that extra passes universally outperform larger models or improve every task.
Direct source - PEER-REVIEWED
Reasoning with Latent Thoughts: On the Power of Looped Transformers
Analyzes looped models on synthetic reasoning and one-billion-parameter language-model experiments, with theory relating effective depth and iterative algorithms. It distinguishes parameter count from effective depth and offers controlled evidence for reasoning-oriented inductive bias.
Limitation: Its theoretical simulation result does not mean a trained model’s hidden states automatically form a readable chain of thought.
Direct source - PEER-REVIEWED
Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
Uses several probing methods on Huginn arithmetic behavior and reports inconsistent evidence for a clean, token-like latent reasoning trajectory. It directly cautions against drawing a readable hidden chain-of-thought from recurrence alone.
Limitation: The analysis is limited to one model family, arithmetic tasks, and the probes used; it does not settle every interpretability question.
Direct source - PEER-REVIEWED
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Reports systematic-generalization and depth-extrapolation results in controlled multi-hop tasks, including gains from increasing recurrence at inference. It directly studies the train-short, test-deeper question and documents an overthinking failure mode.
Limitation: The models are trained from scratch on controlled tasks; excessive recurrence can degrade predictions.
Direct source - PREPRINT
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Combines residual-scaling choices with a convergence-based halting rule intended to adapt loop count to task difficulty. It treats stability and stopping as first-class architecture questions rather than assuming a fixed pass count.
Limitation: Evaluation centers on Sudoku, maze, state-tracking, and ARC-style tasks, not broad production language use.
Direct source - PREPRINT
DeepLoop: Depth Scaling for Looped Transformers
Derives residual-scaling rules that account for repeated visits to shared parameters and tests them at GPT-2 small and medium scale. It identifies a training-stability issue specific to tied recurrent depth.
Limitation: The empirical evidence is limited to the studied scales and does not establish one universal normalization rule.
Direct source - PREPRINT
Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?
Adapts a Jacobian-lens analysis to looped architectures and compares representation access patterns in Ouro and Huginn with an untied baseline. It is a current example of recurrence-specific mechanistic inquiry available immediately before this review cutoff.
Limitation: The work is a new preprint, studies selected models, and its proposed workspace framing remains subject to replication and debate.
Direct source
14 / Terminology note
One phrase, related contexts.
No single architecture owns every use of “recurrent depth.”
Earlier RNN usage
A 2016 paper used recurrent depth as a formal architecture-complexity measure for nonlinear computation across recurrent time steps.
Current Transformer usage
Current discussion often concerns repeatedly applying a shared Transformer-like block to changing hidden representations, increasing executed serial computation.
Related labels
Depth-recurrent, recurrent-depth, looped, recurrent, shared-weight, latent-recurrence, and test-time latent-computation labels overlap without always naming identical implementations.
15 / Glossary
Terms without fog.
Brief, responsible definitions for the concepts used throughout this guide.
- Adaptive recurrence
- A schedule that changes the number of repeated passes according to the input or the evolving state. Return to the main explanation
- Block
- A reusable unit of neural-network computation, commonly containing attention, normalization, residual paths, and feed-forward operations. Return to the main explanation
- Coda
- The post-recurrence computation that turns the final hidden representation into the next model representation or prediction. Return to the main explanation
- Computational depth
- The number of sequential transformations along a computation path. Return to the main explanation
- Depth recurrence
- Repeated application of computation along the model-depth axis rather than across generated tokens. Return to the main explanation
- Dynamic stopping
- A rule that decides at runtime when additional recurrent passes should end. Return to the main explanation
- Executed depth
- The number of block applications actually performed during a run. Return to the main explanation
- Feedforward depth
- Depth along a direct input-to-output path without counting recurrence across repeated steps. Return to the main explanation
- Inference
- Running a trained model to produce a prediction or output. Return to the main explanation
- Latent computation
- Computation carried out in internal numerical representations instead of visible output tokens. Return to the main explanation
- Latent reasoning
- A research label for multi-step computation in hidden representations; it does not itself prove human-like or interpretable reasoning. Return to the main explanation
- Looped Transformer
- A Transformer that applies one block or group of blocks repeatedly; details vary across papers. Return to the main explanation
- Parameter count
- The number of learned numerical values stored by a model. Return to the main explanation
- Parameter sharing
- Using the same learned values in more than one layer position or recurrent pass. Return to the main explanation
- Prelude
- The computation that prepares the initial hidden representation before the recurrent core. Return to the main explanation
- Recurrent core
- The shared computational block or block group applied repeatedly. Return to the main explanation
- Recurrent pass
- One execution of the recurrent core. Return to the main explanation
- Recurrent depth
- A family of related ideas involving repeated computation; in current Transformer discussions, usually shared computation applied repeatedly to changing hidden states. Return to the main explanation
- Serial computation
- Operations that depend on earlier operations and therefore cannot all be completed in parallel. Return to the main explanation
- Test-time computation
- The computational work allocated when a trained model is being used. Return to the main explanation
- Token-based reasoning
- Intermediate computation externalized through generated tokens. Return to the main explanation
- Weight sharing
- Another name for reusing the same parameter values across layers, positions, or passes. Return to the main explanation
24 terms shown
16 / FAQ
Direct answers.
Twenty-five common questions, answered without hiding the qualification.
Occasional, material updates
Track Recurrent Depth
Receive occasional research updates when significant papers, technical releases, or verified developments appear.