全文目录Contents
- 0 · The Project at a Glance, and a Map of the Code
- 1 · The Gateway Layer
- 2 · Identity and Session Routing
- 3 · The Main Loop ★
- 3.1 The Loop Entry: A Triple Budget Gate
- 3.2 Per-Turn Preparation: build_turn_context
- 3.3 Mid-Turn Interjection: /steer
- 3.4 The Wall-Clock Budget Wrap-Up Reminder
- 3.5 The Interrupt Check
- 3.6 The Step Callback: An Observation Point for the Gateway
- 3.7 Other State Inside the Loop
- 3.8 Why the Loop Exited
- 3.9 An Optional Bypass: The Codex App Server Runtime
- 4 · The Tool System
- 4.1 The Most Important Design Decision: Implementation Separated from Exposure
- 4.2 Composing and Resolving Toolsets
- 4.3 Central Tool Dispatch
- 4.4 The Argument Coercion Layer
- 4.5 Sanitizing Tool Error Messages
- 4.6 Observing Tool Results, and Hooks
- 4.7 Recognizing a Delegation Context
- 4.8 Caching Tool Definitions
- 5 · Approval and Safety Red Lines ★
- 5.1 The Overall Structure
- 5.2 The 12 Hard Red Lines
- 5.3 The Real Difficulty: Telling “Command” from “Data”
- 5.4 Quote Masking: But Leave a Way In for “the Part That Really Executes”
- 5.5 Quoting Is Not a Bypass
- 5.6 Sensitive Paths and Write Targets
- 5.7 The sudo Stdin Guard
- 5.8 Performance: Why Precompile
- 5.9 Retaining Blocked Commands
- 5.10 Smart Approval: An Optional Model Judgment
- 5.11 Context Awareness: Different Situations, Different Policies
- 5.12 Where This Layer Sits: Outermost, and Thinnest
- 6 · Execution Environments
- 7 · The Context Engine ★
- 7.1 What It Defines
- 7.2 Lifecycle
- 7.3 The Three Methods You Must Implement
- 7.4 The Sharpest Design Decision: select and compress Are Two Orthogonal Verbs
- 7.5 The Post-Turn Observation Hook
- 7.6 Other Optional Hooks
- 7.7 Default Parameter Values
- 7.8 Control Over User-Visible Status
- 7.9 Size Comparison of the Built-in Implementation
- 8 · The Memory System ★
- 8.1 The Memory Provider Interface
- 8.2 Lifecycle and Hooks
- 8.3 The Interface's Versioned Contract
- 8.4 The Trivial-Prompt Filter
- 8.5 The Memory-Usage Indicator
- 8.6 The Built-in Holographic Memory
- 8.7 The Storage Layer and Trust Scores
- 8.8 The SQLite State Layer
- 8.9 Summing Up the Division of Labor Among Three Kinds of Memory
- 9 · The Plugin System
- 9.1 Three Discovery Sources
- 9.2 What a Plugin Can Provide
- 9.3 The Most Important Design Decision: “Stackable Capabilities” vs. “Mutually Exclusive Strategies”
- 9.4 Plugin Storage
- 9.5 How Plugins Tie into Toolsets
- 9.6 Plugin Hooks
- 9.7 MCP: The Other Extension Path
- 9.8 The Overall Shape of This Extension System
- 10 · Delegation and Multi-Agent
- 11 · Model Providers and the Credential Pool
- 12 · Scheduled Tasks (Cron)
- 12.1 What “cron” Is
- 12.2 Scenarios for Scheduled Agents
- 12.3 The Single Most Important Class: CronPromptInjectionBlocked
- 12.4 Narrowing the Toolset for Scheduled Tasks
- 12.5 Failure Handling
- 12.6 Preventing Duplicate Runs
- 12.7 Why scheduler.py Is 367 KB
- 12.8 Where Scheduled Tasks Sit in the Overall Architecture
- 13 · The Skill System
- 13.1 What a Skill Is
- 13.2 The Front Matter, Field by Field
- 13.3 Progressive Disclosure: The Core Mechanism of the Skill System
- 13.4 The Infrastructure Around Skills
- 13.5 Skills vs. Tools vs. Plugins
- 13.6 An Implicit Design in the Skill System: Composability
- 13.7 The 15 Skill Categories
- 13.8 Looking Back Across the Book: The Overall Shape of Hermes
10 · 委派与多智能体
tools/delegate_tool.py，5,071 行。这是 Hermes 里最长的单个工具文件 —— 比整个审批系统的核心还长。这一章讲一个智能体怎么派另一个智能体去干活。
10.1 为什么需要委派
委派的本质是「上下文分区」。
它不是为了「并行更快」（虽然确实更快），核心价值是：让每个子任务在一个干净、专注、不会被无关信息污染的上下文里执行，而主智能体只承担协调成本。
10.2 深度限制：只允许一层
MAX_DEPTH = 1这一行是整个多智能体系统里最重要的一个常量。它的意思是：主智能体可以派子智能体，但子智能体不能再派孙智能体。
如果不限制深度会怎样
这是一个「用最简单的手段消灭一整类问题」的典型例子。
想做「智能地限制递归」很难：要估算成本、要判断任务复杂度、要有熔断机制、要有预算传递……
而 MAX_DEPTH = 1 一行代码就让整类问题不存在了。代价是失去了「深层任务分解」的能力 —— 但实践中，一层委派已经覆盖了绝大多数场景，而两层带来的复杂度是指数级的。
在做架构设计时，先问「能不能用一个硬限制消灭这类问题」，再考虑「怎么智能地处理这类问题」。
10.3 并发限制
_DEFAULT_MAX_CONCURRENT_CHILDREN = 10
_RECENT_SUBAGENTS_CAP = 200| 常量 | 作用 |
|---|---|
_DEFAULT_MAX_CONCURRENT_CHILDREN = 10 |
同时最多跑 10 个子智能体。第 11 个要排队。防止一次性打爆模型供应商的速率限制，也防止本机内存和文件句柄耗尽 |
_RECENT_SUBAGENTS_CAP = 200 |
「最近的子智能体」记录最多保留 200 条。这是给「查看子智能体状态」这类功能用的历史缓冲，超出就丢弃最老的。防止长时间运行的会话把内存吃光 |
这两个数值配合 MAX_DEPTH = 1，把整个多智能体系统的资源占用锁在一个可预测的范围内：任意时刻最多 1 + 10 = 11 个智能体在跑，历史记录最多 200 条。
10.4 子智能体的工具限制
DELEGATE_BLOCKED_TOOLS子智能体不能使用某些工具。最重要的一条是：子智能体不能再调用 delegate 工具 —— 这是 MAX_DEPTH = 1 在工具层面的强制实现。
注意这是「双重保险」：
· 逻辑层：MAX_DEPTH = 1 在委派时检查深度
· 能力层：DELEGATE_BLOCKED_TOOLS 让子智能体根本看不到 delegate 这个工具
第二层更彻底 —— 模型连「我可以委派」这个念头都不会有，因为工具列表里没有。不给能力，比给了能力再拦截更可靠。这和第 4 章「webhook 只投放 4 个只读工具」是同一个思路。
10.5 子智能体的审批策略
def _subagent_auto_deny(...)
def _subagent_auto_approve(...)这里有一个必须解决的问题：子智能体跑起来后要求审批，谁来批？
| 函数 | 语义 |
|---|---|
_subagent_auto_deny |
自动拒绝。子智能体收到「被拒绝」的结果，它可以换个方式做，或者报告失败。安全但可能卡住任务 |
_subagent_auto_approve |
自动批准。不问人直接放行。能跑通但风险大 |
这是自动化系统里最难的一个权衡。
自动拒绝是安全的默认，但会让很多合法任务失败 —— 而且失败方式很隐蔽（子智能体报告「我做不到」，但真实原因是权限被拒）。
自动批准能跑通，但意味着红线之外的所有操作在无人监督下执行。第 5 章那 12 条红线依然生效（那是绝对禁止的），但「需要确认」这一档就被跳过了。
正确的做法是：让主智能体在委派时显式声明子智能体的权限档位，而不是有一个全局默认。做「只读分析」的子智能体应该自动拒绝一切写操作；做「批量重构」的子智能体则需要预先授权写文件。
10.6 运行中的控制
子智能体不是「发出去就不管了」。有三个控制接口：
def interrupt_subagent(...) # 中断某个子智能体
def steer_subagent(...) # 向运行中的子智能体注入指令
def set_spawn_paused(...) # 暂停/恢复新子智能体的派生| 接口 | 用途 |
|---|---|
interrupt_subagent |
发现某个子智能体走偏了 / 卡住了 / 在烧钱，单独把它停掉，不影响其他 9 个 |
steer_subagent |
不打断，但插一句话。对应第 3 章讲过的 /steer 机制 —— 把新指令注入到最后一条工具消息里，让智能体在下一轮就能看到。比如：「顺便也检查一下类型注解」 |
set_spawn_paused |
暂停派生新的子智能体，但已在跑的继续。用途：发现整批任务方向不对时，先止血 —— 不再派新的，让已经跑起来的自然结束，然后重新规划 |
set_spawn_paused 这个设计值得单独说。
最朴素的做法只有「全部继续」和「全部杀掉」两种。但实际场景里最常见的是第三种：「别再开新的了，让手上的跑完」。
这在运维上叫「排空」（drain）—— 优雅停机、滚动更新、限流降级都是这个模式。一个成熟的并发系统必须区分「停止接受新工作」和「终止现有工作」。
10.7 亲缘关系检查
def _is_descendant_of(..., max_hops: int = 8)「判断智能体 A 是不是智能体 B 的后代」。用途：
- 中断一个智能体时，要连带中断它的所有后代
- 统计成本时，要把后代的花费算到祖先头上
- 权限检查：某些操作只允许对自己的后代做
那个 max_hops = 8 是防御性的。既然 MAX_DEPTH = 1，理论上最多只需要查 1 跳。设成 8 是为了：
- 兼容将来可能放宽的深度限制
- 更重要的：万一数据里出现了环（A 的父亲是 B，B 的父亲是 A），这个上限保证函数一定会返回，而不是无限循环
这是「即使不变量被破坏，程序也不能挂死」的写法。
正常情况下永远不会走到第 8 跳。但如果某个 bug 导致父子关系成了环，有这个上限的版本会返回一个（可能错误的）答案并继续跑，没有上限的版本会把整个进程挂死。
在遍历任何「理论上应该是树，但数据由运行时构造」的结构时，都要加这样一个跳数上限。
10.8 看板：智能体之间的协作
plugins/kanban/ 提供了另一种多智能体模式 —— 不是「派下去等结果」，而是「共享一块任务板」。
kanban_create_task 创建任务
kanban_claim_task 认领任务
kanban_update_task 更新进度
kanban_complete_task 完成任务
kanban_list_tasks 查看任务列表
kanban_heartbeat ★ 心跳| 委派模式 | 看板模式 | |
|---|---|---|
| 关系 | 父子 —— 主智能体明确指派 | 对等 —— 谁有空谁认领 |
| 谁决定做什么 | 主智能体 | 各个智能体自己 |
| 生命周期 | 子智能体做完就结束 | 智能体长期存在，持续认领新任务 |
| 适合 | 已知的、可分解的批量任务 | 持续的、来源不定的工作流 |
kanban_heartbeat 为什么必须存在
任何「认领 - 执行 - 完成」的分布式任务系统，都必须有心跳或租约（lease）机制。
否则「认领了但没做完就死掉」的任务会永久卡住。这在消息队列、任务调度器、分布式锁里是同一个问题，解法也一样：认领是有时效的，需要持续续期。
10.9 委派系统为什么有 5,071 行
回到开头那个数字。真正做「派一个子智能体」的核心逻辑可能只要 200 行。剩下 4,800 行在做什么？
| 类别 | 内容 |
|---|---|
| 生命周期管理 | 创建、启动、监控、中断、清理、超时、僵尸回收 |
| 并发控制 | 并发上限、排队、暂停派生、优先级 |
| 结果聚合 | 收集 10 个子智能体的结果、处理部分失败、超时的怎么算 |
| 状态查询 | 「现在有几个在跑」「花了多少钱」「卡在哪一步」 |
| 控制通道 | 中断、注入指令、暂停 —— 每个都要跨进程/跨线程安全地送达 |
| 安全边界 | 工具屏蔽、审批策略、深度检查、亲缘关系 |
| 可观测性 | 每个子智能体的日志、成本、耗时都要单独记录并能关联回父任务 |
| 失败处理 | 子智能体崩溃、模型报错、上下文爆炸、无限循环 —— 每种都要有对策 |
「让智能体调用智能体」的 demo 是 20 行，生产系统是 5,000 行。
这个 250 倍的差距全部来自「出问题时怎么办」。多智能体系统的难点从来不是「怎么派」，而是「派出去的东西失控了怎么收场」。
面试里如果被问到多智能体，能说清楚这一点，比能画出漂亮的架构图有用得多。
10 · Delegation and Multi-Agent
tools/delegate_tool.py, 5,071 lines. This is the longest single tool file in Hermes — longer than the core of the entire approval system. This chapter covers how one agent sends another agent off to do work.
10.1 Why Delegation Is Needed
Delegation is fundamentally “context partitioning.”
It isn't about “parallelism makes it faster” (though it does). The core value is: every subtask runs in a clean, focused context that can't be polluted by unrelated information, while the main agent bears only the coordination cost.
10.2 The Depth Limit: One Level Only
MAX_DEPTH = 1This one line is the single most important constant in the whole multi-agent system. It means: the main agent may spawn subagents, but a subagent may not spawn grandchild agents.
What happens without a depth limit
This is a textbook case of “killing an entire class of problems with the simplest possible means.”
“Intelligently limiting recursion” is hard: you'd need to estimate cost, judge task complexity, build a circuit breaker, pass budgets down the tree…
And MAX_DEPTH = 1, one line of code, makes the whole class of problems disappear. The cost is losing “deep task decomposition” — but in practice, one level of delegation covers the vast majority of scenarios, and two levels bring exponential complexity.
When designing architecture, first ask “can a hard limit eliminate this class of problems,” and only then consider “how do we handle this class intelligently.”
10.3 Concurrency Limits
_DEFAULT_MAX_CONCURRENT_CHILDREN = 10
_RECENT_SUBAGENTS_CAP = 200| Constant | Role |
|---|---|
_DEFAULT_MAX_CONCURRENT_CHILDREN = 10 |
At most 10 subagents run at once. The 11th waits in line. This prevents blowing through the model provider's rate limit in one shot, and also keeps the local machine from running out of memory and file handles |
_RECENT_SUBAGENTS_CAP = 200 |
The “recent subagents” record keeps at most 200 entries. This is a history buffer for features like “view subagent status”; beyond the cap, the oldest entries are dropped. It keeps long-running sessions from eating all the memory |
Together with MAX_DEPTH = 1, these two numbers lock the resource footprint of the whole multi-agent system into a predictable range: at any moment, at most 1 + 10 = 11 agents are running, and the history holds at most 200 records.
10.4 Tool Restrictions on Subagents
DELEGATE_BLOCKED_TOOLSSubagents cannot use certain tools. The most important rule: a subagent cannot call the delegate tool — this is MAX_DEPTH = 1 enforced at the tool layer.
Note that this is “belt and suspenders”:
· Logic layer: MAX_DEPTH = 1 checks the depth at delegation time
· Capability layer: DELEGATE_BLOCKED_TOOLS means the subagent never even sees the delegate tool
The second layer is more thorough — the model never gets the idea “I could delegate,” because the tool isn't on the list. Not granting a capability is more reliable than granting it and then intercepting. It is the same thinking as chapter 4's “webhooks get only 4 read-only tools.”
10.5 Approval Policy for Subagents
def _subagent_auto_deny(...)
def _subagent_auto_approve(...)There is a problem here that has to be solved: when a running subagent asks for approval, who approves?
| Function | Semantics |
|---|---|
_subagent_auto_deny |
Automatically deny. The subagent receives a “denied” result; it can try another way or report failure. Safe, but may stall the task |
_subagent_auto_approve |
Automatically approve. Let it through without asking anyone. Gets the job done, but risky |
This is the hardest trade-off in any automated system.
Auto-deny is the safe default, but it makes many legitimate tasks fail — and fail in a hidden way (the subagent reports “I couldn't do it,” when the real reason is a denied permission).
Auto-approve gets things done, but it means every operation outside the red lines runs unsupervised. The 12 red lines from chapter 5 still hold (those are absolute prohibitions), but the “needs confirmation” tier gets skipped.
The right approach: have the main agent explicitly declare the subagent's permission tier at delegation time, instead of relying on one global default. A subagent doing “read-only analysis” should auto-deny every write; a subagent doing “bulk refactoring” needs write access pre-authorized.
10.6 In-Flight Control
Subagents aren't “fire and forget.” There are three control interfaces:
def interrupt_subagent(...) # interrupt a specific subagent
def steer_subagent(...) # inject an instruction into a running subagent
def set_spawn_paused(...) # pause/resume spawning of new subagents| Interface | Purpose |
|---|---|
interrupt_subagent |
When one subagent has gone off track / gotten stuck / is burning money, stop just that one without affecting the other 9 |
steer_subagent |
Don't interrupt, but slip in a line. This maps to the /steer mechanism from chapter 3 — the new instruction is injected into the last tool message so the agent sees it on its next turn.For example: “while you're at it, check the type annotations too” |
set_spawn_paused |
Pause spawning new subagents, but let the running ones continue. Use case: when you realize the whole batch is headed in the wrong direction, stop the bleeding first — spawn nothing new, let the ones already running finish naturally, then re-plan |
The set_spawn_paused design deserves its own mention.
The naive approach offers only “continue everything” and “kill everything.” But the most common real-world need is a third option: “don't start anything new; let what's in flight finish.”
In operations this is called “draining” — graceful shutdown, rolling updates, and load-shedding all follow this pattern. A mature concurrent system must distinguish “stop accepting new work” from “terminate existing work.”
10.7 The Lineage Check
def _is_descendant_of(..., max_hops: int = 8)“Determine whether agent A is a descendant of agent B.” Uses:
- When interrupting an agent, interrupt all of its descendants along with it
- When tallying cost, charge the descendants' spending to their ancestor
- Permission checks: some operations are only allowed on your own descendants
That max_hops = 8 is defensive. Given MAX_DEPTH = 1, in theory you only ever need to check 1 hop. Setting it to 8 is for:
- Compatibility with a depth limit that might be relaxed in the future
- More importantly: if a cycle ever shows up in the data (A's parent is B, B's parent is A), this cap guarantees the function returns instead of looping forever
This is how you write “even if the invariant is broken, the program must not hang.”
Under normal conditions you never reach hop 8. But if some bug turns the parent-child relationship into a cycle, the capped version returns a (possibly wrong) answer and keeps running, while the uncapped version hangs the entire process.
Whenever you traverse a structure that “should theoretically be a tree, but whose data is built at runtime,” add a hop limit like this.
10.8 Kanban: Collaboration Between Agents
plugins/kanban/ offers a different multi-agent model — not “hand it down and wait for the result,” but “share one task board.”
kanban_create_task create a task
kanban_claim_task claim a task
kanban_update_task update progress
kanban_complete_task complete a task
kanban_list_tasks view the task list
kanban_heartbeat ★ heartbeat| Delegation mode | Kanban mode | |
|---|---|---|
| Relationship | Parent-child — the main agent assigns explicitly | Peers — whoever is free claims the work |
| Who decides what to do | The main agent | Each agent for itself |
| Lifecycle | A subagent ends when its work is done | Agents are long-lived and keep claiming new tasks |
| Suited to | Known, decomposable batch tasks | Ongoing workflows with unpredictable sources of work |
Why kanban_heartbeat has to exist
Any distributed task system built on “claim – execute – complete” must have a heartbeat or lease mechanism.
Otherwise, tasks that were “claimed but died before finishing” stay stuck forever. It's the same problem in message queues, task schedulers, and distributed locks, and the fix is the same: a claim has a time limit and must be continually renewed.
10.9 Why the Delegation System Is 5,071 Lines
Back to the number from the top. The core logic that actually “spawns a subagent” probably needs 200 lines. What are the other 4,800 doing?
| Category | Contents |
|---|---|
| Lifecycle management | Create, start, monitor, interrupt, clean up, time out, reap zombies |
| Concurrency control | Concurrency caps, queuing, paused spawning, priorities |
| Result aggregation | Collecting results from 10 subagents, handling partial failures, deciding how to count the ones that timed out |
| Status queries | “How many are running right now,” “how much has been spent,” “which step is it stuck on” |
| Control channels | Interrupt, inject instructions, pause — each has to be delivered safely across processes/threads |
| Safety boundaries | Tool blocking, approval policy, depth checks, lineage |
| Observability | Every subagent's logs, cost, and elapsed time must be recorded separately and linkable back to the parent task |
| Failure handling | Subagent crashes, model errors, context blowups, infinite loops — each needs a countermeasure |
The “agents calling agents” demo is 20 lines; the production system is 5,000.
That 250x gap comes entirely from “what to do when things go wrong.” The hard part of multi-agent systems has never been “how to spawn,” but “how to clean up when what you spawned runs out of control.”
If you're asked about multi-agent systems in an interview, being able to explain this clearly is worth far more than being able to draw a pretty architecture diagram.