本章目录In this chapter
- 11 · Model Providers and the Credential Pool
- 11.1 Why You Can't Support Just One
- 11.2 The Provider Adapter Roster
- 11.3 The Credential Pool: The Core Mechanism
- 11.4 PooledCredential: The State of a Single Credential
- 11.5 Why “Persistent” Is Necessary
- 11.6 How It Works with Other Mechanisms
- 11.7 How Deep the Provider Abstraction Should Go
11 · 模型供应商与凭据池
agent/credential_pool.py + plugins/model-providers/。这一章讲系统怎么和多家模型供应商打交道，以及一个 API key 用完了怎么办。
11.1 为什么不能只支持一家
| 原因 | 说明 |
|---|---|
| 能力差异 | 不同模型擅长的事不一样。写代码、写文案、做视觉理解，各有强弱 |
| 成本差异 | 同样一个任务，用最贵的模型和用便宜模型的价差可能有十倍 |
| 可用性 | 任何一家都会宕机、限流、改价格、改条款 |
| 合规 | 某些企业只能用云厂商托管的版本（数据不出自己的云账户） |
| 地域 | 不同地区可访问的服务不同 |
11.2 供应商适配器清单
agent/providers/
├── anthropic.py Anthropic 官方 API
├── bedrock.py AWS Bedrock（在自己的 AWS 账户里调 Claude）
├── vertex.py Google Cloud Vertex AI（在 GCP 里调 Claude）
├── azure.py Azure OpenAI
├── gemini_native.py Google Gemini 原生接口
└── codex_responses.py OpenAI Codex 的 responses 接口anthropic / bedrock / vertex 背后可能是同一个 Claude 模型，但：
- 认证方式不同 —— API key vs AWS 签名 vs Google 服务账号
- 请求格式有差异 —— 字段名、嵌套结构、必填项都不完全一样
- 错误码不同 —— 同样是「限流」，三家返回的状态码和错误体格式都不同
- 可用功能不同 —— 提示词缓存、扩展思考这些特性，各平台支持的版本和参数可能落后于官方
所以「支持 Claude」不是一件事，是三件事。每一条路径都要单独实现、单独测试、单独跟进版本变化。
11.3 凭据池：核心机制
agent/credential_pool.py
"""Persistent multi-credential pool for same-provider failover."""译：「用于同一供应商内部故障转移的持久化多凭据池」。逐词拆开：
| 词 | 含义 |
|---|---|
| multi-credential 多凭据 | 你有多个 API key（多个账号、多个组织、多个付费计划） |
| same-provider 同一供应商 | 这些 key 都是同一家的。不是「Anthropic 挂了切到 OpenAI」，而是「Anthropic 的 key A 限流了切到 key B」 |
| failover 故障转移 | 一个不行了自动换下一个，对上层透明 |
| persistent 持久化 | 状态存到磁盘 —— 重启后还记得哪个 key 已经用完了 |
为什么需要它
11.4 PooledCredential：单个凭据的状态
class PooledCredential:
...
_exhausted_ttl # 「耗尽」状态的存活时长
priority # 优先级排序_exhausted_ttl：按 HTTP 状态码决定「冷却多久」
TTL = Time To Live（存活时间）。这里指的是「这个 key 被标记为不可用后，多久之后再试一次」。
关键设计：冷却时长不是固定的，而是根据 API 返回的错误码来决定的。
| 状态码 | 含义 | 合理的冷却策略 |
|---|---|---|
429 |
限流（Too Many Requests） | 短冷却（几十秒到几分钟）。额度是按时间窗口重置的，等一会儿就恢复 |
401 |
认证失败（key 无效） | 很长冷却，或者直接永久剔除。key 错了，等多久都不会自己变对 |
402 |
需要付费（余额不足） | 长冷却（小时级）。要人工去充值，短时间重试没意义 |
403 |
无权限 | 长冷却。可能是这个 key 没开通某个模型的权限 |
5xx |
服务端错误 | 很短冷却。不是 key 的问题，是供应商临时故障 |
为什么不能用统一的冷却时长？
统一设短（比如 30 秒）→ 一个 401 的错误 key 会每 30 秒被重试一次，永远浪费请求，而且日志里全是噪声。
统一设长（比如 1 小时）→ 一个只是被短暂限流的好 key，白白闲置 1 小时。你花钱买的额度用不上。
「用错误的类型来决定重试策略」是所有重试逻辑的核心。不区分错误类型的重试，要么太急要么太懒，没有中间状态。
priority：优先级排序
凭据不是平等的。典型的排序理由：
- 成本 —— 有的是包月账户（边际成本 0），有的是按量计费。优先用包月的
- 额度 —— 有的账户额度高，有的是备用小号
- 速度 —— 有的账户在更高的服务等级上
所以选凭据的逻辑是：从可用的凭据里，按优先级取第一个。不可用的（在冷却中的）直接跳过。
11.5 「持久化」为什么必要
持久化的另一个价值：多进程共享。
Hermes 可能同时有：网关进程、定时任务进程、命令行会话进程。如果状态只在内存里，三个进程会各自撞墙三次。写到磁盘（或数据库）后，一个进程发现 key-A 满了，另外两个立刻就知道。
11.6 与其他机制的配合
和第 3 章的预算闸门
预算闸门管的是「这一轮对话花了多少钱」，凭据池管的是「用哪个 key 去花」。两者正交：预算闸门决定「还能不能花」，凭据池决定「从哪个口袋掏」。
和第 3 章的降级
回顾第 3 章：模型返回错误时会尝试降级到备用模型。现在可以看到完整的失败处理链条：
注意这四层的顺序不能乱：
换凭据是最便宜的（同一个模型、同样的上下文，只是换个身份）。
降级模型有质量代价。
压缩上下文有信息损失。
所以从代价最小的手段开始尝试。如果反过来，一遇到 429 就先压缩上下文，那就是白白丢了信息 —— 而问题根本不在上下文长度上。
11.7 供应商抽象要抽象到哪一层
这是设计多供应商支持时最容易做错的决策。
| 抽象层次 | 做法 | 问题 |
|---|---|---|
| 太薄 | 只统一「发消息」这个动作，其余暴露原始差异 | 上层代码里到处是 if provider == "bedrock"，加一家供应商要改十处 |
| 太厚 | 抽象出一个「最大公约数」接口，只保留所有供应商都支持的功能 | 丢失特性。提示词缓存、扩展思考、结构化输出这些差异化能力全用不上 —— 而这些恰恰是最有价值的 |
| 刚好 | 统一核心流程（消息、工具调用、流式），把差异化能力做成可查询的能力位 | 上层写 if provider.supports_prompt_caching() 而不是 if provider == "anthropic" |
「能力查询」这个模式在第 1 章的平台适配器里已经出现过一次。
BasePlatformAdapter 有 supports_threads() / supports_reactions() / supports_editing()，让上层代码问「你能不能做 X」而不是「你是谁」。
这里是完全相同的模式，用在了不同的领域。两个地方都面对「一群做同一件事但能力不同的外部系统」，解法一致：抽象「做什么」，查询「能做什么」，永远不要在业务代码里判断「你是谁」。
这是一个可以直接搬到任何项目里的模式 —— 支付渠道、短信通道、对象存储、推送服务，全都适用。
11 · Model Providers and the Credential Pool
agent/credential_pool.py + plugins/model-providers/. This chapter covers how the system deals with multiple model providers, and what happens when an API key runs dry.
11.1 Why You Can't Support Just One
| Reason | Explanation |
|---|---|
| Capability differences | Different models are good at different things. Writing code, writing copy, visual understanding — each has strengths and weaknesses |
| Cost differences | For the same task, the price gap between the most expensive model and a cheap one can be tenfold |
| Availability | Every provider goes down, throttles, changes prices, and changes terms |
| Compliance | Some enterprises may only use versions hosted by their cloud vendor (data never leaves their own cloud account) |
| Geography | Different regions can reach different services |
11.2 The Provider Adapter Roster
agent/providers/
├── anthropic.py Anthropic's official API
├── bedrock.py AWS Bedrock (calling Claude inside your own AWS account)
├── vertex.py Google Cloud Vertex AI (calling Claude inside GCP)
├── azure.py Azure OpenAI
├── gemini_native.py Google Gemini's native interface
└── codex_responses.py OpenAI Codex's responses interfaceBehind anthropic / bedrock / vertex may sit the very same Claude model, but:
- Authentication differs — API key vs. AWS signature vs. Google service account
- Request formats differ — field names, nesting, and required fields don't fully line up
- Error codes differ — for the same “rate limited,” all three return different status codes and error-body formats
- Available features differ — features like prompt caching and extended thinking may lag behind the official API in the versions and parameters each platform supports
So “supporting Claude” isn't one job; it's three. Each path has to be implemented separately, tested separately, and tracked separately as versions change.
11.3 The Credential Pool: The Core Mechanism
agent/credential_pool.py
"""Persistent multi-credential pool for same-provider failover."""In plain terms: “a persistent pool of multiple credentials, for failover within a single provider.” Taking it word by word:
| Term | Meaning |
|---|---|
| multi-credential several credentials | You have multiple API keys (multiple accounts, multiple organizations, multiple billing plans) |
| same-provider a single provider | All of these keys belong to the same provider. This isn't “Anthropic is down, switch to OpenAI”; it's “Anthropic key A is rate-limited, switch to key B” |
| failover automatic switchover | When one stops working, automatically move to the next, transparently to the layers above |
| persistent survives restarts | State is saved to disk — after a restart, it still remembers which key is used up |
Why it's needed
11.4 PooledCredential: The State of a Single Credential
class PooledCredential:
...
_exhausted_ttl # how long the "exhausted" state lives
priority # priority ordering_exhausted_ttl: “how long to cool down” depends on the HTTP status code
TTL = Time To Live. Here it means “after this key is marked unusable, how long until we try it again.”
The key design decision: the cooldown isn't fixed; it's determined by the error code the API returned.
| Status code | Meaning | Sensible cooldown policy |
|---|---|---|
429 |
Rate limited (Too Many Requests) | Short cooldown (tens of seconds to a few minutes). Quotas reset on a time window; wait a bit and it recovers |
401 |
Authentication failed (invalid key) | Very long cooldown, or just remove it permanently. A wrong key won't fix itself no matter how long you wait |
402 |
Payment required (insufficient balance) | Long cooldown (hours). A human has to top up the account; retrying soon is pointless |
403 |
Forbidden | Long cooldown. This key probably hasn't been granted access to a particular model |
5xx |
Server-side error | Very short cooldown. Not the key's fault; the provider is having a transient failure |
Why can't you use one uniform cooldown?
Uniformly short (say, 30 seconds) → a bad key returning 401 gets retried every 30 seconds, wasting requests forever, and the logs fill up with noise.
Uniformly long (say, 1 hour) → a perfectly good key that was only briefly throttled sits idle for an hour. Quota you paid for goes unused.
“Let the type of error decide the retry policy” is the heart of all retry logic. Retries that don't distinguish error types are either too eager or too lazy, with no middle ground.
priority: priority ordering
Credentials aren't equal. Typical reasons to rank them:
- Cost — some are flat-rate subscriptions (marginal cost zero), others are pay-as-you-go. Use the flat-rate ones first
- Quota — some accounts have high limits; others are small backup accounts
- Speed — some accounts sit on a higher service tier
So the credential selection logic is: from the available credentials, take the first by priority. Unavailable ones (in cooldown) are skipped outright.
11.5 Why “Persistent” Is Necessary
Persistence has another payoff: sharing across processes.
Hermes may have several processes running at once: the gateway process, the scheduled-task process, and CLI session processes. If the state lives only in memory, three processes each hit the wall three times. Written to disk (or a database), one process discovers key-A is full and the other two know immediately.
11.6 How It Works with Other Mechanisms
With the budget gate from chapter 3
The budget gate governs “how much has this conversation turn cost”; the credential pool governs “which key to spend with.” The two are orthogonal: the budget gate decides “can we still spend,” the credential pool decides “which pocket to pay from.”
With the fallback from chapter 3
Recall chapter 3: when the model returns an error, the system tries to fall back to a backup model. Now the complete failure-handling chain comes into view:
Note that the order of these four tiers must not be shuffled:
Switching credentials is the cheapest (same model, same context, just a different identity).
Falling back to another model has a quality cost.
Compacting the context has an information cost.
So you start with the least costly remedy. Done the other way round — compacting the context the moment you see a 429 — you'd throw away information for nothing, when the problem had nothing to do with context length.
11.7 How Deep the Provider Abstraction Should Go
This is the decision most often gotten wrong when designing multi-provider support.
| Abstraction level | Approach | Problem |
|---|---|---|
| Too thin | Unify only the act of “sending a message”; expose every other raw difference | Upstream code is littered with if provider == "bedrock"; adding one provider means touching ten places |
| Too thick | Abstract a “greatest common denominator” interface, keeping only features every provider supports | Lost features. Differentiating capabilities like prompt caching, extended thinking, and structured output go unused — and those are exactly the most valuable ones |
| Just right | Unify the core flow (messages, tool calls, streaming) and turn the differentiating capabilities into queryable capability flags | Upstream code writes if provider.supports_prompt_caching() instead of if provider == "anthropic" |
The “capability query” pattern already appeared once, in chapter 1's platform adapters.
BasePlatformAdapter has supports_threads() / supports_reactions() / supports_editing(), so upstream code asks “can you do X” rather than “who are you.”
This is exactly the same pattern, applied in a different domain. Both places face “a group of external systems that do the same job but with different capabilities,” and the solution is the same: abstract “what to do,” query “what can you do,” and never test “who are you” in business logic.
This is a pattern you can lift straight into any project — payment channels, SMS gateways, object storage, push notification services, all of it applies.