7 Commits
Author SHA1 Message Date
chakcy b1fed6f4b5 modify: inline 插件进程内直调快路径(external 不受影响,含 inline/external 一致性测试) 2026-10-13 01:59:08 +08:00
chakcy c87b7baa64 modify: JWT 结果加进程内一级缓存(L1),免每请求共享缓存同步往返(+45~68% 吞吐) 2026-10-13 01:59:05 +08:00
chakcy c92b34a55f modify: 访问日志中间件改纯 ASGI,高并发下回收 30~50% 吞吐(含异常补记 500) 2026-10-13 01:59:05 +08:00
chakcy 2df1e7266c modify: 集成 granian/uvicorn(gunicorn 保留),启动参数保持现状用函数转换 2026-10-13 01:59:00 +08:00
chakcy 3a359fa485 modify: 微服务演进 2026-10-02 11:54:46 +08:00
chakcy a0a65093c4 modify chongming-worker 2026-09-18 15:36:43 +08:00
chakcy b4a5985fe6 modify: inline 插件配置按模式对齐(网关侧兜底,兼容未升级的 worker 包)
现象:cache 模式下应用重启后插件仍按 toml 跑,而前端(读缓存)显示的是缓存值。
根因:跑起来的产物里 chongming-worker 是索引里的发布版(uv run build / uv sync 会把
uv pip install -e utils/worker 的可编辑安装替换回锁文件版本),旧版 Plugin.create_workers
在 on_init 时只拿到 config/workers/<type>/config.toml 展平出来的配置。

改动(两层对齐,规则与 SDK 一致且幂等):
- WorkerManager.align_inline_config:创建 inline 副本之前按 [cache].launch_worker_count
  对齐 [workers.config]——cache 模式取缓存值(缺失才用 toml 引导并写回),toml 模式以
  文件为准并把该段覆盖写入缓存;未配置缓存后端时原样返回。网关侧这层让网关与 worker
  包版本解耦:worker 包仍是旧版时 cache 模式同样生效。
- _discover_plugins 透出 [workers.config] 展平前的原始段;load_workers 在注册 starter/
  创建副本之前完成对齐(插件实例 on_init 用的就是这份配置)。
- test/test_inline_config_align.py:两模式 + 无缓存/无段的边界用例,并校验缓存键与 SDK 一致。
- README:补打包陷阱(可编辑安装会被 uv sync 替换回发布版)与对齐时机说明。
2026-09-18 15:23:13 +08:00
37 changed files with 2607 additions and 4745 deletions
+2 -14
View File
@@ -43,17 +43,5 @@ lib
auto.crt auto.crt
auto.key auto.key
# 机器码授权 *.pem
# 签发私钥是授权的唯一信任根,绝不入库(丢了无法再签发,泄漏则人人可签发); *.key
# license.lic 是设备运行期文件,公钥由构建期从环境变量/文件注入。
license_private*.pem
license_public.key
license.lic
# 台账库(默认在 ~/.chongming/,若显式放到项目内也不应提交)
licenses.db
licenses.db-wal
licenses.db-shm
授权台账.csv
.trae
.ruff_cache
.pytest_cache
+42 -55
View File
@@ -11,7 +11,7 @@
| 特性 | 说明 | | 特性 | 说明 |
| --------------------- | -------------------------------------------------------------------- | | --------------------- | -------------------------------------------------------------------- |
| **FastAPI + Uvicorn** | 高性能异步 Web 框架,自动生成 OpenAPI 文档 | | **FastAPI + Uvicorn** | 高性能异步 Web 框架,自动生成 OpenAPI 文档 |
| **Zenoh 服务发现** | 基于 liveliness 的 Worker 注册与 RPC 调用,负载均衡 + 超时重试 | | **Zenoh 服务发现** | 基于 liveliness 的 Worker 注册与 RPC 调用,负载均衡 + 实例定向 + 超时重试 |
| **MQTT 集成** | amqtt Broker(内嵌或外部),MQTT ↔ Worker 双向桥接 | | **MQTT 集成** | amqtt Broker(内嵌或外部),MQTT ↔ Worker 双向桥接 |
| **动态路由生成** | 扫描 `config/workers/*/*.toml` 自动注册 HTTP 路由 | | **动态路由生成** | 扫描 `config/workers/*/*.toml` 自动注册 HTTP 路由 |
| **Worker 框架** | `@rpc_method` 装饰器 + `Plugin` 基类统一生命周期 + Pydantic 模型绑定 | | **Worker 框架** | `@rpc_method` 装饰器 + `Plugin` 基类统一生命周期 + Pydantic 模型绑定 |
@@ -29,7 +29,6 @@
| **构建打包** | PyInstaller / Nuitka 一键打包为独立可执行文件 | | **构建打包** | PyInstaller / Nuitka 一键打包为独立可执行文件 |
| **Vue 3 前端** | Element Plus UI、Pinia 状态管理、类型安全的 API 客户端 | | **Vue 3 前端** | Element Plus UI、Pinia 状态管理、类型安全的 API 客户端 |
| **Worker 运维管理** | 前端经 Zenoh 向单个副本下发 restart/stop/改配置指令(超管),前端可视化列表 + 分页管理 | | **Worker 运维管理** | 前端经 Zenoh 向单个副本下发 restart/stop/改配置指令(超管),前端可视化列表 + 分页管理 |
| **机器码授权** | 离线 Ed25519 签名 license:绑定机器指纹 + 插件白名单 + 有效期,未授权插件不加载 |
--- ---
@@ -114,11 +113,22 @@ uv sync # utils/* 由 uv 按 pyproject 的 git 源自动安装,无需手
**修改 `utils/<name>`(工具包)**: **修改 `utils/<name>`(工具包)**:
1. 在 `utils/<name>` 内提交并 push 到对应远程 1. 在 `utils/<name>` 内提交并 push 到对应远程(并发布新版本,见下)
2. 回主仓库执行 `uv lock --upgrade-package <包名>` 刷新版本 pin 2. 回主仓库执行 `uv lock --upgrade-package <包名>` 刷新版本 pin
3. `uv sync` 后运行相关测试 3. `uv sync` 后运行相关测试
4. 主仓库提交 `uv.lock` 4. 主仓库提交 `uv.lock`
> ⚠️ **打包陷阱("改了 utils/* 但产物行为没变"几乎都是它)**:`uv run build` 会先按
> 锁文件同步环境,**`uv pip install -e utils/<name>` 的可编辑安装会被替换回锁文件里的
> 发布版**。因此本地改了工具包后,必须走上面 1-4 步(push + 发版 → 刷新 pin → sync),
> 否则 `build/` 产物里仍是旧版代码。仅本地联调时可临时用可编辑安装,但打包前要确认
> venv 指向的是准备发布的版本:
>
> ```bash
> uv pip install -e utils/worker --no-deps # 仅联调:让 venv 用本地源码
> python -c "import chongming_worker, inspect; print(inspect.getsourcefile(chongming_worker))"
> ```
**修改 `src/plugins/` / `config/` / `src/chongming-web/`**: **修改 `src/plugins/` / `config/` / `src/chongming-web/`**:
1. 在对应子仓库内提交并 push 1. 在对应子仓库内提交并 push
@@ -128,6 +138,8 @@ uv sync # utils/* 由 uv 按 pyproject 的 git 源自动安装,无需手
- [ ] 所有子仓库 `git status` 干净且已 push - [ ] 所有子仓库 `git status` 干净且已 push
- [ ] `uv.lock` 与各 git 源最新提交一致 - [ ] `uv.lock` 与各 git 源最新提交一致
- [ ] 打包前确认 venv 里 `chongming-worker` 等工具包为目标版本(改了 `utils/*` 却仍是旧版时,
产物行为不会变,见上一节打包陷阱)
- [ ] 配置中无硬编码凭据——密钥一律走环境变量: - [ ] 配置中无硬编码凭据——密钥一律走环境变量:
`VOLC_ACCESSKEY`(豆包 Ark Key)、`AIBOX_MQTT_URI`(AI Box 外部 Broker URI) `VOLC_ACCESSKEY`(豆包 Ark Key)、`AIBOX_MQTT_URI`(AI Box 外部 Broker URI)
- [ ] `uv run python test_mqtt_e2e.py` 等测试通过 - [ ] `uv run python test_mqtt_e2e.py` 等测试通过
@@ -182,7 +194,7 @@ graph TB
MQTT[MQTT 客户端] -->|chongming/+/+| BRIDGE[MQTT Bridge<br/>纯客户端模式] MQTT[MQTT 客户端] -->|chongming/+/+| BRIDGE[MQTT Bridge<br/>纯客户端模式]
REG[RouteRegistry<br/>订阅 chongming/routes/**] -->|动态挂载/摘除 HTTP 路由| API REG[RouteRegistry<br/>订阅 chongming/routes/**] -->|动态挂载/摘除 HTTP 路由| API
API --> WM[WorkerManager<br/>随机负载均衡 + 超时重试] API --> WM[WorkerManager<br/>随机负载均衡 / 实例定向 + 超时重试]
BRIDGE --> WM BRIDGE --> WM
subgraph GW[网关进程 chongming] subgraph GW[网关进程 chongming]
@@ -298,6 +310,8 @@ worker/<type>/<worker_id>/__admin__
两种模式都不回写 `config.toml`:管理接口的改动只进缓存。`cache` 模式下启动日志会打印 toml 与缓存的差异键(以缓存为准);如需让 toml 的改动生效,先删除 `worker:cm:configs:<type>`(运行时配置)或 `worker:cm:counts:<type>`(副本数)缓存键再重启,或改用 `"toml"` 模式。`GET /admin/workers/<type>/config` 的响应带 `source`(cache/toml)与 `mode`,前端据此标注展示值的来路。 两种模式都不回写 `config.toml`:管理接口的改动只进缓存。`cache` 模式下启动日志会打印 toml 与缓存的差异键(以缓存为准);如需让 toml 的改动生效,先删除 `worker:cm:configs:<type>`(运行时配置)或 `worker:cm:counts:<type>`(副本数)缓存键再重启,或改用 `"toml"` 模式。`GET /admin/workers/<type>/config` 的响应带 `source`(cache/toml)与 `mode`,前端据此标注展示值的来路。
- **对齐时机(两层,规则一致且幂等)**:inline 插件由**网关**在创建副本之前对齐——`WorkerManager.align_inline_config()` 在收到展平配置与 `[workers.config]` 原始段后按模式取缓存/写缓存,再交给插件的 `create_workers`;worker SDK 的 `Plugin.create_workers` 在 `_bootstrap()`/`on_init()` 之前用 `resolve_startup_config()` 再解析一次。网关侧这层是为了让网关与 worker 包版本解耦:worker 包仍是旧版时 cache 模式同样生效(否则旧版 `on_init` 只拿到 toml 展平配置,表现为"前端显示缓存值、实际跑 toml")。**external 插件**没有网关这层兜底,由插件进程内的 SDK 解析,需要 worker 包为含该改动的版本。
- **进程插件(external)实例粒度参与分摊**:网关只对 **inline** 插件用 cache 解析启动副本数(见「启动流程」第 5 步 `load_workers`),external 插件的启动副本数由外部编排按自身配置决定;但**运行时的缩放**上,external 并不退化为“每进程全额”,而是同样在**共享 cache 上登记实例参与方**来分摊,避免多实例把副本翻倍: - **进程插件(external)实例粒度参与分摊**:网关只对 **inline** 插件用 cache 解析启动副本数(见「启动流程」第 5 步 `load_workers`),external 插件的启动副本数由外部编排按自身配置决定;但**运行时的缩放**上,external 并不退化为“每进程全额”,而是同样在**共享 cache 上登记实例参与方**来分摊,避免多实例把副本翻倍:
- **实例登记 + 心跳租约**:每个 external 进程启动时向 `worker:cm:instances:<type>` 登记自身参与方(`[instance_id, ts]`),并每 15s 心跳续约;超过租约(60s)未心跳的死实例会被清理。`divmod` 按存活参与方 N 均分全局 count(余量补靠前实例),`N==1` 时退化为全额(单部署零影响)。 - **实例登记 + 心跳租约**:每个 external 进程启动时向 `worker:cm:instances:<type>` 登记自身参与方(`[instance_id, ts]`),并每 15s 心跳续约;超过租约(60s)未心跳的死实例会被清理。`divmod` 按存活参与方 N 均分全局 count(余量补靠前实例),`N==1` 时退化为全额(单部署零影响)。
- **实例标识默认唯一**:`--instance` / 环境变量 `CHONGMING_WORKER_INSTANCE` 显式指定(跨重启稳定、逐实例精细扩缩可持续命中);缺省回退 `{hostname}:{pid}`——**同一机器并行多个未指定 `--instance` 的进程也能靠 pid 区分**,不会因为 id 相同被并成一个参与方导致副本翻倍或前端只显示一个实例。注意缺省 id 随进程重启变化。 - **实例标识默认唯一**:`--instance` / 环境变量 `CHONGMING_WORKER_INSTANCE` 显式指定(跨重启稳定、逐实例精细扩缩可持续命中);缺省回退 `{hostname}:{pid}`——**同一机器并行多个未指定 `--instance` 的进程也能靠 pid 区分**,不会因为 id 相同被并成一个参与方导致副本翻倍或前端只显示一个实例。注意缺省 id 随进程重启变化。
@@ -314,6 +328,30 @@ worker/<type>/<worker_id>/__admin__
- **REST / UI**:`POST /admin/workers/<type>/scale_instance`(body `{instance_id, count}`)下发;`GET /admin/workers/<type>/instances` 供前端按实例分组展示当前副本数。前端用「调整数量」(全局对称)和「逐实例精细扩缩」表格(实例 id / 模式 / 当前副本数 / 目标 + 应用)两个入口。 - **REST / UI**:`POST /admin/workers/<type>/scale_instance`(body `{instance_id, count}`)下发;`GET /admin/workers/<type>/instances` 供前端按实例分组展示当前副本数。前端用「调整数量」(全局对称)和「逐实例精细扩缩」表格(实例 id / 模式 / 当前副本数 / 目标 + 应用)两个入口。
- **协调前提**:与外部广播缩放一致,需共享 `[cache]`,且外部进程用稳定的 `--instance` / `CHONGMING_WORKER_INSTANCE`(缺省 `{hostname}:{pid}` 虽唯一但重启会变,跨重启持续命中请显式指定)。 - **协调前提**:与外部广播缩放一致,需共享 `[cache]`,且外部进程用稳定的 `--instance` / `CHONGMING_WORKER_INSTANCE`(缺省 `{hostname}:{pid}` 虽唯一但重启会变,跨重启持续命中请显式指定)。
### 实例定向调用(把请求固定到某个进程实例)
默认情况下,同 type 的多个副本(含 inline 与 external 进程)逐请求**随机负载均衡**(`random.choice`),这是绝大多数无状态插件想要的。若某插件持有**只存在于某个进程本地**的状态(本地 SQLite/文件、串口或其他设备连接),需要请求固定打到那个进程上,可用**实例定向**:
- **指定方式(三种入口,语义一致)**:
- HTTP:请求头 `X-Chongming-Instance: <实例id>`
- MQTT:信封载荷顶层加控制字段 `target_instance`(网关读后**摘除**,业务 side 看不到该字段)
- 程序内调用:`worker_manager.call(...)` / `call_raw(...)` / `call_stream(...)` 的 `instance_id=` 参数(`utils/request` 的独立 Client 不支持定向)
- **实例 id 取值**:worker 侧按 `--instance` → `CHONGMING_WORKER_INSTANCE` → `CHONGMING_GATEWAY_ID` → `{hostname}:{pid}` 的顺序解析(`current_instance_id()`);网关侧定向寻址 inline 副本时用网关自身实例 id(`CHONGMING_GATEWAY_ID`,缺省 hostname)。两侧缺省值在容器环境(K8s 给网关进程注入 `CHONGMING_GATEWAY_ID`)下**自动一致**,无需额外配置;裸机同一台跑多个网关时请显式指定,否则默认实例 id 里的 pid 不同会导致对不上。人可读的取值列表见 `GET /api/v1/admin/workers/<type>/instances`。
- **粒度**:定向到**进程实例**;若该实例内有多个副本,请求仍在这几个副本之间随机均衡(即“实例固定、副本仍均衡”)。
- **实例离线时严格报错**:返回 HTTP 503 并附已识别实例列表,**不会静默回退到其它实例**——回退会让粘滞状态的插件跑在错误副本上,产生难查的数据错乱。MQTT 同理(响应主题收到 `{"error": ...}`)。
- **重试不跨实例**:调用失败重试只在同实例内换副本;实例内已无可换副本时立即失败,不重复打同一个副本。流式端点本身不重试,实例离线即失败。
- **缓存按实例隔离**:定向且 `cacheable=true` 时缓存键带实例前缀,避免取到其它实例的结果。
- **身份来源与开销**:副本的实例身份取自其 admin status 回复的 `instance_id`(Python/Rust 两种运行时都会返回),网关**按需查询并缓存**——未使用定向功能时零额外开销,不定向的请求路径与行为完全不变;首次定向到某实例时会对该类型的新副本各发一次轻量 status 查询(每个副本只查一次,inline 副本无需查询)。副本重启后 id 变化,会重新查询一次。
- **版本要求**:worker 端需运行带 `instance_id` 上报的运行时(当前 Python/Rust SDK 均满足);读不出身份的副本会被排除在定向之外(不定向的调用不受影响)。
```bash
# 等价于把这次请求固定到实例 edge-01 的那个进程
curl -H 'X-Chongming-Instance: edge-01' http://127.0.0.1:8000/api/v1/demo/state
# MQTT:载荷顶层带 target_instance(网关转发前会摘掉它)
# topic: chongming/demo/state payload: {"meta":{}, "data":{}, "target_instance":"edge-01"}
```
### inline worker 宿主级领导选举(gunicorn 多 worker 兼容) ### inline worker 宿主级领导选举(gunicorn 多 worker 兼容)
inline worker 的生命周期交给**基于分布式锁的宿主级领导选举**(取代“每 gunicorn worker 各自拉起”),key `inline_owner:<gateway_id>` 按网关隔离: inline worker 的生命周期交给**基于分布式锁的宿主级领导选举**(取代“每 gunicorn worker 各自拉起”),key `inline_owner:<gateway_id>` 按网关隔离:
@@ -357,56 +395,6 @@ external 副本走同一 `__admin__` Queryable,路由发现、指令下发与
--- ---
## 机器码授权
按客户设备「机器码」签发授权:**一份 license 绑定一台机器 + 一批插件 + 有效期**,
未授权的插件不加载、不注册路由、管理接口禁止操作。**纯离线** —— 设备用内置厂商公钥验签,
运行期零联网(与 Zenoh localhost、Caddy `tls internal` 的 air-gapped 形态一致)。
> 📖 **完整指南见 [docs/license.md](docs/license.md)** —— 密钥管理、激活流程、
> 命令行参考、可视化工具、机器指纹与容错判定、强制执行点、安全边界、
> 故障排查(含原因码全表)。下面只保留最小上手路径。
>
> 厂商侧有**两种等价工具**(共用同一份实现):不习惯命令行的话用可视化工具
> —— `cd utils/license-gui && uv sync && uv run chongming-license-gui`(签发 / 密钥 / 校验三页)。
### 最小上手路径
```bash
# ① 厂商侧:生成密钥对(一次性)。私钥离线保管并备份,绝不进仓库/产物
uv run chongming-license keygen -o license_private.pem
# → 输出公钥,按提示注入构建(CHONGMING_LICENSE_PUBKEY 环境变量 / license_public.key 文件)
# → 打包时缺少公钥会直接失败:缺公钥的产物会拒绝加载所有插件,等于砖头
# ② 设备侧:浏览器打开 http://<设备IP>:8000/static/license,登录后「复制申请码」
# ③ 厂商侧:签发
uv run chongming-license issue --request-code "CM1-客户发来的..." \
--plugins aibox_sophon_adapter,ip_sound_column \
--customer "某某科技" --days 365 -k license_private.pem -o license.lic
# ④ 设备侧:同一页面「导入并校验」→「重启网关」生效
```
签发会**自动记入台账**(`~/.chongming/licenses.db`,含机器指纹)。续期时直接复用,
**无需再向客户索取申请码**:
```bash
uv run chongming-license list # 台账检索(按到期日升序)
uv run chongming-license renew --lic-id L2026... --days 365 -k license_private.pem
uv run chongming-license export --out ./archive --csv ./台账.csv --by-customer
```
### 三条需要记住的约束
- **申请码 vs 机器码**:机器码是短哈希,只报它只能签出**精确匹配**的授权(换网卡即失效)。
申请码带回各来源原始值,才能签出支持换硬件的**容错**授权。
- **开发态默认不执行拒绝**:`[license] enabled` 默认 `false`,改配置也绕不过(打包产物恒为强制校验,
依据 `sys.frozen`/`__compiled__` 判定)。开发时想验证门禁需手动改成 `true`。
- **设备侧没有签发命令**:`chongming-cli` 是开发依赖,不随产物发布 —— 签发能力只存在于厂商机器上。
---
## 端点类型与路由注册 ## 端点类型与路由注册
### 端点类型 ### 端点类型
@@ -910,6 +898,5 @@ uv pip install -e utils/<name> # 安装/更新工具包子包
- [配置说明](config/README.md) - [配置说明](config/README.md)
- [插件开发指南](src/plugins/README.md) - [插件开发指南](src/plugins/README.md)
- [进程插件开发指南(external 模式,从开发到部署)](docs/process-plugin.md) - [进程插件开发指南(external 模式,从开发到部署)](docs/process-plugin.md)
- [机器码授权指南(密钥管理 / 激活流程 / 故障排查)](docs/license.md)
- [前端开发指南](src/chongming-web/README.md) - [前端开发指南](src/chongming-web/README.md)
- [Worker 框架](utils/worker/README.md) - [Worker 框架](utils/worker/README.md)
+32 -34
View File
@@ -11,8 +11,8 @@ upload_path = "uploads"
json5_path = "./config/zenoh_config.json5" json5_path = "./config/zenoh_config.json5"
[cache] [cache]
backend = "sqlite" # "sqlite"(默认,本机共享) | "redis"(跨主机) # backend = "sqlite" # "sqlite"(默认,本机共享) | "redis"(跨主机)
# backend = "redis" backend = "redis"
# path = "./cache_data/cache.db" # path = "./cache_data/cache.db"
# 配置优先来源模式(同时决定 [workers.config] 与 count 在启动时的取舍): # 配置优先来源模式(同时决定 [workers.config] 与 count 在启动时的取舍):
@@ -38,22 +38,6 @@ redis_prefix = "chongming:"
algorithm = "HS256" algorithm = "HS256"
secret_key = "your-secret-key-change-in-production" secret_key = "your-secret-key-change-in-production"
[license]
# 机器码授权(插件白名单)。未授权插件不加载、不注册路由、管理接口禁止操作。
#
# enabled 只在**非打包态**(uv run serve / 测试)决定是否执行拒绝:打包产物
# (Nuitka/PyInstaller)恒为强制校验 —— 部署目录的 config.toml 是明文可写的,
# 若以配置开关为准,改一行就能绕过授权。
enabled = false
# 授权文件路径(相对网关工作目录)
path = "license.lic"
# 厂商公钥(base64 裸 32 字节,或 PEM)。公钥可明文放置:换公钥只会验签失败,
# 无法据以伪造授权(伪造需要厂商私钥)。构建时由 build.py 从
# CHONGMING_LICENSE_PUBKEY / license_public.key 注入。
public_key = ""
# 指纹命中阈值(N-of-M)。4 个可信来源下取 3 表示容忍单块网卡或单颗 eMMC 更换。
min_match = 3
[development] [development]
debug = true debug = true
env = "development" env = "development"
@@ -94,8 +78,8 @@ port = 1883
# true (默认)= 网关自带 amqtt Broker,host/port 为 Broker 监听地址 # true (默认)= 网关自带 amqtt Broker,host/port 为 Broker 监听地址
# false = 使用外部 MQTT Broker,网关仅启动 Bridge 客户端连接 # false = 使用外部 MQTT Broker,网关仅启动 Bridge 客户端连接
# host/port 需指向外部 Broker 地址(如 "192.168.1.10") # host/port 需指向外部 Broker 地址(如 "192.168.1.10")
use_embedded_broker = true # use_embedded_broker = true
# use_embedded_broker = false use_embedded_broker = false
# MQTT Broker 认证(可选,默认匿名): # MQTT Broker 认证(可选,默认匿名):
# - 内嵌 Broker(use_embedded_broker=true):配置后要求所有客户端(含网关 Bridge) # - 内嵌 Broker(use_embedded_broker=true):配置后要求所有客户端(含网关 Bridge)
# 用该账号密码连接,未配置则保持匿名 # 用该账号密码连接,未配置则保持匿名
@@ -128,9 +112,6 @@ base = [
"chongming_worker.routes_publisher", "chongming_worker.routes_publisher",
"chongming_request", "chongming_request",
"chongming_auth", "chongming_auth",
"chongming_license", # 机器码授权(指纹采集/验签/判定)
"cryptography", # Ed25519 验签
"cryptography.hazmat.primitives.asymmetric.ed25519",
"server", "server",
"module_bank", # 模块化打包工具 "module_bank", # 模块化打包工具
"sqlite_vfs", "sqlite_vfs",
@@ -142,6 +123,27 @@ base = [
"fastapi.middleware.cors", "fastapi.middleware.cors",
"fastapi.middleware.gzip", "fastapi.middleware.gzip",
"uvicorn", "uvicorn",
"granian", # 备选服务器(APP_SERVER=granian,见 public/main.py),含 Rust 内核
"granian._compat",
"granian._futures",
"granian._granian",
"granian._imports",
"granian._internal",
"granian._loops",
"granian._signals",
"granian._types",
"granian.asgi",
"granian.constants",
"granian.errors",
"granian.http",
"granian.log",
"granian.net",
"granian.rsgi",
"granian.server",
"granian.server.common",
"granian.server.mp",
"granian.server.mt",
"granian.wsgi",
"chongming_cache", "chongming_cache",
"chongming_cache.backends", "chongming_cache.backends",
"chongming_cache.backends.sqlite_", "chongming_cache.backends.sqlite_",
@@ -168,20 +170,16 @@ base = [
"aiofiles", "aiofiles",
"openai", "openai",
] ]
linux = ["gunicorn.glogging", "gunicorn.app", "gunicorn.app.wsgiapp"] linux = [
"gunicorn.glogging",
"gunicorn.app",
"gunicorn.app.wsgiapp",
]
windows = [] windows = []
[production.module_system] [production.module_system]
path = [ path = [
[ ["resources/app.mbank", "src/chongming/app", "app"],
"resources/app.mbank", ["resources/plugins.mbank", "src/plugins", "plugins"],
"src/chongming/app",
"app",
],
[
"resources/plugins.mbank",
"src/plugins",
"plugins",
],
] ]
type = "module-bank" type = "module-bank"
+7 -2
View File
@@ -16,6 +16,10 @@
# #
# 可选:/etc/chongming.env 中写 APP_WORKERS=<N> 覆盖 worker 数(默认取 # 可选:/etc/chongming.env 中写 APP_WORKERS=<N> 覆盖 worker 数(默认取
# config.toml [production.server].workers,打包版通常为 4)。 # config.toml [production.server].workers,打包版通常为 4)。
# 可选:APP_SERVER=granian|uvicorn|gunicorn 切换 HTTP 服务器(默认 Linux 下
# gunicorn,Windows 下 uvicorn);三者的启动参数同取自 [production.server]。
# 可选:CHONGMING_RPC_LOCAL=0 关闭 inline 插件的进程内直调快路径(默认开启;
# 仅影响本进程托管的 inline 副本,external 插件始终走 Zenoh)。
[Unit] [Unit]
Description=Chongming Zenoh Gateway (FastAPI + Zenoh + MQTT) Description=Chongming Zenoh Gateway (FastAPI + Zenoh + MQTT)
@@ -29,7 +33,8 @@ StartLimitBurst=5
[Service] [Service]
Type=simple Type=simple
# 打包应用入口:Nuitka 编译产物,前台运行 gunicorn master(Linux 分支) # 打包应用入口:Nuitka 编译产物,前台运行所选服务器 master(默认 Linux 分支为
# gunicorn,可用 APP_SERVER 切到 granian/uvicorn)
ExecStart=/data/chongming/chongming ExecStart=/data/chongming/chongming
# 所有相对路径(config.toml / config/ / resources/*.mbank / static.svfs / # 所有相对路径(config.toml / config/ / resources/*.mbank / static.svfs /
# data/ / logs/ / uploads/)均相对工作目录解析,必须指向安装目录 # data/ / logs/ / uploads/)均相对工作目录解析,必须指向安装目录
@@ -48,7 +53,7 @@ Environment=PYTHONUNBUFFERED=1
# 私有动态库,再补一条 Environment=LD_LIBRARY_PATH=/opt/sophon/sophon-ffmpeg-latest/lib # 私有动态库,再补一条 Environment=LD_LIBRARY_PATH=/opt/sophon/sophon-ffmpeg-latest/lib
Environment=PATH=/opt/sophon/sophon-ffmpeg-latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Environment=PATH=/opt/sophon/sophon-ffmpeg-latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# 优雅停机:SIGTERM 由 gunicorn 排空 worker,等在途请求/告警上传完成 # 优雅停机:SIGTERM 由服务器 master 排空 worker,等在途请求/告警上传完成
KillSignal=SIGTERM KillSignal=SIGTERM
KillMode=control-group KillMode=control-group
TimeoutStopSec=30 TimeoutStopSec=30
-1183
View File
File diff suppressed because it is too large Load Diff
-858
View File
@@ -1,858 +0,0 @@
# 机器码授权指南
> 机器码授权按客户设备签发 license:**一份 license 绑定一台机器 + 一批插件 + 有效期**。
> 未授权的插件**不加载、不注册路由、管理接口禁止操作**。
> **纯离线** —— 设备用内置厂商公钥验签,运行期零联网,与现有 air-gapped 部署形态
> (Zenoh 仅 localhost、Caddy `tls internal`、无 phone-home)完全一致。
>
> 本文档覆盖 **密钥生成 → 设备导出申请码 → 厂商签发 → 设备导入 → 验证 → 故障排查** 全流程。
---
## 1. 原理与角色
### 1.1 为什么这么设计
| 决策 | 原因 |
| ---- | ---- |
| **非对称签名(Ed25519)** | 设备只需内置公钥即可验签;公钥可明文放置(换公钥只会验签失败,无法据以伪造)。对称方案不可行 —— 见 §9.1 |
| **纯离线** | 现场设备常无外网,且现有部署形态本就 air-gapped;联网激活需要新建厂商侧服务,与本项目架构冲突 |
| **N-of-M 指纹容错** | 把所有硬件标识哈希成一个值的话,换一块网卡或换一颗 eMMC 就失效。改为记录逐来源指纹、按命中数判定 |
| **打包态决定是否强制** | 部署目录的 `config.toml` 是明文可写的,若以配置开关为准,客户改一行就绕过了 |
| **未授权 = 不加载** | 代码不下发到现场,比"加载后返回 403"更彻底 |
### 1.2 三方角色
| 角色 | 持有 | 做什么 | 用什么工具 |
| ---- | ---- | ------ | ---------- |
| **厂商** | 签发**私钥** | 生成密钥对、签发 license | **可视化工具**(推荐)/ `uv run chongming-license`(CLI) |
| **设备** | 内置**公钥** | 导出申请码、导入 license、验签放行 | 浏览器 `/static/license`(网页) |
| **客户** | —— | 把申请码转给厂商、把 license 文件放回设备 | —— |
> **设备上没有签发命令**:`chongming-cli` 是开发依赖,不随打包产物发布。这是刻意的 ——
> 签发能力只存在于厂商机器上,设备端不携带。设备侧一律走网页。
>
> 厂商侧有**两种等价的工具**,共用同一份授权实现(`chongming-license` 库):
> 单台/少量、需要人工核对信息时用可视化工具;批量签发(几十上百台)用 CLI 脚本化。
> 详见 §4 与 §12。
### 1.3 全流程
```mermaid
sequenceDiagram
participant V as 厂商<br/>(私钥)
participant D as 设备<br/>(公钥)
participant C as 客户
Note over V: ① keygen 生成密钥对(一次性)<br/>公钥注入构建产物
D->>D: ② 采集硬件指纹
D->>C: ③ 申请码 CM1-xxxx-yyyy
C->>V: 转交申请码
V->>V: ④ 私钥签发 license.lic
V->>C: 交付 license.lic
C->>D: ⑤ 网页上传 / scp 到工作目录
D->>D: ⑥ 重启网关,按白名单加载插件
```
---
## 2. 密钥管理(厂商侧)
### 2.1 生成密钥对(一次性)
```bash
cd <项目根>
uv run chongming-license keygen -o license_private.pem
```
输出:
```
✓ 私钥已写入 license_private.pem(权限 0600)
厂商公钥(填入部署产物的 [license] 段):
public_key = "yGHeVrFjajV4XgtBkiE9X9KZ4uy0IzF56GkQ0wwqHG0="
公钥指纹:7a9a7c9d427ee864
```
- 私钥文件权限自动置 `0600`;已存在时需显式 `--force` 覆盖(覆盖后旧 license 仍可用旧公钥校验,
但无法再用旧私钥签新授权)。
- 记下 **公钥指纹**(前 16 位十六进制):设备端、日志、`/license/info` 都会显示它,
用于确认"设备内置的是哪把公钥"。
### 2.2 私钥保管要求
> ⚠️ 私钥是授权的**唯一信任根**。
>
> - **丢失** → 再也无法签发新授权(已签发的仍有效,因为校验只需公钥)。
> - **泄漏** → 任何人都能签发授权,整套机器码绑定形同虚设。
>
> 请离线保管、做好备份,**绝不提交仓库或打进部署产物**。
已内置的防护:
- `.gitignore` 忽略 `license_private*.pem` / `license_public.key` / `license.lic`
- `uv run build` 的 **授权私钥守卫**阶段按**内容**扫描 `build/`(跨 1MB 块边界也能检出),
发现私钥立即终止构建
- `keygen` 每次都会打印保管警示
### 2.3 公钥注入构建
构建时按以下顺序解析公钥,注入产物 `config.toml` 的 `[license] public_key`:
1. 环境变量 `CHONGMING_LICENSE_PUBKEY`
2. 环境变量 `CHONGMING_LICENSE_PUBKEY_FILE` 指定的文件
3. 项目根的 `license_public.key`
```bash
# 方式 A:环境变量(CI 推荐)
export CHONGMING_LICENSE_PUBKEY="yGHeVrFjajV4XgtBkiE9X9KZ4uy0IzF56GkQ0wwqHG0="
# 方式 B:文件
echo "yGHeVrFjajV4XgtBkiE9X9KZ4uy0IzF56GkQ0wwqHG0=" > license_public.key
# 方式 C:直接编辑 public/config.toml 的 [license] 段
```
> **打包时缺少公钥会直接失败** —— 这是刻意设计。缺公钥的产物会拒绝加载**全部**插件
> (谁都无法证明被授权),等于把设备变成砖头,而这种故障往往到客户现场才暴露,
> 代价远大于构建失败。仅内部演示可用 `CHONGMING_LICENSE_ALLOW_NO_KEY=1` 跳过(会打印警告)。
### 2.4 密钥轮换
公钥指纹进入判定,因此轮换是显式操作:
1. `keygen -o new_private.pem` 生成新密钥对
2. 用新公钥重新构建并分发产物
3. 已发出去的 license 需用**新私钥**重签(旧公钥不再被内置)
---
## 3. 激活流程
### 3.1 设备侧:导出申请码
浏览器打开 `http://<设备IP>:8000/static/license`,用 `admin` 登录(默认账号已在设备上 seed),
点「复制申请码」。
页面上同时可见:
- **机器码**(短哈希,如 `BC60-D20E-F863-7236-68E2-6A24`)—— 适合口头/日志核对
- **申请码**(单行文本,`CM1-<base64>-<校验和>`)—— 发给厂商签发
- **硬件指纹来源明细** —— 哪个来源采集成功、哪个缺失,装机时用来核对
> **为什么用申请码而不是只报机器码**:机器码是单向哈希,**反推不出各来源指纹**,
> 而 N-of-M 容错匹配需要逐来源原始值。只报机器码也能签(`issue --machine-code ...`),
> 但那样的授权是**精确匹配** —— 换一块网卡就失效。
命令行等价物(供调试;设备上通常没有 CLI):
```bash
uv run chongming-license show --request-code # 只输出申请码
uv run chongming-license show --sources # 各来源明细表
uv run chongming-license show --json # 全量 JSON(可存文件传给厂商)
```
### 3.2 厂商侧:签发
```bash
uv run chongming-license issue \
--request-code "CM1-eyJzb3VyY2VzIjp7..." \
--plugins aibox_sophon_adapter,ip_sound_column \
--customer "某某科技" --days 365 \
-k license_private.pem -o license.lic
```
输出:
```
✓ 已签发 license.lic
授权编号 L20260921-334B33
客户 某某科技
机器码 BC60-D20E-F863-7236-68E2-6A24
指纹来源 mac(不参与判定:machine_id);容错阈值 3
授权插件 ['aibox_sophon_adapter', 'ip_sound_column']
有效期 立即生效 → 2027-09-21
公钥指纹 3d956a50db56b53c
```
要签发给客户的三种入参(优先级从高到低):
| 入参 | 来源 | 容错 |
| ---- | ---- | ---- |
| `--request-code CM1-...` | 设备网页「复制申请码」 | ✅ 支持换硬件容错 |
| `--sources-json <文件>` | `show --json` 的导出文件(批量/邮件附件场景) | ✅ |
| `--machine-code XXXX-...` | 客户只读了短码 | ❌ 精确匹配,换网卡即失效 |
### 3.3 设备侧:导入 license
在同一个「授权」页点「选择 license 文件」→「导入并校验」。
- 文件落盘位置由 `[license] path` 决定,默认网关工作目录下的 `license.lic`
(systemd 部署即 `/data/chongming/license.lic`)。
- **校验不过不会覆盖现有文件** —— 现场误传坏文件不至于把已经能用的授权弄丢。
- 若改用 scp 手工放置,记得在页面上点「从磁盘重读」刷新状态。
### 3.4 重启生效
新授权的插件**需重启网关**才会加载(插件发现只在启动期执行)。导入成功后页面会提示
需要重启、以及哪几个插件待加载;接口响应里的 `restart_required` / `pending_plugins` 同理。
重启后日志:
```
[license] ✓ 授权有效(本机机器码 BC60-D20E-F863-7236-68E2-6A24)
[license] 编号 L20260921-334B33,客户 某某科技,到期 2027-09-21,授权插件 ['aibox_sophon_adapter', 'ip_sound_column']
[license] 厂商公钥指纹 3d956a50db56b53c
```
未授权插件会明确记日志:
```
[license] 插件 aibox_sophon_adapter 未获授权:该插件不在授权白名单内
授权门禁已拒绝 1 个插件(不加载、不注册路由、管理接口禁止操作): ['aibox_sophon_adapter']
```
---
## 4. 命令行参考
命令注册在 `chongming-cli`(根仓 dev 依赖),入口 `uv run chongming-license`。
**退出码**:`0` 成功/校验通过,`1` 失败(便于脚本判断)。
> 不习惯命令行的同学可用可视化工具,功能对等 —— 见 §12。
### 4.1 `keygen` — 生成密钥对(厂商侧,一次性)
| 参数 | 默认 | 说明 |
| ---- | ---- | ---- |
| `-o, --output` | `license_private.pem` | 私钥输出路径 |
| `--kid` | —— | 建议的密钥标识,仅提示用途 |
| `--force` | —— | 覆盖已存在的私钥文件 |
### 4.2 `show` — 导出机器码/申请码(设备侧)
| 参数 | 说明 |
| ---- | ---- |
| `--request-code` | 只输出申请码(单行,供客户粘贴回传) |
| `--sources` | 打印各硬件来源明细表(现场核对用) |
| `--json` | 以 JSON 输出全部内容 |
### 4.3 `issue` — 签发(厂商侧)
| 参数 | 必填 | 说明 |
| ---- | ---- | ---- |
| `--request-code` / `--sources-json` / `--machine-code` | 三选一 | 签发对象(见 §3.2 表) |
| `--plugins` | ✅ | 授权插件 type,逗号分隔 |
| `-k, --key` | | 私钥路径,默认 `license_private.pem` |
| `-o, --output` | | 输出路径,默认 `license.lic` |
| `--customer` | | 客户名称(仅展示/审计) |
| `--lic-id` | | 授权编号,默认 `L<日期>-<随机>` |
| `--kid` | | 密钥标识(写入文件,仅提示) |
| `--not-before` | | 生效日期 ISO8601,默认立即生效 |
| `--expires` | | 到期日期 ISO8601,**含当天** |
| `--days` | | 有效期天数(与 `--expires` 二选一) |
| `--note` | | 备注(合同号等) |
| `--min-match` | | 容错阈值(默认 3,仅影响提示文字) |
> 日期一律按 **UTC** 计算,与校验端保持一致 —— 否则时区跨日时会差一天
> (出现"刚签发即过期"或"到期后仍可用一天")。
### 4.4 `inspect` — 查看内容(可选验签)
```bash
uv run chongming-license inspect license.lic # 只看内容,不验签
uv run chongming-license inspect license.lic --key <公钥> # 顺带验签
uv run chongming-license inspect license.lic --public-key-file pub.key
```
公钥解析顺序:`--key` > `--public-key-file` > 根 `config.toml` 的 `[license] public_key`。
三者都没有时按"只看内容、不验签"处理(打印原始 JSON + 警告),不会报错退出。
### 4.5 `verify` — 按指定环境完整校验
```bash
# 在本机校验(含指纹比对与时钟防线)
uv run chongming-license verify license.lic --key <公钥>
# 顺带判定某插件是否在授权白名单内
uv run chongming-license verify license.lic --key <公钥> --plugin doubao
# 模拟另一台机器(厂商侧发货前自检:确认这份 license 只能在目标机器上用)
uv run chongming-license verify license.lic --key <公钥> \
--request-code "CM1-目标机器的申请码..."
```
`--machine-code` / `--sources-json` / `--request-code` 都是"用指定的机器指纹代替现场采集",
用于模拟目标设备。
### 4.6 台账命令(`list` / `renew` / `export`)
**每次 `issue` 都会自动把授权记入台账**(连同机器指纹),这是后续续期与检索的基础。
用 `--no-registry` 可跳过。台账库默认在 `~/.chongming/licenses.db`,
可用 `--db` 或环境变量 `CHONGMING_LICENSE_DB` 指定。
```bash
# 检索(默认按到期日升序,最急的排最前)
uv run chongming-license list # 全部 + 概览计数
uv run chongming-license list --status expiring # 只看即将到期
uv run chongming-license list --customer 某某科技 # 按客户
uv run chongming-license list --machine-code BC60 # 按机器码
uv run chongming-license list --plugin aibox_sophon_adapter
uv run chongming-license list --json # 供脚本处理
# 续期:**复用台账里的机器指纹,无需再向客户索取申请码**
uv run chongming-license renew --lic-id L20260921-584426 --days 365 -k license_private.pem
# 换机器续期(新机器要给出新指纹)
uv run chongming-license renew --lic-id L20260921-584426 --days 365 \
--request-code "CM1-新机器的申请码..." -k license_private.pem
# 导出归档:.lic 文件(可按客户分目录)+ CSV 清单
uv run chongming-license export --out ./archive --csv ./台账.csv --by-customer
uv run chongming-license export --out ./expired --status expired
```
`list` 输出示例:
```
台账库: /home/user/.chongming/licenses.db
总计 3 条 —— 有效 1、即将到期 1、已过期 1、永久 0、未生效 0
符合条件 3 条:
授权编号 客户 机器码 状态 到期 插件
------------------------------------------------------------------------------------------------------------
L20260921-8FAA47 异机客户 AAAA-BBBB-CCCC-DDDD-EEEE-FFFF 已过期(-263d) 2026-01-01 doubao
L20260921-584426 某某科技 BC60-D20E-F863-7236-68E2-6A24 即将到期(20d) 2026-10-11 aibox_sophon_adapter,ip_sound_column
L20260921-76E11D 另一客户 BC60-D20E-F863-7236-68E2-6A24 有效(400d) 2027-10-26 ip_sound_column
```
---
## 5. 配置参考
```toml
[license]
path = "license.lic" # 授权文件路径(相对网关工作目录)
public_key = "" # 厂商公钥(base64 裸 32 字节,或 PEM),构建时注入
min_match = 3 # 指纹命中阈值 N-of-M
# enabled = false # 仅非打包态生效;打包产物恒为强制校验
```
| 键 | 默认 | 说明 |
| -- | ---- | ---- |
| `path` | `license.lic` | 相对网关工作目录(systemd 部署即 `/data/chongming/license.lic`) |
| `public_key` | `""` | base64 裸 32 字节或 PEM。为空时所有插件都会被拒(`no_public_key`) |
| `min_match` | `3` | 见 §6.3 |
| `enabled` | `false` | **仅非打包态**读取。打包产物恒为强制校验(见 §7.2) |
相关环境变量:
| 变量 | 作用 |
| ---- | ---- |
| `CHONGMING_LICENSE_MODE` | 强制开启/关闭校验(`on`/`off` 等)。**仅非打包态生效** |
| `CHONGMING_LICENSE_PUBKEY` | 构建期注入的公钥 |
| `CHONGMING_LICENSE_PUBKEY_FILE` | 构建期从文件读取公钥 |
| `CHONGMING_LICENSE_ALLOW_NO_KEY` | 设为 `1` 允许无公钥构建(仅内部演示) |
---
## 6. 机器指纹与容错判定
### 6.1 采集来源
按可信度排序(`utils/license/src/chongming_license/machine.py` 的 `SOURCES`):
| 来源 | 位置 | 说明 |
| ---- | ---- | ---- |
| `board_serial` | `/proc/device-tree/serial-number`、DMI `board_serial` | 逐机唯一,最稳定 |
| `emmc_cid` | `/sys/block/mmcblk*/device/cid` | 芯片出厂唯一,不随系统重装变化 |
| `cpu_serial` | `/proc/cpuinfo` 的 `Serial` | ARM 常见;部分型号全 0(会被剔除) |
| `mac` | `/sys/class/net/*/address` | 取**字典序最小**的物理网卡,见下 |
| `product_uuid` | `/sys/class/dmi/id/product_uuid` | x86 有效,ARM 通常缺失 |
| `machine_id` | `/etc/machine-id` | ⚠ **weak**,见 §6.2 |
取值规则:
- 归一化为小写、仅保留 `[0-9a-z]`(`AA:BB:CC` 与 `aa-bb-cc` 归一一致)
- 剔除占位/无效值:空、`unknown`/`none`/`to be filled by oem` 等、全 `0`、全 `f`、长度 < 4
- `mac` 取**字典序最小**而非"第一块网卡":这样插入/移除其它网卡(USB 网卡、调试接口)
以及接口重命名(`eth0` ↔ `enp1s0`)都不影响指纹,只有最小值那块网卡本身被换才会变
- 回环及其别名(`lo`、`loopback0`)无条件排除 —— `loopback0` 带 `device` 符号链接,
会骗过"物理网卡"判定(WSL 上实测到)
IP 地址**不参与**指纹(会变)。
### 6.2 weak 来源:为什么不参与判定
`/etc/machine-id` 在**同镜像批量刷机**场景下,若未在首次启动重新生成,
**所有机器的该值完全相同**。若让它计票,等于给"把整盘镜像拷到另一台机器"
白送一票 —— 阈值越高越危险。
因此它被标记为 `weak`:
- **仍会记录**进 license(便于诊断、也便于售后核对"这台机器确实是我签的那台")
- **不计入判定阈值**
- 前端/CLI 展示时带 `*` 标注
仅当双方**都只采集到 weak 来源**时才降级纳入,并在结果里标记 `degraded: true`
(此时安全性等同"只靠 machine_id 绑定",调用方应告警)。
### 6.3 N-of-M 判定
只比对**双方都有**的来源(license 签发时某来源缺失、或本机现在读不到该来源,
都不计入分母),命中数达到阈值即视为同一台机器:
```
阈值 = max(1, min(min_match, 可比较来源数))
```
在真实硬件(4 个可信来源 + 阈值 3)下的实测行为:
| 场景 | 命中 | 结果 |
| ---- | ---- | ---- |
| 同一台机器 | 4/4 | ✅ 通过 |
| 换一块网卡 | 3/4 | ✅ 通过 |
| 换一颗 eMMC | 3/4 | ✅ 通过 |
| 同时换网卡 + eMMC | 2/4 | ❌ 拒绝 |
| 整盘拷贝到另一台机器 | 0/4 | ❌ 拒绝(weak 来源不计票) |
### 6.4 上机验证(**部署前必做**)
这是唯一无法在开发机上确认的环节。在**真实硬件**(如 Sophon BM1684)上:
```bash
uv run chongming-license show --sources
```
要确认两件事:
1. **两台机器的机器码必须不同** —— 若相同,说明所有可信来源都重复了
(典型原因:同镜像刷机且 `machine-id` 是唯一来源),需要重新评估来源清单
2. **同一台机器重启后机器码必须一致** —— 若变化,说明某个来源不稳定,要从清单里换掉
同时检查来源明细:可信来源越多,容错越好。若只有 1 个可信来源或大量"缺失",
说明该硬件的序列号不可读,此时换网卡就会失授权 —— 应在上线前解决。
调整来源只需改 `utils/license/src/chongming_license/machine.py` 的 `SOURCES` 元组与对应 reader。
---
## 7. 强制执行点
### 7.1 门禁位置
| 位置 | 作用 | 为什么必须在这里 |
| ---- | ---- | ---------------- |
| `lifespan_setup._discover_plugins()` | **唯一收口**:未授权插件从插件列表剔除,`load_workers` / `validate_contracts` / `register_routes` / MQTT 主题汇总随之全部一致 | 契约校验会把孤儿 TOML 路由当**硬性错误并终止启动**。若实现成"跳过 import 但保留在列表里",该插件的 TOML 路由就无人注册 → **设备起不来** |
| `RouteRegistry.set_denied_types()` | 忽略未授权类型的路由描述符(与 `load_mode` 无关) | 否则把插件 `config.toml` 的 `load_mode` 改成 `external`,网关就不加载它、`set_inline_types` 也不再抑制它,其独立进程推来的描述符会被正常挂载 = 绕过授权 |
| `worker_admin._ensure_licensed()` | 未授权插件的扩缩容/重启/读改配置一律 403(7 个端点全部覆盖) | 否则可借扩缩容把未授权插件重新拉起,或改其配置 |
| `/api/v1/license/*` | 管理接口,仅超管;导入**先校验后落盘** | 误传坏文件不得覆盖现有可用授权 |
| 打包态检测 | 决定**是否执行拒绝** | 见 §7.2 |
### 7.2 为什么改配置绕不过
部署目录的 `config.toml` 是**明文可写**的,任何 `enabled = false` / `mode = "off"`
都能被客户一键改掉。因此"是否执行拒绝"不以配置为准,而以**打包形态**为准:
```python
is_packaged() # sys.frozen(PyInstaller)或 __compiled__(Nuitka)
```
- **打包产物** → 恒为强制校验。此时若读到"关闭授权"的配置/环境变量,会留下告警日志
(陷阱日志)但**不予采纳**
- **非打包态**(`uv run serve` / 测试)→ 默认不执行拒绝,以免影响开发流程;
需显式 `[license] enabled = true` 或 `CHONGMING_LICENSE_MODE=on` 才生效
> ⚠️ 该检测假设按 `deploy/chongming.service` 用**编译产物**部署。
> 若现场改用 `uv run` 启动,则检测不生效 —— 这也意味着不能用"未打包"的方式投产。
---
## 8. 管理接口与前端
### 8.1 API(均需超级管理员 Bearer token)
| 端点 | 说明 |
| ---- | ---- |
| `GET /api/v1/license/info` | 机器码、申请码、授权状态、插件授权全景、各来源明细、告警 |
| `POST /api/v1/license/reload` | 从磁盘重读授权(手工替换文件后无需重启刷新状态) |
| `POST /api/v1/license/file` | 上传导入 license(multipart;先校验后落盘,上限 64KB) |
`/license/info` 关键字段:
```jsonc
{
"enforce": true, // 是否执行拒绝
"enforce_forced_by_packaging": true, // 是否由打包态强制
"license_path": "license.lic",
"min_match": 3,
"request_code": "CM1-...", // 直接复制发给厂商
"status": {
"ok": true, "code": "ok", "message": "授权有效",
"machine_code": "BC60-D20E-...",
"public_key_fingerprint": "3d956a50db56b53c",
"match": { "matched": 4, "total": 4, "degraded": false, "reason": "..." },
"payload": { "lic_id": "...", "customer": "...", "plugins": [...] },
"licensed_plugins": ["aibox_sophon_adapter", "ip_sound_column"]
},
"plugins": {
"discovered": [...], // 盘扫到的
"licensed": [...], // 已授权
"denied": { "doubao": { "code": "plugin_not_licensed", "message": "..." } },
"loaded": [...] // 当前已加载
},
"sources_detail": [ { "name": "mac", "present": true, "weak": false, "value": "..." } ],
"warnings": [...]
}
```
### 8.2 前端「授权」页
地址:`http://<设备IP>:8000/static/license`(前端 `base = /static/`,顶部导航有「授权」入口)
- **机器码**大字号展示,一键复制
- **申请码**一键复制(附输入框便于手动选中)
- **授权信息**:授权文件路径、是否有校验、授权编号/客户/签发时间/有效期、指纹比对结论、公钥指纹、备注
- **插件授权表**:每个插件的 授权 / 已加载 / 说明(未授权原因)
- **硬件指纹来源表**:来源 / 说明 / 采集状态 / 值 / 细节,weak 来源带 `*`
- **导入授权文件**:选择文件 + 导入并校验 + 从磁盘重读
登录复用「管理端 token 单例」(`cm_admin_token`),在 Worker 管理页登录过则无需重复登录。
---
## 9. 安全边界与已知缺口
### 9.1 诚实的边界说明
**离线 + root 的天花板是"阻止随手拷贝",不是"阻止逆向"。**
有 root 权限的人可以改二进制跳过校验,或用 `LD_PRELOAD` 伪造指纹文件读取。这是所有
纯离线硬件绑定方案的共同边界,不应对外承诺更强。
**时间防线同理**:`clock` 模块用共享缓存里的高水位标记检测系统时间回拨
(默认容差 1 天;水位超前当前时间 90 天以上判为 RTC 异常并重新起算,避免把设备永久卡死)。
删掉缓存文件即可失效 —— 它只是抬高"随手改系统时间续期"的门槛。
**真正难以绕过的是签名**:伪造 license 需要厂商私钥。
### 9.2 `.mbank` 加密不能当授权信任根
根仓 `resources/*.key` 那套 `.mbank` 验签用的是 **HMAC 对称算法**
(`sqlite_vfs/core.py` 的 `sign` → `hmac.new`),且密钥与银行同目录发布。
它只能算**防篡改/混淆**,**不能**作为厂商信任根 —— 授权必须用非对称。
`test/test_license_build.py::test_mbank_signing_is_not_a_trust_root` 固化了这一认知:
若将来 `.mbank` 改为非对称,该测试会提醒更新文档。
### 9.3 已知缺口(本轮范围外)
| 缺口 | 影响 | 彻底封堵需要 |
| ---- | ---- | ------------ |
| **external/进程插件** | 网关侧只拦住了 HTTP 暴露;直接拉起 `chongming-worker --type X` 进程仍会运行并可能操作 MQTT/定时任务 | 在 `chongming-worker` SDK 的 `process.py` / `plugin.py` 加启动自检(需发布新 wheel,外部插件容器升级后生效) |
| **Rust 进程插件** | 同上,且若未覆盖会成为绕过点 | 同步改 `utils/worker-rs` |
| **动态加载新授权插件** | 导入新 license 后需重启才生效(插件发现只在启动期执行) | 实现运行期插件加载,是独立功能 |
| **打包态检测** | 依赖"用编译产物部署"这一前提 | 无(部署方式约束) |
---
## 10. 故障排查
### 10.1 按现象定位
| 现象 | 原因码 | 处理 |
| ---- | ------ | ---- |
| 所有插件都不加载 | `license_missing` | license 文件不在 `[license] path` 指定位置;网页导入或放好文件后重启 |
| 所有插件都不加载 | `no_public_key` | 产物未注入公钥(构建漏了)。用 `CHONGMING_LICENSE_PUBKEY` 重新构建 |
| 所有插件都不加载 | `signature_invalid` | 文件被篡改,或不是本厂商公钥对应私钥签的。核对 `public_key_fingerprint` |
| 所有插件都不加载 | `machine_mismatch` | 授权不是为本机签发的,或换了硬件。看 `/license/info` 的 `status.match.reason` |
| 所有插件都不加载 | `expired` | 授权过期,需续签 |
| 所有插件都不加载 | `not_yet_valid` | 授权尚未生效(`not_before` 晚于当前日期) |
| 所有插件都不加载 | `clock_rollback` | 系统时间被回拨。检查 RTC/时区/NTP |
| 所有插件都不加载 | `fingerprint_unavailable` | 本机读不到任何硬件指纹。用 `show --sources` 排查 |
| 所有插件都不加载 | `license_malformed` | JSON 损坏或载荷版本高于程序(提示升级网关) |
| 某个插件不加载 | `plugin_not_licensed` | 该插件不在白名单。用 `--plugins` 加上后**重签** |
| 某个插件不加载 | 其他原因码 | 整份 license 无效时**所有**插件都被拒,逐个看日志 |
| 管理接口 403 | —— | 该插件未授权,或当前用户不是超管 |
| 配置了但完全没反应 | `disabled` | 非打包态默认不执行拒绝。把 `[license] enabled` 改成 `true`(或设 `CHONGMING_LICENSE_MODE=on`) |
| 打包时报"缺少厂商授权公钥" | —— | 见 §2.3,按三种方式之一提供公钥 |
| 打包时报"发现私钥" | —— | 产物里混入了私钥(`.pem` 或被改名的私钥内容)。移出 `build/` 与 `resources/` 后重建 |
### 10.2 原因码全表
| 代码 | 含义 |
| ---- | ---- |
| `ok` | 授权有效 |
| `disabled` | 校验未启用(开发/测试模式) |
| `license_missing` | 未找到授权文件 |
| `license_malformed` | 格式错误或内容损坏 |
| `signature_invalid` | 签名校验失败(被篡改或非本厂商签发) |
| `unsupported_alg` | 不支持的签名算法(仅支持 `ed25519`) |
| `no_public_key` | 未配置厂商公钥 |
| `machine_mismatch` | 机器码不匹配(非本机授权) |
| `not_yet_valid` | 授权尚未生效 |
| `expired` | 授权已过期 |
| `clock_rollback` | 检测到系统时间被回拨 |
| `plugin_not_licensed` | 该插件不在授权白名单内 |
| `fingerprint_unavailable` | 本机无法采集硬件指纹 |
### 10.3 常见困惑
**"为什么我配了 license 却完全没反应?"**
非打包态默认 `enabled = false`(不干扰日常开发)。打包产物恒为强制校验,改配置关不掉。
开发时想验证门禁,必须手动把 `[license] enabled` 改成 `true`。
**"我机器上只认出 1 个指纹来源,是不是坏了?"**
虚拟机(WSL/容器)没有设备树、eMMC、DMI,所以只剩 `mac`(虚拟 MAC)与 `machine_id`(weak,不计票)。
真实硬件上 `board_serial` / `emmc_cid` / `cpu_serial` 都会出现。见 §6.4 上机验证。
**"我只报机器码行不行?"**
行,但授权是精确匹配,换网卡即失效。建议一律用申请码。
**"导入新 license 后为什么还要重启?"**
插件发现只在启动期执行。页面会提示需要重启及待加载的插件。
**"能不能签一份授权给多台机器用?"**
当前不支持 —— 判定必然走"逐来源指纹容错匹配"或"机器码精确匹配"其中之一,两者都绑定单机。
批量场景的处理方式是:每台设备各导出申请码,逐台签发(`issue` 可脚本化,
`--lic-id` 可指定编号便于对账)。若确需"一授权多机",需要扩展载荷格式
(例如加入机器码数组),属新功能。
---
## 11. 相关文件与测试
### 11.1 文件索引
| 路径 | 说明 |
| ---- | ---- |
| `utils/license/`(`chongming-license`) | 库:`machine.py`(指纹/机器码/申请码)、`models.py`(载荷格式)、`issue.py`(载荷拼装/签名/写盘)、`verify.py`(规范化/验签/判定)、`clock.py`(时钟防线)、`registry.py`(**已签发台账**)、`__init__.py`(`LicenseManager` 门面) |
| `utils/cli/src/chongming_cli/license.py` | 厂商侧 CLI(`chongming-license`,属 dev 依赖) |
| `src/chongming/app/core/licensing.py` | 网关侧适配:配置读取、打包态判定、门禁单例、启动横幅 |
| `src/chongming/app/core/lifespan_setup.py` | `_scan_plugins()`(盘扫,不过滤)/ `_discover_plugins()`(**门禁收口**)/ `denied_plugin_types()` / `log_license_banner()` |
| `src/chongming/app/core/route_registry.py` | `set_denied_types()` + `_handle_sample` 中的拒绝分支 |
| `src/chongming/app/routes/licensing.py` | `/license/*` 管理端点 |
| `src/chongming/app/routes/worker_admin.py` | `_ensure_licensed()` 及 7 处调用 |
| `src/chongming-web/src/views/LicenseView.vue` | 前端「授权」页 |
| `src/chongming-web/src/api/license.ts` | 前端 API 封装 |
| `utils/cli/src/chongming_cli/build.py` | 公钥注入 `_set_license_pubkey()` + 私钥守卫 `_stage_license_guard()` |
| `utils/license-gui/`(`chongming-license-gui`) | **可视化工具**(厂商侧):签发 / 台账 / 密钥 / 查看校验四页,独立 venv 不污染网关环境 |
| `config.toml` / `public/config.toml` | `[license]` 配置段(`public/` 为产物模板) |
| `config.toml` → `[pyinstaller_build_import]` | PyInstaller 白名单含 `chongming_license` / `cryptography` |
### 11.2 license 文件格式
```jsonc
{
"payload": {
"v": 1,
"lic_id": "L20260921-334B33",
"customer": "某某科技",
"machine_code": "BC60-D20E-F863-7236-68E2-6A24",
"sources": { "board_serial": "...", "emmc_cid": "...", "mac": "..." },
"plugins": ["aibox_sophon_adapter", "ip_sound_column"],
"not_before": null, // 含当天;null = 立即生效
"expires_at": "2027-09-21", // 含当天;null = 永久
"issued_at": "2026-09-21T01:30:59Z",
"note": "合同 HT-2026-0912"
},
"alg": "ed25519",
"kid": "vendor-2026",
"sig": "<base64,覆盖 payload 的规范化 JSON>"
}
```
**签名覆盖 payload 原始字节的规范化形式**(`canonical_json`:键排序、无空白),
因此签发与校验**必须共用同一个函数**(`chongming_license.verify.canonical_json`),
且载荷内**不允许浮点数**(float 的文本表示存在多种合法写法,会破坏字节级一致性)。
校验顺序是**先验签、后解析模型** —— 反过来会让默认值填充/字段顺序变化改变字节导致验签失败。
### 11.3 测试
| 文件 | 覆盖 | 断言数 |
| ---- | ---- | ------ |
| `test/test_license_verify.py` | 库层:归一化、机器码、容错匹配(含 weak 不计票)、验签/篡改/换公钥/字段重排、有效期边界、机器码-only 回退、申请码、时钟防线 | 52 |
| `test/test_license_gate.py` | 网关门禁:剔除未授权插件、**契约校验不硬失败**、完全未授权、过期、管理接口 403、路由描述符拦截、HTTP 端点与导入安全性 | 28 |
| `test/test_license_build.py` | 公钥注入(段内替换、注释保留)、解析优先级、私钥守卫(含跨块边界、证书不误判)、缺公钥构建失败、`.mbank` 非信任根 | 22 |
| `test/test_license_registry.py` | 台账:记录派生、五种状态(含到期当天边界)、检索与排序、**续期复用指纹**、备注不被重导入覆盖、导出(.lic/CSV/中文 BOM)、结构版本、并发打开 | 53 |
```bash
# 三个测试都支持独立运行(无需 pytest)
python test/test_license_verify.py
python test/test_license_gate.py
python test/test_license_build.py
python test/test_license_registry.py
# 或
uv run pytest test/test_license_*.py
```
回归关注点:`test_route_registry.py`(确认 `set_denied_types` 默认为空集时不影响既有行为)、
`test_worker_isolation.py`(确认未破坏进程插件隔离)。
---
## 12. 可视化工具(PySide6,厂商侧)
供商务、售后等不习惯命令行的同学使用,功能与 §4 的 CLI **完全对等**。
所有授权逻辑调用同一个 `chongming-license` 库 —— 界面不重复实现密码学、载荷格式或规范化
(一旦有两处实现,就会出现"自己签的自己验不过")。
### 12.1 安装与运行
```bash
cd utils/license-gui
uv sync
uv run chongming-license-gui
```
> **独立环境,不污染网关**:本包有自己的 `pyproject.toml` 与 `.venv`,
> **不是**根仓 workspace 成员,因此根仓 `uv sync` 不会拉入 PySide6(约 167MB)。
> 主仓 venv 里不会出现 PySide6。
无图形环境(SSH/CI)下可做冒烟检查:
```bash
cd utils/license-gui
QT_QPA_PLATFORM=offscreen uv run python smoke_test.py # 25 项断言
```
### 12.2 三个页面
| 页面 | 内容 |
| ---- | ---- |
| **签发**(默认) | 签名私钥状态与公钥指纹 → 签发依据(申请码 / 来源 JSON / 仅机器码)→ 勾选授权插件 → 客户/编号/备注/有效期 → **载荷预览** → 签发并保存(默认自动记入台账) |
| **台账** | 检索已签发授权(客户/机器码/插件/状态/关键字/即将到期)→ **续期**、编辑备注、查看详情、删除 → 导出 `.lic`(可按客户分目录)与 CSV 清单。见 §13 |
| **密钥** | 生成密钥对(私钥 `0600`,公钥自动进剪贴板);加载现有私钥并显示对应公钥与指纹 |
| **查看 / 校验** | 打开 `.lic` 看全部字段;填公钥验签;**指纹比对基准**可选"本机"或"指定机器(填申请码)" |
设计上刻意做的几件事:
- **所见即所签**:预览用的是与签名**同一个载荷对象**(`build_payload`),
不是另拼一份 —— 按下签发前就能看清"给谁、哪台机器、哪些插件、多久"。
- **窗口标题带公钥指纹**:同时开多个实例或截图沟通时,能一眼确认"当前用的是哪把密钥"。
- **签发前强制核对指纹**:加载私钥后立即显示公钥指纹,与设备端显示比对;
拿错密钥的代价是重签并追回已发出的文件,必须在这一步拦住。
- **"指定机器"校验**:厂商机器自己的指纹当然对不上客户设备。不指定目标机器时
只能验出"签名有效",验不出"机器码是否真的匹配" —— 这是**发货前自检**的关键。
- **weak 来源如实标注**:解析申请码后明确列出"可信来源 N 个 / 不参与判定(weak):…",
避免操作者误以为来源越多越保险。
- **仅机器码模式带警示**:界面直说"换一块网卡就会失授权,建议改用申请码"。
### 12.3 与 CLI 的对应
| GUI 操作 | CLI |
| -------- | --- |
| 密钥 → 生成密钥对 | `chongming-license keygen` |
| 密钥 → 加载私钥 | `chongming-license inspect --key` / `keygen` 的输出 |
| 签发 → 签发并保存 | `chongming-license issue` |
| 查看 / 校验 → 仅查看内容 | `chongming-license inspect` |
| 查看 / 校验 → 指定机器 | `chongming-license verify --request-code` |
批量签发(一次几十台)建议直接用 CLI 脚本化(`--lic-id` 可指定编号便于对账);
GUI 面向单台/少量、需要人工核对信息的场景。
### 12.4 使用注意
- GUI 会在 `QSettings` 里记住**私钥路径、客户名、备注、输出目录、有效期天数**
(便于连续签发),**不会**记住私钥内容本身。
- 界面里的信息框/确认框是模态的(会阻塞等待点击)。因此 GUI 的自动化测试
(`smoke_test.py`)必须先替换这些弹窗,否则会挂住 —— 这是该测试唯一需要 monkeypatch 的地方。
---
## 13. 已签发台账
### 13.1 为什么需要台账
签出去的 license 会散落在邮件、聊天记录、下载目录里。等到要续期时,常见困境是:
- 不知道当初给哪台机器签过、签的哪些插件
- **拿不到那台机器的硬件指纹** —— 而续期需要它;现场还可能已经联系不上客户
- 到期了没人知道,客户突然反馈"插件不能用了"才发现
台账把每次签发**连同机器指纹一起**存下来,于是:按客户/机器码/插件/有效期检索、
到期前主动列出该续期的清单、**续期直接复用台账指纹**(最大价值)、原样导出当初的
文件(审计/补发)、识别"哪份是密钥轮换前的旧公钥签的"。
### 13.2 位置与安全
| 项 | 说明 |
| -- | ---- |
| 默认路径 | `~/.chongming/licenses.db` |
| 覆盖方式 | `--db <路径>` 或环境变量 `CHONGMING_LICENSE_DB`;GUI 在「台账」页选择并记住 |
| 存储 | 单文件 SQLite(纯标准库,无服务、可直接进备份),WAL 模式支持多窗口并发 |
| 版本 | 库内 `PRAGMA user_version`;不匹配时**明确报错**而不是让查询诡异失败 |
| 敏感度 | 含客户设备的硬件指纹与授权信息,属**厂商内部数据**,不宜外发 |
刻意放在用户主目录而非项目内:台账必须能跨项目、跨工具长期留存,放在仓库里容易被
清理脚本删掉或换机器时随项目一起丢掉。
### 13.3 自动入库
`issue` 与 GUI 的「签发并保存」默认把授权写入台账(GUI 上是一个默认勾选的复选框):
```
✓ 已签发 license.lic
授权编号 L20260921-584426
...
已入台账 /home/user/.chongming/licenses.db
续期:`chongming-license renew --lic-id L20260921-584426 --days 365`
(复用台账指纹,无需客户再报申请码)
```
入库失败**不会**把"已签发"变成失败 —— 文件已经生成好了,只是台账没记上,
此时会明确告警(避免误以为需要重签)。
### 13.4 状态与字段
状态按优先级判定:`not_yet_valid`(未到生效日)> `expired` > `expiring`(窗口内)
> `forever`(无到期日)/ `active`。判定**复用同一个 `status()` 方法**,
因此各状态互斥、计数之和等于总数(用独立的 SQL 条件会因重叠而算错)。
关键字段:授权编号、客户、机器码、授权插件、**各来源指纹**、生效/到期日、签发时间、
密钥标识、**签发时的公钥指纹**、原始 license 文件内容、备注。
**备注**是本台账唯一允许人工编辑的字段("已电话确认续期"、"设备已返修"…),
**重新导入同一份 license 不会覆盖它** —— 载荷里的 `note` 是签名内容的一部分,
台账的 `note` 是厂商自己的批注,二者同名但不同用途。
### 13.5 续期(本功能的核心)
```
台账 → 选中一条 → 「续期选中…」→ 只改要变的部分(有效期/插件/客户)→ 续期并保存
```
- **指纹复用自台账**,无需再向客户索取申请码 —— 这是台账最大的价值
- 默认**生成新授权编号**,新旧都留在台账里,可追溯;也可选"沿用原授权编号"
- 旧记录会被自动加上 `已续期 → <新编号>` 的批注,从任一编号都能串起来
- 原授权若当初是"仅机器码精确匹配"签的(没有来源指纹),续期会明确拒绝并提示
重新索取申请码 —— 而不是悄悄签出一份会在设备上因指纹不符被拒的授权
换机器续期时用 `--request-code` 给出新机器的指纹:
```bash
uv run chongming-license renew --lic-id L20260921-584426 --days 365 \
--request-code "CM1-新机器的申请码..." -k license_private.pem
```
### 13.6 导出与归档
```bash
uv run chongming-license export --out ./archive --csv ./台账.csv --by-customer
```
- `.lic` 文件名形如 `<授权编号>_<机器码>.lic`,**内容原样**(可直接补发给客户)
- `--by-customer` 按客户名分子目录,便于打包发给对应客户
(客户名里的路径非法字符会被安全替换)
- CSV 以 **UTF-8 BOM** 写入:否则 Excel 在中文 Windows 下按 GBK 解码会乱码;
插件列表和备注含逗号时按 CSV 规范加引号转义
### 13.7 使用注意
- 台账**只删记录,不会吊销已发出的 license** —— 客户手里的文件仍然有效,
因为离线方案里没有吊销通道(见 §9.3)。删除前请确认这一点。
- 台账记录了签发时的公钥指纹,密钥轮换后可以用它筛出"需要重签"的历史授权。
- 台账规模(几百到几千条)完全在 SQLite 的舒适区;筛选逻辑放在 Python 侧以复用
`status()`,避免 SQL 与 Python 两套状态判定漂移。
+2 -17
View File
@@ -28,28 +28,13 @@ launch_worker_count = "cache"
algorithm = "HS256" algorithm = "HS256"
secret_key = "your-secret-key-change-in-production" secret_key = "your-secret-key-change-in-production"
[license]
# 机器码授权(插件白名单)。未授权插件不加载、不注册路由、管理接口禁止操作。
#
# 打包产物(Nuitka/PyInstaller)**恒为强制校验**,不存在可被配置绕过的开关:
# 部署目录的 config.toml 是明文可写的,任何 enabled/mode 开关都能被一键改掉。
# 只有非打包态(开发/测试)才会读取 enabled,默认关闭以免影响开发流程。
# enabled = false
# 授权文件路径(相对网关工作目录)
path = "license.lic"
# 厂商公钥(base64 裸 32 字节,或 PEM)。公钥可明文放置:换公钥只会验签失败,
# 无法据以伪造授权(伪造需要厂商私钥)。
# 构建时由 build.py 从 CHONGMING_LICENSE_PUBKEY 或 license_public.key 注入;
# 缺失时构建会失败(避免产出无法验签、所有插件都加载不了的"砖头"包)。
public_key = ""
# 指纹命中阈值(N-of-M)。4 个可信来源下取 3 表示容忍单块网卡或单颗 eMMC 更换。
min_match = 3
[production] [production]
debug = false debug = false
env = "production" env = "production"
[production.server] [production.server]
# 启动参数对各服务器通用(由 main.py 的转换函数映射,无需按服务器改配置);
# 服务器/副本数可用环境变量覆盖:APP_SERVER=granian|uvicorn|gunicorn、APP_WORKERS=<N>
access_log = false access_log = false
host = "0.0.0.0" host = "0.0.0.0"
port = 8000 port = 8000
+127 -44
View File
@@ -3,23 +3,126 @@
""" """
FastAPI chongming 示例 FastAPI chongming 示例
主入口文件,适配 chongming 打包系统 主入口文件,适配 chongming 打包系统
服务器选择(APP_SERVER 环境变量,默认见 DEFAULT_SERVER):
- gunicorn:现状保留(Linux 默认,uvloop worker + preload 多进程)
- uvicorn:Windows 默认(gunicorn 依赖 fork,Windows 不可用)
- granian:Rust HTTP 内核,多进程
启动参数保持 config.toml `[<env>.server]` 现状(host / port / workers /
reload / access_log / timeout_keep_alive),由 `_to_uvicorn`、`_to_granian`、
`_to_gunicorn_args` 转换为对应服务器的参数。
""" """
import asyncio # noqa: F401
import os import os
import sys
import platform import platform
from utils.launch import launch import sys
from multiprocessing import freeze_support from multiprocessing import freeze_support
from pathlib import Path
import server # noqa: F401 # 预加载:master 内完成模块银行装载,fork 出的 worker 共享
import uvloop # noqa: F401 # 显式导入保证打包时收集 uvloop(uvicorn/granian loop 依赖)
from utils.launch import launch
from uvicorn.workers import UvicornWorker from uvicorn.workers import UvicornWorker
import server
import uvloop
import asyncio
use_shell = platform.system() == "Windows" use_shell = platform.system() == "Windows"
# 未设置 APP_SERVER 时的平台默认(保持现状)
DEFAULT_SERVER = "uvicorn" if use_shell else "gunicorn"
SUPPORTED_SERVERS = ("gunicorn", "uvicorn", "granian")
# 热重载(reload=true)监听的源码目录,避免 granian 默认扫 cwd 时被 logs/ 写日志触发
RELOAD_PATHS = [Path("src/chongming"), Path("src/plugins")]
class UvloopUvicornWorker(UvicornWorker): class UvloopUvicornWorker(UvicornWorker):
CONFIG_KWARGS = {"loop": "uvloop"} CONFIG_KWARGS = {"loop": "uvloop"}
def _server_params(app_config: dict) -> dict:
"""把 `[<env>.server]` 的现状参数整理为各服务器共用的启动参数。
只做缺省兜底与运行环境覆盖(打包环境强制禁 reload、APP_WORKERS 覆盖副本数),
键名保持 config.toml 现状,改名交给下面的转换函数。
"""
server_config = app_config.get("server", {})
# 打包环境强制禁用重载
is_frozen = getattr(sys, "frozen", False)
# 支持通过环境变量 APP_WORKERS 覆盖 worker 数量(方便嵌入式设备调优)
default_workers = server_config.get("workers", 4)
workers = int(os.environ.get("APP_WORKERS", str(default_workers)))
return {
"host": server_config.get("host", "0.0.0.0"),
"port": server_config.get("port", 8000),
"workers": workers,
"reload": False if is_frozen else server_config.get("reload", False),
"access_log": server_config.get("access_log", False),
"timeout_keep_alive": server_config.get("timeout_keep_alive", 5),
"log_level": "debug" if app_config.get("debug", False) else "info",
}
def _to_uvicorn(params: dict) -> dict:
"""共用参数 → uvicorn.run 关键字参数(键名一致)。"""
return {
"host": params["host"],
"port": params["port"],
"reload": params["reload"],
"workers": params["workers"],
"log_level": params["log_level"],
"access_log": params["access_log"],
"timeout_keep_alive": params["timeout_keep_alive"],
}
def _to_granian(params: dict) -> dict:
"""共用参数 → granian.Granian 关键字参数。
仅两处改名:host → address、access_log → log_access;
timeout_keep_alive 在 granian 无对应项(HTTP/1 keep-alive 只有开关、无空闲
超时配置),忽略。
"""
return {
"address": params["host"],
"port": params["port"],
"interface": "asgi",
"workers": params["workers"],
"reload": params["reload"],
"log_level": params["log_level"],
"log_access": params["access_log"],
}
def _to_gunicorn_args(params: dict) -> list:
"""共用参数 → gunicorn 命令行参数(保持现状:uvloop worker + preload)。"""
return [
"gunicorn",
"server:app",
"-k",
"__main__.UvloopUvicornWorker",
"--preload",
"-w",
str(params["workers"]),
"-b",
f"{params['host']}:{params['port']}",
]
def _select_server(logger) -> str:
"""按 APP_SERVER 环境变量选择服务器,非法值/平台不兼容时回退默认。"""
server_kind = (os.environ.get("APP_SERVER") or DEFAULT_SERVER).strip().lower()
if server_kind not in SUPPORTED_SERVERS:
logger.warning(
f"未知 APP_SERVER={server_kind!r}(可选: {', '.join(SUPPORTED_SERVERS)}),"
f"回退 {DEFAULT_SERVER}"
)
server_kind = DEFAULT_SERVER
if server_kind == "gunicorn" and use_shell:
logger.warning("Windows 不支持 gunicorn(依赖 fork),回退 uvicorn")
server_kind = "uvicorn"
return server_kind
def app_run(app_config: dict, default_config: dict): def app_run(app_config: dict, default_config: dict):
""" """
应用运行函数 - 适配模块银行环境 应用运行函数 - 适配模块银行环境
@@ -35,18 +138,8 @@ def app_run(app_config: dict, default_config: dict):
logger = get_logger("app") logger = get_logger("app")
# 获取服务器配置 params = _server_params(app_config)
server_config = app_config.get("server", {}) server_kind = _select_server(logger)
host = server_config.get("host", "0.0.0.0")
port = server_config.get("port", 8000)
timeout_keep_alive = server_config.get("timeout_keep_alive", 5)
# 打包环境强制禁用重载
is_frozen = getattr(sys, "frozen", False)
reload = False if is_frozen else server_config.get("reload", False)
# 支持通过环境变量 APP_WORKERS 覆盖 worker 数量(方便嵌入式设备调优)
default_workers = server_config.get("workers", 4)
workers = int(os.environ.get("APP_WORKERS", str(default_workers)))
# 显示启动信息 # 显示启动信息
env = app_config.get("env", "development") env = app_config.get("env", "development")
@@ -54,41 +147,31 @@ def app_run(app_config: dict, default_config: dict):
logger.info(f"🚀 启动 {default_config.get('app.name', 'FastAPI 应用')}") logger.info(f"🚀 启动 {default_config.get('app.name', 'FastAPI 应用')}")
logger.info(f"📦 版本: {default_config.get('app.version', '1.0.0')}") logger.info(f"📦 版本: {default_config.get('app.version', '1.0.0')}")
logger.info(f"🌍 环境: {env}") logger.info(f"🌍 环境: {env}")
logger.info(f"📍 地址: http://{host}:{port}") logger.info(f"🖥️ 服务器: {server_kind}")
logger.info(f"📚 文档: http://{host}:{port}/docs") logger.info(f"📍 地址: http://{params['host']}:{params['port']}")
logger.info(f"📚 文档: http://{params['host']}:{params['port']}/docs")
logger.info(f"🔧 调试: {'启用' if app_config.get('debug', False) else '禁用'}") logger.info(f"🔧 调试: {'启用' if app_config.get('debug', False) else '禁用'}")
logger.info(f"🔄 重载: {'启用' if reload else '禁用'}") logger.info(f"🔄 重载: {'启用' if params['reload'] else '禁用'}")
logger.info(f"👥 Workers: {workers}") logger.info(f"👥 Workers: {params['workers']}")
logger.info("=" * 50) logger.info("=" * 50)
# 根据操作系统选择不同的服务器启动方式 # 根据所选服务器启动(默认平台分支保持现状)
if use_shell: if server_kind == "granian":
import granian
granian.Granian(
target="server:app",
**_to_granian(params),
reload_paths=RELOAD_PATHS,
).serve()
elif server_kind == "uvicorn":
import uvicorn import uvicorn
uvicorn.run( uvicorn.run("server:app", **_to_uvicorn(params))
"server:app",
host=host,
port=port,
reload=reload,
workers=workers,
log_level="debug" if app_config.get("debug", False) else "info",
access_log=False,
timeout_keep_alive=timeout_keep_alive,
)
else: else:
from gunicorn.app.wsgiapp import run from gunicorn.app.wsgiapp import run
sys.argv = [ sys.argv = _to_gunicorn_args(params)
"gunicorn",
"server:app",
"-k",
"__main__.UvloopUvicornWorker",
"--preload",
"-w",
str(workers),
"-b",
f"{host}:{port}",
]
run() run()
+2 -10
View File
@@ -2,7 +2,6 @@ from .config import ConfigLoader
import os import os
from module_bank import PythonToSQLite from module_bank import PythonToSQLite
from typing import Callable from typing import Callable
from pathlib import Path
# module_bank 依赖在每次加载 PyArmor 加密模块时都会 print 一行 # module_bank 依赖在每次加载 PyArmor 加密模块时都会 print 一行
# "已部署 PyArmor 运行时",启动期会刷屏(约 50 行)。依赖包不便改动, # "已部署 PyArmor 运行时",启动期会刷屏(约 50 行)。依赖包不便改动,
@@ -37,15 +36,8 @@ def init_module_bank(config):
] ]
# 遍历所有打包器,安装导入器(模块清单不再打印,避免启动刷屏) # 遍历所有打包器,安装导入器(模块清单不再打印,避免启动刷屏)
for packer in packer_list: for packer in packer_list:
key_file_path = Path(packer.db_path).with_suffix(".key") key = packer.get_key()
with open(key_file_path, "r", encoding="utf-8") as f: packer.install_importer(key=key)
key = f.read().strip()
packer.install_importer(
key=key,
signature_key="my_key",
signature_key_id="ci-2026",
allow_unsigned=False,
)
def launch(run_fun: Callable[[dict, dict], None], run_mode: str): def launch(run_fun: Callable[[dict, dict], None], run_mode: str):
+10 -2
View File
@@ -9,6 +9,7 @@ dependencies = [
"eclipse-zenoh", "eclipse-zenoh",
"fastapi[uvicorn]", "fastapi[uvicorn]",
"gunicorn", "gunicorn",
"granian[reload,uvloop]>=2.8.4",
"module-bank", "module-bank",
"openai", "openai",
"python-multipart", "python-multipart",
@@ -28,7 +29,6 @@ dependencies = [
"bytedance", "bytedance",
"chongming-cache", "chongming-cache",
"chongming-auth", "chongming-auth",
"chongming-license",
] ]
description = "基于 modules-bank 的 fastapi 打包脚手架" description = "基于 modules-bank 的 fastapi 打包脚手架"
name = "chongming-zenoh" name = "chongming-zenoh"
@@ -37,7 +37,14 @@ requires-python = ">=3.12"
version = "0.1.0" version = "0.1.0"
[dependency-groups] [dependency-groups]
dev = ["chongming-cli", "nuitka", "pyarmor", "pyinstaller", "ruff"] dev = [
"chongming-cli",
"greenlet>=3.5.6",
"nuitka",
"pyarmor",
"pyinstaller",
"ruff",
]
[tool.hatch.build.targets.wheel] [tool.hatch.build.targets.wheel]
packages = ["src/chongming"] packages = ["src/chongming"]
@@ -50,5 +57,6 @@ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
[project.scripts] [project.scripts]
gunicorn = "chongming:gunicorn_serve" gunicorn = "chongming:gunicorn_serve"
granian = "chongming:granian_serve"
serve = "chongming:serve" serve = "chongming:serve"
worker_serve = "chongming_worker.process:worker_serve" worker_serve = "chongming_worker.process:worker_serve"
+89 -2
View File
@@ -1,7 +1,66 @@
"""chongming 网关启动入口(uvicorn / granian / gunicorn)。
config.toml 中 `[<env>.server]` 的启动参数保持现状(host / port / workers /
reload / access_log / timeout_keep_alive),由 `_to_uvicorn`、`_to_granian` 在
启动时转换为对应服务器的关键字参数,切换服务器不必改配置格式。
"""
from collections.abc import Mapping
from pathlib import Path
from typing import Any
import uvicorn import uvicorn
from chongming_config import load_config from chongming_config import Server, load_config
from .app.core.logger import get_logger from .app.core.logger import get_logger
# 开发期热重载(reload=true)监听的源码目录(granian 用;uvicorn 走其默认扫描)
RELOAD_PATHS = [Path("src/chongming"), Path("src/plugins")]
def _server_config_view(server_config: Server) -> Mapping[str, Any]:
"""[<env>.server] 的宽松视图:实际 TOML 可能带 Server 之外的键
(如 timeout_keep_alive,chongming_config.Server 尚未声明)。"""
return server_config
def _to_uvicorn(server_config: Server) -> dict:
"""把 [<env>.server] 的现状参数转为 uvicorn.run 的关键字参数。
uvicorn 的参数名与配置项一致,这里只做缺省兜底。
"""
cfg = _server_config_view(server_config)
converted: dict = {
"host": cfg.get("host", "0.0.0.0"),
"port": cfg.get("port", 8000),
"reload": cfg.get("reload", False),
}
for key in ("workers", "access_log", "timeout_keep_alive"):
if key in cfg:
converted[key] = cfg[key]
return converted
def _to_granian(server_config: Server) -> dict:
"""把 [<env>.server] 的现状参数转为 granian.Granian 的关键字参数。
仅两处改名:host → address、access_log → log_access;
timeout_keep_alive 在 granian 无对应项(HTTP/1 keep-alive 只有开关、无空闲
超时配置),忽略。
"""
cfg = _server_config_view(server_config)
converted: dict = {
"address": cfg.get("host", "0.0.0.0"),
"port": cfg.get("port", 8000),
"interface": "asgi",
}
for key in ("reload", "workers"):
if key in cfg:
converted[key] = cfg[key]
if "access_log" in cfg:
converted["log_access"] = cfg["access_log"]
return converted
def serve(): def serve():
""" """
@@ -20,7 +79,34 @@ def serve():
logger = get_logger("app") logger = get_logger("app")
logger.info(f"启动服务: {server_config}") logger.info(f"启动服务: {server_config}")
logger.info(f"启动日志: {log_config}") logger.info(f"启动日志: {log_config}")
uvicorn.run("chongming.app:app", **server_config) uvicorn.run("chongming.app:app", **_to_uvicorn(server_config))
def granian_serve():
"""
使用 granian 启动服务
从配置文件中加载服务器配置和日志配置,然后使用 granian 启动应用服务。
该函数适用于开发环境或单进程部署场景。
Returns:
None: 该函数不返回值,会阻塞运行直到服务停止
"""
# 延迟导入:未选用 granian 的部署环境不必安装它
import granian
config = load_config()
env = config["default"]["env"]
server_config = config[env]["server"]
log_config = config[env]["logging"]
logger = get_logger("app")
logger.info(f"启动服务: {server_config}")
logger.info(f"启动日志: {log_config}")
granian.Granian(
target="chongming.app:app",
**_to_granian(server_config),
reload_paths=RELOAD_PATHS,
).serve()
def gunicorn_serve(): def gunicorn_serve():
@@ -36,6 +122,7 @@ def gunicorn_serve():
None: 该函数不返回值,会阻塞运行直到服务停止 None: 该函数不返回值,会阻塞运行直到服务停止
""" """
import sys import sys
from gunicorn.app.wsgiapp import run from gunicorn.app.wsgiapp import run
config = load_config() config = load_config()
+50 -35
View File
@@ -19,17 +19,11 @@ from fastapi.middleware.gzip import GZipMiddleware
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from chongming_config import get_config from chongming_config import get_config
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from .core.logger import get_logger from .core.logger import get_logger
# from .core.static_files import SVFSStaticFiles
from sqlite_vfs.contrib.fastapi_static import SVFSStaticFiles from sqlite_vfs.contrib.fastapi_static import SVFSStaticFiles
from .core import lifespan_setup, call_stats from .core import lifespan_setup, call_stats
from .routes import ( from .routes import stats_router, worker_admin_router, auth_router
storage_router,
stats_router,
worker_admin_router,
auth_router,
license_router,
)
config = get_config() config = get_config()
@@ -69,9 +63,6 @@ async def lifespan(app: FastAPI):
is_inline_owner = manager.acquire_inline_owner() is_inline_owner = manager.acquire_inline_owner()
app.state.inline_owner_locked = is_inline_owner app.state.inline_owner_locked = is_inline_owner
# 4.3 机器码授权状态横幅(需 WorkerManager 就绪:时钟水位存共享 cache)
lifespan_setup.log_license_banner(app)
# 5. 插件加载(混合模式:inline 进程内加载;external 由独立进程运行, # 5. 插件加载(混合模式:inline 进程内加载;external 由独立进程运行,
# 其路由经 Zenoh 动态注册——进程编排交给 Docker/K8s,网关不拉起进程) # 其路由经 Zenoh 动态注册——进程编排交给 Docker/K8s,网关不拉起进程)
worker_tasks, custom_routers, inline_types, external_types = ( worker_tasks, custom_routers, inline_types, external_types = (
@@ -208,41 +199,67 @@ def _should_log_access(status_code: int, sample_rate: float = ACCESS_SAMPLE_RATE
return next(_access_log_counter) % denom == 0 return next(_access_log_counter) % denom == 0
@app.middleware("http") def _log_access(scope: Scope, status_code: int, start: float) -> None:
async def access_log_middleware(request: Request, call_next): """按采样规则写一条访问日志(成功按 access_sample_rate,错误 >=400 全量)"""
if not _should_log_access(status_code):
return
client = scope.get("client")
logger.info(
"%s %s status=%s duration=%.2fms client=%s",
scope.get("method", "-"), scope.get("path", "-"), status_code,
(time.perf_counter() - start) * 1000, client[0] if client else "-",
extra={"status_code": status_code},
)
class AccessLogMiddleware:
""" """
HTTP访问日志中间件,记录每个请求的详细信息。 HTTP访问日志中间件(纯 ASGI 实现),记录每个请求的详细信息。
记录的信息包括: 记录的信息包括:
- 请求方法和路径 - 请求方法和路径
- 响应状态码 - 响应状态码
- 请求处理时长(毫秒) - 请求处理时长(毫秒,按慢路径:从进入中间件到响应头产出)
- 客户端IP地址 - 客户端IP地址
高吞吐优化: 高吞吐优化:
- 采样:成功请求按 access_sample_rate 记录,错误(>=400)全量 - 采样:成功请求按 access_sample_rate 记录,错误(>=400)全量
- 异步:% 参数化消息 + 队列日志(QueueHandler),格式化与文件 I/O - 异步:% 参数化消息 + 队列日志(QueueHandler),格式化与文件 I/O
在后台监听线程执行,主线程仅入队 在后台监听线程执行,主线程仅入队
- 不用 BaseHTTPMiddleware(@app.middleware("http")):它会为每请求建 anyio
参数: task group 并把响应体经内存流过一手,压测实测高并发下吃掉 30~50% 吞吐;
request: FastAPI请求对象 这里只包一层 send 截获 response.start,无缓冲、无额外任务
call_next: 下一个中间件或路由处理函数
返回:
Response: 处理后的响应对象
""" """
start = time.perf_counter()
response = await call_next(request)
duration = (time.perf_counter() - start) * 1000
client = request.client.host if request.client else "-"
if _should_log_access(response.status_code): def __init__(self, app: ASGIApp) -> None:
logger.info( self.app = app
"%s %s status=%s duration=%.2fms client=%s",
request.method, request.url.path, response.status_code, duration, client, async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
extra={"status_code": response.status_code}, if scope["type"] != "http":
) await self.app(scope, receive, send)
return response return
start = time.perf_counter()
logged = False
async def send_wrapper(message: Message) -> None:
nonlocal logged
# 响应头产出即记录(流式响应因此只计到首字节,不计整段流)
if message["type"] == "http.response.start" and not logged:
logged = True
_log_access(scope, message["status"], start)
await send(message)
try:
await self.app(scope, receive, send_wrapper)
except Exception:
# 未产出响应就异常(外层 500):补一条全量错误日志后原样抛出
if not logged:
_log_access(scope, 500, start)
raise
app.add_middleware(AccessLogMiddleware)
@app.get("/", summary="根路径") @app.get("/", summary="根路径")
@@ -320,8 +337,6 @@ if vfs_db_path := config[env].get("file_system", {}).get("path", None):
static_files_handler = SVFSStaticFiles(vfs_db_path=vfs_db_path) static_files_handler = SVFSStaticFiles(vfs_db_path=vfs_db_path)
app.mount("/static", static_files_handler, name="static") app.mount("/static", static_files_handler, name="static")
app.include_router(storage_router)
app.include_router(stats_router) app.include_router(stats_router)
app.include_router(worker_admin_router) app.include_router(worker_admin_router)
app.include_router(auth_router) app.include_router(auth_router)
app.include_router(license_router)
+61 -5
View File
@@ -1,5 +1,7 @@
import hashlib import hashlib
import os
import time import time
from collections import OrderedDict
from fastapi import Depends, HTTPException, status, Request from fastapi import Depends, HTTPException, status, Request
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
@@ -21,6 +23,45 @@ _jwt_cache = Cache.from_config(get_config()["cache"])
# 安全性由命中时的 exp 复核保证,TTL 不承担过期判断 # 安全性由命中时的 exp 复核保证,TTL 不承担过期判断
_JWT_CACHE_TTL = 60 _JWT_CACHE_TTL = 60
# ── 一级:进程内 JWT 结果缓存(LRU + TTL)────────────────────────────
# 鉴权依赖是同步依赖(def),每次请求都要走一遍共享缓存——backend=redis 时
# 即"每请求一次同步网络往返 + FastAPI 线程池 hop"。同一 token 在 TTL 内被
# 反复使用是常态,故在进程内加一层 L1:命中直接返回,不碰网络。
# 语义与 L2 完全一致:key 含密钥/算法指纹;命中仍复核 exp(过期=拒绝而非放行);
# 条目按 TTL 惰性清理并限长。CHONGMING_JWT_L1=0 可关闭(便于 A/B 与排障)。
_JWT_L1_ENABLED = os.environ.get("CHONGMING_JWT_L1", "1") != "0"
_JWT_L1_MAX = 1024
# 存 orjson 字节而非 dict:命中时反序列化出新副本,保证"调用方修改返回值
# 不污染缓存"这一 L2 既有契约(L2 同样是 JSON 往返后返回新对象)
_jwt_l1: "OrderedDict[str, tuple[bytes, float, float]]" = OrderedDict() # key → (payload_json, exp, 写入时刻)
def _l1_get(key: str):
"""命中返回 (payload_json: bytes, exp);条目超过共享缓存 TTL(运行时可调,便于测试/运维)视为过期"""
entry = _jwt_l1.get(key)
if entry is None:
return None
_payload_json, _exp, ts = entry
if time.time() - ts >= _JWT_CACHE_TTL:
_jwt_l1.pop(key, None)
return None
_jwt_l1.move_to_end(key)
return entry
def _l1_set(key: str, payload_json: bytes, exp: float) -> None:
_jwt_l1[key] = (payload_json, exp, time.time())
_jwt_l1.move_to_end(key)
while len(_jwt_l1) > _JWT_L1_MAX:
_jwt_l1.popitem(last=False)
def _l1_sweep(now: float) -> None:
"""惰性清理:超 TTL 的整条丢弃(token 永不重复时防堆积),只在表较大时调用"""
stale = [k for k, v in _jwt_l1.items() if now - v[2] >= _JWT_CACHE_TTL]
for k in stale:
_jwt_l1.pop(k, None)
def _cache_key(token: str, secret: str, algorithm: str) -> str: def _cache_key(token: str, secret: str, algorithm: str) -> str:
"""缓存 key:token 内容 + 密钥/算法指纹,不同密钥/算法不串用""" """缓存 key:token 内容 + 密钥/算法指纹,不同密钥/算法不串用"""
@@ -28,20 +69,33 @@ def _cache_key(token: str, secret: str, algorithm: str) -> str:
def _cached_decode(token: str, secret: str, algorithm: str) -> tuple: def _cached_decode(token: str, secret: str, algorithm: str) -> tuple:
"""JWT 解密结果缓存((payload, exp)),共享缓存避免高频请求重复密码学运算。 """JWT 解密结果缓存((payload, exp)):一级进程内 LRU,二级共享缓存。
- 命中且未过期 -> 直接返回,跳过 RSA/ECDSA 验证与 JSON 解析 - 一级命中且未过期 -> 直接返回(零网络、零密码学);过期 -> 直接拒绝
- 命中但 exp 已过 -> 直接拒绝,不重解密(条目保留至 TTL 到期, - 一级未命中 -> 查二级共享缓存;命中且未过期 -> 回填一级后返回
过期 token 的重复请求零密码学开销;缓存不延长 token 有效期) - 二级也未命中 -> 完整解码(RSA/ECDSA 验证)后写二级 + 一级(TTL=60s/30s)
- 未命中 -> 完整解码后写入共享缓存(TTL=60s)
- decode 失败(无效 token)不缓存,每次都完整验证后拒绝 - decode 失败(无效 token)不缓存,每次都完整验证后拒绝
""" """
key = _cache_key(token, secret, algorithm) key = _cache_key(token, secret, algorithm)
now = time.time()
if _JWT_L1_ENABLED:
hit = _l1_get(key)
if hit is not None:
payload_json, exp, _ts = hit
if not exp or exp >= now:
# 反序列化出新副本:调用方修改不影响缓存条目(与 L2 契约一致)
return orjson.loads(payload_json), exp
raise ValueError("Token 已过期")
if len(_jwt_l1) > _JWT_L1_MAX // 2: # 仅大表时惰性清理,避免常态 O(n)
_l1_sweep(now)
entry = _jwt_cache.get(key) entry = _jwt_cache.get(key)
if entry is not None: if entry is not None:
payload = entry["payload"] payload = entry["payload"]
exp = entry.get("exp", 0) exp = entry.get("exp", 0)
if not exp or exp >= time.time(): if not exp or exp >= time.time():
if _JWT_L1_ENABLED:
_l1_set(key, orjson.dumps(payload), exp)
return payload, exp return payload, exp
raise ValueError("Token 已过期") raise ValueError("Token 已过期")
jwt = JWT(secret, algorithm) jwt = JWT(secret, algorithm)
@@ -49,6 +103,8 @@ def _cached_decode(token: str, secret: str, algorithm: str) -> tuple:
payload = orjson.loads(payload_str.encode("utf-8")) payload = orjson.loads(payload_str.encode("utf-8"))
exp = payload.get("exp", 0) exp = payload.get("exp", 0)
_jwt_cache.set(key, {"payload": payload, "exp": exp}, expire=_JWT_CACHE_TTL) _jwt_cache.set(key, {"payload": payload, "exp": exp}, expire=_JWT_CACHE_TTL)
if _JWT_L1_ENABLED:
_l1_set(key, orjson.dumps(payload), exp)
return payload, exp return payload, exp
-142
View File
@@ -1,142 +0,0 @@
"""网关侧机器码授权门禁。
职责:把 `chongming_license` 的判定接进网关生命周期 —— 读配置、建管理器、
在插件发现阶段拒绝未授权插件、输出启动横幅。
**为什么"是否执行拒绝"不读配置开关**:部署目录的 `config.toml` 是明文可写的,
任何 `enabled = false` / `mode = "off"` 都能被一键绕开。因此以**打包形态**为准
(Nuitka/PyInstaller 产物 `sys.frozen`/`__compiled__`):打包即强制校验,且
改配置无法伪造打包与否。开发/测试(非打包)默认放行,需显式用
``[license] enabled = true`` 或 ``CHONGMING_LICENSE_MODE=on`` 打开才能验证门禁逻辑。
安全边界说明:授权信任根是**厂商私钥**(Ed25519 签名),公钥可明文放置 ——
换公钥只会验签失败,无法据以伪造。注意根仓 `resources/*.key` 那套 `.mbank`
验签是 HMAC **对称**且密钥随包发布,只能算防篡改,**不能**作为授权信任根。
"""
from __future__ import annotations
import os
import sys
from chongming_cache import Cache
from chongming_config import get_config
from chongming_license import (
DEFAULT_MIN_MATCH,
LicenseManager,
LicenseStatus,
PluginDecision,
)
from .logger import get_logger
logger = get_logger("license")
# 默认授权文件路径(相对网关工作目录,与 config/、data/ 同级)
DEFAULT_LICENSE_PATH = "license.lic"
# 强制开启/关闭的环境变量(仅非打包态生效;打包态恒为强制校验)
ENV_MODE = "CHONGMING_LICENSE_MODE"
_TRUTHY = {"on", "enforce", "enabled", "true", "1", "yes"}
_manager: LicenseManager | None = None
def is_packaged() -> bool:
"""当前是否运行在打包产物中(Nuitka / PyInstaller)。
Nuitka 在模块全局注入 ``__compiled__``,PyInstaller 设置 ``sys.frozen``。
"""
return bool(getattr(sys, "frozen", False) or globals().get("__compiled__"))
def license_conf() -> dict:
"""``[license]`` 配置段(缺省为空 dict)。"""
return get_config().get("license", {}) or {}
def should_enforce() -> bool:
"""是否真正执行拒绝。
打包产物 **恒为 True**(见模块文档);非打包态由配置/环境变量决定,
默认 False 以免影响现有开发流程与测试。
"""
conf = license_conf()
env_mode = os.environ.get(ENV_MODE, "").strip().lower()
if is_packaged():
if conf.get("enabled") is False or env_mode in ("off", "disable", "disabled", "0"):
# 陷阱日志:打包态却看到"关闭授权"的意图 —— 不予采纳,但要留痕
logger.warning(
"[license] 检测到关闭授权的配置/环境变量,但当前为打包产物,"
"授权校验强制开启(该配置不生效)。若确需关闭请联系研发。"
)
return True
if env_mode:
return env_mode in _TRUTHY
return bool(conf.get("enabled", False))
def _build_manager() -> LicenseManager:
conf = license_conf()
return LicenseManager(
license_path=str(conf.get("path") or DEFAULT_LICENSE_PATH),
public_key=str(conf.get("public_key") or ""),
min_match=int(conf.get("min_match", DEFAULT_MIN_MATCH)),
# 时钟水位存共享缓存(sqlite 本机 / redis 跨主机),不加 TTL:
# 水位一旦过期被淘汰,回拨检测就失效了
clock_store=Cache.from_config(get_config()["cache"]),
enforce=should_enforce(),
logger=logger,
)
def get_manager() -> LicenseManager:
"""授权管理器单例(惰性创建)。"""
global _manager
if _manager is None:
_manager = _build_manager()
return _manager
def reset_manager() -> None:
"""丢弃单例(测试用;换配置或换 license 文件后重新判定)。"""
global _manager
_manager = None
def reload() -> LicenseStatus:
"""重新读取 license 文件(管理接口导入新文件后调用)。"""
return get_manager().reload()
def check_plugin(plugin_type: str) -> PluginDecision:
"""该插件是否获授权运行。"""
return get_manager().check_plugin(plugin_type)
def denied_types(plugin_types) -> set[str]:
"""从给定插件类型里筛出未获授权的(即不允许加载的)。"""
return get_manager().denied_types(plugin_types)
def describe() -> dict:
"""给 ``GET /license/info`` 与前端「授权」页的完整信息。"""
data = get_manager().describe()
data["enforce_forced_by_packaging"] = is_packaged()
data["license_path"] = get_manager().license_path
return data
def log_banner() -> None:
"""启动横幅:机器码、授权状态、授权插件、公钥指纹。"""
try:
for line in get_manager().banner():
logger.info(line)
if get_manager().enforce and not get_manager().status.ok:
logger.error(
"[license] 未授权状态下所有插件均不会加载;"
"请在管理页「授权」导入 license 文件,或联系厂商签发(需提供本机机器码)。"
)
except Exception as exc: # 授权模块异常不应阻断启动流程
logger.error("[license] 授权状态检查异常: %s", exc)
+11 -82
View File
@@ -326,16 +326,6 @@ async def setup_route_registry(
registry = RouteRegistry(app, manager, prefix=prefix) registry = RouteRegistry(app, manager, prefix=prefix)
if inline_types: if inline_types:
registry.set_inline_types(set(inline_types)) registry.set_inline_types(set(inline_types))
# 授权门禁:未授权类型一律忽略其路由描述符。此处独立重算(不依赖 load_workers
# 的调用顺序),避免模块级状态带来的隐式时序耦合。
from .licensing import denied_types as _denied_types, get_manager as _get_license
if _get_license().enforce:
denied = _denied_types({p["type"] for p in _scan_plugins()})
registry.set_denied_types(denied)
if denied:
logger.warning(
"路由注册表已拒绝未授权插件描述符: %s", sorted(denied)
)
await registry.start(session) await registry.start(session)
app.state.route_registry = registry app.state.route_registry = registry
logger.info( logger.info(
@@ -345,9 +335,9 @@ async def setup_route_registry(
return registry return registry
def _scan_plugins() -> list[dict]: def _discover_plugins() -> list[dict]:
""" """
扫描全部插件配置(**不做授权过滤**,授权门禁见 _discover_plugins)。 发现所有插件配置。
优先从各插件的独立配置文件 config/workers/<type>/config.toml 的 [workers] 节读取, 优先从各插件的独立配置文件 config/workers/<type>/config.toml 的 [workers] 节读取,
兼容旧的 config/workers/config.toml 全局配置。 兼容旧的 config/workers/config.toml 全局配置。
@@ -409,6 +399,9 @@ def _scan_plugins() -> list[dict]:
"load_mode": load_mode, "load_mode": load_mode,
"mqtt_topics": mqtt_topics, "mqtt_topics": mqtt_topics,
"config": runtime_config, "config": runtime_config,
# [workers.config] 展平前的原始段:供 load_workers 按「配置优先来源模式」
# 引导/覆盖共享缓存,使 inline 插件初始化即拿到缓存值(cache 模式)
"config_section": dict(workers_section.get("config") or {}),
}) })
# 2. 兼容旧写法: config/workers/config.toml(仅加载尚未发现的插件) # 2. 兼容旧写法: config/workers/config.toml(仅加载尚未发现的插件)
@@ -432,75 +425,6 @@ def _scan_plugins() -> list[dict]:
return plugins return plugins
# 最近一次授权门禁的拒绝明细:插件类型 -> 拒绝原因
_denied_plugin_types: dict[str, str] = {}
def denied_plugin_types() -> dict[str, str]:
"""最近一次插件发现中被授权门禁拒绝的类型 -> 原因(供管理接口展示)。"""
return dict(_denied_plugin_types)
def log_license_banner(app: FastAPI | None = None) -> None:
"""启动横幅:机器码、授权状态、授权插件、公钥指纹。
在 WorkerManager 就绪后调用(时钟水位写入共享 cache)。异常不阻断启动 ——
授权模块自身出问题时仍应让网关起来并留下日志,而不是变成一个起不来的黑盒。
"""
from .licensing import log_banner
try:
log_banner()
except Exception as exc:
logger.error("[license] 授权状态横幅输出失败: %s", exc)
def _discover_plugins() -> list[dict]:
"""发现插件并应用**授权门禁**,返回可加载的插件列表。
这里是机器码授权的**唯一收口**:load_workers / validate_contracts /
register_routes(以及旧写法下的 MQTT 主题汇总)全部经由本函数取插件列表,
因此在此剔除未授权插件即可让三处自动保持一致。
必须收口于此而不是"跳过 import":契约校验(``validate_contracts``)会把
该插件的 TOML 路由视为"缺少 @rpc_method 注册"的硬性错误并**终止启动** ——
只跳过 import 会让设备直接起不来。在此剔除后,该插件连同其 TOML 路由、
自定义路由、MQTT 主题、副本数一起从启动流程中消失,契约校验自然跳过。
external 插件的路由由 RouteRegistry 动态挂载,不经本函数,故另有
``RouteRegistry.set_denied_types`` 封堵(否则把 load_mode 改成 external
即可绕过门禁)。
"""
plugins = _scan_plugins()
from .licensing import check_plugin, get_manager
if not get_manager().enforce:
# 未启用(开发/测试):不剔除任何插件
_denied_plugin_types.clear()
return plugins
allowed: list[dict] = []
denied: dict[str, str] = {}
for p in plugins:
decision = check_plugin(p["type"])
if decision.allowed:
allowed.append(p)
else:
denied[p["type"]] = decision.message
_denied_plugin_types.clear()
_denied_plugin_types.update(denied)
for plugin_type in sorted(denied):
logger.error("[license] %s", denied[plugin_type])
if denied:
logger.warning(
"授权门禁已拒绝 %d 个插件(不加载、不注册路由、管理接口禁止操作): %s",
len(denied), sorted(denied),
)
return allowed
async def load_workers(app: FastAPI, session: zenoh.Session, manager: WorkerManager, async def load_workers(app: FastAPI, session: zenoh.Session, manager: WorkerManager,
is_inline_owner: bool = True): is_inline_owner: bool = True):
""" """
@@ -554,7 +478,12 @@ async def load_workers(app: FastAPI, session: zenoh.Session, manager: WorkerMana
# 否则用配置值引导并写回 cache(多网关共享同一 cache 时自动对齐)。 # 否则用配置值引导并写回 cache(多网关共享同一 cache 时自动对齐)。
count = wc["count"] count = wc["count"]
worker_type = wc["type"] worker_type = wc["type"]
worker_config = wc["config"] # [workers.config] 单一事实来源:按配置优先来源模式对齐(cache 模式取缓存值、
# toml 模式以文件为准并覆盖缓存),必须在注册 starter/创建副本之前完成——
# 插件实例(on_init 重建客户端等)用的就是这份配置。
worker_config = manager.align_inline_config(
worker_type, wc["config"], wc.get("config_section")
)
if not module_name or not factory_name: if not module_name or not factory_name:
logger.warning(f"Worker 配置不完整: {wc}") logger.warning(f"Worker 配置不完整: {wc}")
+23 -6
View File
@@ -169,6 +169,15 @@ class MQTTBridge:
except orjson.JSONDecodeError: except orjson.JSONDecodeError:
return return
# 实例定向控制字段:与 HTTP 的 X-Chongming-Instance 头同义,指定请求发往的
# worker 进程实例。必须在下面归一化之前取出——归一化只保留 meta/data 两键,
# 之后控制字段就不在载荷里了。
target_instance: Optional[str] = None
if isinstance(pl, dict):
ti = pl.pop("target_instance", None)
if isinstance(ti, str) and ti.strip():
target_instance = ti.strip()
# 信封协议: {"meta": {...}, "data": {...}} # 信封协议: {"meta": {...}, "data": {...}}
# 兼容非信封载荷(如第三方设备原始报文): 整包视为业务数据、meta 置空, # 兼容非信封载荷(如第三方设备原始报文): 整包视为业务数据、meta 置空,
# 与 worker 运行时 runtime.py 的容错语义一致(method 默认 GET)—— # 与 worker 运行时 runtime.py 的容错语义一致(method 默认 GET)——
@@ -194,7 +203,7 @@ class MQTTBridge:
for ct, compiled in self._compiled: for ct, compiled in self._compiled:
matched = compiled.match(t) matched = compiled.match(t)
if matched: if matched:
await self._route_custom(ct, matched, pl, t) await self._route_custom(ct, matched, pl, t, target_instance)
return return
# 2. 内置接口 chongming/{worker}/{operation} # 2. 内置接口 chongming/{worker}/{operation}
@@ -204,7 +213,7 @@ class MQTTBridge:
wt, op = parts[1], parts[2] wt, op = parts[1], parts[2]
subj = f"{wt}.{op}" subj = f"{wt}.{op}"
rt = f"{MQTT_TOPIC_PREFIX}/{wt}/{op}/response" rt = f"{MQTT_TOPIC_PREFIX}/{wt}/{op}/response"
await self._route(wt, subj, pl, rt) await self._route(wt, subj, pl, rt, target_instance)
def _fill(self, template: str, matched: re.Match) -> str: def _fill(self, template: str, matched: re.Match) -> str:
"""用匹配到的通配符值填充模板中的 {1} {2}... 占位符""" """用匹配到的通配符值填充模板中的 {1} {2}... 占位符"""
@@ -214,7 +223,8 @@ class MQTTBridge:
template = template.replace("{" + str(i) + "}", g) template = template.replace("{" + str(i) + "}", g)
return template return template
async def _route_custom(self, ct: dict, matched: re.Match, pl: dict, topic: str): async def _route_custom(self, ct: dict, matched: re.Match, pl: dict, topic: str,
target_instance: Optional[str] = None):
worker = ct.get("worker", "") worker = ct.get("worker", "")
# subject 默认 "<worker>.<topic 模板最后一段>",与内置 <插件>.<操作> 约定一致 # subject 默认 "<worker>.<topic 模板最后一段>",与内置 <插件>.<操作> 约定一致
subject_tpl = ct.get("subject") subject_tpl = ct.get("subject")
@@ -225,15 +235,22 @@ class MQTTBridge:
# response_topic 默认 "<实际匹配主题>/response" # response_topic 默认 "<实际匹配主题>/response"
rt_tpl = ct.get("response_topic") rt_tpl = ct.get("response_topic")
rt = self._fill(rt_tpl, matched) if rt_tpl else f"{topic}/response" rt = self._fill(rt_tpl, matched) if rt_tpl else f"{topic}/response"
await self._route(worker, subject, pl, rt) await self._route(worker, subject, pl, rt, target_instance)
async def _route(self, wt: str, subj: str, pl: dict, rt: str): async def _route(self, wt: str, subj: str, pl: dict, rt: str,
target_instance: Optional[str] = None):
"""把一次 MQTT 请求转成 worker RPC。
target_instance 为实例定向目标(来自载荷顶层的 target_instance 控制字段,
已在 _handle 归一化前摘除);缺省时跨实例随机均衡,行为与以往一致。
"""
logger.info(f"MQTT -> {subj}") logger.info(f"MQTT -> {subj}")
meta = pl.get("meta") or {} meta = pl.get("meta") or {}
try: try:
r = await self._wm.call(worker_type=wt, subject=subj, payload=pl, r = await self._wm.call(worker_type=wt, subject=subj, payload=pl,
timeout=float(meta.get("timeout", 30)), timeout=float(meta.get("timeout", 30)),
cacheable=bool(meta.get("cacheable", False))) cacheable=bool(meta.get("cacheable", False)),
instance_id=target_instance or None)
if r is not None: if r is not None:
await self._pub(rt, r) await self._pub(rt, r)
except RuntimeError as e: except RuntimeError as e:
+20
View File
@@ -24,6 +24,21 @@ from .call_stats import spawn, record_call
logger = get_logger("route_generator") logger = get_logger("route_generator")
# 实例定向请求头:给定后该请求固定发往指定 worker 进程实例(同一实例下的多个副本
# 之间仍随机均衡),不参与跨实例的随机负载均衡。取值 = 实例身份(外部进程启动参数
# --instance / CHONGMING_WORKER_INSTANCE;inline 副本为网关自身实例 id)。
# 未携带该头时行为与以往完全一致(全类型随机均衡)。
INSTANCE_HEADER = "X-Chongming-Instance"
def _instance_from_request(request: Request) -> Optional[str]:
"""从请求头读取定向实例 id;空串/未携带视为不定向。"""
value = request.headers.get(INSTANCE_HEADER)
if value:
value = value.strip()
return value or None
# 简单类型映射 # 简单类型映射
TYPE_MAP: Dict[str, Any] = { TYPE_MAP: Dict[str, Any] = {
"float": float, "float": float,
@@ -348,6 +363,7 @@ def _make_body_endpoint(worker_manager: WorkerManager, rc: RouteConfig, worker_t
payload=payload_bytes, payload=payload_bytes,
timeout=timeout, timeout=timeout,
cacheable=cacheable, cacheable=cacheable,
instance_id=_instance_from_request(request),
) )
duration_ms = (time.perf_counter() - start) * 1000 duration_ms = (time.perf_counter() - start) * 1000
_spawn_call_record( _spawn_call_record(
@@ -409,6 +425,7 @@ def _make_raw_body_endpoint(worker_manager: WorkerManager, rc: RouteConfig, work
payload=payload, payload=payload,
timeout=timeout, timeout=timeout,
cacheable=cacheable, cacheable=cacheable,
instance_id=_instance_from_request(request),
) )
duration_ms = (time.perf_counter() - start) * 1000 duration_ms = (time.perf_counter() - start) * 1000
_spawn_call_record( _spawn_call_record(
@@ -472,6 +489,7 @@ def _make_query_endpoint(worker_manager: WorkerManager, rc: RouteConfig, worker_
payload=payload_bytes, payload=payload_bytes,
timeout=timeout, timeout=timeout,
cacheable=cacheable, cacheable=cacheable,
instance_id=_instance_from_request(request),
) )
duration_ms = (time.perf_counter() - start) * 1000 duration_ms = (time.perf_counter() - start) * 1000
_spawn_call_record( _spawn_call_record(
@@ -515,6 +533,7 @@ def _make_query_endpoint(worker_manager: WorkerManager, rc: RouteConfig, worker_
payload=payload_bytes, payload=payload_bytes,
timeout=timeout, timeout=timeout,
cacheable=cacheable, cacheable=cacheable,
instance_id=_instance_from_request(request),
) )
duration_ms = (time.perf_counter() - start) * 1000 duration_ms = (time.perf_counter() - start) * 1000
_spawn_call_record( _spawn_call_record(
@@ -648,6 +667,7 @@ def _make_stream_endpoint(
payload=payload, payload=payload,
timeout=timeout, timeout=timeout,
idle_timeout=idle_timeout, idle_timeout=idle_timeout,
instance_id=_instance_from_request(request),
) )
except RuntimeError as e: except RuntimeError as e:
duration_ms = (time.perf_counter() - start) * 1000 duration_ms = (time.perf_counter() - start) * 1000
-28
View File
@@ -74,10 +74,6 @@ class RouteRegistry:
self._mqtt_bridge = None self._mqtt_bridge = None
# 网关进程内加载(inline)的插件类型:忽略其描述符,防止与盘扫注册重复挂载 # 网关进程内加载(inline)的插件类型:忽略其描述符,防止与盘扫注册重复挂载
self._inline_types: set = set() self._inline_types: set = set()
# 被授权门禁拒绝的插件类型:忽略其描述符(未获授权的插件不得暴露任何 HTTP 端点)
self._denied_types: set = set()
# 已告警过的拒绝类型(避免 worker 周期性重发布描述符导致日志刷屏)
self._denied_logged: set = set()
# 将动态 schema 注入 OpenAPI(provider 实时读取本注册表) # 将动态 schema 注入 OpenAPI(provider 实时读取本注册表)
_inject_schemas_to_openapi( _inject_schemas_to_openapi(
@@ -91,20 +87,6 @@ class RouteRegistry:
"""设置 inline 插件类型集合(混合模式防双注册)。""" """设置 inline 插件类型集合(混合模式防双注册)。"""
self._inline_types = set(types) self._inline_types = set(types)
def set_denied_types(self, types: set):
"""设置被授权门禁拒绝的插件类型集合,其路由描述符一律忽略。
必要性:门禁在网关侧只能拦住"盘扫加载"这条路径。若把某插件 config.toml
的 ``load_mode`` 改成 ``external``,网关就不再加载它、``set_inline_types``
也不再抑制它,其独立进程推来的路由描述符就会被正常挂载 —— 等于绕过了
授权。因此这里对未授权类型再做一道拦截,且**与 load_mode 无关**。
局限(如实记录):这只拦截 HTTP 暴露,被直接拉起的 external 进程自身
仍会运行。彻底封堵需在 chongming-worker SDK 侧加门禁(另行规划)。
"""
self._denied_types = set(types)
self._denied_logged &= self._denied_types
async def start(self, session: zenoh.Session): async def start(self, session: zenoh.Session):
"""订阅 chongming/routes/**,开始接收路由描述符。""" """订阅 chongming/routes/**,开始接收路由描述符。"""
self._loop = asyncio.get_running_loop() self._loop = asyncio.get_running_loop()
@@ -169,16 +151,6 @@ class RouteRegistry:
# 该插件由网关进程内加载(盘扫注册路由),忽略外部描述符防双注册 # 该插件由网关进程内加载(盘扫注册路由),忽略外部描述符防双注册
return return
if wtype in self._denied_types:
# 授权门禁拒绝的插件:不下发任何 HTTP 端点(与 load_mode 无关)
if wtype not in self._denied_logged:
self._denied_logged.add(wtype)
logger.error(
"拒绝未授权插件 %s 的路由描述符(key=%s):该插件不在授权白名单内,"
"其端点不会被挂载", wtype, key,
)
return
if kind == zenoh.SampleKind.DELETE: if kind == zenoh.SampleKind.DELETE:
self._remove(wtype, route_id) self._remove(wtype, route_id)
return return
+324 -43
View File
@@ -14,6 +14,11 @@ from .logger import get_logger
logger = get_logger("worker_manager") logger = get_logger("worker_manager")
# ── 插件运行时配置(chongming-cache 单一事实来源)──
# 键与 worker SDK(chongming_worker.runtime._configs_cache_key)一致,全局唯一一份
# [workers.config];读写规则见 WorkerManager.align_inline_config。
_CONFIGS_KEY = "worker:cm:configs:{type}"
# ── 全局副本数(chongming-cache 单一事实来源)── # ── 全局副本数(chongming-cache 单一事实来源)──
# 多网关共用同一 cache(单实例=本地 diskcache,多实例=共享卷/redis), # 多网关共用同一 cache(单实例=本地 diskcache,多实例=共享卷/redis),
# 「count」语义 = 该类 worker 的**全局目标总量**,各网关按在线数量分摊。 # 「count」语义 = 该类 worker 的**全局目标总量**,各网关按在线数量分摊。
@@ -83,7 +88,40 @@ class _MemLRU:
self._d.popitem(last=False) self._d.popitem(last=False)
# ── 进程内直调(inline 快路径)─────────────────────────────────────────────
# inline 副本由本进程托管(与网关同事件循环),业务请求却仍要绕
# "zenoh 查询 → router → queryable 回调 → 跨线程回本 loop"。分派逻辑已统一到
# chongming_worker.dispatch,故此处可直接 await 同一个 dispatcher,省掉整圈往返。
# 语义不变:信封/错误通道/超时/重试/换副本/实例定向/in-flight 与 drain 全部沿用原逻辑;
# 非本地副本(external 插件、w>1 时非托管进程)与旧版 worker 包自动回退 zenoh。
_RPC_LOCAL = os.environ.get("CHONGMING_RPC_LOCAL", "1") != "0"
try: # 旧版 chongming-worker(<0.5)无 dispatch 模块:保持纯 zenoh 行为
from chongming_worker.dispatch import get_dispatcher as _get_local_dispatcher
except Exception: # noqa: BLE001
_get_local_dispatcher = None
def _local_dispatcher_for(key_expr: str) -> Optional[Any]:
"""'worker/<type>/<id>' → 本进程托管该副本的 dispatcher;非本地/不可用返回 None"""
if not _RPC_LOCAL or _get_local_dispatcher is None:
return None
parts = key_expr.split("/")
if len(parts) != 3 or parts[0] != "worker":
return None
return _get_local_dispatcher(parts[1], parts[2])
def _err_text(payload: Any) -> str:
"""把错误载荷归一为文本(兼容 zenoh ZBytes 与直调的 bytes)"""
if payload is None:
return "unknown error"
if isinstance(payload, (bytes, bytearray)):
return bytes(payload).decode("utf-8", "replace") or "unknown error"
return str(payload) or "unknown error"
class WorkerManager: class WorkerManager:
""" """
统一 Worker 管理器 统一 Worker 管理器
@@ -107,6 +145,15 @@ class WorkerManager:
self._launch_worker_count = launch_worker_count self._launch_worker_count = launch_worker_count
self._workers_cache: Dict[str, List[str]] = {} # type -> list of worker key_exprs self._workers_cache: Dict[str, List[str]] = {} # type -> list of worker key_exprs
self._local_workers: Dict[str, List[str]] = {} # type -> list of locally registered workers self._local_workers: Dict[str, List[str]] = {} # type -> list of locally registered workers
# ── 实例级定向(把请求固定到某个进程实例而非随机负载均衡)──
# 副本经 liveliness 只暴露 worker/<type>/<id>,实例身份取自副本 admin status
# 的 instance_id(Python/Rust 运行时均返回),故定向索引按需查询并缓存:
# _instance_index[type][instance_id] = [worker/<type>/<id>, ...]
# _instance_unknown[type] = [已查询但身份不可得的副本 key, ...]
# 只缓存已查询过的副本;每次按需刷新时整表重建并对新副本补查询、对离线副本剪枝。
self._instance_index: Dict[str, Dict[str, List[str]]] = {}
self._instance_unknown: Dict[str, List[str]] = {}
self._instance_locks: Dict[str, asyncio.Lock] = {}
self._lock = asyncio.Lock() self._lock = asyncio.Lock()
self._refresh_interval = 15 # seconds self._refresh_interval = 15 # seconds
# 当前网关心跳标识:优先环境变量(多实例需唯一,尤其同宿主多网关须显式指定), # 当前网关心跳标识:优先环境变量(多实例需唯一,尤其同宿主多网关须显式指定),
@@ -134,6 +181,51 @@ class WorkerManager:
""" """
return "cache" if str(self._launch_worker_count).lower() == "cache" else "toml" return "cache" if str(self._launch_worker_count).lower() == "cache" else "toml"
def align_inline_config(
self,
worker_type: str,
worker_config: Dict[str, Any],
config_section: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""inline 插件初始化前,按配置优先来源模式对齐其 [workers.config](网关侧兜底)。
worker SDK 的 Plugin.create_workers 会做同样的事(权威实现,见
chongming_worker.runtime.resolve_startup_config);网关侧再兜一层,是为了让
网关(随本仓库发布)与 worker 包版本解耦:worker 包仍是旧版时,旧版 on_init
只拿到 config/workers/<type>/config.toml 展平出来的配置,会出现“前端显示缓存值、
实际跑 toml”。规则与 SDK 一致:
- "cache":缓存里有该段就用它覆盖传入配置(toml 仅作缓存缺失时的引导并写回,
之后以缓存为准、跨重启保持);
- "toml" :以 config.toml 为准,把该段覆盖写入缓存(前端读缓存 → 随之显示文件值);
- 未配置缓存后端(cache 为 None)时保持传入配置不动。
Args:
worker_type: 插件类型(如 aibox_sophon_adapter)
worker_config: _discover_plugins 展平出来的运行时配置
config_section: 该插件 config.toml 的 [workers.config] 原始段(引导/覆盖缓存用)
"""
if self.cache is None:
return worker_config
key = _CONFIGS_KEY.format(type=worker_type)
section = config_section if isinstance(config_section, dict) else None
if self.config_mode == "cache":
cached = self.cache.get(key)
if isinstance(cached, dict):
logger.info(
"%s [workers.config] 取自缓存(cache 模式,%d 键)",
worker_type, len(cached),
)
return {**worker_config, **cached}
if section:
self.cache[key] = dict(section) # 缓存缺失:用 toml 引导并写回
logger.info("%s [workers.config] 缓存缺失,已用 config.toml 引导", worker_type)
return worker_config
if section is not None:
self.cache[key] = dict(section) # 以文件为准:覆盖缓存,前端随之显示文件值
logger.info("%s [workers.config] 以 config.toml 为准并覆盖缓存(toml 模式)", worker_type)
return worker_config
# ── 全局副本数协调(chongming-cache 单一事实来源)──────────── # ── 全局副本数协调(chongming-cache 单一事实来源)────────────
# count/quota 的读写与分摊已收归 worker 层(chongming_worker.runtime): # count/quota 的读写与分摊已收归 worker 层(chongming_worker.runtime):
# 网关不再直写这些缓存键,仅维护「在线网关列表」(register/heartbeat/unregister)。 # 网关不再直写这些缓存键,仅维护「在线网关列表」(register/heartbeat/unregister)。
@@ -356,23 +448,112 @@ class WorkerManager:
logger.error(f"刷新 worker 列表失败: {e}") logger.error(f"刷新 worker 列表失败: {e}")
await asyncio.sleep(self._refresh_interval) await asyncio.sleep(self._refresh_interval)
async def get_worker(self, worker_type: str) -> Optional[str]: def _all_workers(self, worker_type: str) -> List[str]:
"""获取一个可用的 worker 地址(简单随机负载均衡) """该类型当前全部候选副本(liveliness 列表优先,启动初期回退本地注册表)。"""
workers = list(self._workers_cache.get(worker_type, []))
if not workers:
workers = list(self._local_workers.get(worker_type, []))
return workers
async def _query_instance_id(self, worker_type: str, worker_key: str) -> Optional[str]:
"""问单个副本要它的实例身份(admin status 的 instance_id)。
复用单副本管理协议 worker/<type>/<id>/__admin__;查询失败或副本运行旧版
运行时(回复无 instance_id)返回 None,由调用方归入 unknown 组。
"""
wid = worker_key.split("/")[2] if "/" in worker_key else worker_key
try:
reply = await self._send_admin_cmd(worker_type, wid, "query", timeout=2.0)
except Exception as e: # noqa: BLE001 - 单副本查询失败不影响其它副本
logger.warning("[instance] %s/%s 身份查询失败: %s", worker_type, wid, e)
return None
iid = reply.get("instance_id") if isinstance(reply, dict) else None
return str(iid) if iid else None
async def _refresh_instance_map(self, worker_type: str) -> Dict[str, List[str]]:
"""重建某类型的「实例 → 副本」索引,并按需补查询新副本。
只对从未查询过的副本发一次 admin status(本地注册的 inline 副本直接返回
网关自身实例,零查询);已查询过的副本沿用缓存,因此常规路径是纯内存查询,
仅新上线副本会引入一次额外 RPC。离线副本在重建时被剪枝。
"""
workers = self._all_workers(worker_type)
known = self._instance_index.get(worker_type, {})
unknown = set(self._instance_unknown.get(worker_type, []))
# 反查已有归属:副本 -> 实例(副本 key 唯一,一个副本只属一个实例)
owner = {w: iid for iid, ws in known.items() for w in ws}
index: Dict[str, List[str]] = {}
still_unknown: List[str] = []
queried = owner.keys() | unknown
for w in workers:
if w in queried:
iid = owner.get(w)
else:
# 本地注册(inline)副本天然属于本网关实例,无需查询
iid = self._gateway_id if w in self._local_workers.get(worker_type, []) else None
if iid is None:
iid = await self._query_instance_id(worker_type, w)
if iid is None:
still_unknown.append(w)
if iid:
index.setdefault(iid, []).append(w)
self._instance_index[worker_type] = index
self._instance_unknown[worker_type] = still_unknown
return index
async def _instance_workers(self, worker_type: str, instance_id: str) -> List[str]:
"""取某实例下该类型的在线副本 key 列表(无则空表)。"""
index = self._instance_index.get(worker_type)
if index is None:
# 同类型并发请求只让一个跑刷新,其余等结果,避免重复查询同一批副本
lock = self._instance_locks.setdefault(worker_type, asyncio.Lock())
async with lock:
index = self._instance_index.get(worker_type)
if index is None:
index = await self._refresh_instance_map(worker_type)
return index.get(instance_id, [])
async def get_worker(
self, worker_type: str, instance_id: Optional[str] = None
) -> Optional[str]:
"""获取一个可用的 worker 地址。
instance_id 为 None 时按简单随机负载均衡挑选(默认,行为不变);
指定实例时只在该实例的在线副本中挑选,把请求固定到那个进程实例。
实例无在线副本(或该实例在旧版运行时下无法识别身份)返回 None。
读路径无锁:_workers_cache 由 refresh_workers 整表原子替换, 读路径无锁:_workers_cache 由 refresh_workers 整表原子替换,
_local_workers 由同步方法修改,单事件循环内读取安全。 _local_workers 由同步方法修改,单事件循环内读取安全。
""" """
workers = self._workers_cache.get(worker_type, []) if instance_id:
# 如果缓存为空,回退到本地注册表(适用于启动初期 refresh_workers 尚未执行时) return await self._pick_from_instance(worker_type, instance_id)
if not workers: workers = self._all_workers(worker_type)
local = self._local_workers.get(worker_type, []) return random.choice(workers) if workers else None
if local:
return random.choice(local) async def _pick_from_instance(
self, worker_type: str, instance_id: str
) -> Optional[str]:
"""在指定实例的在线副本中随机挑一个(定向路径,永不跨实例回退)。
定向的语义就是"必须打在同一个进程实例上",因此实例离线时返回 None 让
调用方报错,而不是悄悄换成别的实例——后者会让粘滞状态类的插件跑在错误
副本上,产生难查的数据错乱。
"""
candidates = await self._instance_workers(worker_type, instance_id)
if candidates:
return random.choice(candidates)
logger.warning(
"[instance] 目标实例 %s 下无 %s 的在线副本(已识别实例: %s;%d 个副本读不出身份)",
instance_id, worker_type, sorted(self._instance_index.get(worker_type, {})),
len(self._instance_unknown.get(worker_type, [])),
)
return None return None
return random.choice(workers)
async def _pick_other_worker( async def _pick_other_worker(
self, worker_type: str, exclude: List[str] self, worker_type: str, exclude: List[str],
instance_id: Optional[str] = None,
) -> Optional[str]: ) -> Optional[str]:
""" """
选择一个不同于已尝试列表的 worker。 选择一个不同于已尝试列表的 worker。
@@ -380,10 +561,17 @@ class WorkerManager:
Args: Args:
worker_type: Worker 类型 worker_type: Worker 类型
exclude: 已尝试(需排除)的 worker key_expr 列表 exclude: 已尝试(需排除)的 worker key_expr 列表
instance_id: 定向实例;给定时只在同实例内换副本,绝不跨实例
Returns: Returns:
可用的 worker key_expr,若无其他可用则返回 None 可用的 worker key_expr,若无其他可用则返回 None
""" """
if instance_id:
candidates = [
w for w in await self._instance_workers(worker_type, instance_id)
if w not in exclude
]
return random.choice(candidates) if candidates else None
workers = self._workers_cache.get(worker_type, []) workers = self._workers_cache.get(worker_type, [])
candidates = [w for w in workers if w not in exclude] candidates = [w for w in workers if w not in exclude]
if candidates: if candidates:
@@ -405,24 +593,37 @@ class WorkerManager:
return False return False
return True return True
async def _raise_no_worker(self, worker_type: str) -> None: async def _no_worker_message(
self, worker_type: str, instance_id: Optional[str] = None
) -> str:
""" """
无可用 Worker 时的统一诊断:列出当前已知的所有 worker 类型并抛 RuntimeError。 无可用 Worker 时的统一诊断文案。
供 call() / call_stream() 共用,保证两类调用错误信息一致。 供 call() / call_stream() 共用,保证两类调用错误信息一致;定向(instance_id)
调用额外列出当前已识别实例,便于区分"实例名写错"与"实例离线"。
""" """
all_known = list(self._workers_cache.keys()) all_known = list(self._workers_cache.keys())
local_known = list(self._local_workers.keys()) local_known = list(self._local_workers.keys())
all_types = sorted(set(all_known + local_known)) all_types = sorted(set(all_known + local_known))
msg = f"类型 '{worker_type}' 没有可用 Worker。"
if instance_id:
msg = ( msg = (
f"类型 '{worker_type}' 没有可用 Worker。" f"类型 '{worker_type}' 的实例 '{instance_id}' 没有在线副本(定向调用不"
f"跨实例回退)。已识别实例: "
f"{sorted(self._instance_index.get(worker_type, {})) or '(无)'}。"
f"请检查: 1) 该实例 id 是否写错(取值见 GET "
f"/admin/workers/{worker_type}/instances 或副本启动时的 --instance)"
f" 2) 该实例进程是否在线 3) 该实例的 worker 是否运行旧版运行时"
f"(不支持实例身份上报)"
)
else:
msg += (
f"当前已知的 Worker 类型: {all_types if all_types else '(无)'}。" f"当前已知的 Worker 类型: {all_types if all_types else '(无)'}。"
f"请检查: 1) config/workers/config.toml 是否配置了该类型 " f"请检查: 1) config/workers/config.toml 是否配置了该类型 "
f"2) 插件初始化是否成功(查看启动日志中的 '加载 Worker 插件失败' 错误)" f"2) 插件初始化是否成功(查看启动日志中的 '加载 Worker 插件失败' 错误)"
f"3) Worker 进程是否正常运行" f"3) Worker 进程是否正常运行"
) )
logger.warning(msg) return msg
raise RuntimeError(msg)
async def call_stream( async def call_stream(
self, self,
@@ -431,6 +632,7 @@ class WorkerManager:
payload: Dict[str, Any], payload: Dict[str, Any],
timeout: float = 300.0, timeout: float = 300.0,
idle_timeout: float = 10.0, idle_timeout: float = 10.0,
instance_id: Optional[str] = None,
) -> AsyncGenerator[bytes, None]: ) -> AsyncGenerator[bytes, None]:
""" """
流式调用:消费 Worker 的多回复流,逐块产出 JSON 字节。 流式调用:消费 Worker 的多回复流,逐块产出 JSON 字节。
@@ -449,21 +651,53 @@ class WorkerManager:
payload: 请求参数(信封格式,由路由层构造) payload: 请求参数(信封格式,由路由层构造)
timeout: 流总寿命上限(秒),需大于业务最长生成时长 timeout: 流总寿命上限(秒),需大于业务最长生成时长
idle_timeout: 空闲超时(秒),超时抛 StreamIdleTimeoutError idle_timeout: 空闲超时(秒),超时抛 StreamIdleTimeoutError
instance_id: 定向进程实例(None=随机负载均衡);流本身不可重试,
故实例离线时直接报错,不会中途改派到别的实例
Yields: Yields:
bytes: 每个回复块的 JSON 字节 bytes: 每个回复块的 JSON 字节
Raises: Raises:
RuntimeError: 无可用 worker RuntimeError: 无可用 worker(含定向实例无在线副本)
StreamIdleTimeoutError: 空闲超时 StreamIdleTimeoutError: 空闲超时
""" """
# 急切解析 worker:无可用时立即抛错,让路由层映射 HTTP 503 # 急切解析 worker:无可用时立即抛错,让路由层映射 HTTP 503
# (异步生成器函数体在首次迭代才执行,故这里用普通 async 函数返回内部生成器) # (异步生成器函数体在首次迭代才执行,故这里用普通 async 函数返回内部生成器)
worker_expr = await self.get_worker(worker_type) worker_expr = await self.get_worker(worker_type, instance_id=instance_id)
if not worker_expr: if not worker_expr:
await self._raise_no_worker(worker_type) raise RuntimeError(await self._no_worker_message(worker_type, instance_id))
full_key = f"{worker_expr}/{subject}" full_key = f"{worker_expr}/{subject}"
# 进程内直调(inline 快路径):与 zenoh 路径同语义(逐块产出、错误块终止、无重试)
local = _local_dispatcher_for(worker_expr)
if local is not None and local.has_subject(subject):
async def _local_stream():
agen = local.invoke_stream(subject, orjson.dumps(payload))
try:
while True:
try:
data = await asyncio.wait_for(agen.__anext__(), timeout=idle_timeout)
except StopAsyncIteration:
break
except asyncio.TimeoutError:
raise StreamIdleTimeoutError(
f"Worker {worker_expr} 流式空闲超时 ({idle_timeout}s 无数据)"
) from None
if not data:
continue
decoded = orjson.loads(data)
if isinstance(decoded, dict) and decoded.get("error"):
# Worker 错误收尾块:转发后终止流
yield data
return
yield data
finally:
# 消费方提前退出(SSE 断连/网关放弃):停流并取消在途派发
await agen.aclose()
return _local_stream()
reply_iter = self.session.get( reply_iter = self.session.get(
full_key, full_key,
payload=orjson.dumps(payload), payload=orjson.dumps(payload),
@@ -552,6 +786,7 @@ class WorkerManager:
timeout: float = 3.0, timeout: float = 3.0,
retries: int = 1, retries: int = 1,
cacheable: bool = False, cacheable: bool = False,
instance_id: Optional[str] = None,
) -> Any: ) -> Any:
""" """
统一调用入口(dict 语义,MQTT Bridge 等调用方使用) 统一调用入口(dict 语义,MQTT Bridge 等调用方使用)
@@ -563,6 +798,8 @@ class WorkerManager:
timeout: 单次调用超时(秒) timeout: 单次调用超时(秒)
retries: 失败重试次数 retries: 失败重试次数
cacheable: 是否允许缓存结果(若为 False 则跳过缓存读写) cacheable: 是否允许缓存结果(若为 False 则跳过缓存读写)
instance_id: 定向进程实例;给定时请求固定发往该实例(不参与随机
负载均衡,重试也只在该实例内换副本),None=默认随机均衡
Returns: Returns:
Dict[str, Any]: 调用结果(worker 回复解析后的 dict) Dict[str, Any]: 调用结果(worker 回复解析后的 dict)
@@ -570,10 +807,18 @@ class WorkerManager:
# cacheable=False 时不计算 cache_key,省一次排序序列化 # cacheable=False 时不计算 cache_key,省一次排序序列化
cache_key = None cache_key = None
if cacheable: if cacheable:
cache_key = f"{worker_type}:{subject}:{orjson.dumps(payload, option=orjson.OPT_SORT_KEYS).decode()}" # 定向调用的缓存键含实例名:定向通常用于读取该实例的本地状态
# (本地库/设备连接),共用缓存会拿到其它实例的结果
cache_key = (
f"{worker_type}:{subject}"
f":{orjson.dumps(payload, option=orjson.OPT_SORT_KEYS).decode()}"
)
if instance_id:
cache_key = f"{instance_id}@{cache_key}"
reply_bytes = await self._call_impl( reply_bytes = await self._call_impl(
worker_type, subject, orjson.dumps(payload), worker_type, subject, orjson.dumps(payload),
timeout=timeout, retries=retries, cacheable=cacheable, cache_key=cache_key, timeout=timeout, retries=retries, cacheable=cacheable, cache_key=cache_key,
instance_id=instance_id,
) )
return orjson.loads(reply_bytes) if reply_bytes else {} return orjson.loads(reply_bytes) if reply_bytes else {}
@@ -585,6 +830,7 @@ class WorkerManager:
timeout: float = 3.0, timeout: float = 3.0,
retries: int = 1, retries: int = 1,
cacheable: bool = False, cacheable: bool = False,
instance_id: Optional[str] = None,
) -> bytes: ) -> bytes:
""" """
统一调用入口(bytes 语义,HTTP 端点透传用):返回 worker 回复的原始字节。 统一调用入口(bytes 语义,HTTP 端点透传用):返回 worker 回复的原始字节。
@@ -602,6 +848,7 @@ class WorkerManager:
timeout: 单次调用超时(秒) timeout: 单次调用超时(秒)
retries: 失败重试次数 retries: 失败重试次数
cacheable: 是否允许缓存结果(缓存原始字节) cacheable: 是否允许缓存结果(缓存原始字节)
instance_id: 定向进程实例(见 call());None=默认随机均衡
Returns: Returns:
bytes: worker 回复的原始字节(空回复兜底 b"{}",与 call() 的 {} 语义一致) bytes: worker 回复的原始字节(空回复兜底 b"{}",与 call() 的 {} 语义一致)
@@ -610,9 +857,12 @@ class WorkerManager:
cache_key = None cache_key = None
if cacheable: if cacheable:
cache_key = f"{worker_type}:{subject}:{hashlib.sha256(payload).hexdigest()}" cache_key = f"{worker_type}:{subject}:{hashlib.sha256(payload).hexdigest()}"
if instance_id:
cache_key = f"{instance_id}@{cache_key}"
return await self._call_impl( return await self._call_impl(
worker_type, subject, payload, worker_type, subject, payload,
timeout=timeout, retries=retries, cacheable=cacheable, cache_key=cache_key, timeout=timeout, retries=retries, cacheable=cacheable, cache_key=cache_key,
instance_id=instance_id,
) )
async def _call_impl( async def _call_impl(
@@ -624,11 +874,13 @@ class WorkerManager:
retries: int = 1, retries: int = 1,
cacheable: bool = False, cacheable: bool = False,
cache_key: Optional[str] = None, cache_key: Optional[str] = None,
instance_id: Optional[str] = None,
) -> bytes: ) -> bytes:
""" """
调用实现:缓存 > 负载均衡 > 超时重试,返回回复的原始字节。 调用实现:缓存 > 选副本(随机均衡/实例定向)> 超时重试,返回回复的原始字节。
call() / call_raw() 共用本实现,仅入参出参形态不同(dict ↔ bytes)。 call() / call_raw() 共用本实现,仅入参出参形态不同(dict ↔ bytes)。
instance_id 给定时选副本与重试都被约束在该实例内(定向语义,不跨实例回退)。
""" """
# 1. 缓存检查(仅在 cacheable=True 时检查) # 1. 缓存检查(仅在 cacheable=True 时检查)
if cacheable and cache_key is not None: if cacheable and cache_key is not None:
@@ -645,22 +897,54 @@ class WorkerManager:
logger.debug(f"命中共享缓存: {cache_key}") logger.debug(f"命中共享缓存: {cache_key}")
return disk_val return disk_val
# 2. 获取可用 worker # 2. 获取可用 worker(instance_id 给定时固定到该实例,否则随机均衡)
worker_expr = await self.get_worker(worker_type) worker_expr = await self.get_worker(worker_type, instance_id=instance_id)
if not worker_expr: if not worker_expr:
await self._raise_no_worker(worker_type) raise RuntimeError(await self._no_worker_message(worker_type, instance_id))
full_key = f"{worker_expr}/{subject}" full_key = f"{worker_expr}/{subject}"
last_error = None last_error = None
tried_workers: List[str] = [] tried_workers: List[str] = []
async def _switch_worker(why: str) -> bool:
"""换一个副本重试;返回是否值得继续(定向调用只在同实例内换副本)。
换完更新闭包外层的 worker_expr / full_key。定向实例内已无未试副本时
返回 False —— 继续循环只会重复打同一个坏副本,白等一整轮超时。
"""
nonlocal worker_expr, full_key, last_error
new_worker = await self._pick_other_worker(
worker_type, tried_workers, instance_id=instance_id
)
if new_worker:
if new_worker != worker_expr:
logger.info(f"{why}切换 Worker: {worker_expr} -> {new_worker}")
worker_expr = new_worker
full_key = f"{worker_expr}/{subject}"
return True
# 没有任何未试过的副本可换:继续循环只会重复打同一个副本(若它已死,
# 白等一整轮超时;若它已回业务错误,结果必然相同),直接结束重试。
# 保留具体失败原因(last_error 已由调用方按场景写入),便于定位。
scope = f"实例 '{instance_id}' 内" if instance_id else ""
last_error = f"{scope}已无其它未试副本可换({why}):{last_error}"
return False
for attempt in range(retries + 1): for attempt in range(retries + 1):
# 记录已尝试的 worker # 记录已尝试的 worker
if worker_expr not in tried_workers: if worker_expr not in tried_workers:
tried_workers.append(worker_expr) tried_workers.append(worker_expr)
try: try:
local = _local_dispatcher_for(worker_expr)
if local is not None and local.has_subject(subject):
# 进程内直调(inline 快路径):省 zenoh 往返,语义与下面 zenoh 路径一致
logger.debug(f"[local-rpc] {worker_expr}/{subject} 进程内直调")
local_res = await asyncio.wait_for(
local.invoke_unary(subject, payload_bytes), timeout=timeout
)
reply_ok, reply_payload = local_res.ok, local_res.payload
else:
reply_iter = self.session.get( reply_iter = self.session.get(
full_key, full_key,
payload=payload_bytes, payload=payload_bytes,
@@ -685,16 +969,20 @@ class WorkerManager:
f"Worker {worker_expr} 无回复,尝试重试 {attempt+1}/{retries+1}" f"Worker {worker_expr} 无回复,尝试重试 {attempt+1}/{retries+1}"
) )
last_error = f"Worker {worker_expr} 无回复" last_error = f"Worker {worker_expr} 无回复"
# 换一个 worker 重试 if not await _switch_worker("无回复"):
new_worker = await self._pick_other_worker(worker_type, tried_workers) break
if new_worker:
worker_expr = new_worker
full_key = f"{worker_expr}/{subject}"
continue continue
reply = first_reply reply = first_reply
if reply.ok: # type: ignore if reply.ok: # type: ignore
reply_bytes = bytes(reply.ok.payload) if reply.ok.payload else b"{}" reply_ok = True
reply_payload = bytes(reply.ok.payload) if reply.ok.payload else b"{}"
else:
reply_ok = False
reply_payload = reply.err.payload # type: ignore
if reply_ok:
reply_bytes = bytes(reply_payload) if reply_payload else b"{}"
# 缓存写入(仅在 cacheable=True 时):解析一次做成功检查, # 缓存写入(仅在 cacheable=True 时):解析一次做成功检查,
# 但缓存与返回均保留原始字节(网关侧零二次序列化)。 # 但缓存与返回均保留原始字节(网关侧零二次序列化)。
# 先写内存一级,再异步写共享二级(executor,不阻塞响应返回) # 先写内存一级,再异步写共享二级(executor,不阻塞响应返回)
@@ -705,16 +993,12 @@ class WorkerManager:
await self._cache_set(cache_key, reply_bytes) await self._cache_set(cache_key, reply_bytes)
return reply_bytes return reply_bytes
else: else:
error_payload = reply.err.payload # type: ignore error_msg = _err_text(reply_payload)
error_msg = str(error_payload) if error_payload else "unknown error"
logger.warning(f"Worker 返回错误: {error_msg} (尝试 {attempt+1}/{retries+1})") logger.warning(f"Worker 返回错误: {error_msg} (尝试 {attempt+1}/{retries+1})")
last_error = f"Worker {worker_expr} 返回错误: {error_msg}" last_error = f"Worker {worker_expr} 返回错误: {error_msg}"
# 错误回复(含 Zenoh 超时)也换一个 worker 重试 # 错误回复(含 Zenoh 超时)也换一个 worker 重试
new_worker = await self._pick_other_worker(worker_type, tried_workers) if not await _switch_worker("错误后"):
if new_worker: break
logger.info(f"错误后切换 Worker: {worker_expr} -> {new_worker}")
worker_expr = new_worker
full_key = f"{worker_expr}/{subject}"
continue # 重试 continue # 重试
except asyncio.TimeoutError: except asyncio.TimeoutError:
logger.warning( logger.warning(
@@ -722,11 +1006,8 @@ class WorkerManager:
) )
last_error = f"Worker {worker_expr} 超时 (>{timeout}s)" last_error = f"Worker {worker_expr} 超时 (>{timeout}s)"
# 超时后换一个 worker 重试,避免打到仍在处理中的同一个 worker # 超时后换一个 worker 重试,避免打到仍在处理中的同一个 worker
new_worker = await self._pick_other_worker(worker_type, tried_workers) if not await _switch_worker("超时后"):
if new_worker: break
logger.info(f"超时后切换 Worker: {worker_expr} -> {new_worker}")
worker_expr = new_worker
full_key = f"{worker_expr}/{subject}"
continue continue
except Exception as e: except Exception as e:
logger.error(f"调用 worker 异常: {e}") logger.error(f"调用 worker 异常: {e}")
@@ -734,7 +1015,7 @@ class WorkerManager:
raise RuntimeError( raise RuntimeError(
f"Worker '{worker_type}' 调用失败 (已尝试 {len(tried_workers)} 个 Worker, " f"Worker '{worker_type}' 调用失败 (已尝试 {len(tried_workers)} 个 Worker, "
f"共 {retries + 1} 次, timeout={timeout}s): {last_error}" f"实际尝试 {attempt + 1} 次/上限 {retries + 1} 次, timeout={timeout}s): {last_error}"
) )
# ------------------------------------------------------------------ # ------------------------------------------------------------------
+1 -9
View File
@@ -1,13 +1,5 @@
from .storage import router as storage_router
from .stats import router as stats_router from .stats import router as stats_router
from .worker_admin import router as worker_admin_router from .worker_admin import router as worker_admin_router
from .auth import router as auth_router from .auth import router as auth_router
from .licensing import router as license_router
__all__ = [ __all__ = ["stats_router", "worker_admin_router", "auth_router"]
"storage_router",
"stats_router",
"worker_admin_router",
"auth_router",
"license_router",
]
-208
View File
@@ -1,208 +0,0 @@
"""机器码授权管理端点(查看授权状态 / 展示机器码 / 导入 license 文件)。
设计取舍:
- **仅超级管理员可用**:机器码与各硬件来源指纹属设备标识信息,且导入 license 会
改变整个网关的插件加载行为,与 `worker_admin` 同级风险,统一按超管门禁。
设备上默认已 seed admin/admin,现场装机人员登录后即可取机器码,不额外加门槛。
- **导入先验证后落盘**:非法/验签失败/机器码不符的文件一律拒绝且不覆盖现有文件,
避免"上传了一个坏文件导致原本可用的授权也没了"。
- **导入不即时加载插件**:`_discover_plugins` 在启动期完成,新授权的插件要等
网关重启才会真正加载;响应里的 `restart_required` 如实告知,前端据此提示。
路径为 ``/api/v1/license/...``(跟随 ``[default] prefix``)。
"""
from __future__ import annotations
import json
import os
import shutil
import tempfile
from pathlib import Path
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile
from chongming_config import get_config
from ..core import licensing
from ..core.auth import jwt_auth_dependency
from ..core.lifespan_setup import _scan_plugins, denied_plugin_types
from ..core.logger import get_logger
config = get_config()
logger = get_logger("license")
router = APIRouter(prefix=config["default"]["prefix"], tags=["license"])
# license 文件大小上限:正常文件仅数 KB,给足余量的同时避免被当作上传通道
MAX_LICENSE_BYTES = 64 * 1024
def require_superuser(
payload: dict = Depends(jwt_auth_dependency),
) -> dict:
"""仅超级管理员可访问(与 worker_admin 一致的门禁口径)。"""
if not payload.get("is_superuser"):
raise HTTPException(status_code=403, detail="需要超级管理员权限 (is_superuser)")
return payload
def _loaded_types(request: Request) -> set[str]:
"""当前已实际加载的插件类型(inline + external 观测集合)。"""
return set(getattr(request.app.state, "inline_types", None) or []) | set(
getattr(request.app.state, "external_types", None) or []
)
def _plugin_overview(request: Request) -> dict:
"""插件视角的授权全景:盘扫到的 / 已授权 / 被拒 / 当前已加载。"""
discovered = [p["type"] for p in _scan_plugins()]
manager = licensing.get_manager()
enforce = manager.enforce
licensed, denied = [], {}
for plugin_type in discovered:
decision = manager.check_plugin(plugin_type)
if decision.allowed:
licensed.append(plugin_type)
else:
denied[plugin_type] = {"code": decision.code, "message": decision.message}
return {
"discovered": sorted(discovered),
# 未启用校验时不存在"授权与否",如实按"全部可用"展示,避免误导
"licensed": sorted(licensed) if enforce else sorted(discovered),
"denied": denied if enforce else {},
"loaded": sorted(_loaded_types(request)),
"denied_from_last_scan": denied_plugin_types(),
}
@router.get("/license/info", summary="授权状态 / 机器码 / 申请码")
async def license_info(
request: Request,
_admin: dict = Depends(require_superuser),
):
"""返回本机机器码、授权状态、各插件授权情况与硬件来源明细。
现场申请授权时,把响应中的 ``request_code``(或 ``machine_code``)发给厂商即可。
"""
info = licensing.describe()
# 申请码:把各来源指纹打包成一段可抄写文本,方便直接贴进邮件 ——
# 只报机器码只能签出"精确匹配"的授权(无换硬件容错),故优先给申请码
request_code = ""
try:
from chongming_license import encode_request_code
request_code = encode_request_code()
except Exception as exc:
logger.warning("生成申请码失败:%s", exc)
overview = _plugin_overview(request)
return {
**info,
"request_code": request_code,
"plugins": overview,
}
@router.post("/license/reload", summary="重新读取授权文件")
async def license_reload(
_admin: dict = Depends(require_superuser),
):
"""从磁盘重新读取 license(手工替换文件后无需重启即可刷新状态)。
注意:插件是否加载仍以启动期判定为准,刷新状态不会加载新授权的插件。
"""
status = licensing.reload()
logger.info("授权文件已重新读取: %s", status.message)
return {"success": True, "status": status.as_dict()}
@router.post("/license/file", summary="导入 license 文件(覆盖现有授权)")
async def license_upload(
request: Request,
file: UploadFile = File(..., description="厂商签发的 license.lic"),
_admin: dict = Depends(require_superuser),
):
"""上传并安装 license 文件。
流程:读入 → 大小/JSON 校验 → **验签 + 机器码 + 有效期校验** → 通过才落盘
(临时文件 + 原子替换)→ 刷新授权状态。
未通过校验的文件一律拒绝,且**不会覆盖**现有可用授权 —— 现场误传坏文件不至于
把设备搞成未授权。校验通过但白名单含此前未授权的插件时,需**重启网关**才会
加载它们(`_discover_plugins` 只在启动期执行)。
"""
raw = await file.read()
if not raw:
raise HTTPException(status_code=422, detail="上传内容为空")
if len(raw) > MAX_LICENSE_BYTES:
raise HTTPException(
status_code=413,
detail=f"license 文件过大({len(raw)} 字节,上限 {MAX_LICENSE_BYTES})",
)
try:
json.loads(raw)
except ValueError as exc:
raise HTTPException(status_code=422, detail=f"不是合法的 JSON:{exc}") from exc
manager = licensing.get_manager()
target = Path(manager.license_path)
# 先落到同目录的临时文件再校验:read_license 需要真实路径,
# 且同目录保证后续 move 是原子 rename(不会读到半截文件)
tmp_dir = target.parent if str(target.parent) else Path(".")
tmp_dir.mkdir(parents=True, exist_ok=True)
fd, tmp_name = tempfile.mkstemp(prefix=".license-", suffix=".tmp", dir=str(tmp_dir))
try:
with os.fdopen(fd, "wb") as f:
f.write(raw)
# 用与运行期完全相同的判定逻辑校验(含机器码与有效期)
from chongming_license import read_license
status = read_license(
tmp_name,
manager.public_key,
min_match=manager.min_match,
sources=manager.collect_sources(),
)
if not status.ok:
raise HTTPException(
status_code=422,
detail=f"授权文件校验未通过:{status.message}(原因码 {status.code})",
)
shutil.move(tmp_name, target)
tmp_name = "" # 已移走,finally 不再删除
os.chmod(target, 0o644)
finally:
if tmp_name and os.path.exists(tmp_name):
os.unlink(tmp_name)
after = licensing.reload()
logger.info(
"已导入授权文件 %s:%s(授权插件 %s)",
target, after.message, sorted(after.licensed_plugins) or "无",
)
# 本次授权后"已授权但尚未加载"的插件 —— 这些需要重启网关才会生效
pending = sorted(
(set(_plugin_overview(request)["discovered"]) & after.licensed_plugins)
- _loaded_types(request)
)
return {
"success": True,
"license_path": str(target),
"status": after.as_dict(),
"licensed_plugins": sorted(after.licensed_plugins),
"restart_required": bool(pending),
"pending_plugins": pending,
"restart_hint": (
f"以下插件已获授权但尚未加载,重启网关后生效:{pending}"
if pending else "无新增插件需要加载,无需重启。"
),
}
-53
View File
@@ -1,53 +0,0 @@
# src/chongming/app/routes/storage.py
import os
import uuid
import urllib.parse
import aiofiles
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, Request
from fastapi.responses import FileResponse
router = APIRouter(prefix="/api/v1/storage", tags=["Storage"])
@router.post("/upload", summary="上传文件")
async def upload_file(
request: Request,
file: UploadFile = File(...),
worker_type: str = Form(...),
):
"""上传文件到本地存储"""
upload_root = request.app.state.upload_root
object_key = f"{worker_type}/{uuid.uuid4()}/{file.filename}"
dest = os.path.join(upload_root, object_key)
os.makedirs(os.path.dirname(dest), exist_ok=True)
async with aiofiles.open(dest, "wb") as afp:
while chunk := await file.read(1024 * 1024):
await afp.write(chunk)
file_size = os.path.getsize(dest)
return {
"file_key": object_key,
"bucket": "local",
"filename": file.filename,
"content_type": file.content_type,
"size": file_size,
}
@router.get("/download/{file_key:path}")
async def download_file(request: Request, file_key: str):
"""从本地存储下载文件"""
upload_root = request.app.state.upload_root
path = os.path.join(upload_root, file_key)
if not os.path.isfile(path):
raise HTTPException(404, "文件不存在")
raw_filename = file_key.split('/')[-1]
encoded = urllib.parse.quote(raw_filename, safe='')
return FileResponse(
path,
filename=raw_filename,
headers={"Content-Disposition": f"attachment; filename*=UTF-8''{encoded}"},
)
+2 -48
View File
@@ -19,7 +19,6 @@ from chongming_config import get_config
from ..core.auth import jwt_auth_dependency from ..core.auth import jwt_auth_dependency
from ..core.logger import get_logger from ..core.logger import get_logger
from ..core import licensing
config = get_config() config = get_config()
@@ -40,26 +39,6 @@ def require_admin(
return payload return payload
def _ensure_licensed(worker_type: str) -> None:
"""授权门禁:未授权的插件禁止任何管理操作(扩缩容 / 重启 / 读改配置)。
未授权插件在启动期就不加载(见 lifespan_setup._discover_plugins),因此在
离线设备上通常表现为"不存在";但网关可能因设备重启后
license 到期、或被手动改过配置而留下残存副本,故此处必须独立拦截 ——
否则可以借扩缩容把未授权插件重新拉起、或经 config 修改其行为。
每个管理端点在入口显式调用(而非做成依赖),便于审计"哪些入口受门禁保护"。
"""
if not licensing.get_manager().enforce:
return
decision = licensing.check_plugin(worker_type)
if not decision.allowed:
raise HTTPException(
status_code=403,
detail=f"{decision.message};该插件不在授权白名单内,管理操作已拒绝",
)
def _get_manager(request: Request): def _get_manager(request: Request):
"""从应用状态获取 WorkerManager;未初始化时(lifespan 外)报 503。""" """从应用状态获取 WorkerManager;未初始化时(lifespan 外)报 503。"""
manager = getattr(request.app.state, "worker_manager", None) manager = getattr(request.app.state, "worker_manager", None)
@@ -73,27 +52,9 @@ async def list_workers(
request: Request, request: Request,
_admin: dict = Depends(require_admin), _admin: dict = Depends(require_admin),
): ):
"""返回全部插件实例(worker_type / worker_id / mode / alive),供前端渲染与选择。 """返回全部插件实例(worker_type / worker_id / mode / alive),供前端渲染与选择。"""
每个实例附 `licensed` 标注:未授权插件即使有残存副本也不应被操作,
前端据此禁用按钮。
"""
manager = _get_manager(request) manager = _get_manager(request)
workers = manager.list_workers() return {"workers": manager.list_workers(), "success": True}
enforce = licensing.get_manager().enforce
for w in workers:
w["licensed"] = (
True if not enforce
else licensing.check_plugin(w["worker_type"]).allowed
)
return {
"workers": workers,
"success": True,
"license_enforced": enforce,
"denied_plugin_types": (
licensing.denied_types({w["worker_type"] for w in workers}) if enforce else set()
),
}
@router.post("/admin/workers/restart", summary="重启单个 Worker 副本") @router.post("/admin/workers/restart", summary="重启单个 Worker 副本")
@@ -108,7 +69,6 @@ async def restart_worker(
- worker_type: 插件类型(如 example) - worker_type: 插件类型(如 example)
- worker_id: 副本 id(如 worker-0-1-12345),由 GET /admin/workers 提供 - worker_id: 副本 id(如 worker-0-1-12345),由 GET /admin/workers 提供
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
try: try:
result = await manager.restart_worker(worker_type, worker_id) result = await manager.restart_worker(worker_type, worker_id)
@@ -143,7 +103,6 @@ async def restart_all(
注意:网关进程内加载的 inline 插件副本经本接口重启,仅重建 worker 协程, 注意:网关进程内加载的 inline 插件副本经本接口重启,仅重建 worker 协程,
不会卸载插件;external 插件同理逐个副本重启。 不会卸载插件;external 插件同理逐个副本重启。
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
candidates = [ candidates = [
w for w in manager.list_workers() w for w in manager.list_workers()
@@ -187,7 +146,6 @@ async def scale_workers(
- inline 直接调整本进程 Worker 池;external 经 Zenoh 下发 rescale 到存活副本, - inline 直接调整本进程 Worker 池;external 经 Zenoh 下发 rescale 到存活副本,
由其进程内池扩缩并写回自身配置。 由其进程内池扩缩并写回自身配置。
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
try: try:
result = await manager.scale_workers(worker_type, body.count) result = await manager.scale_workers(worker_type, body.count)
@@ -228,7 +186,6 @@ async def list_worker_instances(
每个实例 = 一个可独立承载副本的执行单元(inline=本网关进程/host, 每个实例 = 一个可独立承载副本的执行单元(inline=本网关进程/host,
external=独立进程/容器),含当前副本数与可选中的 worker_id。 external=独立进程/容器),含当前副本数与可选中的 worker_id。
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
try: try:
instances = await manager.list_instances(worker_type) instances = await manager.list_instances(worker_type)
@@ -253,7 +210,6 @@ async def scale_worker_instance(
经该实例任一副本下发 rescale{scale, instance_id, absolute:true},由其进程内 经该实例任一副本下发 rescale{scale, instance_id, absolute:true},由其进程内
WorkerPool 直接扩缩并记入逐实例覆盖表;重启后仍保持该绝对数。 WorkerPool 直接扩缩并记入逐实例覆盖表;重启后仍保持该绝对数。
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
try: try:
result = await manager.scale_instance( result = await manager.scale_instance(
@@ -300,7 +256,6 @@ async def read_worker_config(
- worker_type: 插件类型(如 example) - worker_type: 插件类型(如 example)
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
try: try:
info = await manager.get_worker_config(worker_type) info = await manager.get_worker_config(worker_type)
@@ -350,7 +305,6 @@ async def update_worker_config(
- worker_type: 插件类型(如 example) - worker_type: 插件类型(如 example)
- body.updates: 待更新的键值,如 {"data_dir": "./data2"} - body.updates: 待更新的键值,如 {"data_dir": "./data2"}
""" """
_ensure_licensed(worker_type)
manager = _get_manager(request) manager = _get_manager(request)
if not body.updates: if not body.updates:
raise HTTPException(status_code=422, detail="updates 不能为空") raise HTTPException(status_code=422, detail="updates 不能为空")
+10 -3
View File
@@ -22,7 +22,7 @@ import httpx
import orjson import orjson
from fastapi import FastAPI from fastapi import FastAPI
REPO_ROOT = Path(__file__).resolve().parent REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT / "src")) sys.path.insert(0, str(REPO_ROOT / "src"))
sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src")) sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src"))
@@ -53,6 +53,11 @@ _cs_mod = importlib.util.module_from_spec(_cs_spec)
sys.modules["chongming.app.core.call_stats"] = _cs_mod sys.modules["chongming.app.core.call_stats"] = _cs_mod
_cs_spec.loader.exec_module(_cs_mod) # type: ignore[union-attr] _cs_spec.loader.exec_module(_cs_mod) # type: ignore[union-attr]
# 审计采样率固定为全量:生产 config 的 access_sample_rate=0.1 下,成功请求的审计行
# 本就是按概率写的,断言"几条请求 = 几条审计行"会随机失败。本测试验证的是审计链路
# 是否工作,与采样策略无关,故固定为 1.0。
_cs_mod._AUDIT_SAMPLE_RATE = 1.0
# 加载真实 route_generator # 加载真实 route_generator
_rg_path = REPO_ROOT / "src" / "chongming" / "app" / "core" / "route_generator.py" _rg_path = REPO_ROOT / "src" / "chongming" / "app" / "core" / "route_generator.py"
_rg_spec = importlib.util.spec_from_file_location("chongming.app.core.route_generator", _rg_path) _rg_spec = importlib.util.spec_from_file_location("chongming.app.core.route_generator", _rg_path)
@@ -95,12 +100,14 @@ class FakeWorkerManager:
self.reply = reply self.reply = reply
self.error = error self.error = error
async def call_raw(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False): async def call_raw(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False,
instance_id=None):
if self.error == "no_worker": if self.error == "no_worker":
raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。") raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。")
return self.reply return self.reply
async def call_stream(self, worker_type, subject, payload, timeout=300.0, idle_timeout=10.0): async def call_stream(self, worker_type, subject, payload, timeout=300.0, idle_timeout=10.0,
instance_id=None):
if self.error == "no_worker": if self.error == "no_worker":
raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。") raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。")
+163
View File
@@ -0,0 +1,163 @@
"""
inline 插件 [workers.config] 对齐(网关侧兜底)单测 —— WorkerManager.align_inline_config。
网关侧在创建 inline 副本之前,按「配置优先来源模式」(根 config.toml 的
[cache].launch_worker_count)对齐插件配置:cache 模式取共享缓存里的段,
toml 模式以 config/workers/<type>/config.toml 为准并覆盖缓存。目的是让网关与
worker 包版本解耦——worker 包仍是旧版时,旧版 on_init 只拿到 toml 展平配置,
会出现“前端显示缓存值、实际跑 toml”。
用 dict 作缓存桩,验证两种模式下插件拿到的配置与缓存/文件两侧的取值一致。
"""
import os
import sys
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(REPO, "src"))
os.chdir(REPO) # 网关模块导入期读根 config.toml / 日志配置
from chongming.app.core.worker_manager import _CONFIGS_KEY, WorkerManager # noqa: E402
PASS = 0
FAIL = 0
WT = "aibox_sophon_adapter"
# _discover_plugins 展平出来的形态(toml 值)
FLAT = {"type": WT, "count": 1, "data_dir": "./data", "mediamtx_addr": "rtsp://toml:8554"}
# 该插件 config.toml 的 [workers.config] 原始段
SECTION = {"data_dir": "./data", "mediamtx_addr": "rtsp://toml:8554", "heartbeat": {"timeout": 15}}
# 前端下发后缓存里的段(改过 mediamtx_addr 与 heartbeat)
CACHED = {"data_dir": "./data", "mediamtx_addr": "rtsp://cache:9999", "heartbeat": {"timeout": 30}}
def report(name, cond, detail=""):
global PASS, FAIL
if cond:
PASS += 1
print(f"✅ {name}" + (f" — {detail}" if detail else ""))
else:
FAIL += 1
print(f"❌ {name}" + (f" — {detail}" if detail else ""))
def manager(cache, mode):
m = WorkerManager.__new__(WorkerManager) # 免会话构造:仅用到 cache + config_mode
m.cache = cache
m._launch_worker_count = mode
return m
def run() -> int:
key = _CONFIGS_KEY.format(type=WT)
# ── 1. cache 模式:缓存有段 → 插件拿到缓存值(本次修复目标)──────────
cache = {key: dict(CACHED)}
got = manager(cache, "cache").align_inline_config(WT, dict(FLAT), dict(SECTION))
report(
"cache 模式:插件配置取缓存值(覆盖 toml 同名键)",
got.get("mediamtx_addr") == "rtsp://cache:9999"
and got.get("heartbeat") == {"timeout": 30},
f"mediamtx_addr={got.get('mediamtx_addr')}",
)
report("cache 模式:不覆盖已存在的缓存段", cache[key] == CACHED, f"cache={cache[key]}")
# ── 2. cache 模式:缓存缺失 → 用 toml 段引导并写回 ──────────────────
cache2 = {}
got2 = manager(cache2, "cache").align_inline_config(WT, dict(FLAT), dict(SECTION))
report(
"cache 模式:缓存缺失用 config.toml 引导并写回",
cache2.get(key) == SECTION and got2.get("mediamtx_addr") == "rtsp://toml:8554",
f"cache={cache2.get(key)}",
)
# ── 3. toml 模式:以文件为准并覆盖缓存(前端随之显示文件值)─────────
cache3 = {key: dict(CACHED)}
got3 = manager(cache3, "toml").align_inline_config(WT, dict(FLAT), dict(SECTION))
report(
"toml 模式:插件配置为 toml 值",
got3.get("mediamtx_addr") == "rtsp://toml:8554",
f"mediamtx_addr={got3.get('mediamtx_addr')}",
)
report(
"toml 模式:启动时用 toml 覆盖缓存(前端与运行值一致)",
cache3.get(key) == SECTION,
f"cache={cache3.get(key)}",
)
# ── 4. 边界:无缓存后端 / 无段(旧 config/workers/config.toml 兼容路径)──
cache4 = {key: dict(CACHED)}
m4 = manager(cache4, "cache")
m4.cache = None
report(
"无缓存后端:原样返回(经典 toml 行为)",
m4.align_inline_config(WT, dict(FLAT), dict(SECTION)) == FLAT
and cache4[key] == CACHED,
)
cache5 = {}
got5 = manager(cache5, "toml").align_inline_config(WT, dict(FLAT), None)
report("无 [workers.config] 段(旧写法):不写缓存、原样返回", got5 == FLAT and not cache5)
# ── 5. 与 SDK 侧一致:缓存键必须相同;SDK 为本地版本时才比对解析结果 ──
# 注意:venv 里的 chongming-worker 可能是索引里的发布版(uv sync 会把可编辑安装
# 换回锁文件版本),新版才有 resolve_startup_config / 正确的 toml 分支语义。
sys.path.insert(0, os.path.join(REPO, "utils", "worker", "src"))
import chongming_worker.runtime as r # noqa: E402
report(
"缓存键与 SDK 一致(两端必须读写同一键)",
r._configs_cache_key(WT) == _CONFIGS_KEY.format(type=WT),
f"sdk={r._configs_cache_key(WT)}",
)
local_src = os.path.join(REPO, "utils", "worker", "src") in os.path.abspath(r.__file__)
if not local_src:
print(f"⚠️ 跳过解析结果比对:当前 SDK 为 {r.__file__}(非本地源码,缺本次改动)")
else:
c = {r._configs_cache_key(WT): dict(CACHED)}
sdk = r.resolve_runtime_config(WT, dict(FLAT), c, "cache")
report(
"cache 模式:网关侧兜底与 SDK 解析结果一致",
sdk.get("mediamtx_addr") == CACHED["mediamtx_addr"],
f"sdk={sdk.get('mediamtx_addr')}",
)
# ── 6. 端到端:网关对齐 + 插件初始化(部署组合:网关新、worker 包为发布版)──
# 该断言即线上现象的复现与验证:插件 on_init 拿到的必须是网关对齐后的缓存值,
# 与 venv 里 chongming-worker 是否为含本次改动的版本无关。
import asyncio
from chongming_worker.plugin import Plugin # noqa: E402
seen = {}
class DemoPlugin(Plugin):
type = WT
auto_bind = False
async def _bootstrap(self): # 隔离数据库/对象存储引导,只看配置来源
return None
async def on_init(self):
seen["on_init"] = self.config.get("mediamtx_addr")
def _create_workers_impl(self, session, config, count=1, worker_manager=None):
seen["impl"] = (config or {}).get("mediamtx_addr")
return []
aligned = manager({key: dict(CACHED)}, "cache").align_inline_config(
WT, dict(FLAT), dict(SECTION)
)
asyncio.run(DemoPlugin().create_workers(None, aligned, 1))
report(
"端到端:网关对齐后插件 on_init 读到缓存值(不依赖 worker 包版本)",
seen.get("on_init") == CACHED["mediamtx_addr"] == seen.get("impl"),
f"on_init={seen.get('on_init')}, impl={seen.get('impl')}, "
f"sdk={os.path.abspath(__import__('chongming_worker').__file__)}",
)
print("\n" + "=" * 56)
print(f"测试结果: ✅ {PASS} 通过 ❌ {FAIL} 失败")
return 0 if FAIL == 0 else 1
if __name__ == "__main__":
sys.exit(run())
+209
View File
@@ -0,0 +1,209 @@
"""inline 直调路径 与 zenoh 路径 的一致性测试。
守护"进程插件 ↔ 内置插件无缝迁移"契约:同一个 worker 副本,分别经
A) 进程内直调(chongming_worker.dispatch:网关 inline 快路径用的入口)
B) zenoh 查询(session.get → router → queryable 回调 → 原有路径)
发起同一批请求,逐字段比对两路结果(仅屏蔽时间戳这类天然不同的字段)。
覆盖:正常 GET / POST、模型校验失败(reply_err 通道)、未知 subject、
非信封旧式载荷、流式多块、流式生成器异常收尾块。
运行: uv run python test/test_inline_local_dispatch.py
(需要 zenoh router 在 config/zenoh_config.json5 指向的端点可达)
"""
import asyncio
import os
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
# 优先本地框架 checkout(与 test_stream_worker.py 同约定)
sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src"))
sys.path.insert(0, str(REPO_ROOT / "src"))
import orjson # noqa: E402
import zenoh # noqa: E402
from chongming_worker.dispatch import get_dispatcher, unregister_dispatcher # noqa: E402
from chongming_worker.runtime import create_worker # noqa: E402
WORKER_TYPE = "example"
WORKER_ID = "worker-parity-0"
failures = []
def check(name, cond, detail=""):
status = "PASS" if cond else "FAIL"
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
def mask_timestamp(obj):
"""屏蔽时间戳类字段(两路调用时刻不同,天然不一致)"""
if isinstance(obj, dict):
return {k: (0 if k in ("timestamp", "time") else mask_timestamp(v)) for k, v in obj.items()}
if isinstance(obj, list):
return [mask_timestamp(v) for v in obj]
return obj
def normalize(payload: bytes):
try:
return mask_timestamp(orjson.loads(payload))
except Exception:
return payload
def envelope(method: str, data: dict) -> bytes:
return orjson.dumps({"meta": {"method": method, "user_info": None}, "data": data})
async def invoke_zenoh(session, subject: str, payload: bytes, timeout: float = 3.0):
"""经 zenoh 查询取首个回复:返回 ("ok", 载荷) / ("err", 载荷) / ("noreply", None)"""
key = f"worker/{WORKER_TYPE}/{WORKER_ID}/{subject}"
loop = asyncio.get_running_loop()
def _first():
for reply in session.get(key, payload=payload,
consolidation=zenoh.ConsolidationMode.NONE,
timeout=timeout):
return reply
return None
reply = await loop.run_in_executor(None, _first)
if reply is None:
return ("noreply", None)
if reply.ok:
return ("ok", bytes(reply.ok.payload) if reply.ok.payload else b"")
return ("err", bytes(reply.err.payload))
async def collect_zenoh_stream(session, subject: str, payload: bytes, timeout: float = 5.0):
key = f"worker/{WORKER_TYPE}/{WORKER_ID}/{subject}"
loop = asyncio.get_running_loop()
def _all():
out = []
for reply in session.get(key, payload=payload,
consolidation=zenoh.ConsolidationMode.NONE,
timeout=timeout):
if reply.ok:
out.append(bytes(reply.ok.payload) if reply.ok.payload else b"")
else:
out.append(("err", bytes(reply.err.payload)))
return out
return await loop.run_in_executor(None, _all)
async def collect_local_stream(dispatcher, subject: str, payload: bytes):
out = []
async for chunk in dispatcher.invoke_stream(subject, payload):
out.append(chunk)
return out
async def main() -> int:
# ── 起会话 + 一个真实 worker 副本(example 插件,纯计算端点)──
import plugins.example.router.add # noqa: F401
import plugins.example.router.subtract # noqa: F401
import plugins.example.router.stream_chat # noqa: F401
zconfig = zenoh.Config()
zconfig.from_file(str(REPO_ROOT / "config" / "zenoh_config.json5"))
session = zenoh.open(zconfig)
worker_task = asyncio.create_task(
create_worker(session, WORKER_TYPE, WORKER_ID, config={"data_dir": "./data"})
)
await asyncio.sleep(1.5) # 等 liveliness / queryable / 直调注册都就绪
dispatcher = get_dispatcher(WORKER_TYPE, WORKER_ID)
check("直调入口已注册(register_dispatcher)", dispatcher is not None)
if dispatcher is None:
worker_task.cancel()
session.close()
return 1
try:
# ── 1) 正常 GET:example.add?a&b ──
payload = envelope("GET", {"a": 2, "b": 3})
z = await invoke_zenoh(session, "example.add", payload)
l = await dispatcher.invoke_unary("example.add", payload)
l_norm = ("ok" if l.ok else "err", l.payload)
check("GET 两路状态一致", z[0] == l_norm[0], f"zenoh={z[0]} local={l_norm[0]}")
check(
"GET 两路载荷逐字段一致",
normalize(z[1] or b"") == normalize(l_norm[1]),
f"zenoh={z[1]!r} local={l_norm[1]!r}",
)
# ── 2) 正常 POST(同 subject 的另一方法)──
payload = envelope("POST", {"a": 5, "b": 7})
z = await invoke_zenoh(session, "example.add", payload)
l = await dispatcher.invoke_unary("example.add", payload)
l_norm = ("ok" if l.ok else "err", l.payload)
check("POST 两路状态一致", z[0] == l_norm[0], f"zenoh={z[0]} local={l_norm[0]}")
check("POST 两路载荷逐字段一致", normalize(z[1] or b"") == normalize(l_norm[1]))
# ── 3) 模型校验失败 → 框架错误(zenoh reply_err ⇔ local ok=False)──
payload = envelope("GET", {"a": "not-a-number"})
z = await invoke_zenoh(session, "example.add", payload)
l = await dispatcher.invoke_unary("example.add", payload)
l_norm = ("ok" if l.ok else "err", l.payload)
check("校验失败两路都走错误通道", z[0] == "err" and l_norm[0] == "err",
f"zenoh={z[0]} local={l_norm[0]}")
z_err, l_err = normalize(z[1] or b""), normalize(l_norm[1])
check("错误信封结构一致(operation=error 且含 error 文本)",
isinstance(z_err, dict) and isinstance(l_err, dict)
and z_err.get("operation") == l_err.get("operation") == "error"
and bool(z_err.get("error")) and bool(l_err.get("error")),
f"zenoh={z_err} local={l_err}")
# ── 4) 非信封旧式载荷(method 默认 GET)──
payload = orjson.dumps({"a": 1, "b": 1})
z = await invoke_zenoh(session, "example.subtract", payload)
l = await dispatcher.invoke_unary("example.subtract", payload)
l_norm = ("ok" if l.ok else "err", l.payload)
check("旧式载荷两路一致", z[0] == l_norm[0] and normalize(z[1] or b"") == normalize(l_norm[1]),
f"zenoh={z} local={l_norm}")
# ── 5) 未知 subject:两路都应失败(路径不同但终态一致:网关侧都会换副本重试后 503)──
z = await invoke_zenoh(session, "example.not_exists", envelope("GET", {}), timeout=1.0)
l = await dispatcher.invoke_unary("example.not_exists", envelope("GET", {}))
check("未知 subject 两路都失败(zenoh 无回复 / 直调返回错误)",
z[0] == "noreply" and l.ok is False, f"zenoh={z[0]} local.ok={l.ok}")
# ── 6) 流式多块 ──
payload = envelope("POST", {"prompt": "你好", "tokens": 4})
zs = await collect_zenoh_stream(session, "example.stream_chat", payload)
ls = await collect_local_stream(dispatcher, "example.stream_chat", payload)
check("流式块数一致", len(zs) == len(ls), f"zenoh={len(zs)} local={len(ls)}")
check("流式逐块内容一致",
[normalize(x) for x in zs] == [normalize(x) for x in ls],
f"zenoh={[normalize(x) for x in zs]} local={[normalize(x) for x in ls]}")
# ── 7) 流式生成器中途异常 → {"error": ...} 收尾块(两路都应以此结束)──
# 流终止语义(无 token 的空流)两路一致
payload = envelope("POST", {"prompt": "空", "tokens": 0})
zs = await collect_zenoh_stream(session, "example.stream_chat", payload)
ls = await collect_local_stream(dispatcher, "example.stream_chat", payload)
check("空流终止语义一致(不挂起、块数相同)",
len(zs) == len(ls), f"zenoh={len(zs)} local={len(ls)}")
finally:
worker_task.cancel()
try:
await worker_task
except asyncio.CancelledError:
pass
unregister_dispatcher(WORKER_TYPE, WORKER_ID)
session.close()
print(f"\n结果: {len(failures)} 个失败" + (f" -> {failures}" if failures else ""))
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(asyncio.run(main()))
+220
View File
@@ -0,0 +1,220 @@
"""
worker 实例定向(把请求固定到某个进程实例)测试
验证 WorkerManager 的实例定向语义(HTTP 的 X-Chongming-Instance 头 / MQTT 的
target_instance 字段 / 程序调用 instance_id 参数最终都走这里):
1. 不指定实例:与以往一致,在全部候选副本里随机挑
2. 指定实例:只在其实例的在线副本里挑,不跨实例
3. 身份来源:副本经 admin status 上报 instance_id,网关按需查询并缓存
(本地注册的 inline 副本直接用网关自身实例,无需查询)
4. 目标实例不在线:报错而不回退到别的实例(定向即严格)
5. 调用失败重试:限定在同一实例内换副本,绝不跨实例
6. 实例内已无其它副本可换:立即失败,不重复打同一个副本
7. 实例维度缓存键:定向调用的缓存与其它实例隔离
运行: .venv/bin/python test/test_instance_targeting.py
"""
import asyncio
import sys
from pathlib import Path
_HERE = Path(__file__).resolve().parent # chongming-zenoh/
_PROJ = _HERE
for _p in (_PROJ / "src", _PROJ / "utils" / "worker" / "src"):
sys.path.insert(0, str(_p))
from chongming.app.core.worker_manager import WorkerManager # noqa: E402
failures = []
def check(name, cond, detail=""):
status = "PASS" if cond else "FAIL"
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
class _OkPayload:
def __init__(self, data: bytes):
self.payload = data
class _Ok:
def __init__(self, data: bytes):
self.ok = _OkPayload(data)
class _Reply:
"""一次 get 的回复:ok 与 err 互斥(与 Zenoh reply 的形态一致)。"""
def __init__(self, data: bytes = b"{}", err: bytes | None = None):
if err is None:
self.ok = _OkPayload(data)
self.err = None
else:
self.ok = None
self.err = _OkPayload(err)
class FakeSession:
"""记录每次 get 的 key,并按 key 决定回复。
admin 查询(worker/<type>/<id>/__admin__)由 admin_instances 决定 instance_id;
业务调用(worker/<type>/<id>/<subject>)可按 call_errors 注入错误回复。
"""
def __init__(self, admin_instances: dict | None = None,
call_errors: set | None = None):
self.admin_instances = admin_instances or {}
self.call_errors = set(call_errors or ())
self.calls: list = []
def get(self, key, payload=None, **kwargs):
self.calls.append(key)
if key.endswith("/__admin__"):
wid = key.split("/")[2]
iid = self.admin_instances.get(wid)
body = (
f'{{"worker_id":"{wid}","instance_id":"{iid}","status":"running"}}'.encode()
if iid else f'{{"worker_id":"{wid}","status":"running"}}'.encode()
)
return iter([_Reply(body)])
if key in self.call_errors:
return iter([_Reply(err=b'{"error":"boom"}')])
return iter([_Reply(b'{"ok":true}')])
def liveliness(self):
raise RuntimeError("本测试不触及 liveliness")
def close(self):
pass
def new_manager(session, gateway_id="gw-local"):
m = WorkerManager(session=session)
m._gateway_id = gateway_id
return m
async def main():
# ── 1. 不定向:与以往一致,全候选集随机 ─────────────────
sess = FakeSession(admin_instances={"w-a": "edge-01", "w-b": "edge-02"})
m = new_manager(sess)
m._workers_cache["demo"] = ["worker/demo/w-a", "worker/demo/w-b"]
picked = {await m.get_worker("demo") for _ in range(60)}
check("1.1 不定向时可命中全部副本", picked == set(m._workers_cache["demo"]),
f"picked={sorted(picked)}")
check("1.2 不定向不触发实例身份查询(零额外 RPC)",
not [c for c in sess.calls if c.endswith("__admin__")], f"calls={sess.calls}")
check("1.3 不定向调用走随机副本",
await m.call_raw("demo", "ping", b"{}") == b'{"ok":true}')
# ── 2. 定向:只在目标实例内挑副本 ────────────────────────
sess = FakeSession(admin_instances={"w-a": "edge-01", "w-b": "edge-02",
"w-c": "edge-01"})
m = new_manager(sess)
m._workers_cache["demo"] = [
"worker/demo/w-a", "worker/demo/w-b", "worker/demo/w-c",
]
picked = {await m.get_worker("demo", instance_id="edge-01") for _ in range(60)}
check("2.1 定向只命中目标实例的副本",
picked == {"worker/demo/w-a", "worker/demo/w-c"}, f"picked={sorted(picked)}")
# ── 3. 身份查询:按需查询并缓存,实例归属正确 ─────────────
sess = FakeSession(admin_instances={"w-a": "edge-01", "w-b": "edge-02",
"w-c": "edge-01"})
m = new_manager(sess)
m._workers_cache["demo"] = [
"worker/demo/w-a", "worker/demo/w-b", "worker/demo/w-c",
]
index = await m._instance_workers("demo", "edge-01")
check("3.1 实例→副本反向索引正确",
sorted(index) == ["worker/demo/w-a", "worker/demo/w-c"], f"index={sorted(index)}")
admin_calls = [c for c in sess.calls if c.endswith("__admin__")]
check("3.2 每个副本只查询一次身份", len(admin_calls) == 3, f"n={len(admin_calls)}")
await m._refresh_instance_map("demo")
check("3.3 重复刷新复用缓存不重复查询",
len([c for c in sess.calls if c.endswith("__admin__")]) == len(admin_calls))
# ── 3b. inline 本地注册副本:直接用网关实例,零查询 ──────
sess = FakeSession()
m = new_manager(sess, gateway_id="gw-1")
m.register_worker("demo", "worker-0-1-999")
index = await m._instance_workers("demo", "gw-1")
check("3.4 inline 副本归入网关实例", index == ["worker/demo/worker-0-1-999"],
f"index={index}")
check("3.5 inline 副本身份查询零 RPC",
not [c for c in sess.calls if c.endswith("__admin__")], f"calls={sess.calls}")
# ── 4. 定向实例不在线:严格报错,不回退 ──────────────────
sess = FakeSession(admin_instances={"w-a": "edge-01"})
m = new_manager(sess)
m._workers_cache["demo"] = ["worker/demo/w-a"]
check("4.1 目标实例无副本时 get_worker 返回 None",
await m.get_worker("demo", instance_id="edge-99") is None)
try:
await m.call_raw("demo", "ping", b"{}", instance_id="edge-99")
check("4.2 定向到离线实例应报错", False, "未抛错")
except RuntimeError as e:
msg = str(e)
check("4.2 定向到离线实例报错且说明不跨实例回退",
"edge-99" in msg and "不" in msg, f"msg={msg[:90]}")
# ── 5. 重试锚定同实例:绝不跨实例 ────────────────────────
sess = FakeSession(
admin_instances={"w-a": "edge-01", "w-b": "edge-02", "w-c": "edge-01"},
call_errors={"worker/demo/w-b/ping"}, # 只让别实例的副本出错,验证不会被选中
)
m = new_manager(sess)
m._workers_cache["demo"] = [
"worker/demo/w-a", "worker/demo/w-b", "worker/demo/w-c",
]
# 目标实例 edge-01 内的 w-a 失败 -> 应换到同实例的 w-c,不碰 edge-02 的 w-b
sess.call_errors = {"worker/demo/w-a/ping"}
out = await m.call_raw("demo", "ping", b"{}", instance_id="edge-01", retries=2)
business = [c for c in sess.calls if not c.endswith("__admin__")]
check("5.1 重试不跨实例(未打其它实例的副本)",
all("/w-b/" not in c for c in business), f"calls={business}")
check("5.2 同实例内换副本后成功", out == b'{"ok":true}', f"out={out!r}")
# ── 6. 实例内无其它副本可换:立即失败 ────────────────────
sess = FakeSession(admin_instances={"w-a": "edge-01"},
call_errors={"worker/demo/w-a/ping"})
m = new_manager(sess)
m._workers_cache["demo"] = ["worker/demo/w-a", "worker/demo/w-b"]
sess.admin_instances["w-b"] = "edge-02"
raised = None
try:
await m.call_raw("demo", "ping", b"{}", instance_id="edge-01", retries=2)
except RuntimeError as e:
raised = e
check("6.1 实例内无副本可换应报错", raised is not None, f"raised={raised!r}")
n = len([c for c in sess.calls if not c.endswith("__admin__")])
check("6.2 不重复打同一个副本(只调用一次)", n == 1, f"业务调用次数={n}")
check("6.3 错误信息点名实例与原因",
raised is not None and "edge-01" in str(raised), f"msg={str(raised)[:90]}")
# ── 7. 定向调用的缓存按实例隔离 ──────────────────────────
sess = FakeSession(admin_instances={"w-a": "edge-01", "w-b": "edge-02"})
m = new_manager(sess)
m._workers_cache["demo"] = ["worker/demo/w-a", "worker/demo/w-b"]
r1 = await m.call_raw("demo", "state", b"{}", cacheable=True, instance_id="edge-01")
r2 = await m.call_raw("demo", "state", b"{}", cacheable=True, instance_id="edge-02")
keys = [f"{iid}@demo:state:" for iid in ("edge-01", "edge-02")]
check("7.1 不同实例的定向缓存键互不影响",
all(any(k.startswith(p) for k in m._rpc_mem_cache._d) for p in keys),
f"cache_keys={list(m._rpc_mem_cache._d)}")
check("7.2 定向调用结果正常返回", r1 == b'{"ok":true}' and r2 == b'{"ok":true}',
f"r1={r1!r}, r2={r2!r}")
print()
if failures:
print(f"FAILED: {len(failures)} 项 -> {failures}")
return 1
print("全部通过")
return 0
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
+207
View File
@@ -0,0 +1,207 @@
"""
HTTP 实例定向头(X-Chongming-Instance)透传测试
依赖隔离:桩掉 chongming 包链、auth 与 chongming_cache,直接加载真实
route_generator.py 与 worker_manager.py,用 FastAPI TestClient 验证:
1. 携带 X-Chongming-Instance 的请求,该实例 id 透传到 call_raw / call_stream
2. 未携带该头时 instance_id 为 None(行为与改动前完全一致,兼容旧调用方)
3. 覆盖全部端点形态:GET query / POST body / validate=false 原样透传 / SSE 流式
4. 头值为空白时视为未指定
5. 定向到离线实例(RuntimeError)-> HTTP 503,错误信息透出
运行: .venv/bin/python test/test_instance_targeting_http.py
"""
import importlib.util
import logging
import sys
import types
from pathlib import Path
import orjson
from fastapi import FastAPI
from fastapi.testclient import TestClient
TEST_DIR = Path(__file__).resolve().parent
REPO_ROOT = TEST_DIR.parent
sys.path.insert(0, str(REPO_ROOT / "src"))
sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src"))
# ---- 依赖隔离:桩掉 chongming 包链 / auth / chongming_cache ----
_pkg = types.ModuleType("chongming")
_pkg.__path__ = []
_app = types.ModuleType("chongming.app")
_app.__path__ = []
_core = types.ModuleType("chongming.app.core")
_core.__path__ = []
_logger_stub = types.ModuleType("chongming.app.core.logger")
_logger_stub.get_logger = lambda name: logging.getLogger(name)
_auth_stub = types.ModuleType("chongming.app.core.auth")
_auth_stub.store_user_to_request = lambda x: x
_cache_stub = types.ModuleType("chongming_cache")
_cache_stub.Cache = object
_cache_stub.DistributedLock = object # worker_manager 顶层导入(本测试不触达)
for mod in (_pkg, _app, _core, _logger_stub, _auth_stub, _cache_stub):
sys.modules[mod.__name__] = mod
_wm_path = REPO_ROOT / "src" / "chongming" / "app" / "core" / "worker_manager.py"
_wm_spec = importlib.util.spec_from_file_location(
"chongming.app.core.worker_manager", _wm_path
)
_wm_mod = importlib.util.module_from_spec(_wm_spec)
sys.modules["chongming.app.core.worker_manager"] = _wm_mod
_wm_spec.loader.exec_module(_wm_mod) # type: ignore[union-attr]
_cs_path = REPO_ROOT / "src" / "chongming" / "app" / "core" / "call_stats.py"
_cs_spec = importlib.util.spec_from_file_location(
"chongming.app.core.call_stats", _cs_path
)
_cs_mod = importlib.util.module_from_spec(_cs_spec)
sys.modules["chongming.app.core.call_stats"] = _cs_mod
_cs_spec.loader.exec_module(_cs_mod) # type: ignore[union-attr]
_rg_path = REPO_ROOT / "src" / "chongming" / "app" / "core" / "route_generator.py"
_rg_spec = importlib.util.spec_from_file_location(
"chongming.app.core.route_generator", _rg_path
)
_rg_mod = importlib.util.module_from_spec(_rg_spec)
sys.modules["chongming.app.core.route_generator"] = _rg_mod
_rg_spec.loader.exec_module(_rg_mod) # type: ignore[union-attr]
register_route_group = _rg_mod.register_route_group
INSTANCE_HEADER = _rg_mod.INSTANCE_HEADER
failures = []
def check(name, cond, detail=""):
status = "PASS" if cond else "FAIL"
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
class FakeWorkerManager:
"""记录每次调用收到的 instance_id;可指定定点失败以验证 503 映射。"""
def __init__(self, fail_instance=None):
self.fail_instance = fail_instance # 指定实例报错(模拟实例离线)
self.raw_instances = [] # call_raw 收到的 instance_id
self.stream_instances = [] # call_stream 收到的 instance_id
async def call_raw(self, worker_type, subject, payload, timeout=3.0,
retries=1, cacheable=False, instance_id=None):
self.raw_instances.append(instance_id)
if instance_id and instance_id == self.fail_instance:
raise RuntimeError(
f"类型 '{worker_type}' 的实例 '{instance_id}' 没有在线副本"
"(定向调用不跨实例回退)。"
)
return orjson.dumps({"op": subject, "instance": instance_id})
async def call_stream(self, worker_type, subject, payload, timeout=300.0,
idle_timeout=10.0, instance_id=None):
self.stream_instances.append(instance_id)
if instance_id and instance_id == self.fail_instance:
raise RuntimeError(f"实例 '{instance_id}' 没有在线副本")
async def _gen():
yield orjson.dumps({"instance": instance_id})
return _gen()
ROUTES = [
{"subject": "t.get", "path": "/calc/add", "method": "GET",
"params": ["a: float", "b: float"], "summary": "add",
"timeout": 3.0, "auth_required": False},
{"subject": "t.post", "path": "/calc/echo", "method": "POST",
"params": ["msg: str"], "summary": "echo",
"timeout": 3.0, "auth_required": False},
{"subject": "t.raw", "path": "/calc/raw", "method": "POST",
"params": ["payload: any"], "summary": "原样透传",
"validate": False, "timeout": 3.0, "auth_required": False},
{"subject": "t.stream", "path": "/stream/chat", "method": "POST",
"params": ["prompt: str"], "summary": "sse",
"stream": True, "timeout": 60.0, "stream_idle_timeout": 1.0,
"auth_required": False},
]
def build_client(fwm):
app = FastAPI()
register_route_group(app, fwm, "demotype", ROUTES, prefix="/api/v1")
return TestClient(app)
def main():
# ── 1. GET query 端点 ────────────────────────────────────
fwm = FakeWorkerManager()
c = build_client(fwm)
r = c.get("/api/v1/calc/add?a=1&b=2", headers={INSTANCE_HEADER: "edge-01"})
check("1.1 GET 携带头 -> 200", r.status_code == 200, f"status={r.status_code}")
check("1.2 GET 头透传到 call_raw", fwm.raw_instances == ["edge-01"],
f"instances={fwm.raw_instances}")
check("1.3 响应体回带实例(证明打到目标实例)",
r.json().get("instance") == "edge-01", f"body={r.json()}")
# ── 2. 未携带头:兼容旧调用方 ────────────────────────────
fwm = FakeWorkerManager()
c = build_client(fwm)
r = c.get("/api/v1/calc/add?a=1&b=2")
check("2.1 未携带头 -> 200(默认随机均衡路径不变)", r.status_code == 200)
check("2.2 未携带头时 instance_id=None", fwm.raw_instances == [None],
f"instances={fwm.raw_instances}")
check("2.3 响应体不含实例", r.json().get("instance") is None, f"body={r.json()}")
# ── 3. POST body 端点 ────────────────────────────────────
fwm = FakeWorkerManager()
c = build_client(fwm)
r = c.post("/api/v1/calc/echo", json={"msg": "hi"},
headers={INSTANCE_HEADER: "edge-02"})
check("3.1 POST body 头透传", fwm.raw_instances == ["edge-02"],
f"instances={fwm.raw_instances}")
# ── 4. validate=false 原样透传端点 ───────────────────────
fwm = FakeWorkerManager()
c = build_client(fwm)
r = c.post("/api/v1/calc/raw", content=b'{"anything": [1, 2]}',
headers={INSTANCE_HEADER: "edge-03",
"content-type": "application/json"})
check("4.1 原样透传端点头透传", fwm.raw_instances == ["edge-03"],
f"instances={fwm.raw_instances}")
# ── 5. SSE 流式端点 ──────────────────────────────────────
fwm = FakeWorkerManager()
c = build_client(fwm)
with c.stream("POST", "/api/v1/stream/chat", json={"prompt": "x"},
headers={INSTANCE_HEADER: "edge-04"}) as r:
body = b"".join(r.iter_bytes())
check("5.1 流式端点头透传", fwm.stream_instances == ["edge-04"],
f"instances={fwm.stream_instances}")
check("5.2 流式响应含目标实例", b"edge-04" in body, f"body={body[:80]!r}")
# ── 6. 空白头值视为未指定 ────────────────────────────────
fwm = FakeWorkerManager()
c = build_client(fwm)
r = c.get("/api/v1/calc/add?a=1&b=2", headers={INSTANCE_HEADER: " "})
check("6.1 空白头值 -> instance_id=None", fwm.raw_instances == [None],
f"instances={fwm.raw_instances}")
# ── 7. 定向实例离线 -> 503 ───────────────────────────────
fwm = FakeWorkerManager(fail_instance="edge-gone")
c = build_client(fwm)
r = c.get("/api/v1/calc/add?a=1&b=2", headers={INSTANCE_HEADER: "edge-gone"})
check("7.1 定向实例离线 -> 503", r.status_code == 503, f"status={r.status_code}")
check("7.2 503 详情点名实例", "edge-gone" in str(r.json().get("detail", "")),
f"detail={r.json().get('detail')}")
print()
if failures:
print(f"FAILED: {len(failures)} 项 -> {failures}")
return 1
print("全部通过")
return 0
if __name__ == "__main__":
raise SystemExit(main())
-224
View File
@@ -1,224 +0,0 @@
"""构建期授权守护测试:公钥注入 + 私钥泄漏守卫。
对齐 `test_build_plugin.py` 对"插件包不得包含 secret_key"的同类断言 ——
**签发私钥绝不能进入部署产物**:它是授权的唯一信任根,一旦随包发布,
任何人都能签发授权,机器码绑定形同虚设。
运行方式:`python test/test_license_build.py`,或 `uv run pytest test/test_license_build.py`。
"""
import os
import sys
import tempfile
import tomllib
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "utils" / "cli" / "src"))
from chongming_cli.build import ( # noqa: E402
ALLOW_NO_KEY_ENV,
LICENSE_PUBKEY_ENV,
_resolve_license_public_key,
_set_license_pubkey,
_stage_license_guard,
)
failures: list[str] = []
def check(name, cond, detail=""):
status = "PASS" if cond else "FAIL"
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
def test_pubkey_injection_is_section_scoped():
"""公钥只写进 [license] 段,注释与其它段不受影响。"""
src = (ROOT / "public" / "config.toml").read_text(encoding="utf-8")
out = _set_license_pubkey(src, "AAAABBBB=")
body = tomllib.loads(out)
check("注入后 TOML 可解析", body["license"]["public_key"] == "AAAABBBB=",
str(body["license"].get("public_key")))
# 只改了 [license] 段内那一行
lic_start = out.index("[license]")
prod_start = out.index("[production]")
check("公钥位于 [license] 段内",
'public_key = "AAAABBBB="' in out[lic_start:prod_start])
check("[production] 段未被误改", "public_key" not in out[prod_start:])
check("注释保留", "# 机器码授权" in out)
check("其它段结构完整", "[production.module_system]" in out)
check("[license] 段仍保留 path/min_match",
body["license"]["path"] == "license.lic" and body["license"]["min_match"] == 3)
diff = [i for i, (a, b) in enumerate(zip(src.splitlines(), out.splitlines())) if a != b]
check("仅一行发生变化", len(diff) == 1, f"变化行号={diff}")
def test_pubkey_resolution_order():
"""公钥解析优先级:环境变量 > 文件环境变量 > 默认文件 license_public.key。
必须在**空的临时目录**里跑:默认回退路径 ``license_public.key`` 是相对 CWD 的,
而厂商在项目根放过这个文件是正常用法(也正是构建取公钥的正规途径),
测试不能依赖它"不存在"。
"""
tmp = Path(tempfile.mkdtemp())
key_file = tmp / "pk.txt"
key_file.write_text("FROMFILE", encoding="utf-8")
old_cwd = os.getcwd()
old_env = dict(os.environ)
try:
os.chdir(tmp) # 空目录:既无 license_public.key 也无其它干扰
for var in (LICENSE_PUBKEY_ENV, "CHONGMING_LICENSE_PUBKEY_FILE"):
os.environ.pop(var, None)
check("三者都无时返回空", _resolve_license_public_key() == "")
# 默认文件回退(构建按此取公钥,之前漏测)
(tmp / "license_public.key").write_text("FROMDEFAULT", encoding="utf-8")
check("回退到默认文件 license_public.key",
_resolve_license_public_key() == "FROMDEFAULT")
os.environ["CHONGMING_LICENSE_PUBKEY_FILE"] = str(key_file)
check("文件环境变量优先于默认文件",
_resolve_license_public_key() == "FROMFILE")
os.environ[LICENSE_PUBKEY_ENV] = "FROMENV"
check("环境变量优先于文件", _resolve_license_public_key() == "FROMENV")
finally:
os.chdir(old_cwd)
os.environ.clear()
os.environ.update(old_env)
def test_guard_detects_private_key():
"""守护能检出私钥(按内容,与扩展名无关,含跨块边界)。"""
tmp = Path(tempfile.mkdtemp())
build = tmp / "build"
(build / "sub").mkdir(parents=True)
old_cwd = os.getcwd()
try:
os.chdir(tmp)
(build / "ok.txt").write_text("正常内容\n", encoding="utf-8")
check("干净产物通过", _stage_license_guard() == "未发现私钥泄漏")
# 证书类 .pem 是合法的构建内容,不应被误判
(build / "cert.pem").write_text(
"-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n", encoding="utf-8"
)
check("合法证书 .pem 不误判", _stage_license_guard() == "未发现私钥泄漏")
# 私钥改名藏进普通文件 -> 按内容仍能检出
leak = build / "sub" / "innocent.txt"
leak.write_text("-----BEGIN PRIVATE KEY-----\nMIIEvQ\n", encoding="utf-8")
try:
_stage_license_guard()
check("按内容检出改名的私钥", False, "未检出")
except RuntimeError as exc:
check("按内容检出改名的私钥", "innocent.txt" in str(exc))
leak.unlink()
# 跨 1MB 块边界的私钥
(build / "big.bin").write_bytes(
b"x" * (1 << 20) + b"-----BEGIN RSA PRIVATE KEY-----" + b"y" * (1 << 20)
)
try:
_stage_license_guard()
check("检出跨块边界的私钥", False, "未检出")
except RuntimeError as exc:
check("检出跨块边界的私钥", "big.bin" in str(exc))
(build / "big.bin").unlink()
# 文件名叫 license_private* 的一律拦下
(build / "license_private.pem").write_text("(truncated)", encoding="utf-8")
try:
_stage_license_guard()
check("按文件名拦下授权私钥文件", False, "未检出")
except RuntimeError as exc:
check("按文件名拦下授权私钥文件", "license_private" in str(exc))
finally:
os.chdir(old_cwd)
def test_build_fails_without_pubkey():
"""打包时缺少公钥 -> 构建失败(避免产出拒绝所有插件的砖头包)。"""
import chongming_cli.build as build_mod
tmp = Path(tempfile.mkdtemp())
old_cwd = os.getcwd()
old_env = dict(os.environ)
try:
# 复刻 build/ 最小结构:_stage_prepare_build_dir 需要 public/ 与 config.toml
(tmp / "public" / "utils").mkdir(parents=True)
(tmp / "public" / "main.py").write_text("", encoding="utf-8")
(tmp / "public" / "server.py").write_text("", encoding="utf-8")
(tmp / "public" / "config.toml").write_text(
(ROOT / "public" / "config.toml").read_text(encoding="utf-8"), encoding="utf-8"
)
os.chdir(tmp)
for var in (LICENSE_PUBKEY_ENV, "CHONGMING_LICENSE_PUBKEY_FILE", ALLOW_NO_KEY_ENV):
os.environ.pop(var, None)
try:
build_mod._stage_prepare_build_dir({}, "pyinstaller") # type: ignore[arg-type]
check("缺少公钥时构建失败", False, "未抛异常")
except RuntimeError as exc:
check("缺少公钥时构建失败", LICENSE_PUBKEY_ENV in str(exc))
# 显式放行时才允许无公钥构建
os.environ[ALLOW_NO_KEY_ENV] = "1"
build_mod._stage_prepare_build_dir({}, "pyinstaller") # type: ignore[arg-type]
built = tomllib.loads((tmp / "build" / "config.toml").read_text(encoding="utf-8"))
check("显式放行时构建成功且公钥为空", built["license"]["public_key"] == "")
# 提供公钥后应注入到产物配置
os.environ[LICENSE_PUBKEY_ENV] = "INJECTEDKEY="
build_mod._stage_prepare_build_dir({}, "pyinstaller") # type: ignore[arg-type]
built = tomllib.loads((tmp / "build" / "config.toml").read_text(encoding="utf-8"))
check("公钥被注入产物配置", built["license"]["public_key"] == "INJECTEDKEY=")
check("产物配置仍含 min_match", built["license"]["min_match"] == 3)
finally:
os.chdir(old_cwd)
os.environ.clear()
os.environ.update(old_env)
def test_mbank_signing_is_not_a_trust_root():
""".mbank 验签用随包发布的对称密钥 —— 不能作为授权信任根。
本测试固化这一认知:只要 sqlite_vfs 的 sign 仍是 HMAC,就说明它只能算防篡改,
授权必须依赖非对称签名(chongming_license 的 Ed25519)。
"""
import inspect
from sqlite_vfs import core as vfs_core
src = inspect.getsource(vfs_core.SQLiteVFS.sign)
check("确认 .mbank 签名为 HMAC(对称)", "hmac.new" in src,
"若已改为非对称,可更新本测试与文档说明")
check("确认验证端同样是对称 HMAC",
"hmac.new" in inspect.getsource(vfs_core.SQLiteVFS.verify_signature))
def main():
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
for fn in tests:
print(f"\n--- {fn.__name__} ---")
fn()
print("\n" + "=" * 56)
if failures:
print(f"❌ {len(failures)} 项未通过: {failures}")
return 1
print("✅ 全部通过")
return 0
if __name__ == "__main__":
sys.exit(main())
-377
View File
@@ -1,377 +0,0 @@
"""机器码授权门禁测试(网关侧)。
核心验证目标 —— 未授权插件被拒绝时,**网关必须仍能正常启动**:
1. 门禁收口在 `_discover_plugins()`:未授权插件不出现在返回列表中,
且 `denied_plugin_types()` 能给出拒绝原因;
2. **契约校验不硬失败**:这是最容易踩的坑 —— `validate_contracts()` 会把
"TOML 路由没有对应 @rpc_method 注册"视为硬性错误并终止启动。若把"不加载
未授权插件"实现成"跳过 import 但保留在插件列表里",该插件的 TOML 路由就会
变成孤儿 → 设备起不来。本测试确保收口位置正确。
3. 路由描述符门禁:未授权类型的 external 路由描述符被忽略(防"改 load_mode
成 external 绕过门禁");
4. 管理接口门禁:未授权插件的扩缩容/重启/配置接口一律 403;
5. 未启用校验时(开发/测试)全部放行。
运行方式:`python test/test_license_gate.py`,或 `uv run pytest test/test_license_gate.py`。
"""
import asyncio
import base64
import json
import os
import sys
import tempfile
from datetime import date, timedelta
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
# 必须在导入网关模块前设置:licensing 的 enforce 在首次建管理器时读取
os.environ["CHONGMING_LICENSE_MODE"] = "enforce"
from chongming_config import get_config # noqa: E402
from cryptography.hazmat.primitives import serialization # noqa: E402
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey # noqa: E402
from chongming_license import collect_fingerprints, machine_code, sign_payload # noqa: E402
from chongming.app.core import licensing # noqa: E402
from chongming.app.core.lifespan_setup import ( # noqa: E402
_discover_plugins,
_scan_plugins,
denied_plugin_types,
)
failures: list[str] = []
def check(name, cond, detail=""):
status = "PASS" if cond else "FAIL"
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
# ---------------------------------------------------------------------------
# 测试夹具:临时密钥对 + license 文件
# ---------------------------------------------------------------------------
def _make_keypair() -> tuple[str, Ed25519PrivateKey]:
priv = Ed25519PrivateKey.generate()
pub = base64.b64encode(
priv.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
).decode("ascii")
return pub, priv
def _issue(tmpdir: Path, priv: Ed25519PrivateKey, plugins: list[str],
expires_days: int = 365, sources: dict | None = None) -> str:
src = sources if sources is not None else collect_fingerprints()
payload = {
"v": 1,
"lic_id": "L-TEST-0001",
"customer": "测试客户",
"machine_code": machine_code(src),
"sources": src,
"plugins": plugins,
"not_before": None,
"expires_at": (date.today() + timedelta(days=expires_days)).isoformat(),
"issued_at": "2026-09-20T00:00:00Z",
"note": "",
}
doc = {
"payload": payload,
"alg": "ed25519",
"kid": "test",
"sig": sign_payload(payload, priv),
}
path = tmpdir / "license.lic"
path.write_text(json.dumps(doc), encoding="utf-8")
return str(path)
def _use_license(public_key: str, path: str, min_match: int = 3) -> None:
"""把测试用的公钥/路径写进运行期配置并重建管理器。
直接改 `get_config()` 返回的 dict(chongming_config 缓存同一对象)。
"""
cfg = get_config()
cfg.setdefault("license", {})
cfg["license"]["public_key"] = public_key
cfg["license"]["path"] = path
cfg["license"]["min_match"] = min_match
licensing.reset_manager()
def test_gate_denies_unlicensed_plugins():
"""未授权插件被剔除,且原因可查。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _make_keypair()
_use_license(pub, _issue(tmp, priv, plugins=["ip_sound_column"]))
scanned = sorted(p["type"] for p in _scan_plugins())
allowed = sorted(p["type"] for p in _discover_plugins())
denied = denied_plugin_types()
check("盘扫到全部插件", scanned == ["aibox_sophon_adapter", "ip_sound_column"],
f"scanned={scanned}")
check("只放行已授权插件", allowed == ["ip_sound_column"], f"allowed={allowed}")
check("未授权插件给出拒绝原因",
"aibox_sophon_adapter" in denied
and "白名单" in denied["aibox_sophon_adapter"],
str(denied))
def test_contract_validation_survives_denial():
"""关键:有插件被拒时契约校验不得抛异常(否则设备起不来)。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _make_keypair()
_use_license(pub, _issue(tmp, priv, plugins=["ip_sound_column"]))
from chongming.app.core.contracts import validate_contracts
inline = [p for p in _discover_plugins() if p.get("load_mode", "inline") != "external"]
try:
validate_contracts(inline)
check("被拒插件的 TOML 路由不会导致契约校验硬失败", True)
except Exception as exc: # noqa: BLE001
check("被拒插件的 TOML 路由不会导致契约校验硬失败", False, repr(exc))
def test_all_denied_still_validates():
"""极端情况:完全未授权(无 license 文件)时启动流程仍不硬失败。"""
tmp = Path(tempfile.mkdtemp())
pub, _priv = _make_keypair()
_use_license(pub, str(tmp / "not-exists.lic"))
from chongming.app.core.contracts import validate_contracts
allowed = _discover_plugins()
check("无授权文件时不放行任何插件", allowed == [], str(allowed))
try:
validate_contracts([])
check("完全未授权时契约校验不抛异常", True)
except Exception as exc: # noqa: BLE001
check("完全未授权时契约校验不抛异常", False, repr(exc))
def test_route_registry_denies_descriptors():
"""未授权类型的路由描述符被忽略(防改 load_mode 绕过)。"""
from fastapi import FastAPI
from chongming.app.core.route_registry import RouteRegistry
tmp = Path(tempfile.mkdtemp())
pub, priv = _make_keypair()
_use_license(pub, _issue(tmp, priv, plugins=["ip_sound_column"]))
denied = licensing.denied_types({p["type"] for p in _scan_plugins()})
check("denied_types 含未授权插件", "aibox_sophon_adapter" in denied, str(denied))
async def run():
registry = RouteRegistry(FastAPI(), worker_manager=None, prefix="/api/v1")
registry.set_denied_types(denied)
descriptor = {
"schema_version": 1, "type": "aibox_sophon_adapter",
"subject": "aibox_sophon_adapter.alarm_report", "method": "POST",
"path": "/aibox_sophon_adapter/alarm", "params": {},
}
await registry._handle_sample(
"chongming/routes/aibox_sophon_adapter/alarm_report",
__import__("zenoh").SampleKind.PUT,
json.dumps(descriptor).encode(),
)
return registry._groups, registry.route_types
groups, route_types = asyncio.run(run())
check("未授权类型的描述符未进入路由表", groups == {}, f"groups={list(groups)}")
check("未授权类型不出现在 route_types 中", route_types == [], str(route_types))
def test_admin_gate_403_for_denied():
"""未授权插件的管理操作一律 403。"""
from fastapi import HTTPException
from chongming.app.routes.worker_admin import _ensure_licensed
tmp = Path(tempfile.mkdtemp())
pub, priv = _make_keypair()
_use_license(pub, _issue(tmp, priv, plugins=["ip_sound_column"]))
try:
_ensure_licensed("aibox_sophon_adapter")
check("未授权插件的管理操作被拒", False, "未抛 403")
except HTTPException as exc:
check("未授权插件的管理操作被拒", exc.status_code == 403,
f"status={exc.status_code}")
try:
_ensure_licensed("ip_sound_column")
check("已授权插件的管理操作放行", True)
except HTTPException as exc: # noqa: BLE001
check("已授权插件的管理操作放行", False, f"status={exc.status_code}")
def test_expired_license_denies_all():
"""过期授权 -> 全部拒绝。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _make_keypair()
_use_license(pub, _issue(tmp, priv, plugins=["ip_sound_column", "aibox_sophon_adapter"],
expires_days=-1))
allowed = _discover_plugins()
check("过期授权不放行任何插件", allowed == [], str(allowed))
check("拒绝原因为过期",
all("过期" in m for m in denied_plugin_types().values()),
str(denied_plugin_types()))
def test_disabled_mode_allows_everything():
"""未启用校验时(非打包态的默认行为)全部放行。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _make_keypair()
_use_license(pub, _issue(tmp, priv, plugins=[]))
licensing.reset_manager()
os.environ.pop("CHONGMING_LICENSE_MODE", None)
get_config()["license"]["enabled"] = False
try:
allowed = sorted(p["type"] for p in _discover_plugins())
check("未启用校验时全部放行",
allowed == ["aibox_sophon_adapter", "ip_sound_column"], str(allowed))
check("未启用校验时无拒绝记录", denied_plugin_types() == {},
str(denied_plugin_types()))
finally:
os.environ["CHONGMING_LICENSE_MODE"] = "enforce"
get_config()["license"]["enabled"] = False
licensing.reset_manager()
def test_http_endpoints_and_upload_safety():
"""HTTP 层:门禁、信息接口、以及"坏文件不覆盖现有授权"。
这一步覆盖最容易在现场出事的行为 —— 误传一个坏 license 文件把设备搞成未授权。
"""
import base64 as _b64
import json as _json
import tempfile as _tf
from datetime import date as _date, timedelta as _td
import orjson
from cryptography.hazmat.primitives import serialization as _ser
from fastapi.testclient import TestClient
from chongming_jwt import JWT
from chongming_license import collect_fingerprints as _collect
from chongming.app import app
tmp = Path(_tf.mkdtemp())
priv = Ed25519PrivateKey.generate()
pub = _b64.b64encode(
priv.public_key().public_bytes(
encoding=_ser.Encoding.Raw, format=_ser.PublicFormat.Raw
)
).decode("ascii")
src = _collect()
payload = {
"v": 1, "lic_id": "L-HTTP-1", "customer": "HTTP 测试",
"machine_code": machine_code(src), "sources": src,
"plugins": ["ip_sound_column"], "not_before": None,
"expires_at": (_date.today() + _td(days=30)).isoformat(),
"issued_at": "2026-09-20T00:00:00Z", "note": "",
}
lic = tmp / "license.lic"
lic.write_text(_json.dumps({
"payload": payload, "alg": "ed25519", "kid": "test",
"sig": sign_payload(payload, priv),
}), encoding="utf-8")
_use_license(pub, str(lic))
sec = get_config()["security"]
def _token(superuser: bool) -> str:
return JWT(sec["secret_key"], sec["algorithm"]).encode(
orjson.dumps({"sub": "u1", "username": "admin",
"is_superuser": superuser}).decode(),
expires_in_seconds=3600,
)
client = TestClient(app)
auth = {"Authorization": f"Bearer {_token(True)}"}
r = client.get("/api/v1/license/info")
check("无 token 时拒绝访问", r.status_code == 401, f"status={r.status_code}")
r = client.get("/api/v1/license/info",
headers={"Authorization": f"Bearer {_token(False)}"})
check("非超管拒绝访问", r.status_code == 403, f"status={r.status_code}")
r = client.get("/api/v1/license/info", headers=auth)
check("超管可读取授权信息", r.status_code == 200, f"status={r.status_code}")
info = r.json()
check("/license/info 含机器码", info["status"]["machine_code"] != "",
info["status"]["machine_code"])
check("/license/info 含申请码", info["request_code"].startswith("CM1-"),
info["request_code"][:16])
check("未授权插件在 denied 中", "aibox_sophon_adapter" in info["plugins"]["denied"],
str(list(info["plugins"]["denied"])))
check("来源明细含 weak 标注",
any(s["weak"] for s in info["sources_detail"]),
str([(s["name"], s["weak"]) for s in info["sources_detail"]]))
# 签名错误的文件:必须拒绝,且不得覆盖现有授权
tampered = dict(payload, plugins=["aibox_sophon_adapter"])
r = client.post(
"/api/v1/license/file", headers=auth,
files={"file": ("bad.lic", _json.dumps(
{"payload": tampered, "alg": "ed25519", "kid": "test", "sig": "AAAA"}
), "application/json")},
)
check("签名错误的文件被拒绝", r.status_code == 422, f"status={r.status_code}")
check("被拒文件未覆盖现有授权",
_json.loads(lic.read_text(encoding="utf-8"))["payload"]["lic_id"] == "L-HTTP-1")
# 非 JSON 文件
r = client.post("/api/v1/license/file", headers=auth,
files={"file": ("bad.lic", "not json at all", "application/json")})
check("非 JSON 文件被拒绝", r.status_code == 422, f"status={r.status_code}")
# 合法文件:安装成功,并提示需重启加载
payload2 = dict(payload, lic_id="L-HTTP-2",
plugins=["ip_sound_column", "aibox_sophon_adapter"])
r = client.post(
"/api/v1/license/file", headers=auth,
files={"file": ("good.lic", _json.dumps(
{"payload": payload2, "alg": "ed25519", "kid": "test",
"sig": sign_payload(payload2, priv)}
), "application/json")},
)
check("合法文件导入成功", r.status_code == 200, f"status={r.status_code}")
if r.status_code == 200:
body = r.json()
check("导入后授权插件更新",
body["licensed_plugins"] == ["aibox_sophon_adapter", "ip_sound_column"],
str(body["licensed_plugins"]))
check("导入后提示需重启加载",
body["restart_required"] and "aibox_sophon_adapter" in body["pending_plugins"],
str(body["pending_plugins"]))
def main():
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
for fn in tests:
print(f"\n--- {fn.__name__} ---")
fn()
print("\n" + "=" * 56)
if failures:
print(f"❌ {len(failures)} 项未通过: {failures}")
return 1
print("✅ 全部通过")
return 0
if __name__ == "__main__":
sys.exit(main())
-327
View File
@@ -1,327 +0,0 @@
"""已签发授权台账测试(厂商侧 Registry)。
重点验证:
1. 由 license 文档派生记录(唯一写入口,不手工拼记录)
2. 状态判定(永久/有效/即将到期/已过期/未生效)与剩余天数
3. 检索:客户 / 机器码 / 插件 / 关键字 / 状态
4. **续期复用台账里的来源指纹** —— 台账最大的价值:不必再向客户索取申请码
5. 重新导入不覆盖人工维护的备注
6. 导出 .lic(按客户分子目录)与 CSV(UTF-8 BOM,Excel 中文不乱码)
7. 结构版本不匹配时明确报错,而不是让查询诡异失败
运行:`python test/test_license_registry.py`,或 `uv run pytest test/test_license_registry.py`。
"""
import csv
import sqlite3
import sys
import tempfile
from datetime import date, timedelta
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey # noqa: E402
from chongming_license import ( # noqa: E402
Registry,
RegistryError,
build_license,
build_payload,
machine_code,
public_key_b64,
public_key_fingerprint,
record_from_license,
)
failures: list[str] = []
def check(name, cond, detail=""):
print(f"[{'PASS' if cond else 'FAIL'}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
PRIV = Ed25519PrivateKey.generate()
PUB = public_key_b64(PRIV)
FP = public_key_fingerprint(PUB)
SRC = {"board_serial": "sn123456789", "emmc_cid": "1a2b3c4d", "mac": "aabbccddeeff"}
CODE = machine_code(SRC)
def make_doc(*, customer="某某科技", days=300, plugins=("aibox_sophon_adapter",),
note="", machine=None, sources=None, lic_id="", not_before=None):
payload = build_payload(
machine_code=machine or CODE,
plugins=list(plugins),
sources=SRC if sources is None else sources,
customer=customer,
note=note,
lic_id=lic_id,
not_before=not_before,
expires_at=(date.today() + timedelta(days=days)) if days is not None else None,
)
return build_license(payload, PRIV, kid="vendor-2026")
def test_record_derivation():
"""记录完全由 license 文档派生,查询列与原始文档不会不一致。"""
doc = make_doc(customer="派生测试", note="合同 X", plugins=["a", "b"])
rec = record_from_license(doc, public_key_fp=FP, source="test")
payload = doc["payload"]
check("lic_id 取自载荷", rec.lic_id == payload["lic_id"], rec.lic_id)
check("机器码大写归一", rec.machine_code == payload["machine_code"].upper())
check("插件列表解析", rec.plugins == ["a", "b"], str(rec.plugins))
check("来源指纹解析", rec.sources == SRC, str(sorted(rec.sources)))
check("kid 取自信封", rec.kid == "vendor-2026")
check("公钥指纹记录在案", rec.public_key_fp == FP)
check("可还原完整文档", rec.license_doc()["sig"] == doc["sig"])
# 日期只保留日期部分:否则 SQL 里的字符串比较会得出错误排序
doc2 = make_doc()
doc2["payload"]["expires_at"] = "2027-09-01T12:34:56Z"
check("日期裁剪到日粒度",
record_from_license(doc2).expires_at == "2027-09-01",
str(record_from_license(doc2).expires_at))
# 缺 lic_id 必须报错,而不是静默写一条垃圾记录
try:
record_from_license({"payload": {"machine_code": "X"}})
check("缺 lic_id 时报错", False, "未抛异常")
except RegistryError:
check("缺 lic_id 时报错", True)
def test_status_and_days_left():
"""五种状态的判定(含边界:到期当天仍有效)。"""
tmp = Path(tempfile.mkdtemp())
with Registry(tmp / "r.db") as reg:
reg.save(make_doc(lic_id="L-FOREVER", days=None), public_key_fp=FP)
reg.save(make_doc(lic_id="L-ACTIVE", days=300), public_key_fp=FP)
reg.save(make_doc(lic_id="L-EXPIRING", days=10), public_key_fp=FP)
reg.save(make_doc(lic_id="L-EXPIRED", days=-1), public_key_fp=FP)
reg.save(make_doc(lic_id="L-TODAY", days=0), public_key_fp=FP)
reg.save(make_doc(lic_id="L-FUTURE", days=100,
not_before=(date.today() + timedelta(days=30)).isoformat()),
public_key_fp=FP)
got = {r.lic_id: r.status() for r in reg.query()}
check("无到期日 → forever", got["L-FOREVER"] == "forever", got["L-FOREVER"])
check("远期 → active", got["L-ACTIVE"] == "active", got["L-ACTIVE"])
check("30 天内 → expiring", got["L-EXPIRING"] == "expiring", got["L-EXPIRING"])
check("已过 → expired", got["L-EXPIRED"] == "expired", got["L-EXPIRED"])
check("到期当天仍有效(含当天)", got["L-TODAY"] == "expiring", got["L-TODAY"])
check("未到生效日 → not_yet_valid",
got["L-FUTURE"] == "not_yet_valid", got["L-FUTURE"])
check("剩余天数为负表示已过期",
reg.get("L-EXPIRED").days_left() < 0,
str(reg.get("L-EXPIRED").days_left()))
check("永久授权剩余天数为 None",
reg.get("L-FOREVER").days_left() is None)
st = reg.stats()
check("概览计数一致",
st == {"total": 6, "expired": 1, "expiring": 2, "active": 1,
"forever": 1, "not_yet_valid": 1}, str(st))
def test_query_filters():
"""检索:客户 / 机器码 / 插件 / 关键字 / 状态,以及排序。"""
tmp = Path(tempfile.mkdtemp())
with Registry(tmp / "r.db") as reg:
reg.save(make_doc(lic_id="L-A", customer="阿尔法科技", days=300,
plugins=["doubao"], note="HT-001"), public_key_fp=FP)
reg.save(make_doc(lic_id="L-B", customer="贝塔公司", days=10,
plugins=["ip_sound_column"], note="HT-002"), public_key_fp=FP)
reg.save(make_doc(lic_id="L-C", customer="贝塔公司", days=200,
plugins=["doubao_extended"], note="HT-003"), public_key_fp=FP)
other = "AAAA-BBBB-CCCC-DDDD-EEEE-FFFF"
reg.save(make_doc(lic_id="L-D", customer="伽马", days=300, machine=other,
sources={"mac": "aabbccddeeff"}), public_key_fp=FP)
check("按客户模糊匹配", {r.lic_id for r in reg.query(customer="贝塔")} == {"L-B", "L-C"})
# L-C 装的是 doubao_extended,查 doubao 时不应被命中(前缀同名不算同一插件)
check("按插件匹配不误伤前缀同名",
{r.lic_id for r in reg.query(plugin="doubao")} == {"L-A"},
str([r.lic_id for r in reg.query(plugin="doubao")]))
check("按插件匹配到前缀同名者本身",
{r.lic_id for r in reg.query(plugin="doubao_extended")} == {"L-C"})
check("按机器码匹配", {r.lic_id for r in reg.query(machine_code=CODE[:16])}
== {"L-A", "L-B", "L-C"})
check("按备注关键字匹配", {r.lic_id for r in reg.query(keyword="HT-002")} == {"L-B"})
check("大小写不敏感(机器码)",
{r.lic_id for r in reg.query(machine_code=other.lower())} == {"L-D"})
order = [r.lic_id for r in reg.query()]
check("按到期日升序(急的在前)", order[0] == "L-B", str(order))
# 某台机器的全部历史授权
check("按机器码取全部历史",
len(reg.find_by_machine_code(CODE)) == 3,
str(len(reg.find_by_machine_code(CODE))))
check("limit 生效", len(reg.query(limit=2)) == 2)
def test_renew_reuses_sources_without_request_code():
"""核心能力:续期复用台账里的指纹,无需再向客户索取申请码。"""
tmp = Path(tempfile.mkdtemp())
with Registry(tmp / "r.db") as reg:
old = reg.save(make_doc(lic_id="L-OLD", customer="续期客户", days=5,
plugins=["aibox_sophon_adapter", "ip_sound_column"],
note="合同 HT-100"), public_key_fp=FP)
check("原授权即将到期", old.status() == "expiring", old.status())
# 模拟现场:客户已联系不上,只有台账记录可用
renewed_doc = make_doc(lic_id="L-NEW", customer=old.customer, days=365,
plugins=old.plugins, note=old.note,
machine=old.machine_code, sources=old.sources)
new = reg.save(renewed_doc, public_key_fp=FP, source="renew")
check("续期成功", new.status() == "active", new.status())
check("指纹与原件一致(未索取新申请码)", new.sources == old.sources)
check("机器码不变", new.machine_code == old.machine_code)
check("新旧授权都在台账(可追溯)",
{r.lic_id for r in reg.find_by_machine_code(old.machine_code)}
== {"L-OLD", "L-NEW"})
def test_note_preserved_on_reimport():
"""重新导入同一份 license 不应覆盖人工维护的备注。"""
tmp = Path(tempfile.mkdtemp())
with Registry(tmp / "r.db") as reg:
doc = make_doc(lic_id="L-N", note="原始备注")
reg.save(doc, public_key_fp=FP)
check("备注已入库", reg.get("L-N").note == "原始备注")
reg.update_note("L-N", "人工补充:已电话确认续期")
check("备注可更新", reg.get("L-N").note == "人工补充:已电话确认续期")
created = reg.get("L-N").created_at
reg.save(doc, public_key_fp=FP) # 重新导入(载荷里还是旧备注)
after = reg.get("L-N")
check("重新导入不覆盖人工备注",
after.note == "人工补充:已电话确认续期", after.note)
check("创建时间保持不变", after.created_at == created, after.created_at)
check("更新时间被推进", after.updated_at >= created)
def test_export_and_csv():
"""导出 .lic(可按客户分目录)与 CSV(UTF-8 BOM,Excel 不乱码)。"""
tmp = Path(tempfile.mkdtemp())
with Registry(tmp / "r.db") as reg:
reg.save(make_doc(lic_id="L-X1", customer="甲方/科技", days=100), public_key_fp=FP)
reg.save(make_doc(lic_id="L-X2", customer="乙方", days=-3), public_key_fp=FP)
files = reg.export_licenses(tmp / "arc", subdir_by_customer=True)
check("导出数量正确", len(files) == 2, str([f.name for f in files]))
check("客户名含非法字符时安全替换",
any("甲方" in str(f) for f in files), str([str(f.parent.name) for f in files]))
# 导出的文件应能被校验端直接读取(验证导出内容不是"看着像"而已)
# 注意顺序按到期日升序:files[0] 是已过期的 L-X2,files[1] 是有效的 L-X1
from chongming_license import read_license
valid_file = next(f for f in files if "L-X1" in f.name)
expired_file = next(f for f in files if "L-X2" in f.name)
st = read_license(valid_file, PUB, sources=SRC)
check("导出的有效授权仍可验签通过", st.ok, st.message)
st2 = read_license(expired_file, PUB, sources=SRC)
check("导出的已过期授权被判定为过期(到期日完整保留)",
not st2.ok and st2.code == "expired", f"{st2.code} / {st2.message}")
check("导出内容可原样还原",
valid_file.read_text(encoding="utf-8").lstrip().startswith("{"))
csv_path = reg.export_csv(tmp / "台账.csv")
with csv_path.open(encoding="utf-8-sig", newline="") as f:
rows = list(csv.reader(f))
check("CSV 含表头 + 两条数据", len(rows) == 3, str(len(rows)))
check("CSV 表头含状态与到期日",
"状态" in rows[0] and "到期日期" in rows[0], str(rows[0]))
check("CSV 记录了状态文案",
any("已过期" in r for r in rows[1:]), str(rows[1:]))
check("CSV 以 UTF-8 BOM 写入(Excel 中文不乱码)",
csv_path.read_bytes()[:3] == b"\xef\xbb\xbf")
def test_schema_version_mismatch_is_explicit():
"""结构版本不匹配时明确报错,而不是让后续查询诡异失败。"""
tmp = Path(tempfile.mkdtemp())
path = tmp / "r.db"
with Registry(path) as reg:
reg.save(make_doc(lic_id="L-V"), public_key_fp=FP)
# 手工把版本改成未来值,模拟"被更新版本的工具写过"
conn = sqlite3.connect(str(path))
conn.execute("PRAGMA user_version = 99")
conn.commit()
conn.close()
try:
Registry(path)
check("版本不匹配时拒绝打开", False, "未抛异常")
except RegistryError as exc:
check("版本不匹配时拒绝打开", "99" in str(exc), str(exc).splitlines()[0])
# 已存在但 user_version=0 的库(旧文件)应被平滑接管
path2 = tmp / "old.db"
conn = sqlite3.connect(str(path2))
conn.executescript(
"CREATE TABLE issued_licenses (lic_id TEXT PRIMARY KEY, machine_code TEXT, "
"customer TEXT, note TEXT, plugins_json TEXT, sources_json TEXT, not_before TEXT, "
"expires_at TEXT, issued_at TEXT, kid TEXT, public_key_fp TEXT, license_json TEXT, "
"source TEXT, created_at TEXT, updated_at TEXT);"
)
conn.commit()
conn.close()
with Registry(path2) as reg2:
check("user_version=0 的旧库被平滑接管", reg2.stats()["total"] == 0)
def test_concurrent_open():
"""两个实例同时打开同一库(多个 GUI 窗口 / CLI 并行)不应互相报错。"""
tmp = Path(tempfile.mkdtemp())
path = tmp / "r.db"
a = Registry(path)
b = Registry(path)
try:
a.save(make_doc(lic_id="L-C1", customer="并发A"), public_key_fp=FP)
b.save(make_doc(lic_id="L-C2", customer="并发B"), public_key_fp=FP)
check("两实例各自写入后互相可见",
{r.lic_id for r in a.query()} == {"L-C1", "L-C2"},
str([r.lic_id for r in a.query()]))
check("B 能看到 A 的记录", b.get("L-C1") is not None)
finally:
a.close()
b.close()
def test_delete():
tmp = Path(tempfile.mkdtemp())
with Registry(tmp / "r.db") as reg:
reg.save(make_doc(lic_id="L-DEL"), public_key_fp=FP)
check("删除成功", reg.delete("L-DEL") is True)
check("删除后查不到", reg.get("L-DEL") is None)
check("删除不存在的返回 False", reg.delete("L-DEL") is False)
def main():
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
for fn in tests:
print(f"\n--- {fn.__name__} ---")
fn()
print("\n" + "=" * 56)
if failures:
print(f"❌ {len(failures)} 项未通过: {failures}")
return 1
print("✅ 全部通过")
return 0
if __name__ == "__main__":
sys.exit(main())
-358
View File
@@ -1,358 +0,0 @@
"""chongming-license 库层测试:指纹、机器码、验签、容错匹配、时钟防线。
运行方式:`python test/test_license_verify.py`,或 `uv run pytest test/test_license_verify.py`。
"""
import base64
import json
import sys
import tempfile
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
from cryptography.hazmat.primitives import serialization # noqa: E402
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey # noqa: E402
from chongming_license import ( # noqa: E402
REASON_CLOCK_ROLLBACK,
REASON_EXPIRED,
REASON_MACHINE_MISMATCH,
REASON_NOT_YET_VALID,
REASON_OK,
REASON_PLUGIN_NOT_LICENSED,
REASON_SIGNATURE_INVALID,
ClockGuard,
LicenseManager,
MemoryClockStore,
RequestCodeError,
decode_request_code,
encode_request_code,
machine_code,
match_fingerprints,
normalize,
read_license,
sign_payload,
)
failures: list[str] = []
def check(name, cond, detail=""):
status = "PASS" if cond else "FAIL"
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
if not cond:
failures.append(name)
# ---------------------------------------------------------------------------
# 夹具
# ---------------------------------------------------------------------------
# 模拟真实盒子的指纹:4 个可信来源 + 1 个 weak(同镜像刷机会重复的 machine-id)
HW = {
"board_serial": "sn123456789",
"emmc_cid": "1a2b3c4d",
"cpu_serial": "0000deadbeef",
"mac": "aabbccddeeff",
"machine_id": "sharedbyallunits",
}
def _keypair():
priv = Ed25519PrivateKey.generate()
pub = base64.b64encode(
priv.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
).decode("ascii")
return pub, priv
def _write_license(tmp: Path, priv, sources=None, plugins=None, **over) -> str:
src = HW if sources is None else sources
payload = {
"v": 1,
"lic_id": "L-TEST",
"customer": "测试客户",
# sources 为空表示"只签机器码"的授权(无逐来源容错),此时机器码必须由调用方给定
"machine_code": over.pop("machine_code") if not src else machine_code(src),
"sources": src,
"plugins": plugins if plugins is not None else ["aibox_sophon_adapter"],
"not_before": None,
"expires_at": (date.today() + timedelta(days=365)).isoformat(),
"issued_at": "2026-09-20T00:00:00Z",
"note": "",
}
payload.update(over)
doc = {"payload": payload, "alg": "ed25519", "kid": "test",
"sig": sign_payload(payload, priv)}
path = tmp / "license.lic"
path.write_text(json.dumps(doc), encoding="utf-8")
return str(path)
# ---------------------------------------------------------------------------
# 指纹与机器码
# ---------------------------------------------------------------------------
def test_normalize_rejects_placeholders():
"""占位符/全 0/全 f/过短一律视为无效。"""
for bad in ["", " ", "unknown", "None", "0000000000000000", "ffffffffffff",
"ab", "00000000-0000-0000-0000-000000000000"]:
check(f"normalize 拒绝 {bad!r}", normalize(bad) == "", f"got {normalize(bad)!r}")
check("normalize 保留合法值", normalize("SN-1234-ABCD") == "sn1234abcd",
normalize("SN-1234-ABCD"))
check("normalize 分隔符差异归一", normalize("AA:BB:CC:DD:EE:FF") == normalize("aabbccddeeff"))
def test_machine_code_deterministic_and_weak_excluded():
"""机器码确定,且不受 weak 来源影响。"""
a = machine_code(HW)
b = machine_code(dict(reversed(list(HW.items()))))
check("机器码与来源顺序无关", a == b)
check("机器码分组格式", len(a.split("-")) == 6 and len(a.replace("-", "")) == 24, a)
other = dict(HW, machine_id="anothervalue")
check("weak 来源变化不影响机器码", machine_code(other) == a)
strong_changed = dict(HW, mac="112233445566")
check("可信来源变化会改变机器码", machine_code(strong_changed) != a)
# ---------------------------------------------------------------------------
# 容错匹配
# ---------------------------------------------------------------------------
def test_tolerance_single_component_swap():
"""换一块网卡 / 换一颗 eMMC 仍通过;整盘拷到别的机器不通过。"""
same = match_fingerprints(HW, dict(HW))
check("同机全命中", same.ok and same.matched == 4 and same.total == 4, same.reason)
nic = match_fingerprints(HW, dict(HW, mac="112233445566"))
check("换网卡仍通过(3/4)", nic.ok, nic.reason)
emmc = match_fingerprints(HW, dict(HW, emmc_cid="ffffffff"))
check("换 eMMC 仍通过(3/4)", emmc.ok, emmc.reason)
two = match_fingerprints(HW, dict(HW, mac="112233445566", emmc_cid="ffffffff"))
check("同时换网卡与 eMMC 不通过(2/4)", not two.ok, two.reason)
copied = match_fingerprints(
HW,
{"board_serial": "x1", "emmc_cid": "x2", "cpu_serial": "x3", "mac": "x4",
"machine_id": "sharedbyallunits"},
)
check("整盘拷贝到另一台机器不通过(weak 来源不得计票)",
not copied.ok and copied.matched == 0, copied.reason)
check("判定未降级", not copied.degraded, str(copied.degraded))
def test_tolerance_degraded_when_only_weak():
"""仅有 weak 来源时降级判定,并显式标记。"""
r = match_fingerprints({"machine_id": "shared"}, {"machine_id": "shared"})
check("仅 weak 来源时降级通过", r.ok and r.degraded, r.reason)
# ---------------------------------------------------------------------------
# 验签与判定
# ---------------------------------------------------------------------------
def test_sign_verify_roundtrip_and_tamper():
tmp = Path(tempfile.mkdtemp())
pub, priv = _keypair()
path = _write_license(tmp, priv)
st = read_license(path, pub, sources=HW)
check("正常授权校验通过", st.ok and st.code == REASON_OK, st.message)
check("解析出授权插件", st.licensed_plugins == {"aibox_sophon_adapter"},
str(st.licensed_plugins))
# 篡改白名单(典型的"自己加一个插件")→ 签名失效
doc = json.loads(Path(path).read_text())
doc["payload"]["plugins"] = ["aibox_sophon_adapter", "doubao"]
bad = tmp / "tampered.lic"
bad.write_text(json.dumps(doc))
st2 = read_license(str(bad), pub, sources=HW)
check("篡改白名单被识破", not st2.ok and st2.code == REASON_SIGNATURE_INVALID,
st2.message)
# 换公钥(非本厂商签发)
other_pub, _ = _keypair()
st3 = read_license(path, other_pub, sources=HW)
check("非本厂商公钥验签失败",
not st3.ok and st3.code == REASON_SIGNATURE_INVALID, st3.message)
# 字段重排序不影响验签(签名覆盖规范化 JSON)
doc = json.loads(Path(path).read_text())
doc["payload"] = dict(reversed(list(doc["payload"].items())))
reordered = tmp / "reordered.lic"
reordered.write_text(json.dumps(doc))
st4 = read_license(str(reordered), pub, sources=HW)
check("载荷字段重排序不影响验签", st4.ok, st4.message)
def test_expiry_and_not_before():
tmp = Path(tempfile.mkdtemp())
pub, priv = _keypair()
expired = _write_license(
tmp, priv, expires_at=(date.today() - timedelta(days=1)).isoformat()
)
st = read_license(expired, pub, sources=HW)
check("过期授权被拒", not st.ok and st.code == REASON_EXPIRED, st.message)
# 到期当天仍有效(含首尾当天)
today = _write_license(tmp, priv, expires_at=date.today().isoformat())
st2 = read_license(today, pub, sources=HW)
check("到期当天仍然有效", st2.ok, st2.message)
future = _write_license(
tmp, priv, not_before=(date.today() + timedelta(days=5)).isoformat()
)
st3 = read_license(future, pub, sources=HW)
check("未生效授权被拒", not st3.ok and st3.code == REASON_NOT_YET_VALID, st3.message)
never = _write_license(tmp, priv, expires_at=None)
st4 = read_license(never, pub, sources=HW)
check("无到期日视为永久有效", st4.ok, st4.message)
def test_machine_mismatch_and_plugin_whitelist():
tmp = Path(tempfile.mkdtemp())
pub, priv = _keypair()
path = _write_license(tmp, priv, plugins=["ip_sound_column"])
st = read_license(path, pub, sources=HW)
check("本机授权有效", st.ok, st.message)
mgr = LicenseManager(license_path=path, public_key=pub, min_match=3,
clock_store=MemoryClockStore(), collector=lambda: HW)
check("白名单内插件放行", mgr.check_plugin("ip_sound_column").allowed)
denied = mgr.check_plugin("doubao")
check("白名单外插件拒绝",
not denied.allowed and denied.code == REASON_PLUGIN_NOT_LICENSED, denied.message)
other_machine = dict(HW, mac="999999999999", cpu_serial="888888888888")
st2 = read_license(path, pub, sources=other_machine)
check("机器码不符拒绝",
not st2.ok and st2.code == REASON_MACHINE_MISMATCH, st2.message)
def test_machine_code_only_license_fallback():
"""只签机器码(无逐来源指纹)时退化为精确匹配。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _keypair()
path = _write_license(tmp, priv, sources={}, machine_code=machine_code(HW))
st = read_license(path, pub, sources=HW)
check("仅机器码的授权:本机通过", st.ok, st.message)
st2 = read_license(path, pub, sources=dict(HW, mac="000000000001"))
check("仅机器码的授权:异机拒绝", not st2.ok and st2.code == REASON_MACHINE_MISMATCH,
st2.message)
# ---------------------------------------------------------------------------
# 申请码
# ---------------------------------------------------------------------------
def test_request_code_roundtrip():
code = encode_request_code(HW)
check("申请码前缀", code.startswith("CM1-"), code[:12])
data = decode_request_code(code)
check("申请码解析出全部来源", data["sources"] == HW, str(data["sources"]))
check("申请码含机器码", data["machine_code"] == machine_code(HW))
wrapped = code[:30] + "\n " + code[30:] + " "
check("申请码容忍折行与空白", decode_request_code(wrapped)["sources"] == HW)
try:
decode_request_code(code[:-1] + ("0" if code[-1] != "0" else "1"))
check("申请码校验和可检出抄写错误", False, "未抛异常")
except RequestCodeError as exc:
check("申请码校验和可检出抄写错误", True, str(exc))
try:
decode_request_code("not-a-code")
check("非法申请码被拒", False, "未抛异常")
except RequestCodeError:
check("非法申请码被拒", True)
# ---------------------------------------------------------------------------
# 时钟防线
# ---------------------------------------------------------------------------
def test_clock_guard():
store = MemoryClockStore()
guard = ClockGuard(store, tolerance_seconds=3600)
now = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc)
check("首次检查通过并建立水位", guard.check(now).ok)
later = now + timedelta(hours=5)
check("时间前进正常", guard.check(later).ok)
# 回退 30 分钟:容差内,放行
slight = later - timedelta(minutes=30)
check("容差内回退放行", guard.check(slight).ok)
# 回退 10 小时:超过容差,拒绝
rollback = later - timedelta(hours=10)
r = guard.check(rollback)
check("明显回拨被检出", not r.ok and r.rollback_seconds > 0, r.reason)
# 水位未被回退覆盖
check("回拨后水位未被拉低", guard.check(later).ok)
def test_clock_guard_reseeds_when_far_ahead():
"""水位远超当前时间(RTC 异常)时重新起算,避免把设备永久卡死。"""
store = MemoryClockStore()
guard = ClockGuard(store, tolerance_seconds=3600, reseed_if_ahead_days=90)
guard.check(datetime(2027, 1, 1, tzinfo=timezone.utc))
r = guard.check(datetime(2026, 9, 20, tzinfo=timezone.utc))
check("大幅领先判为时钟异常而非攻击", r.ok and "异常" in r.reason, r.reason)
def test_manager_blocks_on_clock_rollback():
"""时钟回拨时管理器的判定整体中止。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _keypair()
path = _write_license(tmp, priv, plugins=["ip_sound_column"])
store = MemoryClockStore()
far_future = datetime.now(timezone.utc) + timedelta(days=2)
ClockGuard(store, tolerance_seconds=3600).check(far_future)
mgr = LicenseManager(license_path=path, public_key=pub, clock_store=store,
collector=lambda: HW)
st = mgr.reload()
check("时钟回拨时授权判定中止",
not st.ok and st.code == REASON_CLOCK_ROLLBACK, st.message)
check("时钟回拨时插件被拒", not mgr.check_plugin("ip_sound_column").allowed)
def test_manager_disabled_mode_reports_reality():
"""enforce=False(开发/测试)时放行,但状态仍如实反映真实校验结果。"""
tmp = Path(tempfile.mkdtemp())
pub, priv = _keypair()
path = _write_license(tmp, priv, plugins=[])
mgr = LicenseManager(license_path=path, public_key=pub, enforce=False,
collector=lambda: HW)
check("未启用时插件放行", mgr.check_plugin("doubao").allowed)
check("未启用时仍如实报告授权有效", mgr.status.ok, mgr.status.message)
def main():
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
for fn in tests:
print(f"\n--- {fn.__name__} ---")
fn()
print("\n" + "=" * 56)
if failures:
print(f"❌ {len(failures)} 项未通过: {failures}")
return 1
print("✅ 全部通过")
return 0
if __name__ == "__main__":
sys.exit(main())
+39 -1
View File
@@ -33,7 +33,8 @@ class FakeWM:
self.calls = [] self.calls = []
self.result = {"result": "ok"} self.result = {"result": "ok"}
async def call(self, worker_type, subject, payload, timeout=30, cacheable=False): async def call(self, worker_type, subject, payload, timeout=30, cacheable=False,
instance_id=None):
self.calls.append( self.calls.append(
{ {
"worker_type": worker_type, "worker_type": worker_type,
@@ -41,6 +42,7 @@ class FakeWM:
"payload": payload, "payload": payload,
"timeout": timeout, "timeout": timeout,
"cacheable": cacheable, "cacheable": cacheable,
"instance_id": instance_id,
} }
) )
return self.result return self.result
@@ -153,6 +155,41 @@ async def test_flat_payload_routed():
print("✅ 非信封(扁平)载荷按原始报文路由通过") print("✅ 非信封(扁平)载荷按原始报文路由通过")
async def test_target_instance():
"""实例定向:载荷顶层 target_instance 作为控制字段摘除并透传给 worker_manager.call"""
wm = FakeWM()
bridge, published = make_bridge(
wm, [{"topic": "devices/+/telemetry", "worker": "device"}]
)
# 1. 带 target_instance:透传实例名,且控制字段不进业务 payload
await bridge._handle(make_msg(
"devices/d1/telemetry",
{"meta": {}, "data": {"v": 1}, "target_instance": "edge-01"},
))
assert wm.calls[-1]["instance_id"] == "edge-01", wm.calls[-1]
assert "target_instance" not in wm.calls[-1]["payload"], wm.calls[-1]["payload"]
assert wm.calls[-1]["payload"]["data"] == {"v": 1}
# 2. 不带该字段:instance_id 为 None(跨实例随机均衡,与改动前一致)
await bridge._handle(make_msg("devices/d1/telemetry", {"meta": {}, "data": {"v": 1}}))
assert wm.calls[-1]["instance_id"] is None, wm.calls[-1]
# 3. 空白值视为未指定
await bridge._handle(make_msg(
"devices/d1/telemetry",
{"meta": {}, "data": {"v": 1}, "target_instance": " "},
))
assert wm.calls[-1]["instance_id"] is None, wm.calls[-1]
# 4. 内置 chongming/... 主题同样支持
await bridge._handle(make_msg(
f"{MQTT_TOPIC_PREFIX}/example/add",
{"meta": {}, "data": {"a": 1}, "target_instance": "edge-02"},
))
assert wm.calls[-1]["instance_id"] == "edge-02", wm.calls[-1]
print("✅ 实例定向(target_instance 控制字段)透传通过")
async def test_non_json_dropped(): async def test_non_json_dropped():
wm = FakeWM() wm = FakeWM()
bridge, published = make_bridge( bridge, published = make_bridge(
@@ -219,6 +256,7 @@ async def main():
await test_custom_routing() await test_custom_routing()
await test_envelope_meta_fields() await test_envelope_meta_fields()
await test_flat_payload_routed() await test_flat_payload_routed()
await test_target_instance()
await test_non_json_dropped() await test_non_json_dropped()
await test_board_ping_topic() await test_board_ping_topic()
test_plugin_config_driven_collection() test_plugin_config_driven_collection()
+58 -8
View File
@@ -16,12 +16,13 @@
D. WorkerManager 单测(FakeSession): D. WorkerManager 单测(FakeSession):
9. call_raw 零解析返回原始字节;call() 仍解析为 dict(MQTT 语义保留) 9. call_raw 零解析返回原始字节;call() 仍解析为 dict(MQTT 语义保留)
10. cacheable: 缓存原始 bytes、二次命中不再发查询、业务错误不缓存 10. cacheable: 缓存原始 bytes、二次命中不再发查询、业务错误不缓存
11. reply.err -> 重试 -> RuntimeError;空回复兜底 b"{}" 11. reply.err -> 换副本重试/无副本可换时立即 RuntimeError;空回复兜底 b"{}"
运行: uv run python test_passthrough_optimization.py 运行: uv run python test_passthrough_optimization.py
""" """
import asyncio import asyncio
import importlib.util import importlib.util
import random
import logging import logging
import sys import sys
import types import types
@@ -32,7 +33,7 @@ import orjson
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
REPO_ROOT = Path(__file__).resolve().parent REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT / "src")) sys.path.insert(0, str(REPO_ROOT / "src"))
sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src")) sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src"))
@@ -49,6 +50,7 @@ _auth_stub = types.ModuleType("chongming.app.core.auth")
_auth_stub.store_user_to_request = lambda x: x # 仅用于 Depends,桩掉鉴权 _auth_stub.store_user_to_request = lambda x: x # 仅用于 Depends,桩掉鉴权
_cache_stub = types.ModuleType("chongming_cache") _cache_stub = types.ModuleType("chongming_cache")
_cache_stub.Cache = object _cache_stub.Cache = object
_cache_stub.DistributedLock = object # worker_manager 顶层导入(本测试不触达)
for mod in (_pkg, _app, _core, _logger_stub, _auth_stub, _cache_stub): for mod in (_pkg, _app, _core, _logger_stub, _auth_stub, _cache_stub):
sys.modules[mod.__name__] = mod sys.modules[mod.__name__] = mod
@@ -101,7 +103,8 @@ class FakeWorkerManager:
self.error = error self.error = error
self.call_raw_payloads = [] # 捕获的预序列化请求字节 self.call_raw_payloads = [] # 捕获的预序列化请求字节
async def call_raw(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False): async def call_raw(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False,
instance_id=None):
self.call_raw_payloads.append(payload) self.call_raw_payloads.append(payload)
if self.error == "no_worker": if self.error == "no_worker":
raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。") raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。")
@@ -282,16 +285,27 @@ def test_openapi_request_body_injected():
# ---- D. WorkerManager 单测(FakeSession)---- # ---- D. WorkerManager 单测(FakeSession)----
class FakeSession: class FakeSession:
"""模拟 zenoh Session:get() 返回单回复迭代器""" """模拟 zenoh Session:get() 返回单回复迭代器
def __init__(self, payload=b'{"result": 5}', err_payload=None): err_payload 非空时整体报错;err_target 给定时只对该 key 前缀报错(其余正常回复),
用于验证「失败换副本重试」。
"""
def __init__(self, payload=b'{"result": 5}', err_payload=None, err_target=None):
self.payload = payload self.payload = payload
self.err_payload = err_payload self.err_payload = err_payload
self.err_target = err_target
self.get_calls = 0 self.get_calls = 0
self.call_keys = []
def get(self, *args, **kwargs): def get(self, *args, **kwargs):
self.get_calls += 1 self.get_calls += 1
if self.err_payload is not None: key = args[0] if args else kwargs.get("key_expr", "")
self.call_keys.append(str(key))
failing = self.err_payload is not None and (
self.err_target is None or str(key).startswith(self.err_target)
)
if failing:
reply = SimpleNamespace(ok=None, err=SimpleNamespace(payload=self.err_payload)) reply = SimpleNamespace(ok=None, err=SimpleNamespace(payload=self.err_payload))
else: else:
reply = SimpleNamespace(ok=SimpleNamespace(payload=self.payload), err=None) reply = SimpleNamespace(ok=SimpleNamespace(payload=self.payload), err=None)
@@ -341,14 +355,49 @@ async def test_wm_cache_bytes():
async def test_wm_reply_err_retry(): async def test_wm_reply_err_retry():
# 单副本:无其它副本可换 —— 立即失败(不空转重试同一个坏副本),保留具体错误
session = FakeSession(err_payload=b'{"error": "boom"}') session = FakeSession(err_payload=b'{"error": "boom"}')
wm = make_wm(session) wm = make_wm(session)
try: try:
await wm.call_raw("demo", "calc.add", b'{}') await wm.call_raw("demo", "calc.add", b'{}')
check("reply.err 应抛 RuntimeError", False) check("reply.err 应抛 RuntimeError", False)
except RuntimeError as e: except RuntimeError as e:
check("reply.err 重试后抛 RuntimeError", "boom" in str(e), str(e)) check("reply.err 抛 RuntimeError 且保留具体错误", "boom" in str(e), str(e))
check("reply.err 触发重试(2 次查询)", session.get_calls == 2, str(session.get_calls)) check("单副本无其它可换时不空转重试(1 次查询)",
session.get_calls == 1, str(session.get_calls))
async def test_wm_reply_err_switches_worker():
"""多副本:先打的副本报错则换另一个副本重试,且不重复打已失败的副本。
首个副本由 random.choice 决定,故对多个 seed 试跑:必有一个覆盖「先失败后成功」
路径(另一个覆盖「首个即成功」),两条路径都不允许重复打同一个失败的副本。
"""
ok, detail = False, "无 seed 覆盖到失败换副本路径"
for seed in (0, 1):
random.seed(seed)
session = FakeSession(
payload=b'{"result": 5}', err_payload=b'{"error": "boom"}',
err_target="worker/demo/w-dead",
)
wm = make_wm(session)
wm._workers_cache = {"demo": ["worker/demo/w-dead", "worker/demo/w-alive"]}
try:
result = await wm.call_raw("demo", "calc.add", b'{}', retries=2)
except RuntimeError as e: # 未换到正常副本(不该发生)
detail = f"seed={seed} 意外失败: {e}"
continue
assert result == b'{"result": 5}', result
dead_calls = session.call_keys.count("worker/demo/w-dead/calc.add")
if dead_calls > 1:
detail = f"seed={seed} 重复打失败副本: {session.call_keys}"
continue
if session.call_keys == [
"worker/demo/w-dead/calc.add",
"worker/demo/w-alive/calc.add",
]:
ok, detail = True, f"seed={seed} keys={session.call_keys}"
check("失败副本 -> 换副本重试成功(且不重复打失败副本)", ok, detail)
async def test_wm_empty_reply_fallback(): async def test_wm_empty_reply_fallback():
@@ -375,6 +424,7 @@ def run():
asyncio.run(test_wm_call_dict_semantics()) asyncio.run(test_wm_call_dict_semantics())
asyncio.run(test_wm_cache_bytes()) asyncio.run(test_wm_cache_bytes())
asyncio.run(test_wm_reply_err_retry()) asyncio.run(test_wm_reply_err_retry())
asyncio.run(test_wm_reply_err_switches_worker())
asyncio.run(test_wm_empty_reply_fallback()) asyncio.run(test_wm_empty_reply_fallback())
print(f"\n结果: {len(failures)} 个失败") print(f"\n结果: {len(failures)} 个失败")
+4 -2
View File
@@ -24,7 +24,7 @@ from pathlib import Path
import orjson import orjson
import zenoh import zenoh
REPO_ROOT = Path(__file__).resolve().parent REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src")) sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src"))
sys.path.insert(0, str(REPO_ROOT / "src")) sys.path.insert(0, str(REPO_ROOT / "src"))
@@ -39,6 +39,7 @@ _logger_stub = types.ModuleType("chongming.app.core.logger")
_logger_stub.get_logger = lambda name: logging.getLogger(name) _logger_stub.get_logger = lambda name: logging.getLogger(name)
_cache_stub = types.ModuleType("chongming_cache") _cache_stub = types.ModuleType("chongming_cache")
_cache_stub.Cache = object _cache_stub.Cache = object
_cache_stub.DistributedLock = object # worker_manager 顶层导入(本测试不触达)
for mod in (_pkg, _app, _core, _logger_stub, _cache_stub): for mod in (_pkg, _app, _core, _logger_stub, _cache_stub):
sys.modules[mod.__name__] = mod sys.modules[mod.__name__] = mod
@@ -67,7 +68,8 @@ def check(name, cond, detail=""):
failures.append(name) failures.append(name)
LISTEN_PORT = 7447 import os
LISTEN_PORT = int(os.environ.get("CM_STREAM_TEST_PORT", "7447"))
LISTEN = f"tcp/127.0.0.1:{LISTEN_PORT}" LISTEN = f"tcp/127.0.0.1:{LISTEN_PORT}"
+8 -4
View File
@@ -21,7 +21,7 @@ import orjson
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
REPO_ROOT = Path(__file__).resolve().parent REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT / "src")) sys.path.insert(0, str(REPO_ROOT / "src"))
sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src")) sys.path.insert(0, str(REPO_ROOT / "utils" / "worker" / "src"))
@@ -38,6 +38,7 @@ _auth_stub = types.ModuleType("chongming.app.core.auth")
_auth_stub.store_user_to_request = lambda x: x # 仅用于 Depends,桩掉鉴权 _auth_stub.store_user_to_request = lambda x: x # 仅用于 Depends,桩掉鉴权
_cache_stub = types.ModuleType("chongming_cache") _cache_stub = types.ModuleType("chongming_cache")
_cache_stub.Cache = object _cache_stub.Cache = object
_cache_stub.DistributedLock = object # worker_manager 顶层导入(本测试不触达)
for mod in (_pkg, _app, _core, _logger_stub, _auth_stub, _cache_stub): for mod in (_pkg, _app, _core, _logger_stub, _auth_stub, _cache_stub):
sys.modules[mod.__name__] = mod sys.modules[mod.__name__] = mod
@@ -90,18 +91,21 @@ class FakeWorkerManager:
self.call_payloads = [] self.call_payloads = []
self.stream_payloads = [] self.stream_payloads = []
async def call(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False): async def call(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False,
instance_id=None):
self.call_payloads.append((worker_type, subject, payload)) self.call_payloads.append((worker_type, subject, payload))
data = payload["data"] data = payload["data"]
return {"result": data.get("a", 0) + data.get("b", 0)} return {"result": data.get("a", 0) + data.get("b", 0)}
async def call_raw(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False): async def call_raw(self, worker_type, subject, payload, timeout=3.0, retries=1, cacheable=False,
instance_id=None):
# 网关透传路径:payload 为预序列化字节,回复原始字节 # 网关透传路径:payload 为预序列化字节,回复原始字节
self.call_payloads.append((worker_type, subject, orjson.loads(payload))) self.call_payloads.append((worker_type, subject, orjson.loads(payload)))
data = orjson.loads(payload)["data"] data = orjson.loads(payload)["data"]
return orjson.dumps({"result": data.get("a", 0) + data.get("b", 0)}) return orjson.dumps({"result": data.get("a", 0) + data.get("b", 0)})
async def call_stream(self, worker_type, subject, payload, timeout=300.0, idle_timeout=10.0): async def call_stream(self, worker_type, subject, payload, timeout=300.0, idle_timeout=10.0,
instance_id=None):
if self.stream_error == "no_worker": if self.stream_error == "no_worker":
raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。") raise RuntimeError(f"类型 '{worker_type}' 没有可用 Worker。")
self.stream_payloads.append((worker_type, subject, payload)) self.stream_payloads.append((worker_type, subject, payload))
Generated
+848 -525
View File
File diff suppressed because it is too large Load Diff