add: 进程插件打包命令 build_plugin;RouteRegistry 常驻订阅与插件盘扫解耦(external 插件无需网关侧配置)
This commit is contained in:
@@ -208,7 +208,8 @@ uv run serve → FastAPI lifespan:
|
|||||||
4. setup_worker_manager() — WorkerManager + 定期 liveliness 刷新
|
4. setup_worker_manager() — WorkerManager + 定期 liveliness 刷新
|
||||||
5. load_workers() — 按插件 [workers].load_mode 分派:
|
5. load_workers() — 按插件 [workers].load_mode 分派:
|
||||||
inline → 进程内 import 插件 → 启动 Worker 协程
|
inline → 进程内 import 插件 → 启动 Worker 协程
|
||||||
external → 跳过(独立进程运行),启动 RouteRegistry 订阅 chongming/routes/**
|
external → 跳过(独立进程运行,盘扫条目仅观测)
|
||||||
|
5.1 setup_route_registry() — 常驻订阅 chongming/routes/**,external 插件路由动态挂载/摘除
|
||||||
6. validate_contracts() — inline 插件三方契约校验,失败终止启动
|
6. validate_contracts() — inline 插件三方契约校验,失败终止启动
|
||||||
7. register_routes() — inline 插件盘扫生成 HTTP 路由
|
7. register_routes() — inline 插件盘扫生成 HTTP 路由
|
||||||
8. setup_mqtt() — MQTT Bridge(inline 主题静态收集 + external 主题动态推送)
|
8. setup_mqtt() — MQTT Bridge(inline 主题静态收集 + external 主题动态推送)
|
||||||
@@ -282,7 +283,10 @@ load_mode = "external" # 独立进程运行,经 Zenoh 注册路由
|
|||||||
- 路由注册:worker 启动时 put 描述符到 `chongming/routes/<type>/...`,退出时 delete;每 10s 周期重发布自愈
|
- 路由注册:worker 启动时 put 描述符到 `chongming/routes/<type>/...`,退出时 delete;每 10s 周期重发布自愈
|
||||||
- 服务发现:liveliness `worker/<type>/<id>`(与 inline 副本同一机制,网关无需改动)
|
- 服务发现:liveliness `worker/<type>/<id>`(与 inline 副本同一机制,网关无需改动)
|
||||||
- worker 进程自读 TOML 并在**自己进程内**做契约校验(`chongming_worker.contracts`);
|
- worker 进程自读 TOML 并在**自己进程内**做契约校验(`chongming_worker.contracts`);
|
||||||
网关 RouteRegistry 订阅描述符后动态挂载/摘除 HTTP 路由(type 级 2s 防抖)
|
网关 RouteRegistry **常驻订阅**描述符后动态挂载/摘除 HTTP 路由(type 级 2s 防抖)。
|
||||||
|
路由注册与网关侧插件盘扫(`config/workers/*/config.toml`)解耦:**网关容器无需
|
||||||
|
放置 external 插件的配置文件**,盘扫发现的 external 条目仅用于启动日志观测;
|
||||||
|
inline 插件仍按盘扫在网关进程内加载,不受影响
|
||||||
- 代码结构与 inline 插件完全一致(Plugin 基类 + router/ + models/),示例见本地 `workers/demo/`
|
- 代码结构与 inline 插件完全一致(Plugin 基类 + router/ + models/),示例见本地 `workers/demo/`
|
||||||
- 运行:
|
- 运行:
|
||||||
|
|
||||||
@@ -296,6 +300,31 @@ load_mode = "external" # 独立进程运行,经 Zenoh 注册路由
|
|||||||
- 进程插件容器:插件代码 + `chongming-worker` + 依赖
|
- 进程插件容器:插件代码 + `chongming-worker` + 依赖
|
||||||
- 隔离性由 `test_worker_isolation.py` 保证(阻断 chongming 导入 + AST 检查 import 语句)
|
- 隔离性由 `test_worker_isolation.py` 保证(阻断 chongming 导入 + AST 检查 import 语句)
|
||||||
|
|
||||||
|
### 进程插件打包
|
||||||
|
|
||||||
|
将 external 进程插件一键打包为可部署 zip(插件代码 + 配置 + 依赖清单 + 容器入口):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run build_plugin demo # → dist/demo.zip
|
||||||
|
uv run build_plugin demo --output /tmp/pkg
|
||||||
|
```
|
||||||
|
|
||||||
|
产物布局(zip 根即容器工作目录):
|
||||||
|
|
||||||
|
```text
|
||||||
|
workers/<name>/ 插件代码(模块 workers.<name>)
|
||||||
|
config/workers/<name>/ 插件注册 + 端点配置
|
||||||
|
config/zenoh_config.json5 Zenoh 会话配置
|
||||||
|
config.toml 最小根配置(自动生成,仅 [zenoh] 节,不泄露网关配置)
|
||||||
|
pyproject.toml uv 依赖清单(chongming-worker 内网 git 源)
|
||||||
|
start.sh 容器入口:uv sync + worker_serve --type <name>
|
||||||
|
```
|
||||||
|
|
||||||
|
- 插件扩展依赖(可选):在 `workers/<name>/requirements.txt` 中逐行声明,打包时自动合并进 `pyproject.toml` 的 dependencies
|
||||||
|
- 容器内部署:`unzip <name>.zip -d /app && cd /app && ./start.sh`(镜像需预装 uv)
|
||||||
|
- 打包前校验:`workers/<name>/` 与 `config/workers/<name>/config.toml` 必须存在,`load_mode` 非 `external` 时告警
|
||||||
|
- `config/workers/<name>/` 仅部署在**插件容器**(worker 自读);网关侧无需放置(RouteRegistry 常驻订阅 Zenoh 路由描述符,与盘扫解耦)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## MQTT 协议
|
## MQTT 协议
|
||||||
@@ -434,7 +463,7 @@ uv pip install -e utils/<name> # 安装/更新工具包
|
|||||||
- `load_mode = "inline"`(默认):插件在网关进程内加载
|
- `load_mode = "inline"`(默认):插件在网关进程内加载
|
||||||
- `load_mode = "external"`:插件以独立进程运行(进程编排交给 Docker/K8s,
|
- `load_mode = "external"`:插件以独立进程运行(进程编排交给 Docker/K8s,
|
||||||
进程入口 `uv run worker_serve --type <name>`),经 Zenoh 在 `chongming/routes/**`
|
进程入口 `uv run worker_serve --type <name>`),经 Zenoh 在 `chongming/routes/**`
|
||||||
注册路由,网关动态挂载 HTTP 端点
|
注册路由,网关动态挂载 HTTP 端点(RouteRegistry 常驻订阅,无需在网关侧保留插件配置)
|
||||||
- 两种模式可同时存在(混合部署)
|
- 两种模式可同时存在(混合部署)
|
||||||
- **进程插件与网关完全隔离**:进程插件只依赖 `chongming-worker` SDK + zenoh,
|
- **进程插件与网关完全隔离**:进程插件只依赖 `chongming-worker` SDK + zenoh,
|
||||||
与网关(`chongming` 包)零导入关系,仅通过 Zenoh(`chongming/routes/**` 注册路由、
|
与网关(`chongming` 包)零导入关系,仅通过 Zenoh(`chongming/routes/**` 注册路由、
|
||||||
@@ -442,7 +471,8 @@ uv pip install -e utils/<name> # 安装/更新工具包
|
|||||||
**生产部署时进程插件与网关分别打包到不同容器**:
|
**生产部署时进程插件与网关分别打包到不同容器**:
|
||||||
- 网关容器:`chongming`(含 `src/chongming/`)+ 依赖
|
- 网关容器:`chongming`(含 `src/chongming/`)+ 依赖
|
||||||
- 进程插件容器:插件代码(`workers/<name>/`)+ `chongming-worker` + 依赖,
|
- 进程插件容器:插件代码(`workers/<name>/`)+ `chongming-worker` + 依赖,
|
||||||
入口 `chongming-worker --type <name>`(或 `uv run worker_serve --type <name>`)
|
入口 `chongming-worker --type <name>`(或 `uv run worker_serve --type <name>`),
|
||||||
|
打包: `uv run build_plugin <name>` → `dist/<name>.zip`(见「进程插件打包」)
|
||||||
- **Zenoh 配置**:`config/zenoh_config.json5`
|
- **Zenoh 配置**:`config/zenoh_config.json5`
|
||||||
- **插件注册**:`config/workers/<name>/config.toml` 的 `[workers]` 节,定义加载方式、副本数与运行时配置(旧全局 `config/workers/config.toml` 仅作兜底兼容)
|
- **插件注册**:`config/workers/<name>/config.toml` 的 `[workers]` 节,定义加载方式、副本数与运行时配置(旧全局 `config/workers/config.toml` 仅作兜底兼容)
|
||||||
- **路由定义**:`config/workers/<name>/*.toml`,每个端点一个文件
|
- **路由定义**:`config/workers/<name>/*.toml`,每个端点一个文件
|
||||||
@@ -473,6 +503,7 @@ uv run gen_plugin <name> # 生成统一风格插件骨架
|
|||||||
uv run python -m chongming.scripts.gen_models # 生成 Pydantic 模型
|
uv run python -m chongming.scripts.gen_models # 生成 Pydantic 模型
|
||||||
uv run build pyinstaller # PyInstaller 打包
|
uv run build pyinstaller # PyInstaller 打包
|
||||||
uv run build nuitka # Nuitka 打包
|
uv run build nuitka # Nuitka 打包
|
||||||
|
uv run build_plugin <name> # 进程插件打包 → dist/<name>.zip
|
||||||
chongming-worker --type <name> # 独立 Worker 进程(进程插件容器入口)
|
chongming-worker --type <name> # 独立 Worker 进程(进程插件容器入口)
|
||||||
uv pip install -e utils/<name> # 安装/更新工具包子包
|
uv pip install -e utils/<name> # 安装/更新工具包子包
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -64,6 +64,7 @@ chongming-worker = { git = "http://124.71.68.6:3000/chongming/chongming-utils-wo
|
|||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
build = "chongming.scripts.build:build"
|
build = "chongming.scripts.build:build"
|
||||||
|
build_plugin = "chongming.scripts.build_plugin:build_plugin"
|
||||||
gen_models = "chongming.scripts.gen_models:gen_models"
|
gen_models = "chongming.scripts.gen_models:gen_models"
|
||||||
gen_plugin = "chongming.scripts.gen_plugin:gen_plugin"
|
gen_plugin = "chongming.scripts.gen_plugin:gen_plugin"
|
||||||
gunicorn = "chongming:gunicorn_serve"
|
gunicorn = "chongming:gunicorn_serve"
|
||||||
|
|||||||
@@ -52,11 +52,12 @@ async def lifespan(app: FastAPI):
|
|||||||
worker_tasks, custom_routers, inline_types, external_types = (
|
worker_tasks, custom_routers, inline_types, external_types = (
|
||||||
await lifespan_setup.load_workers(app, session, manager)
|
await lifespan_setup.load_workers(app, session, manager)
|
||||||
)
|
)
|
||||||
if external_types:
|
|
||||||
# 注册表为 external 插件服务;inline 类型集合防止误报重复挂载
|
# 5.1 动态路由注册表常驻启动:订阅 chongming/routes/**,external 插件的
|
||||||
await lifespan_setup.setup_route_registry(
|
# 路由经 Zenoh 注册/摘除,无需网关侧保留插件配置(external_types 仅观测)
|
||||||
app, session, manager, inline_types=inline_types
|
await lifespan_setup.setup_route_registry(
|
||||||
)
|
app, session, manager, inline_types=inline_types
|
||||||
|
)
|
||||||
|
|
||||||
# 5.5 契约校验(仅 inline 插件;external 插件在自己进程内校验),失败则启动终止
|
# 5.5 契约校验(仅 inline 插件;external 插件在自己进程内校验),失败则启动终止
|
||||||
lifespan_setup.validate_contracts(app)
|
lifespan_setup.validate_contracts(app)
|
||||||
@@ -88,7 +89,7 @@ async def lifespan(app: FastAPI):
|
|||||||
# 1. 停止 MQTT Bridge
|
# 1. 停止 MQTT Bridge
|
||||||
await lifespan_setup.shutdown_mqtt(app)
|
await lifespan_setup.shutdown_mqtt(app)
|
||||||
|
|
||||||
# 1.5 停止路由注册表订阅(存在 external 插件时)
|
# 1.5 停止路由注册表订阅(常驻,随应用生命周期启停)
|
||||||
registry = getattr(app.state, "route_registry", None)
|
registry = getattr(app.state, "route_registry", None)
|
||||||
if registry is not None:
|
if registry is not None:
|
||||||
registry.stop()
|
registry.stop()
|
||||||
|
|||||||
@@ -229,11 +229,12 @@ async def setup_route_registry(
|
|||||||
inline_types: list[str] | None = None,
|
inline_types: list[str] | None = None,
|
||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
启动动态路由注册表(订阅 chongming/routes/**)。
|
启动动态路由注册表(常驻订阅 chongming/routes/**)。
|
||||||
|
|
||||||
存在 load_mode="external" 的插件时启动:external 插件以独立进程运行
|
网关启动即订阅,不依赖网关侧是否登记 external 插件配置:external 插件
|
||||||
(由 Docker/K8s 等编排拉起,网关不负责进程管理),其路由描述符经 Zenoh
|
以独立进程运行(由 Docker/K8s 等编排拉起,网关不负责进程管理),其路由
|
||||||
动态注册;inline 插件仍在网关进程内加载,两者同时存在(混合模式)。
|
描述符经 Zenoh 动态注册;inline 插件仍在网关进程内加载,两者同时存在
|
||||||
|
(混合模式)。盘扫发现的 external 插件条目仅用于启动日志观测。
|
||||||
|
|
||||||
inline_types: 网关进程内加载的插件类型,注册表忽略其描述符以防重复挂载。
|
inline_types: 网关进程内加载的插件类型,注册表忽略其描述符以防重复挂载。
|
||||||
|
|
||||||
@@ -347,7 +348,8 @@ async def load_workers(app: FastAPI, session: zenoh.Session, manager: WorkerMana
|
|||||||
加载 inline 插件,启动对应的 worker 任务(混合模式)。
|
加载 inline 插件,启动对应的 worker 任务(混合模式)。
|
||||||
|
|
||||||
自动从 config/workers/<type>/config.toml 发现插件;load_mode="external"
|
自动从 config/workers/<type>/config.toml 发现插件;load_mode="external"
|
||||||
的插件不在网关内加载(由独立进程运行,其路由经 RouteRegistry 动态注册)。
|
的插件不在网关内加载(由独立进程运行,其路由经常驻 RouteRegistry 动态
|
||||||
|
注册,无需网关侧保留插件配置,盘扫发现的 external 条目仅用于观测)。
|
||||||
|
|
||||||
同时自动发现插件模块中暴露的 APIRouter(约定:模块顶层名为 `router` 的
|
同时自动发现插件模块中暴露的 APIRouter(约定:模块顶层名为 `router` 的
|
||||||
APIRouter 实例),收集后供 register_custom_routes() 统一注册。
|
APIRouter 实例),收集后供 register_custom_routes() 统一注册。
|
||||||
@@ -554,7 +556,7 @@ async def setup_mqtt(app: FastAPI, worker_manager: WorkerManager) -> MQTTBridge
|
|||||||
)
|
)
|
||||||
app.state.mqtt_bridge = bridge
|
app.state.mqtt_bridge = bridge
|
||||||
await bridge.start()
|
await bridge.start()
|
||||||
# process 模式:把 Bridge 挂接给路由注册表(动态主题 + 补推已到达的元信息)
|
# 把 Bridge 挂接给常驻路由注册表(动态主题 + 补推已到达的元信息)
|
||||||
registry = getattr(app.state, "route_registry", None)
|
registry = getattr(app.state, "route_registry", None)
|
||||||
if registry is not None:
|
if registry is not None:
|
||||||
registry.attach_bridge(bridge)
|
registry.attach_bridge(bridge)
|
||||||
|
|||||||
@@ -0,0 +1,253 @@
|
|||||||
|
"""
|
||||||
|
进程插件打包器 — 将 external 进程插件打包为可部署 zip
|
||||||
|
|
||||||
|
用法:
|
||||||
|
uv run build_plugin <plugin_name> [--output <dir>]
|
||||||
|
|
||||||
|
示例:
|
||||||
|
uv run build_plugin demo # 打包 workers/demo → dist/demo.zip
|
||||||
|
uv run build_plugin demo --output /tmp/pkg
|
||||||
|
|
||||||
|
产物布局(zip 根即容器工作目录,unzip 后直接 ./start.sh 启动):
|
||||||
|
workers/<name>/ 插件代码(模块 workers.<name>)
|
||||||
|
config/workers/<name>/ 插件注册 + 端点配置
|
||||||
|
config/zenoh_config.json5 Zenoh 会话配置
|
||||||
|
config.toml 最小根配置(仅 [zenoh] 节,worker 启动必需)
|
||||||
|
pyproject.toml uv 依赖清单(chongming-worker + 插件扩展依赖)
|
||||||
|
start.sh 容器入口: uv sync + uv run worker_serve --type <name>
|
||||||
|
|
||||||
|
说明:
|
||||||
|
- 插件扩展依赖声明在 workers/<name>/requirements.txt(可选,每行一个包),
|
||||||
|
打包时自动合并进 pyproject.toml dependencies
|
||||||
|
- 根 config.toml 仅提取 [zenoh] 节重新生成,避免把网关侧配置(如 JWT 密钥)泄露进插件容器
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tomllib
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
NAME_RE = re.compile(r"^[a-z][a-z0-9_]*$")
|
||||||
|
|
||||||
|
# chongming-worker 内网 git 源(与网关 pyproject.toml [tool.uv.sources] 保持一致)
|
||||||
|
CHONGMING_WORKER_GIT = "http://124.71.68.6:3000/chongming/chongming-utils-worker.git"
|
||||||
|
PYPI_MIRROR = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
|
||||||
|
|
||||||
|
|
||||||
|
def _find_project_root() -> Path:
|
||||||
|
"""从本文件所在目录向上查找项目根(含 pyproject.toml 的目录)。"""
|
||||||
|
here = Path(__file__).resolve().parent
|
||||||
|
for parent in here.parents:
|
||||||
|
if (parent / "pyproject.toml").exists():
|
||||||
|
return parent
|
||||||
|
return here.parent # 兜底:脚本目录的父目录
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = _find_project_root()
|
||||||
|
WORKERS_DIR = ROOT / "workers"
|
||||||
|
PLUGINS_CONFIG_DIR = ROOT / "config" / "workers"
|
||||||
|
ZENOH_CONFIG = ROOT / "config" / "zenoh_config.json5"
|
||||||
|
ROOT_CONFIG = ROOT / "config.toml"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================
|
||||||
|
# 产物生成
|
||||||
|
# ============================================================
|
||||||
|
|
||||||
|
def _rel(path: Path) -> Path | str:
|
||||||
|
"""相对 ROOT 的路径(不在 ROOT 下时原样返回,供错误消息使用)。"""
|
||||||
|
try:
|
||||||
|
return path.relative_to(ROOT)
|
||||||
|
except ValueError:
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _read_plugin_conf(name: str) -> dict:
|
||||||
|
"""读取 config/workers/<name>/config.toml 完整内容(校验含 [workers] 节)。"""
|
||||||
|
path = PLUGINS_CONFIG_DIR / name / "config.toml"
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
conf = tomllib.load(f)
|
||||||
|
if not conf.get("workers"):
|
||||||
|
raise ValueError(f"{path.relative_to(ROOT)} 缺少 [workers] 节")
|
||||||
|
return conf
|
||||||
|
|
||||||
|
|
||||||
|
def _gen_root_config_toml() -> str:
|
||||||
|
"""生成最小根 config.toml(仅 [zenoh] 节,worker 进程只读它)。"""
|
||||||
|
with open(ROOT_CONFIG, "rb") as f:
|
||||||
|
root_conf = tomllib.load(f)
|
||||||
|
json5_path = root_conf.get("zenoh", {}).get("json5_path", "config/zenoh_config.json5")
|
||||||
|
return f"""# 最小根配置(由 build_plugin 自动生成,仅含 worker 进程所需的 [zenoh] 节)
|
||||||
|
[zenoh]
|
||||||
|
json5_path = "{json5_path}"
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _gen_pyproject_toml(name: str, version: str, extra_deps: list[str]) -> str:
|
||||||
|
"""生成插件 uv 依赖清单(chongming-worker + 插件扩展依赖)。"""
|
||||||
|
deps = [
|
||||||
|
"chongming-worker",
|
||||||
|
"orjson", # chongming-worker 运行时依赖(SDK metadata 未声明,容器内需显式安装)
|
||||||
|
*extra_deps,
|
||||||
|
]
|
||||||
|
deps_lines = "\n".join(f' "{d}",' for d in deps)
|
||||||
|
return f"""# {name} 进程插件依赖清单(由 build_plugin 自动生成)
|
||||||
|
[project]
|
||||||
|
name = "{name}-worker"
|
||||||
|
version = "{version}"
|
||||||
|
description = "{name} 进程插件(external 模式,独立容器部署)"
|
||||||
|
requires-python = ">=3.12"
|
||||||
|
dependencies = [
|
||||||
|
{deps_lines}
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
worker_serve = "chongming_worker.process:worker_serve"
|
||||||
|
|
||||||
|
[[tool.uv.index]]
|
||||||
|
url = "{PYPI_MIRROR}"
|
||||||
|
|
||||||
|
[tool.uv.sources]
|
||||||
|
chongming-worker = {{ git = "{CHONGMING_WORKER_GIT}" }}
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _gen_start_sh(name: str) -> str:
|
||||||
|
"""生成容器入口脚本:uv 引导依赖 + 启动 worker 进程。"""
|
||||||
|
return f"""#!/usr/bin/env bash
|
||||||
|
# {name} 进程插件容器入口(由 build_plugin 自动生成)
|
||||||
|
# 要求: 镜像预装 uv(https://docs.astral.sh/uv/)
|
||||||
|
set -e
|
||||||
|
cd "$(dirname "$0")"
|
||||||
|
|
||||||
|
uv sync --no-dev
|
||||||
|
exec uv run worker_serve --type {name} "$@"
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _read_extra_deps(name: str) -> list[str]:
|
||||||
|
"""读取 workers/<name>/requirements.txt(可选),每行一个依赖。"""
|
||||||
|
path = WORKERS_DIR / name / "requirements.txt"
|
||||||
|
if not path.exists():
|
||||||
|
return []
|
||||||
|
deps = []
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines():
|
||||||
|
dep = line.strip()
|
||||||
|
if dep and not dep.startswith("#"):
|
||||||
|
deps.append(dep)
|
||||||
|
return deps
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================
|
||||||
|
# 打包主逻辑
|
||||||
|
# ============================================================
|
||||||
|
|
||||||
|
def build_plugin_package(name: str, output_dir: str | Path = "dist") -> Path:
|
||||||
|
"""
|
||||||
|
打包进程插件为可部署 zip,返回产物路径。
|
||||||
|
|
||||||
|
失败(插件/配置缺失、契约不合法)抛 FileNotFoundError / ValueError。
|
||||||
|
"""
|
||||||
|
if not NAME_RE.match(name):
|
||||||
|
raise ValueError(f"插件名不合法: '{name}'(需小写字母/数字/下划线,且以字母开头)")
|
||||||
|
|
||||||
|
workers_src = WORKERS_DIR / name
|
||||||
|
if not workers_src.is_dir():
|
||||||
|
raise FileNotFoundError(f"未找到插件代码目录: {_rel(workers_src)}/")
|
||||||
|
|
||||||
|
conf = _read_plugin_conf(name) # 缺配置在此抛 FileNotFoundError
|
||||||
|
workers = conf["workers"]
|
||||||
|
|
||||||
|
plugin_type = workers.get("type", name)
|
||||||
|
load_mode = workers.get("load_mode", "inline")
|
||||||
|
if load_mode != "external":
|
||||||
|
print(
|
||||||
|
f"⚠️ [{name}] load_mode = \"{load_mode}\":网关也会在进程内加载该插件,"
|
||||||
|
"独立进程将与网关内副本重复服务。进程插件请在 [workers] 节设置 "
|
||||||
|
"load_mode = \"external\"。"
|
||||||
|
)
|
||||||
|
version = str(conf.get("worker", {}).get("version", "0.1.0"))
|
||||||
|
|
||||||
|
output = Path(output_dir)
|
||||||
|
stage = output / name
|
||||||
|
if stage.exists():
|
||||||
|
shutil.rmtree(stage)
|
||||||
|
stage.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# ---- 1. 插件代码 ----
|
||||||
|
shutil.copytree(
|
||||||
|
workers_src,
|
||||||
|
stage / "workers" / name,
|
||||||
|
ignore=shutil.ignore_patterns("__pycache__", "*.pyc"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- 2. 插件配置 ----
|
||||||
|
shutil.copytree(
|
||||||
|
PLUGINS_CONFIG_DIR / name,
|
||||||
|
stage / "config" / "workers" / name,
|
||||||
|
ignore=shutil.ignore_patterns("__pycache__", "*.pyc"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- 3. Zenoh 会话配置 ----
|
||||||
|
if not ZENOH_CONFIG.exists():
|
||||||
|
raise FileNotFoundError(f"未找到 Zenoh 配置: {_rel(ZENOH_CONFIG)}")
|
||||||
|
shutil.copy(ZENOH_CONFIG, stage / "config" / "zenoh_config.json5")
|
||||||
|
|
||||||
|
# ---- 4. 生成最小根配置 + 依赖清单 + 启动脚本 ----
|
||||||
|
(stage / "config.toml").write_text(_gen_root_config_toml(), encoding="utf-8")
|
||||||
|
(stage / "pyproject.toml").write_text(
|
||||||
|
_gen_pyproject_toml(name, version, _read_extra_deps(name)), encoding="utf-8"
|
||||||
|
)
|
||||||
|
start_sh = stage / "start.sh"
|
||||||
|
start_sh.write_text(_gen_start_sh(plugin_type), encoding="utf-8")
|
||||||
|
start_sh.chmod(0o755)
|
||||||
|
|
||||||
|
# ---- 5. 压缩(zip 根即容器工作目录)----
|
||||||
|
zip_path = output / f"{name}.zip"
|
||||||
|
zip_path.unlink(missing_ok=True)
|
||||||
|
shutil.make_archive(str(output / name), "zip", root_dir=stage)
|
||||||
|
|
||||||
|
# 暂存目录保留便于排查,也可直接作为容器工作目录
|
||||||
|
return zip_path
|
||||||
|
|
||||||
|
|
||||||
|
def _format_size(path: Path) -> str:
|
||||||
|
size = path.stat().st_size
|
||||||
|
for unit in ("B", "KB", "MB", "GB"):
|
||||||
|
if size < 1024:
|
||||||
|
return f"{size:.2f} {unit}"
|
||||||
|
size /= 1024.0
|
||||||
|
return f"{size:.2f} TB"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description="chongming 进程插件打包器")
|
||||||
|
parser.add_argument("name", help="插件名(如 demo)")
|
||||||
|
parser.add_argument("--output", default="dist", help="产物输出目录(默认 dist)")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
try:
|
||||||
|
zip_path = build_plugin_package(args.name, output_dir=args.output)
|
||||||
|
except (FileNotFoundError, ValueError) as e:
|
||||||
|
print(f"❌ {e}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"✅ 打包完成: {zip_path} ({_format_size(zip_path)})")
|
||||||
|
print("📦 产物布局(zip 根即容器工作目录):")
|
||||||
|
print(f" workers/{args.name}/ 插件代码")
|
||||||
|
print(f" config/workers/{args.name}/ 插件配置")
|
||||||
|
print(" config/zenoh_config.json5 Zenoh 会话配置")
|
||||||
|
print(" config.toml 最小根配置(仅 [zenoh] 节)")
|
||||||
|
print(" pyproject.toml uv 依赖清单")
|
||||||
|
print(" start.sh 容器入口")
|
||||||
|
print(f"\n🚀 容器内部署: unzip {args.name}.zip -d /app && cd /app && ./start.sh")
|
||||||
|
|
||||||
|
|
||||||
|
def build_plugin():
|
||||||
|
"""CLI 入口(供 pyproject.toml [project.scripts] 调用)"""
|
||||||
|
main()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
"""
|
||||||
|
build_plugin 打包命令冒烟测试
|
||||||
|
|
||||||
|
验证进程插件打包产物的完整性与错误路径:
|
||||||
|
1. demo 插件打包成功,zip 内含代码/配置/最小根配置/依赖清单/可执行 start.sh
|
||||||
|
2. 插件代码目录缺失时报 FileNotFoundError
|
||||||
|
3. 插件 config.toml 缺失时报 FileNotFoundError
|
||||||
|
|
||||||
|
运行: uv run python test_build_plugin.py
|
||||||
|
"""
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import tomllib
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SRC = Path(__file__).parent
|
||||||
|
sys.path.insert(0, str(SRC / "src"))
|
||||||
|
|
||||||
|
from chongming.scripts import build_plugin as bp # noqa: E402
|
||||||
|
|
||||||
|
failures = []
|
||||||
|
|
||||||
|
|
||||||
|
def check(name, cond, detail=""):
|
||||||
|
status = "PASS" if cond else "FAIL"
|
||||||
|
print(f"[{status}] {name}" + (f" — {detail}" if detail else ""))
|
||||||
|
if not cond:
|
||||||
|
failures.append(name)
|
||||||
|
|
||||||
|
|
||||||
|
tmp = Path(tempfile.mkdtemp(prefix="build_plugin_test_"))
|
||||||
|
|
||||||
|
|
||||||
|
# ========== 1. demo 插件打包 ==========
|
||||||
|
zip_path = bp.build_plugin_package("demo", output_dir=tmp / "dist")
|
||||||
|
check("zip 产物生成", zip_path.exists(), str(zip_path))
|
||||||
|
|
||||||
|
with zipfile.ZipFile(zip_path) as z:
|
||||||
|
names = set(z.namelist())
|
||||||
|
check("包含插件代码 workers/demo/__init__.py", "workers/demo/__init__.py" in names)
|
||||||
|
check("包含插件配置 config/workers/demo/config.toml", "config/workers/demo/config.toml" in names)
|
||||||
|
check("包含 Zenoh 配置", "config/zenoh_config.json5" in names)
|
||||||
|
check("包含最小根配置 config.toml", "config.toml" in names)
|
||||||
|
check("包含依赖清单 pyproject.toml", "pyproject.toml" in names)
|
||||||
|
check("包含 start.sh", "start.sh" in names)
|
||||||
|
check("不包含 __pycache__", not any("__pycache__" in n for n in names))
|
||||||
|
|
||||||
|
# 最小根配置:仅 [zenoh] 节,不含网关侧敏感配置
|
||||||
|
root_conf = tomllib.loads(z.read("config.toml").decode())
|
||||||
|
check("根配置仅含 [zenoh] 节", set(root_conf.keys()) == {"zenoh"}, str(sorted(root_conf)))
|
||||||
|
check(
|
||||||
|
"根配置不泄露网关 [security] 密钥",
|
||||||
|
"secret_key" not in z.read("config.toml").decode(),
|
||||||
|
)
|
||||||
|
|
||||||
|
# 依赖清单:chongming-worker 内网 git 源 + worker_serve 入口
|
||||||
|
pyproject = tomllib.loads(z.read("pyproject.toml").decode())
|
||||||
|
deps = pyproject["project"]["dependencies"]
|
||||||
|
check("pyproject 依赖包含 chongming-worker", "chongming-worker" in deps)
|
||||||
|
check("pyproject 依赖包含 orjson(SDK 运行时依赖)", "orjson" in deps)
|
||||||
|
check(
|
||||||
|
"pyproject 声明 worker_serve 入口",
|
||||||
|
pyproject["project"]["scripts"].get("worker_serve") == "chongming_worker.process:worker_serve",
|
||||||
|
)
|
||||||
|
check(
|
||||||
|
"pyproject 声明 chongming-worker git 源",
|
||||||
|
pyproject["tool"]["uv"]["sources"]["chongming-worker"]["git"] == bp.CHONGMING_WORKER_GIT,
|
||||||
|
)
|
||||||
|
|
||||||
|
# start.sh:可执行 + 指向正确的插件 type
|
||||||
|
start_sh = z.read("start.sh").decode()
|
||||||
|
check("start.sh 以 --type demo 启动", "--type demo" in start_sh)
|
||||||
|
perms = (z.getinfo("start.sh").external_attr >> 16) & 0o777
|
||||||
|
check("start.sh 可执行(0o755)", perms == 0o755, oct(perms))
|
||||||
|
|
||||||
|
|
||||||
|
# ========== 2. 错误路径:插件代码目录缺失 ==========
|
||||||
|
orig_workers = bp.WORKERS_DIR
|
||||||
|
bp.WORKERS_DIR = tmp / "no_workers"
|
||||||
|
try:
|
||||||
|
bp.build_plugin_package("ghost", output_dir=tmp / "dist")
|
||||||
|
check("缺插件代码目录应抛 FileNotFoundError", False)
|
||||||
|
except FileNotFoundError:
|
||||||
|
check("缺插件代码目录应抛 FileNotFoundError", True)
|
||||||
|
finally:
|
||||||
|
bp.WORKERS_DIR = orig_workers
|
||||||
|
|
||||||
|
|
||||||
|
# ========== 3. 错误路径:插件配置缺失 ==========
|
||||||
|
orig_conf_dir = bp.PLUGINS_CONFIG_DIR
|
||||||
|
bp.PLUGINS_CONFIG_DIR = tmp / "no_config"
|
||||||
|
try:
|
||||||
|
bp.build_plugin_package("demo", output_dir=tmp / "dist")
|
||||||
|
check("缺插件配置应抛 FileNotFoundError", False)
|
||||||
|
except FileNotFoundError:
|
||||||
|
check("缺插件配置应抛 FileNotFoundError", True)
|
||||||
|
finally:
|
||||||
|
bp.PLUGINS_CONFIG_DIR = orig_conf_dir
|
||||||
|
|
||||||
|
|
||||||
|
# ========== 4. 错误路径:插件名不合法 ==========
|
||||||
|
try:
|
||||||
|
bp.build_plugin_package("Bad-Name", output_dir=tmp / "dist")
|
||||||
|
check("非法插件名应抛 ValueError", False)
|
||||||
|
except ValueError:
|
||||||
|
check("非法插件名应抛 ValueError", True)
|
||||||
|
|
||||||
|
|
||||||
|
shutil.rmtree(tmp, ignore_errors=True)
|
||||||
|
|
||||||
|
print("\n" + ("=" * 40))
|
||||||
|
if failures:
|
||||||
|
print(f"❌ {len(failures)} 项未通过: {failures}")
|
||||||
|
sys.exit(1)
|
||||||
|
print("✅ 全部通过")
|
||||||
Reference in New Issue
Block a user