commit f4de43c74303429bcd61c0c9c6c98a953a727af5 Author: chakcy <947105045@qq.com> Date: Thu Oct 8 23:47:50 2026 +0800 初始提交：chongming-task 任务控制库 单任务受控执行（15 态状态机 + 审计轨迹）、DAG 管道、Controller 门面。 执行器三个：线程（默认，含 async def 任务）、内联（同步 / 测试）、进程 （spawn 子进程：硬性停止、绕开 GIL、崩溃隔离）。 调度：并发额度 + 每桶限流 + 具名资源信号量（按标签）+ 优先级排队。 持久层：TaskStore 契约 + 内存 / SQLite / Postgres 实现，契约测试随包发布。 跨进程与多节点：写穿 / 读穿、指令通道与租约、owned_buckets 划界、reconcile。 CLI、12 个可直接运行的示例、docs/ 下 5 份文档。 门禁：ruff / ruff format / mypy(strict) 干净； pytest 563 passed（44 skipped 是需要真实 Postgres 的那批）； 带一台可连的 Postgres 时 607 passed。 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..c4ecd16 --- /dev/null +++ b/.gitignore @@ -0,0 +1,25 @@ +# Python-generated files +__pycache__/ +*.py[oc] +build/ +dist/ +wheels/ +*.egg-info + +# Virtual environments +.venv + +# pytest 的临时目录（默认落在项目根下） +pytest-of-*/ +.pytest_cache/ +.hypothesis/ +.mypy_cache/ +.ruff_cache/ + +# 示例与 CLI 走查产生的库文件（tasks.db / demo.db / crash.db ……） +*.db + +# 测试里用 tempfile 建的隔离目录：跑在项目根下，正常结束会被清掉， +# 被强杀（例如 `timeout` 截断跑批）时会留下残骸 +chongming-store-*/ +chongming-jsonstore-*/ diff --git a/.python-version b/.python-version new file mode 100644 index 0000000..6324d40 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.14 diff --git a/README.md b/README.md new file mode 100644 index 0000000..0f5b97a --- /dev/null +++ b/README.md @@ -0,0 +1,906 @@ +# chongming-task + +精细化状态控制的 Python 管道任务控制工具。 + +任务在这里不是一个"跑着 / 没跑"的布尔值，而是一条**可审计的状态轨迹**：每个中间态 +（排队、启动、运行、暂停中、重试等待、停止中）都能被表达、被观察、被干预，而且 +**结果永远如实上报**。 + +--- + +## 当前状态（务必先读） + +这是一个**正在实现中的库**。已经能用的部分与还没做的部分如下： + +| 能力 | 状态 | +|---|---| +| 任务状态模型与合法转移表（15 个状态） | ✅ 可用 | +| 单任务受控执行（`run_task` / `TaskRun`） | ✅ 可用 | +| 停止 / 暂停 / 恢复 | ✅ 可用（协作式） | +| 失败重试与退避 | ✅ 可用 | +| 三种超时（启动 / 单次尝试 / 静默） | ✅ 可用 | +| 线程执行器（含 `async def` 任务） | ✅ 可用 | +| 内联执行器（同步、零开销） | ✅ 可用 | +| 进程执行器（硬性停止 / 绕开 GIL / 崩溃隔离） | ✅ 可用 | +| 状态迁移历史（可审计） | ✅ 可用 | +| 任务注册表 + 统一管理门面（`Controller`） | ✅ 可用 | +| 并发额度与排队、具名资源信号量（按标签）、优先级、文本表格渲染 | ✅ 可用 | +| 任务分桶：稳定哈希 + 每桶独立限流 + 按桶查询（为分布式预留分配单位） | ✅ 可用 | +| 节点桶归属：`owned_buckets` 划界 + 提交校验 + `reconcile` 范围 | ✅ 可用 | +| 持久层契约 `TaskStore`（内存 / SQLite / Postgres 实现 + 契约测试套件） | ✅ 可用 | +| 跨机持久层 `PostgresTaskStore`（`SKIP LOCKED` 认领 + 连接池 + 跨进程并发用例） | ✅ 可用 | +| 存储接线：写穿 / 读穿 / 跨进程查询与控制 | ✅ 可用 | +| 重启恢复：监督者租约 + `reconcile()` 清理僵尸任务 | ✅ 可用 | +| DAG 管道：依赖、扇出/扇入、三种汇合语义、fail_fast 连坐 | ✅ 可用 | +| `@task` 装饰器、`TaskContext`（进度 / 日志 / 上游数据） | ✅ 可用 | +| 存储与 `Controller` 接线、跨进程查询、重启恢复 | ✅ 全部可用 | +| CLI（`run` / `ps` / `status` / `stop` / `pause` / `resume` / `stats` / `buckets`） | ✅ 可用 | + +> **默认线程执行器，另备进程执行器**（外加测试用的内联执行器）。 +> +> 线程共享内存，可以直接复用连接池、缓存、已加载的模型，也没有参数必须可 pickle 的 +> 约束——闭包、`lambda`、`@task` 装饰器产物都能用。代价是**强制停止做不到**：优雅 +> 窗口用尽后只能放弃等待并记 `lost`（结果未知），绝不谎报"已停止"。 +> +> 需要**硬性停止保证、绕开 GIL、或崩溃隔离**时用 `ProcessExecutor()`——任务跑在独立 +> 子进程里，优雅窗口用尽会 `terminate()` → `kill()`，段错误也只带走子进程。代价是 +> 三条结构性约束：参数与返回值必须可 pickle、目标函数必须在可导入的模块里（且调用方 +> 脚本要有 `if __name__ == "__main__":` 守卫）、停止与暂停有约 20ms 的轮询延迟。 +> 详见[用户指南 §7](docs/user-guide.md#7-执行器)。 +> +> 两种执行器下都请让任务调 `ctx.checkpoint()`，并写成幂等或可断点续传。 + +完整设计（状态表、分层、里程碑、风险）见 [设计文档](docs/task-control-design.md)； +面向使用者的完整说明见 [用户指南](docs/user-guide.md)。 + +--- + +## 安装 + +```bash +uv sync # 开发环境 +# 或 +pip install -e . +``` + +核心包**零第三方运行时依赖**，只用标准库。 + +跨机部署共用一套持久层时，再装一个可选 extra： + +```bash +pip install "chongming-task[postgres]" +``` + +它只带来 `psycopg`（Postgres 驱动）。**不装也能正常 `import chongming_task`**—— +`PostgresTaskStore` 是惰性取的，不装时取它会报出安装命令而不是一个光秃秃的 +`ModuleNotFoundError`（见 [持久层抽象](#持久层抽象)）。 + +--- + +## 快速上手 + +`examples/01_quickstart.py`： + +```python +import time +from chongming_task.run import TaskRun + + +def extract(ctx, source: str) -> list[int]: + """长耗时循环里必须周期性地调用 ctx.checkpoint()。""" + rows: list[int] = [] + for index in range(5): + ctx.checkpoint() # 响应停止/暂停，并刷新心跳 + time.sleep(0.05) + rows.append(index) + return rows + + +run = TaskRun(extract, ("s3://demo/rows",), task_id="demo-1", name="抽取") +run.start() +outcome = run.wait(timeout=30) + +for record in run.transitions: + print(f"{record.from_status.value:>10} -> {record.to_status.value:<10} by {record.trigger}") + +print(outcome.status.value, outcome.value) +``` + +真实输出： + +``` + 抽取完成：s3://demo/rows -> 5 行 + +状态迁移历史（只追加、可审计）： + pending -> queued by orchestrator:dependencies_resolved + queued -> starting by scheduler:dispatch + starting -> running by executor:launched + running -> succeeded by executor:completed + +最终状态：succeeded +返回值 ：[0, 1, 2, 3, 4] +耗时 ：0.25s +``` + +只想跑一个任务、不需要中途干预时，用一行： + +```python +from chongming_task.run import run_task + +outcome = run_task(extract, "s3://demo/rows", task_id="demo-1", name="抽取") +if outcome.ok: + print(outcome.value) +``` + +--- + +## 任务怎么写 + +**唯一约定**：长耗时循环里周期性调用 `ctx.checkpoint()`。 + +```python +def my_task(ctx, arg1, *, kwarg1=1): + for row in source: + ctx.checkpoint() # ← 停止、暂停、心跳都在这里生效 + process(row) + return result +``` + +`ctx` 由执行器注入为**第一个位置参数**，包含： + +| 成员 | 说明 | +|---|---| +| `ctx.checkpoint()` | 协作点：响应停止（抛 `TaskCancelled`）、响应暂停（挂起）、刷新心跳 | +| `ctx.acheckpoint()` | 同上，在 `async def` 任务里 `await`（执行器会把协程真的跑完） | +| `ctx.is_stop_requested` | 只读，是否已请求停止（用于清理逻辑等不便阻塞处） | +| `ctx.is_pause_requested` | 只读，是否已请求暂停 | +| `ctx.task_id` / `ctx.name` / `ctx.attempt` | 身份与重试次数 | +| `ctx.heartbeat` | 心跳对象，`idle` 超时依据它判定 | +| `ctx.artifact_dir` | 本次尝试的产物目录（按尝试隔离，重试不污染上一次） | + +**不调用 `checkpoint()` 的任务，既无法优雅停止，也无法暂停。** 这是协作式设计的 +硬约束——见 [用户指南](docs/user-guide.md#为什么必须调用-checkpoint)。 + +--- + +## 控制任务 + +```python +run = TaskRun(my_task, task_id="job-1") +run.start() + +run.pause() # 任务在下一个 checkpoint 挂起 → paused +run.resume() # → running +run.stop(reason="发布窗口") # → stopping → stopped +outcome = run.wait(timeout=60) +``` + +几个刻意设计的语义，都是"不骗人"的结果： + +- **`pause()` 返回 `False` = 暂停没生效**（任务没到达检查点），此时状态**回滚为 + `running`**，绝不假装已暂停。 +- **`stop()` 停不掉时不会说"已停止"**。线程无法强杀，因此优雅窗口用尽后任务落 + `lost`（结果未知），而不是 `stopped`： + + ``` + == 停不掉的任务：如实上报 == + 终态 ：lost + 失败归类 ：lost + 含义 ：线程无法强杀，任务可能仍在后台产生副作用，因此不敢说'已停止' + ``` + +- **`wait(timeout=...)` 超时只抛 `TimeoutError`，不改变任务状态**。把"我没等到" + 当成"任务失败"是最常见的误用。 +- **`pause()` 不释放并发槽位**。我们无法把任意的 Python 调用栈换出内存再换回来， + 所以暂停是就地挂起；暂停大量任务可能耗尽并发额度。 + +--- + +## 管理一批任务 + +`Controller` 是统一的管理入口：它持有**任务定义白名单**、**运行中的任务**与 +**并发额度**。将来的 CLI（以及任何看板）都只是这一层的薄适配。 + +```python +from chongming_task.controller import Controller, render_table + +controller = Controller(max_parallel=2) # 最多同时跑 2 个 + +controller.register("extract", extract, tags={"io"}) # 先登记，之后只传名字 +controller.register("transform", transform) + +controller.submit("transform", 100, task_id="t0", label="转换-1") +controller.submit("transform", 200, task_id="t1", label="转换-2") +controller.submit("transform", 300, task_id="t2", label="转换-3") +controller.submit("transform", 400, task_id="t3", label="转换-4") + +print(render_table(controller.list_tasks())) # 查询 +controller.stop("t0", reason="手工干预") # 控制 +controller.wait_all(timeout=60) +``` + +真实输出（`examples/04_controller.py`，节选）： + +``` +--- 查询：运行中的视图 --- +TASK_ID NAME STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------- ------- ------- -------- ------- +t0 转换-1 running 1/1 0.4s - - +t1 转换-2 running 1/1 0.4s - - +t2 转换-3 queued 0/1 - - - +t3 转换-4 queued 0/1 - - - + 未结束 4（其中排队 2，占用执行额度 2） +``` + +三个必须知道的设计点： + +1. **只接受名字，绝不接受代码。** `submit("extract", ...)` 里的 `extract` 必须已在 + 启动时 `register` 过，否则抛 `ConfigurationError`。一个能启动任意可调用对象的 + 入口等于**远程代码执行**——这条在接 CLI 之后同样不能破。 +2. **超限是排队，不是拒绝，也不是无限起线程。** 排队的任务状态为 `queued` + （已被受理、等待执行槽），与 `pending`（尚未受理）不同。排队中的任务 `ELAPSED` + 显示 `-`，因为它还没执行过。 +3. **`stop_all()` 返回的是"已发出请求"的数量，不是"已停止"。** 请求发出后任务先进入 + `stopping`，落终态是异步的；要等就接着 `wait_all()`。 + +> **跨进程限制**：`Controller` 管理的是**当前进程内**的任务。另一个进程里的 +> `chongming-task ps` 看不到它们——内存不共享。跨进程查询与持久化需要共享媒介： +> 同机用 `sqlite3`（**它在标准库里**，不引入任何第三方依赖），跨机用 +> `PostgresTaskStore`（可选 extra）。 + +--- + +## 任务分桶 + +桶既是**本地并发隔离**的单位，也是将来**分布式分配**的单位（一台机器拥有一组桶）。 + +```python +controller = Controller(max_parallel=10, max_parallel_per_bucket=1) +controller.submit("sync", "租户A", task_id="a0", bucket_key="tenant-A") +controller.submit("sync", "租户B", task_id="b0", bucket_key="tenant-B") +``` + +真实输出（`examples/05_buckets.py`，节选）——A 桶积压，B 桶照常并行： + +``` +分桶策略：64 个桶 +并发限制：全局 10，但每个桶最多 1 个 + +业务键 -> 桶号（同一个键在任何进程、任何机器上都会落到同一个桶）： + tenant-A -> bucket 7 + tenant-B -> bucket 28 + +--- 查询：A 桶积压，但不影响 B 桶 --- +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------ --------- ------- ------- -------- ------- +a0 A-同步 7 succeeded 1/1 0.4s - - +a1 A-同步 7 running 1/1 0.1s - - +a2 A-同步 7 queued 0/1 - - - +a3 A-同步 7 queued 0/1 - - - +b0 B-同步 28 succeeded 1/1 0.4s - - +b1 B-同步 28 running 1/1 0.1s - - + 占用额度 2，排队 2，A 桶在用 1 +``` + +### 两条硬约束 + +**一、桶数一旦开始用就不能随便改。** `bucket = hash(key) % N`，改 N 会让几乎所有键 +改换桶位。所以任务会**同时记下 `bucket_key`**——将来真要改桶数，只有拿得到原始键 +才能重新分桶。 + +**二、绝不能用内置 `hash()`。** CPython 对 `str` 的 `hash()` 带**进程级随机盐** +（`PYTHONHASHSEED`）：同一个键在不同进程里会落到不同的桶，于是"同桶任务在同一台 +机器上"会**静默**失效——没有任何报错，只是本地性与缓存复用全都不成立。 +`BucketPolicy` 用 `blake2b`，并有真实子进程 + 不同 `PYTHONHASHSEED` 的用例守着。 + +### 多节点部署：自治节点 + 统一平面 + +**每个节点有自己的任务定义与触发方式；节点之间不互相执行对方的任务；但共用同一套存储与 +控制层，靠桶区分"哪一份是我的"。** + +```python +# 节点 A：负责 0..15 号桶（按业务划分，可以多于一个桶） +controller = Controller( + store=SqliteTaskStore("tasks.db"), + owner="node-a", + owned_buckets=range(0, 16), +) +``` + +| 维度 | 自治（每节点自有） | 统一（全局共享） | +|---|---|---| +| 任务定义、触发方式、负责的桶 | ✅ | — | +| 存储（任务状态、审计轨迹、指令）、控制层 | — | ✅ 一套 | + +`owned_buckets` 的作用是**划界**： + +- **提交时校验**——任务落进别人的桶会在提交那一刻报错。否则它会永远停在 `queued`， + 而且**没有任何报错**； +- **`reconcile()` 只看自己的桶**——不去动别的节点的任务。 + +```bash +# 运维：只看"我这一份" +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 ps --all --mine + +# 每个桶多少任务；星号标出本节点负责的桶 +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 buckets +``` + +统一平面给的是**看得见**与**管得着**：一个看板看全部、跨节点 `stop`/`pause`（走已有的指令 +通道）、统一审计轨迹、以及**发现"某个桶没人管、任务在静默堆积"**——这是分散存储做不到的。 + +**跨机器**用 Postgres：SQLite 文件不能跨机共享，把 `store` 换成 `PostgresTaskStore` 即可， +其余代码（`owner`、`owned_buckets`、`reconcile()`、跨节点 `stop`）一行都不用改。 + +```python +from chongming_task import PostgresTaskStore # 需要 pip install "chongming-task[postgres]" + +controller = Controller( + store=PostgresTaskStore("postgresql://user:pw@db.internal:5432/chongming"), + owner="node-a", + owned_buckets=range(0, 16), +) +``` + +CLI 同样认连接串，运维不必在每个节点上找那个 `.db` 文件： + +```bash +chongming-task --db 'postgresql://user:pw@db.internal:5432/chongming' --num-buckets 64 ps --all +``` + +它实现的是**同一个 `TaskStore` 契约**，通过 `StoreContract` 全绿，并额外带着自己的并发证据 +（`SKIP LOCKED` 认领、并发 `save` 恰好一个赢家、持有者被强杀后租约照常到期）。细节与建表 +语句见[持久层抽象](#持久层抽象)与[持久层实现指南](docs/store-guide.md) §6。 + +这个架构的代价是**没有东西替你检查配置**：两个节点配了同一个桶（可能重复执行）或某个桶 +没人配（任务静默堆积）都不会自动报错。缓解手段与其它风险见 +[分布式设计说明](docs/distributed-design.md) §11。 + +> 早先设计过一个真正的集群方案（任意节点可跑任意任务：配方落库、跨节点原子认领、成员表、 +> rendezvous 分配、再均衡、fencing）。它被否决——定时任务是**有归属**的，不是无差别算力池。 +> 被否决的理由与仍然成立的结论见该文 §17。 + +--- + +## 嵌入到你自己的应用 + +**它是库优先的**：`Controller` 就是集成点，CLI 只是可选外壳。不需要单独进程、不需要 +网络层、不需要先起服务。 + +```python +from chongming_task import Controller, ProcessExecutor, ThreadExecutor + +controller = Controller(max_parallel=4, executor=ThreadExecutor()) # 默认：线程 +# 需要硬性停止 / 绕开 GIL / 崩溃隔离时换成子进程： +# controller = Controller(max_parallel=4, executor=ProcessExecutor()) + +controller.register("extract", extract) # 启动时登记一次 +run = controller.submit("extract", "s3://bucket/x", task_id="job-1") + +# 你的 Web 框架 / 定时任务 / 消息消费者里都可以直接用： +controller.list_tasks() +controller.get("job-1") +controller.stop("job-1", reason="用户取消") +controller.stats() +``` + +完整的一条集成主线（含多节点桶划分、启动恢复、跨节点控制、优雅退出）见 +**[examples/11_best_practice.py](examples/11_best_practice.py)** —— 照抄它比照抄本节的 +片段更省事。 + +公开 API 一行导入，`__all__` 里就是全部受支持的表面： + +```python +from chongming_task import ( # noqa: F401 + Controller, + TaskRegistry, + TaskRun, + run_task, + render_table, + TaskStatus, + TaskSnapshot, + TaskStore, + InMemoryTaskStore, + SqliteTaskStore, + RetryPolicy, + ExponentialBackoff, + Timeout, + ThreadExecutor, +) +``` + +## 持久层抽象 + +`TaskStore` 是持久层的**唯一契约**，第三方后端（Redis、对象存储……）只需满足它。内置三个 +实现： + +| 实现 | 适用 | 依赖 | +|---|---|---| +| `InMemoryTaskStore` | 单进程嵌入、单元测试、作为其他实现的**参照物** | 标准库 | +| `SqliteTaskStore` | 跨进程可见 + 持久化（**默认**） | 标准库 `sqlite3` | +| `PostgresTaskStore` | **跨机器**共用一套存储与指令通道 | 可选 `chongming-task[postgres]` | + +```python +from chongming_task import InMemoryTaskStore, Progress, SqliteTaskStore, TaskQuery, TaskStatus + +store = SqliteTaskStore("tasks.db") +store.create(snapshot) +store.save(next_snapshot, expect_version=snapshot.status_version) # 乐观并发 +store.query(TaskQuery(statuses=frozenset({TaskStatus.RUNNING}), tag="io")) +store.touch("job-1", progress=Progress(done=120, total=500)) # 高频旁路 +``` + +跨机器时换成 DSN 就完了——**同一个契约，同一套语义**： + +```python +from chongming_task import PostgresTaskStore + +store = PostgresTaskStore("postgresql://user:pw@db.internal:5432/chongming") +``` + +它建表所需的三张表（`tasks` / `transitions` / `commands`）在首次连接时自动创建； +若当前角色没有 DDL 权限，它检测到表已存在就直接使用——只读运维账号因此也能连。 +`close()` 幂等，关闭后再用会抛 `StoreError` 而不是未定义行为。 + +四类数据刻意分开，因为它们的写入频次与一致性要求完全不同： + +| 类别 | 方法 | 特点 | +|---|---|---| +| 任务快照 | `create` / `save` / `get` / `query` | 参与**乐观并发**（`status_version`） | +| 审计轨迹 | `append` / `history` | **只追加**，重复写入幂等 | +| 高频旁路 | `touch` | 进度与心跳，**不碰版本号**（否则会与状态写入持续冲突） | +| 控制指令 | `submit_command` / `claim_commands` / `complete_command` | 跨进程控制的媒介，带**租约** | + +### 自证契约 + +写一个新后端时别靠读文档猜语义——跑契约套件。**完整的接口规格见 +[持久层实现指南](docs/store-guide.md)**（14 个方法的逐条语义、并发要求、陷阱清单， +以及一个可执行的第三方后端范例）： + +```python +from chongming_task.testing.store_contract import StoreContract + + +class TestMyStore(StoreContract): + def make_store(self): + return MyStore(...) +``` + +套件**随包发布**且不依赖 pytest；内置实现跑的正是同一套用例（内存 / SQLite 文件 / +SQLite 内存 / Postgres，四份实例共用同一批断言——最后一份需要 +`CHONGMING_TEST_POSTGRES` 给出 DSN，没有就干净跳过）。 + +--- + +## 跨进程：一个进程跑，另一个进程看和管 + +`Controller(store=...)` 一旦给出，本实例就成为**可跨进程的一环**： + +| 方向 | 行为 | +|---|---| +| 写 | 每次状态迁移**写穿**到存储（乐观并发，带迁移历史） | +| 读 | `get` / `list_tasks` / `stats` 走存储，因此**能看到别的进程的任务** | +| 控制 | 对不属于本进程的任务，`stop` / `pause` / `resume` 写成一条**指令**，由持有者轮询取走 | + +真实输出（`examples/06_cross_process.py`，节选）： + +``` +--- ops-console 看到的状态（它并不持有这些任务）--- +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------ ------- ------- ------- -------- ------- +job-1 A-同步 7 running 1/1 0.5s - - +job-2 B-同步 28 running 1/1 0.5s - - + 它本地的任务句柄数：0 + +>>> ops-console 跨进程停止 job-1 + 下发结果：True + worker-1 已取走指令并停止该任务 + +--- job-1 的迁移历史（从存储读，可审计）--- + 1 pending -> queued by orchestrator:dependencies_resolved + 2 queued -> starting by scheduler:dispatch + 3 starting -> running by executor:launched + 4 running -> stopping by user:stop + 5 stopping -> stopped by runtime:stop_confirmed +``` + +### 三个刻意的设计点 + +1. **纯查询客户端不轮询指令。** 轮询线程只在**提交过任务**之后才启动。否则一个 + 只读的看板会把别人的停止请求抢走——即便它把指令交还，持续轮询也会让真正的持有者 + 反复错过。 +2. **取到不属于自己的指令是"交还"，不是"完成"。** 存储契约里有 + `release_command`：标记完成会让真正的持有者永远收不到这次请求（一个静默的丢指令）； + 什么都不做、只等租约过期又会被持续续租饿死。所以需要第三种结局。 +3. **存储故障不让任务失败。** 任务本身没问题，坏的只是可观测性。但也绝不静默—— + 写穿失败记进 `controller.persist_failures`，让"看板上的状态是旧的"有迹可循。 + +### 两个使用注意 + +- **跨进程的 `pause()` 返回 `False`。** 那不是失败，是**未确认**——跨进程没有回执 + 通道。要观测结果请轮询 `get()`。 +- **用完一定要关存储。** `controller.close(close_store=True)` 或自己 `store.close()`。 + 没关的 SQLite 连接会泄漏文件句柄，在 Windows 上还会让数据库文件无法删除。 + +--- + +## 命令行控制台 + +CLI 是**另一个进程**，所以它只能通过共享存储看和管——这正是持久层存在的理由。 + +```bash +# 一个终端里跑（前台执行到结束） +chongming-task --db tasks.db run --tasks myapp.tasks sync --arg tenant=A + +# 另一个终端里看和管 +chongming-task --db tasks.db ps +chongming-task --db tasks.db status demo-1 +chongming-task --db tasks.db stop demo-1 --reason "发布窗口" +``` + +真实输出（`examples/tasks_demo.py` + `PYTHONPATH=examples`，节选）： + +``` +$ chongming-task --db tasks.db ps +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------ ------- ------- ------- -------- ------- +demo-1 A-同步 7 running 1/1 7.4s - - + +$ chongming-task --db tasks.db history demo-1 +VER FROM TO AT TRIGGER ACTOR REASON +--- -------- -------- ------------------------- ---------------------------------- ------ ------ +1 pending queued 2026-10-02T14:01:23+00:00 orchestrator:dependencies_resolved system - +2 queued starting 2026-10-02T14:01:23+00:00 scheduler:dispatch system - +3 starting running 2026-10-02T14:01:23+00:00 executor:launched system - +4 running pausing 2026-10-02T14:01:40+00:00 user:pause user - +5 pausing paused 2026-10-02T14:01:40+00:00 runtime:pause_reached system - +6 paused running 2026-10-02T14:01:42+00:00 user:resume user - +7 running stopping 2026-10-02T14:01:44+00:00 user:stop user 发布窗口 +8 stopping stopped 2026-10-02T14:01:44+00:00 runtime:stop_confirmed system 发布窗口 +``` + +第 4–6 行是一次完整的暂停往返，第 7–8 行是停止——**谁按的、为什么按，都在轨迹里**。 + +### 退出码分三档 + +| 码 | 含义 | +|---|---| +| `0` | 成功 | +| `1` | 命令本身失败（参数错、任务不存在、存储打不开……） | +| `3` | 命令成功，但它查询/执行的**任务没有成功** | + +`3` 是给脚本判分支用的——把"运维脚本写错了"和"任务本身失败了"分开，脚本才可能做对 +判断。 + +### 任务定义来自模块，CLI 只传名字 + +```bash +chongming-task tasks --tasks myapp.tasks +``` + +模块必须定义 `register(controller)`。**一个能启动任意可调用对象的命令行入口等于任意 +代码执行**，所以这条约定必须从第一天立好。 + +子命令、错误路径与完整走查见 **[命令行控制台](docs/cli.md)**。 + +--- + +## DAG 管道 + +把多个任务按依赖关系串起来，由一个调度器推进：节点先停在 `blocked` 等上游，上游满足后 +才放行；注定跑不了的下游会被 `skipped`。 + +```python +from chongming_task import Controller, Pipeline +from chongming_task.policies import AggregationPolicy, JoinMode + +pipeline = Pipeline("nightly-etl") + +fetch = pipeline.node("fetch", "2026-10-02", node_id="fetch") +shards = [ + pipeline.node("transform", index, node_id=f"shard{index}", depends_on=[fetch]) + for index in range(3) +] +load = pipeline.node("load", node_id="load", depends_on=shards) # 默认屏障：全部分片成功 +pipeline.node( + "audit", node_id="audit", depends_on=[load], join=JoinMode.ALL_SETTLED +) # 不管成败都要跑 + +run = controller.submit_pipeline(pipeline) +outcome = run.wait(timeout=60) +print(outcome.status) # succeeded / failed / cancelled +``` + +真实输出（`examples/07_pipeline.py`）： + +``` +--- 刚提交时：下游都停在 blocked --- +NODE TASK STATUS DEPENDS_ON JOIN +------ --------- ------- -------------------- ------------- +fetch fetch running - all_succeeded +shard0 transform blocked fetch all_succeeded +shard1 transform blocked fetch all_succeeded +shard2 transform blocked fetch all_succeeded +load load blocked shard0,shard1,shard2 all_succeeded +audit audit blocked load all_settled +``` + +### 依赖由状态机表达，不是调度器自造的 + +`blocked` 是一个**真正的任务状态**（早已在转移表里），不是管道额外发明的标记。所以： + +* `PENDING → BLOCKED`（等待上游）→ `QUEUED`（上游满足）——每一步都走状态机、都写穿到 + 存储、都留下审计轨迹； +* 等待中的节点**不占并发额度**（它不在待执行队列里），一条长管道不会把额度锁死。 + +### 两个不同的轴，别混 + +| 概念 | 回答的问题 | +|---|---| +| `JoinMode`（逐节点） | **这一个节点何时该跑**：`ALL_SUCCEEDED` / `ALL_SETTLED` / `ANY_SUCCEEDED` | +| `AggregationPolicy`（整条管道） | **管道何时算结束、要不要连坐**：`FAIL_FAST` / `ALL_SETTLED` | + +### 三种结局必须分得清 + +| 状态 | 含义 | +|---|---| +| `skipped` | 上游没成功，它**本来就不该跑** | +| `cancelled` | 本来要跑，被 fail_fast **连坐**了 | +| `stopped` | 值班同学**手动停的** | + +把它们混成一个状态，事后就再也分不出"上游挂了"、"被连坐了"和"人停的"。 + +`CANCELLED` 刻意**不作为** fail_fast 的触发条件——它是连坐的*结果*，不是原因。否则用户 +主动取消整条管道会被记成"失败"，被连坐的节点还会再触发一轮连坐。 + +### 诚实的边界 + +* **管道结构在内存里，不落库。** 任务的函数与参数从未持久化，所以进程重启后一条跑到 + 一半的管道无法续跑——它的节点会被 `reconcile()` 如实标记，而不是被假装恢复。 +* **不支持 `quorum` 结算。** quorum 描述的是"同一件事的 N 份副本成功 k 份"，它属于扇出 + 分组而不是一条节点各异的 DAG。硬套上去只能得到一个没人能预测含义的数字，所以 + `submit_pipeline` 会明确拒绝并说明原因。 + +--- + +## 写一个任务：`@task` 与 `TaskContext` + +```python +from chongming_task import RetryPolicy, task + + +@task(tags={"io"}, retry=RetryPolicy(max_attempts=3)) +def extract(ctx, day: str) -> list[str]: + """按天抽取分区列表。""" + partitions = [f"{day}/part{i}" for i in range(6)] + for index, part in enumerate(partitions, start=1): + ctx.checkpoint() # 响应停止/暂停 + 刷新心跳 + ctx.progress(index, total=len(partitions), message=part) + ctx.log(f"处理 {part}") + return partitions +``` + +一次登记整个模块： + +```python +controller.register_module(myapp.tasks) +``` + +`ctx` 是任务与运行时之间**唯一**的协作入口： + +| 成员 | 作用 | +|---|---| +| `ctx.checkpoint()` | 协作点：响应停止与暂停，并刷新心跳。**不调用它就不响应停止** | +| `ctx.progress(done, total=, unit=, message=)` | 上报进度（高频旁路，不碰版本号） | +| `ctx.log(msg)` / `.warning` / `.error` | 走标准库 `logging`，logger 名 `chongming_task.task.` | +| `ctx.input(节点id, 默认值)` / `ctx.upstream` | 取**上游节点**的返回值（管道数据传递） | +| `ctx.artifact_dir` | 本次尝试的产物目录（按 attempt 隔离） | + +**返回值必须可 JSON 序列化**（它会写入存储）。违反时任务以 `result_invalid` **失败**—— +不会静默地变成"存储里停在 running、重启后被判 LOST"。大数据请传指针。 + +进度会进快照、进存储，并出现在 `ps` 的 `PROGRESS` 列： + +``` +TASK_ID NAME STATUS ATTEMPT ELAPSED PROGRESS FAILURE +---------------------- ------------------- --------- ------- ------- -------- ------- +2b4e5e326776:extract daily-etl/extract succeeded 1/2 0.3s 6/6 - +2b4e5e326776:transform daily-etl/transform succeeded 1/1 0.3s 6/6 - +2b4e5e326776:load daily-etl/load succeeded 1/1 0.2s 4/4 - +``` + +### 三个刻意的设计点 + +1. **`@task` 不产生全局注册表。** 它只把规格挂在函数对象上（`__chongming_task__`）， + 谁注册、注册到哪个控制器由你决定。全局注册表意味着 **import 即注册**，而 import + 可能发生在任何地方（包括测试收集阶段），那种隐式行为极难排查。 +2. **`progress()` 不刷新心跳。** 心跳是"我还在推进"的整体信号，进度是"推进到哪了"。 + 把心跳藏进进度里，会让一个只报进度、不做实际工作的任务看起来永远健康——那正是 + `idle` 超时要抓的东西。 +3. **失败的上游不传 `None`。** 只有成功的上游才出现在 `ctx.upstream` 里。塞一个 + `None` 会让下游以为"上游返回了 None"，而不是"上游没有值"。 + +--- + +## 重启恢复 + +进程意外退出后，存储里会留下一批永远停在 `running` 的任务。**新进程启动时先跑一次 +`reconcile()`**： + +```python +controller = Controller(store=SqliteTaskStore("tasks.db"), owner="worker-2") +report = controller.reconcile() +if report.lost: + log.warning("上次有 %d 个任务失去监督者", len(report.lost)) +``` + +```bash +chongming-task --db crash.db reconcile --dry-run # 先看一眼，不写任何状态 +chongming-task --db crash.db reconcile --reason "worker-old 崩溃" +``` + +``` +$ chongming-task --db crash.db reconcile --dry-run +OUTCOME TASK_ID WAS DETAIL +----------- ----------- ------- ----------------- +adoptable job-queued queued 没有执行载体，等待重新提交 +marked-lost job-crashed running 监督者已失联（进程重启后未再续租） +``` + +### 判据是租约，不是心跳 + +每个监督进程都会给自己负责的任务**续租**。租约过期才说明"没人在管了"： + +| 情况 | 处置 | +|---|---| +| 租约未过期 | 有活着的进程在管，**一动不动** | +| 租约过期 + 任务**占着执行资源** | 载体已消失，判 `LOST` | +| 租约过期 + 任务**不占资源**（排队/暂停/退避） | 只是没人接手，**不改状态** | +| 原本的监督者又活过来抢先写入 | 检测到版本冲突，**放手** | + +**绝不能用任务心跳当判据。** `last_heartbeat_at` 由任务自己的 `checkpoint()` 驱动， +一个长时间不调 checkpoint 的健康任务（纯 CPU 计算）心跳同样停滞——拿它判断进程存活， +会把一个正常任务判成 `LOST` 并**杀掉**。这就是租约与心跳必须分成两个字段的原因，也是 +持久层契约里 `save` 不许写租约列的原因。 + +### 它不会重新执行任务 + +这是能力边界，不是遗漏：**存储里记的是状态，不是配方**——任务的函数与参数从未落库。 +所以 `reconcile()` 做的是**如实标记**，让看板不再显示一个假的"运行中"；真正重跑要靠 +调用方自己重新 `submit`。 + +--- + +## 重试与超时 + +```python +from chongming_task.policies import ExponentialBackoff, RetryPolicy, Timeout + +outcome = run_task( + flaky_task, + retry=RetryPolicy(max_attempts=5, backoff=ExponentialBackoff(base=0.05, factor=2)), + timeout=Timeout(per_attempt=30, idle=10), +) +``` + +三档超时相互独立，取最早到期者： + +| 作用域 | 含义 | +|---|---| +| `start` | 从提交到真正开始执行的等待上限（一旦开始即失效） | +| `per_attempt` | 单次尝试的总时长 | +| `idle` | 距最近一次 `checkpoint()` 的静默时长——唯一能发现"卡住但没报错"的机制 | + +真实输出（`examples/03_retry_and_timeout.py`）： + +``` +== 重试：失败两次后成功 == + 终态 ：succeeded + 返回值 ：第 3 次尝试成功 + 尝试次数 ：3/5 + +== 重试：预算耗尽 == + 终态 ：failed + 尝试次数 ：2/2 + 失败归类 ：exception + +== 超时：单次尝试不超过 0.5 秒 == + 终态 ：failed + 失败归类 ：timeout_attempt + 可重试 ：True +``` + +--- + +## 状态模型速览 + +生命周期状态有 15 个，分为非终态与终态： + +``` +非终态 pending blocked queued starting running pausing paused retry_wait stopping +终态 succeeded failed stopped cancelled skipped lost +``` + +关键区分： + +- `stopped`（人工主动停止）与 `cancelled`（编排层级联取消）**责任方不同**，分开记录。 +- `lost` 表示**结果未知**（线程被放弃、载体失联），与"已停止"是两回事。 +- **超时不是状态**，是失败原因（`failure.kind = timeout_attempt` 等）。否则会逼出 + "已超时但正在重试"这类单枚举表达不了的情况。 +- 状态**只由状态机写入**，每次变更都递增 `status_version` 并留下一条带触发源的 + `Transition` 记录。 + +完整转移表与理由见 [设计文档 §2](docs/task-control-design.md)。 + +--- + +## 已知限制 + +1. **不调用 `ctx.checkpoint()` 就无法优雅停止或暂停**，只能等超时：线程执行器下那 + 个线程停不掉，进程执行器下则要等强杀（任务会被真的停掉，但已经产生的那部分副作用 + 无法撤销）。 +2. **线程无法被强制终止**。优雅窗口用尽后任务落 `lost`，且可能仍在后台产生副作用。 + 需要"停得住"的保证时换 `ProcessExecutor()`，见[用户指南 §7.4](docs/user-guide.md#74-进程执行器硬性停止--绕开-gil--崩溃隔离)。 +3. **暂停不释放执行槽位**（就地挂起）。 +4. **`idle` 超时衡量的是 `checkpoint()` 间隔**，不是"CPU 是否在干活"。长时间不 + checkpoint 的重计算会被判定为疑似卡死——这要求任务作者让 checkpoint 频率与 + `idle` 阈值相匹配。 +5. **跨机之后仍没有自动告警**：两个节点配了同一个桶（会重复执行）或某个桶没人管 + （任务静默堆积）都不会报错——缓解手段见 + [分布式设计说明 §11](docs/distributed-design.md)。 +6. **重试总是从头开始，任务不跨节点迁移**：没有用户态检查点 / 断点续传。请把任务写成 + 幂等或可断点续传。 + +--- + +## 文档 + +| 文档 | 内容 | +|---|---| +| [用户指南](docs/user-guide.md) | 概念、任务编写、控制语义、超时、执行器、FAQ | +| [命令行控制台](docs/cli.md) | 子命令、退出码、任务模块约定、完整走查（真实输出） | +| [持久层实现指南](docs/store-guide.md) | **换掉默认持久层要实现哪些接口**、逐方法语义、并发要求、陷阱清单 | +| [分布式设计说明](docs/distributed-design.md) | 多节点部署：边界表、自治节点 + 统一平面架构、风险、Postgres 落地状态 | +| [设计文档](docs/task-control-design.md) | 状态模型、转移表、分层架构、里程碑、风险 | +| [示例](examples/) | 十二个可直接运行的脚本 + 两个 CLI 任务模块（**多节点集成模板见 11 号**） | + +```bash +uv run python examples/11_best_practice.py # ← 想接入自己的应用，从这个开始 +uv run python examples/01_quickstart.py +uv run python examples/02_control.py +uv run python examples/03_retry_and_timeout.py +uv run python examples/04_controller.py +uv run python examples/05_buckets.py +uv run python examples/06_cross_process.py +uv run python examples/07_pipeline.py +uv run python examples/08_task_context.py +uv run python examples/09_long_running.py +uv run python examples/10_worker_walkthrough.py +uv run python examples/12_process_executor.py # 硬性停止 / 绕开 GIL / 崩溃隔离 +``` + +其中 **11 号是"最佳实践"主模板**：一个可以直接照抄的多节点任务服务 —— +部署参数与桶划分（`buckets_for_node` 连续分区）→ 任务定义 → `build_node()` +接线 → 启动 / 提交 / 观测 / 跨节点控制 / 收尾，全流程自带断言校验。 +**12 号**演示进程执行器：不响应停止的任务被真的杀掉、崩溃隔离、以及不可 pickle +的目标在提交那一刻被拦下。其余示例是单点机制演示，按需查阅即可。 + +`examples/tasks_demo.py` 不是可运行脚本，而是**给 CLI 用的任务模块**（演示 +`register(controller)` 约定）： + +```bash +PYTHONPATH=examples uv run chongming-task tasks --tasks tasks_demo +``` + +--- + +## 开发 + +```bash +uv run pytest -q # 测试 +uv run ruff check . # 静态检查 +uv run ruff format . # 格式化 +uv run mypy # 类型检查（strict） +``` + +质量门禁：测试全绿、ruff 无告警、mypy strict 在全部源文件上零问题。 diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..84795a7 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,506 @@ +# 命令行控制台 + +`chongming-task` 是**另一个进程**：它看不到任务进程的内存，只能通过共享存储查询与 +控制。这正是持久层存在的理由——没有它，`ps` 什么也看不到。 + +```bash +# 一个终端里跑任务（前台执行到结束） +chongming-task --db /var/lib/tasks.db run --tasks myapp.tasks sync --arg tenant=A + +# 另一个终端里看和管 +chongming-task --db /var/lib/tasks.db ps +chongming-task --db /var/lib/tasks.db status demo-1 +chongming-task --db /var/lib/tasks.db stop demo-1 --reason "发布窗口" +``` + +--- + +## 1. 命令总览 + +| 命令 | 作用 | +|---|---| +| `tasks --tasks MODULE` | 列出某个任务模块里注册的定义（名字、标签、描述） | +| `run TASK --tasks MODULE [--executor thread\|process]` | 提交并**前台执行**一个任务，直到落终态 | +| `ps [--all] [--status S] [--bucket N] [--pipeline RUN_ID] [--mine] [--limit N]` | 列出任务。默认只显示**未落终态**的 | +| `status TASK_ID` | 单个任务的详情 + 迁移历史 | +| `history TASK_ID` | 只列迁移历史（可审计轨迹） | +| `stats` | 各状态计数 | +| `buckets [--num-buckets N]` | 每个桶的任务数 | +| `reconcile [--dry-run] [--stale-after S] [--reason R]` | **重启恢复**：清理上次进程留下的僵尸任务 | +| `stop TASK_ID [--reason R]` | 请求停止 | +| `pause TASK_ID [--timeout S]` | 请求暂停 | +| `resume TASK_ID` | 恢复被暂停的任务 | + +全局选项： + +| 选项 | 说明 | +|---|---| +| `--db PATH_OR_DSN` | 任务存储：SQLite 文件路径，或 `postgresql://` 连接串（跨机部署；需要 `pip install "chongming-task[postgres]"`）。未给出时读环境变量 `CHONGMING_TASK_DB` | +| `--num-buckets N` | 集群的桶数。**必须与部署一致**——它决定 `--owned-buckets` 的合法性与 `buckets` 视图的行数；不给出按默认值 64 | +| `--owned-buckets SPEC` | 本节点负责的桶，如 `0-15,20,30`。给出后 `reconcile` 只清理这些桶、`ps --mine` 只看这些桶 | +| `--json` | 以 JSON 输出，便于脚本消费 | +| `--owner NAME` | 本客户端的标识，会写进它下发的指令里，便于事后追溯是谁按的停止 | + +### 退出码 + +| 码 | 含义 | +|---|---| +| `0` | 命令成功 | +| `1` | 命令本身失败（参数错、任务不存在、存储打不开、目录不存在……） | +| `3` | 命令成功，但它查询/执行的**任务没有成功** | + +`3` 是给脚本判分支用的： + +```bash +if ! chongming-task --db tasks.db run --tasks myapp.tasks load --arg day=2026-10-02; then + echo "任务没成功，别继续往下走了" >&2 + exit 1 +fi +``` + +> 注意 `1` 与 `3` 的区别：`1` 是"我连命令都没跑成"，`3` 是"命令跑成了，但任务本身 +> 没成功"。混在一起会让脚本无法区分"运维脚本写错了"和"数据质量有问题"。 + +--- + +## 2. 任务定义从哪里来 + +CLI **只传任务名，绝不接受代码**——一个能启动任意可调用对象的命令行入口等于任意 +代码执行。所以必须先有一个模块把定义登记进来： + +```python +# myapp/tasks.py +def sync_tenant(ctx, tenant: str, batches: int = 120) -> str: + """同步一个租户的数据。""" + for index in range(batches): + ctx.checkpoint() + ... + return f"{tenant} 同步完成" + + +def register(controller) -> None: + """约定：CLI 导入模块后会调用它。""" + controller.register("sync", sync_tenant, tags={"io"}) +``` + +然后 `--tasks myapp.tasks`。模块必须在 `PYTHONPATH` 上（或是一个已安装的包）。 + +仓库里的 `examples/tasks_demo.py` 就是一个可直接用的例子： + +```bash +PYTHONPATH=examples chongming-task tasks --tasks tasks_demo +``` + +``` +NAME TAGS MAX_ATTEMPTS DESCRIPTION +------------------------------------------------------------ +add cpu 1 两数相加。 +flaky - 1 前几次故意失败，用来演示重试。 +sync io 1 同步一个租户的数据。 +``` + +模块没有 `register` 时会给一句明确的话，而不是 `AttributeError`： + +``` +错误：任务模块 'json' 必须定义 register(controller) 函数。 +``` + +--- + +## 3. 完整走查 + +以下全部是**真实运行输出**（`examples/tasks_demo.py`，先在一个终端启动，再从另一个 +终端查询与控制）。 + +### 3.1 启动一个长任务（前台） + +```bash +chongming-task --db .cli-demo/tasks.db run --tasks tasks_demo sync \ + --arg tenant=租户A --arg batches=20000 --bucket-key tenant-A \ + --task-id demo-1 --label A-同步 +``` + +`run` 是**前台**的：它自己就是 worker，执行到任务落终态才退出。要后台跑就交给 +`nohup`、`systemd` 或你已有的进程管理器——CLI 不假装自己是守护进程管理器。 + +### 3.2 从另一个进程查询 + +```bash +chongming-task --db .cli-demo/tasks.db ps +``` + +``` +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------ ------- ------- ------- -------- ------- +demo-1 A-同步 7 running 1/1 7.4s - - +``` + +```bash +chongming-task --db .cli-demo/tasks.db status demo-1 +``` + +``` + task_id : demo-1 + name : A-同步 + status : running + bucket : 7 + tags : io + bucket_key : tenant-A + attempt : 1/1 + status_version : 3 + executor : thread + created_at : 2026-10-02T14:01:23+00:00 + started_at : 2026-10-02T14:01:23+00:00 + ended_at : - +last_heartbeat_at : 2026-10-02T14:01:31+00:00 + elapsed : 7.8s + progress : - + failure : - + retryable : - + overridden : False + +迁移历史： +VER FROM TO AT TRIGGER ACTOR REASON +--- -------- --------- ------------------------- ---------------------------------- ------ ------ +1 pending queued 2026-10-02T14:01:23+00:00 orchestrator:dependencies_resolved system - +2 queued starting 2026-10-02T14:01:23+00:00 scheduler:dispatch system - +3 starting running 2026-10-02T14:01:23+00:00 executor:launched system - +``` + +**`last_heartbeat_at` 只有在这里才有意义。** 它由任务自己的 `ctx.checkpoint()` 驱动， +所以一个"卡住但没报错"的任务会显示心跳停滞——这是唯一能发现死锁的信号。 + +`ps` 的 `PROGRESS` 列来自任务里的 `ctx.progress(...)`。它走**高频旁路**（不参与乐观 +并发），由持有任务的进程定期回写，因此可能有最多一个轮询周期的延迟： + +``` +TASK_ID NAME STATUS PROGRESS +---------------------- ------------------- --------- -------- +2b4e5e326776:extract daily-etl/extract succeeded 6/6 +2b4e5e326776:load daily-etl/load succeeded 4/4 +``` + +`stats` 与 `buckets`： + +``` +$ chongming-task --db .cli-demo/tasks.db stats +running 1 +-------------------- +total 1 +unfinished 1 + +$ chongming-task --db .cli-demo/tasks.db buckets --num-buckets 10 +BUCKET TASKS +-------------- + 0 0 + ... + 7 1 + ... +``` + +### 3.3 暂停与恢复 + +```bash +chongming-task --db .cli-demo/tasks.db --owner ops@console pause demo-1 +``` + +``` +已请求暂停 demo-1（用 status 观察是否真的挂起） +暂停**未确认**。若任务由别的进程持有，这是正常的——跨进程没有回执通道，请用 status 观察结果。 +``` + +第一行在 stdout，第二行在 stderr，**退出码仍是 0**。跨进程没有回执通道，所以 +`pause` 只能如实说"未确认"——这不是失败，而是它做不到的事就别假装做到。 + +紧接着 `ps` 能看到它确实停了： + +``` +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------ ------ ------- ------- -------- ------- +demo-1 A-同步 7 paused 1/1 18.4s - - +``` + +`resume` 之后回到 `running`。 + +### 3.4 停止，并留下原因 + +```bash +chongming-task --db .cli-demo/tasks.db --owner ops@console stop demo-1 --reason "发布窗口" +chongming-task --db .cli-demo/tasks.db ps --all +``` + +``` +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------ ------- ------- ------- -------- --------- +demo-1 A-同步 7 stopped 1/1 20.9s - cancelled +``` + +原因会完整落进审计轨迹，并且**区分了是谁按的**： + +``` +$ chongming-task --db .cli-demo/tasks.db history demo-1 +VER FROM TO AT TRIGGER ACTOR REASON +--- -------- -------- ------------------------- ---------------------------------- ------ ------ +1 pending queued 2026-10-02T14:01:23+00:00 orchestrator:dependencies_resolved system - +2 queued starting 2026-10-02T14:01:23+00:00 scheduler:dispatch system - +3 starting running 2026-10-02T14:01:23+00:00 executor:launched system - +4 running pausing 2026-10-02T14:01:40+00:00 user:pause user - +5 pausing paused 2026-10-02T14:01:40+00:00 runtime:pause_reached system - +6 paused running 2026-10-02T14:01:42+00:00 user:resume user - +7 running stopping 2026-10-02T14:01:44+00:00 user:stop user 发布窗口 +8 stopping stopped 2026-10-02T14:01:44+00:00 runtime:stop_confirmed system 发布窗口 +``` + +第 4–6 行是暂停往返，第 7–8 行是停止。**`REASON` 只在人为动作上出现**——机器自己的 +迁移没有原因要记。 + +### 3.5 `--json`：给脚本用 +```bash +chongming-task --db .cli-demo/tasks.db --json ps --all +``` + +```json +[{"attempt": 1, "bucket": 7, "bucket_key": "tenant-A", "created_at": "2026-10-02T14:01:23.570947+00:00", + "ended_at": "2026-10-02T14:01:44.485082+00:00", "executor": "thread", "failure": "{...}", + "last_heartbeat_at": "2026-10-02T14:01:44.430100+00:00", "max_attempts": 1, "name": "A-同步", + "outcome": null, "overridden": 0, "status": "stopped", "status_version": 8, + "tags": "[\"io\"]", "task_id": "demo-1"}] +``` + +JSON 与**存储行同构**（全部是原语），因此脚本消费它不需要了解 `datetime` / 枚举的 +序列化规则。三点约定： + +* 时间是 ISO 字符串（UTC）； +* 枚举是 `.value` 字符串； +* 嵌套结构（`failure` / `outcome` / `tags`）是 JSON **字符串**而不是对象——保持与 + 存储行一致，代价是脚本里要多一次 `json.loads`。 + +错误走 stderr，stdout 保持干净，所以 `--json` 的输出可以直接管道给 `jq`。 + +### 3.6 按管道查看 + +管道里的每个任务都带 `pipeline_run_id`（形如 `ead248368f4a:load` 的 task_id），因此 +可以整条一起看： + +```bash +chongming-task --db tasks.db ps --all --pipeline ead248368f4a +``` + +过滤**下推到存储层**（SQLite 侧有 `idx_tasks_pipeline_run` 索引），不是取回全部再筛。 +`--pipeline` 也可以与 `--status` 组合： + +```bash +# 这条管道里有哪些节点没成功 +chongming-task --db tasks.db ps --status failed --pipeline ead248368f4a +``` + +--- + +## 4. 多节点：`--owned-buckets` 与 `--num-buckets` + +多个节点各跑各的任务时，靠桶区分"哪一份是我的"。两个选项都必须与部署一致： + +```bash +# 只看"我这一份" +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 ps --all --mine + +# 每个桶多少任务；星号标出本节点负责的桶 +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 buckets +``` + +``` + BUCKET TASKS +---------------- +* 0 3 +* 1 0 + 2 0 +``` + +`reconcile` 会**限定在自己负责的桶内**： + +```bash +chongming-task --db tasks.db --owned-buckets 0-15 reconcile --dry-run +``` + +**`run` 同样受归属约束**：限定了 `--owned-buckets` 时，`--bucket-key` 是必需的——没有归属 +桶的任务不会有任何节点认领它，所以 CLI 在提交时就报错，而不是留下一个永远排队的任务： + +``` +$ chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 run \ + --tasks myapp.tasks worker --task-id orders-worker +错误：本节点只负责桶 [0, 1, ..., 15]，因此提交任务 'orders-worker' 时必须给出 bucket_key +——否则它没有归属桶，不会有任何节点认领它。 +``` + +这很重要：统一存储里有别的节点的任务，去动它既越界（那是它的责任）也没有意义（它的租约 +由它自己续）。 + +**桶归属的校验在做任何事之前完成**，包括 `ps` / `stats` / `buckets` 这些不构造 +`Controller`、直接查存储的命令。否则越界的桶号会被静默接受——筛选条件匹配不到任何东西， +看起来就像"这个桶没有任务"： + +``` +$ chongming-task --db tasks.db --owned-buckets 0-999 ps +错误：桶号越界：64；桶数是 64（合法范围 0..63）。若部署的桶数不是 64，请用 --num-buckets 指明。 +``` + +### 4.1 `run --executor process`：能被真的杀掉的执行方式 + +默认的线程执行器在"任务不响应停止"时只能放弃等待（记 `lost`，结果未知）。需要**停得住** +的保证时，让任务跑在独立子进程里： + +```bash +# 前台跑一个可能不配合的任务 +chongming-task --db tasks.db run --tasks tasks_demo sync --arg tenant=acme --executor process --graceful-stop 5 + +# 另一个终端：优雅窗口用尽后会 terminate() → kill()，任务被真的停掉 +chongming-task --db tasks.db stop --reason "跑太久" +``` + +代价写在这里，省得踩了才知道（与库的行为完全一致，见[用户指南 §7.4](user-guide.md#74-进程执行器硬性停止--绕开-gil--崩溃隔离)）： + +- **参数与返回值必须可 pickle**——`lambda`、闭包、局部函数都不行；违反时提交那一刻就报错； +- **任务函数必须在可导入的模块里**（`--tasks` 指定的那个模块正好满足），且该模块的顶层 + 不能有副作用； +- 停止与暂停有约 20ms 的轮询延迟（信号要过进程边界）。 + +`run` 输出里的 `pid` 就是那个子进程的进程号，`ps` 能直接对上号。 + +### 4.2 跨机部署：`--db` 换成连接串 + +SQLite 文件不能跨机共享。多台机器共用一套存储时，给 `--db` 一个 Postgres 连接串即可： + +```bash +pip install "chongming-task[postgres]" + +chongming-task --db 'postgresql://user:pw@db.internal:5432/chongming' \ + --num-buckets 64 --owned-buckets 0-15 ps --all --mine +``` + +**除连接串本身，其余一切都一样**：`--json`、退出码、`--owned-buckets` 校验、`reconcile` +的语义都不变。三张表（`tasks` / `transitions` / `commands`）在首次连接时自动创建，不需要 +手工迁移；当前角色没有建表权限时，它检测到表已存在就直接使用——**只读运维账号因此也能连**。 + +没装可选依赖时给的是安装命令，而不是一个 `ModuleNotFoundError`： + +``` +$ chongming-task --db postgresql://… ps +错误：--db 给的是 Postgres 连接串，但可选依赖未安装：PostgresTaskStore 需要可选依赖 psycopg 与 psycopg-pool。安装： + pip install "chongming-task[postgres]" +或：pip install "psycopg[binary]" psycopg-pool +``` + +连不上时同样只说该说的——DSN 里的**密码不会出现在任何错误信息里**： + +``` +$ chongming-task --db 'postgresql://someuser:s3cret@nope.invalid:5432/x' ps +错误：无法连接 Postgres（someuser@nope.invalid:5432/x）：failed to resolve host 'nope.invalid': [Errno 11001] getaddrinfo failed +``` + +--- + +## 5. 重启恢复：`reconcile` + +进程意外退出后，存储里会留下一批永远停在 `running` 的任务。**新进程启动时先跑一次 +`reconcile`**，否则看板会永远显示一个假的"运行中"。 + +真实输出。先看崩溃后的现场： + +``` +$ chongming-task --db crash.db ps +TASK_ID NAME BUCKET STATUS ATTEMPT ELAPSED PROGRESS FAILURE +----------- ------------ ------ ------- ------- ------- -------- ------- +job-queued 等待中的任务 - queued 0/1 - - - +job-crashed 夜间抽取 - running 1/1 600.4s - - +``` + +`job-crashed` 已经"运行"了 600 秒——它的进程其实十分钟前就没了。 + +**先预演**（这类操作会改状态，看一眼是值得的）： + +```bash +chongming-task --db crash.db reconcile --dry-run +``` + +``` +OUTCOME TASK_ID WAS DETAIL +----------- ----------- ------- ----------------- +adoptable job-queued queued 没有执行载体，等待重新提交 +marked-lost job-crashed running 监督者已失联（进程重启后未再续租） + +预演：有 1 个任务**会被**标记为 lost，1 个等待重新提交。未写入任何状态。 +注意：本命令不会、也无法重新执行任务——存储里记的是状态，不是参数。 +``` + +确认无误后真正执行： + +```bash +chongming-task --db crash.db reconcile --reason "worker-old 崩溃" +``` + +``` +已把 1 个僵尸任务标记为 lost，1 个任务无人接手（状态未改动，需要重新提交）。 +``` + +之后 `ps --all` 里 `job-crashed` 是 `lost`，而 `job-queued` **仍然是 `queued`**—— +它没占执行资源，只是没人接手，把它的排队事实改掉才是错的。原因也进了审计轨迹： + +``` +$ chongming-task --db crash.db history job-crashed +VER FROM TO AT TRIGGER ACTOR REASON +--- ------- ---- ------------------------- ---------------------- ------ ------------- +4 running lost 2026-10-02T14:10:47+00:00 runtime:heartbeat_lost system worker-old 崩溃 +``` + +### 两件必须清楚的事 + +1. **它不会、也无法重新执行任务。** 存储里记的是状态，不是配方：任务的函数与参数从未 + 落库。所以"重跑"只能靠你重新 `run`，而那需要你自己知道参数。 +2. **判据是租约，不是心跳。** 一个长时间不调 `checkpoint()` 的健康任务心跳也会停滞； + 若拿心跳判断进程存活，它会被误判丢失并**杀掉**。所以心跳与租约是两个独立字段。 + +`--stale-after` 只影响"从未续过租的任务"的宽限期（默认取租约时长的两倍），用来避开 +"刚提交、工作线程还没轮到第一次续租"这个窗口。 + +--- + +## 6. 错误路径 + +``` +$ chongming-task ps +错误：未指定任务存储。请用 --db PATH，或设置环境变量 CHONGMING_TASK_DB。 +CLI 是另一个进程，没有共享存储就看不到任何任务。 + +$ chongming-task --db .cli-demo/tasks.db stop typo-task-id +错误：没有找到任务 typo-task-id；无法停止。 + +$ chongming-task --db /nope/tasks.db ps +错误：数据库所在目录不存在：/nope。请先创建它，或改用别的路径。 +``` + +最后一条是刻意加的：SQLite 原本只会报 `unable to open database file`，完全看不出是 +目录不存在——这是最容易浪费时间的错误之一。 + +--- + +## 7. 桌面上的注意事项 +* **`run` 是前台进程。** 它退出时任务就没人管了。要长期跑请交给进程管理器。 +* **`pause` 跨进程返回"未确认"。** 想看结果请 `status`；`stop` 则是"请求已受理"。 +* **`stop` 一个不存在的任务会失败并留下零条指令。** 拼错的 id 不会制造孤儿指令。 +* **`ps` 默认不含已结束的任务。** 想看历史加 `--all`，或按状态过滤 `--status failed`。 +* **`--db` 必须显式给出或设环境变量。** 不会静默在当前目录建一个库——那类"我的任务 + 去哪了"的问题很难查。 + +--- + +## 8. 与库的关系 + +CLI 是 `Controller` 之上的一层**薄适配**：它做的全部事情就是解析参数、调 +`Controller` / `TaskStore`、渲染输出。所以： + +* 库能做的，CLI 不一定都暴露（比如 `wait_all`、`close_store`、`prune`）——直接在代码里用 + [Controller](user-guide.md#8-管理一批任务controller) 更灵活； +* CLI 不引入任何库没有的概念，也不绕过任何不变量（比如它不会直接改状态）。 + +替换持久层的步骤见 [持久层实现指南](store-guide.md)——CLI 会自动用上你的实现，因为它 +只依赖 `TaskStore` 契约。 diff --git a/docs/distributed-design.md b/docs/distributed-design.md new file mode 100644 index 0000000..bb4d73a --- /dev/null +++ b/docs/distributed-design.md @@ -0,0 +1,566 @@ +# 分布式设计说明 + +本文说明**多节点部署**的架构、边界与风险。 + +> **阅读顺序**：§2 的边界表一眼说清今天能做什么、不能做什么；§5 是选定架构；§11 是这个 +> 架构**新增的风险**；§17 记录了被否决的另一个方案及原因。 + +**一句话**：不是分布式集群，而是**多个自治的定时任务节点 + 一套统一的存储与控制平面**， +靠**桶**区分"哪一份是我的"。 + +--- + +## 1. 结论速览 + +| 层次 | 状态 | +|---|---| +| 稳定分片（桶） | ✅ 已实现：同一个键在任何进程、任何机器上落到同一个桶 | +| 节点归属（我负责哪些桶） | ✅ 已实现：`Controller(owned_buckets=...)` | +| 统一存储（多节点共享任务状态） | ✅ 已实现（默认后端是 SQLite **文件**，跨机需要换后端） | +| 统一控制（跨节点停止/暂停/查询） | ✅ 已实现：指令通道 + 统一 CLI | +| 判活与重启恢复 | ✅ 已实现：监督者租约 + `reconcile()` | +| 跨节点**执行**对方的任务 | ❌ **刻意不做**（见 §17） | +| 弹性伸缩 / 负载均衡 | ❌ 不做（定时任务有归属，不是无差别算力） | + +--- + +## 2. 边界表：今天能做什么 + +### ✅ 能做 + +| 能力 | 位置 | +|---|---| +| 同一个键稳定映射到同一个桶（跨进程、跨机器一致） | `policies.BucketPolicy`（blake2b，**不是**内置 `hash()`） | +| 桶号与原始业务键记在每条任务上 | `TaskSnapshot.bucket` / `.bucket_key` | +| 声明本节点负责的桶 | `Controller(owned_buckets=...)` | +| 提交时校验归属（防止任务写进别人的桶） | `Controller._check_bucket_ownership()` | +| 每桶独立限流 | `Controller(max_parallel_per_bucket=...)` | +| 按桶查询，**下推到存储层**且有索引 | `TaskQuery.buckets` / `idx_tasks_bucket` | +| 多进程共享任务状态（同一台机器） | `SqliteTaskStore`（WAL + 乐观并发） | +| 一个进程控制另一个进程的任务 | 指令通道：`submit_command` / `claim_commands` / `complete_command` / `release_command` | +| 崩溃后的清理，**限定在自己的桶内** | `reconcile()` + 监督者租约 | +| 换掉持久层而不改业务代码 | `TaskStore` 契约 + 随包发布的 `StoreContract` | +| 写穿失败后自动追上（不永久冻结） | 写穿缓冲（见 §14） | + +### ❌ 不能做 + +| 做不到的事 | 原因 | +|---|---| +| 两台机器共享 SQLite 文件 | 网络文件系统的锁不可靠。跨机必须换网络后端 | +| 节点执行另一个节点定义的任务 | **架构决定**：任务定义各节点自有，见 §5 / §17 | +| 任务在任意节点间漂移/迁移 | 归属是配置，不是调度结果 | +| 自动发现"某个桶没人负责" | 需要一份中心归属声明，见 §11.3 | +| 检测"两个节点配了同一个桶" | 同上 | + +**这里的关键区分是"共享"与"分工"**：本文解决的是共享（谁都能看到、谁都不会踩到别人）， +分工由**配置**表达——每个节点声明自己的桶，不靠协商。 + +--- + +## 3. 今天的三条基础机制 + +这三条已经跑通，也是后面一切的地基。 + +### 3.1 乐观并发：一个任务只有一个写入者 + +`save(snapshot, expect_version=...)` 是**单键 compare-and-set**。冲突就抛 +`ConcurrencyConflict`，**绝不自动重试**——因为每个任务的写入者唯一（它的持有进程）， +冲突意味着存在第二个写入者，那是设计缺陷。 + +`reconcile()` 直接依赖这条：发现版本冲突时它**放手**，因为"原来的监督者又活过来了"。 + +### 3.2 指令通道：控制意图怎么跨进程 + +一个进程无法给另一个进程的执行载体发信号，所以 `stop` / `pause` / `resume` 变成**指令**： + +``` +控制方 存储 持有者 + stop(T1) ──submit_command──▶ [c1: stop T1] ◀──claim── worker-1 + (租约 30s) │ + [c1: done] ◀──complete── 执行 run.stop() +``` + +取走指令时打**租约**：取走者崩溃后租约到期，别人可以重取。 +取到**不属于自己**的任务时**交还**（`release_command`）而不是标记完成——后者会静默丢 +指令，前者让真正的持有者马上能取到。 + +**这条通道本来就是跨节点的**，只是单机部署时收发双方碰巧同机。所以"统一控制平面"不需要 +任何新机制。 + +### 3.3 监督者租约：谁还在管这个任务 + +每个监督进程定期给自己负责的任务续租（`lease_owner` / `lease_until`，经 `touch` 高频 +旁路写入）。`reconcile()` 据此判定僵尸任务。 + +**它与任务心跳是两件事，不能互相推导：** + +| 字段 | 谁写 | 回答 | +|---|---|---| +| `last_heartbeat_at` | 任务自己的 `checkpoint()` | **任务在推进吗**（停滞 = 卡死） | +| `lease_until` | 监督进程续租 | **还有人在管它吗**（过期 = 监督者没了） | + +用任务心跳判断进程存活，会把一个长时间不调 `checkpoint()` 的**健康**任务误判为失联并 +判 `LOST`——把一个正常任务杀掉。这条在持久层契约里也钉死了：`save` **不得**写租约列。 + +--- + +## 4. 已经就位的地基里，哪些是**为**分布式而做的 + +这些今天就有价值，但它们的存在理由包含"将来要多节点"： + +1. **`blake2b` 而不是内置 `hash()`。** CPython 对 `str` 的 `hash()` 带进程级随机盐， + 同一个键在不同进程会落到不同的桶。后果是"同桶任务在同一台机器上"**静默**失效。 + 有跨进程用例守着（真实子进程 + 不同 `PYTHONHASHSEED`）。 +2. **桶号记在任务上，而不是每次重算。** 万一分桶策略或桶数变化，重算会把一个正在跑的 + 任务挪到别的桶，局部性与隔离假设随之悄悄失效。 +3. **`bucket_key` 留存。** 改桶数或改归属时，只有拿得到原始键才能重新分桶。 +4. **按桶查询下推 + 索引。** "取我拥有的那组桶"是这套架构里最高频的查询。 +5. **持久层契约可换。** 这是跨机部署的第 0 步——SQLite 文件不能跨机共享。 +6. **`owner` 标识**（默认 `主机名:pid`）写进指令里，便于事后追溯是谁按的停止。 + +--- + +## 5. 选定架构：自治节点 + 统一平面 + +**每个节点有自己的任务定义与触发方式；节点之间不互相执行对方的任务；但共用同一套存储与 +控制层，靠桶区分"哪一份是我的"。** + +``` + ┌──────────────────────────────────────────────┐ + │ PostgreSQL（统一存储） │ + │ tasks / transitions / commands │ + └───┬────────────────┬────────────────┬────────┘ + │ │ │ + ┌────────▼──────┐ ┌──────▼────────┐ ┌────▼──────────┐ + │ 节点 A │ │ 节点 B │ │ 运维 / CLI │ + │ 自己的任务定义 │ │ 自己的任务定义 │ │ 统一观测与控制 │ + │ 自己的定时触发 │ │ 自己的定时触发 │ │ --owned-buckets│ + │ 桶 {0-15} │ │ 桶 {16-31} │ │ --num-buckets │ + └───────────────┘ └───────────────┘ └───────────────┘ +``` + +| 维度 | 自治（每节点自有） | 统一（全局共享） | +|---|---|---| +| 任务定义（函数、默认重试/超时、标签） | ✅ | — | +| 触发方式（定时、事件、手动） | ✅ | — | +| 负责的桶（按业务划分，可多个） | ✅ 声明 | — | +| 存储（任务状态、审计轨迹、指令） | — | ✅ 一套 | +| 控制层（API、CLI、停止/暂停、reconcile） | — | ✅ 一套 | + +**没有调度器进程，也没有跨节点的任务派发。** 每个节点只做自己的事；统一平面提供的是 +**看得见**与**管得着**。 + +--- + +## 6. 桶在这个架构里的三个作用 + +1. **划界（核心）**——节点只接受、只清理自己负责的桶： + + * 提交时校验：任务落进别人的桶会**在提交那一刻报错**； + * `reconcile()` 只看自己的桶，不去动别的节点的任务。 + +2. **隔离与公平**——桶内并发限制（`max_parallel_per_bucket`），一个租户的积压不饿死 + 其他租户。这是分桶在**本地**就有的价值，不必等到多节点。 + +3. **将来迁移的最小单位**——`bucket_key` 已留存，改桶数或换归属都有依据。 + +--- + +## 7. 统一平面给了你什么 + +既然节点不跑对方的任务，"统一"的全部回报就是**可观测**与**可控制**： + +| 回报 | 靠什么 | +|---|---| +| 一个看板看全部，不必逐台登录 | 所有任务在一张表里，`ps` / `stats` 直接查 | +| 跨节点控制 | **已有**的指令通道：`stop` / `pause` / `resume` 别的节点正在跑的任务 | +| 统一的审计轨迹 | 所有状态迁移在一张表里：谁的、什么时候、为什么 | +| 重启恢复各自负责，但**对运维可见** | 每个节点只 `reconcile` 自己的桶，而"哪个桶有僵尸"全局可见 | +| **能发现"没人管"的任务** | 见 §11.2——这是统一平面**唯一能做**而分散存储做不到的事 | + +最后一条最重要：分散存储时，"某个桶的节点已经下线，它的任务在静默堆积"是**不可发现**的。 + +--- + +## 8. 节点归属：`owned_buckets`（已实现） + +```python +# 节点 A：负责 0..15 号桶（按业务划分，可以多于一个桶） +controller = Controller( + store=SqliteTaskStore("tasks.db"), + owner="node-a", + owned_buckets=range(0, 16), +) + +# 单机，或不构成划分的节点：不限定（默认） +controller = Controller(store=store) # owned_buckets=None +``` + +行为： + +| 成员 | 说明 | +|---|---| +| `owned_buckets` | 本节点负责的桶；`None` 表示不限定 | +| `owns_all_buckets` | 是否**不构成划分**：不限定，或显式覆盖了全部桶 | +| `owns_bucket(b)` | 某个桶是否归本节点（只回答归属，不回答"无桶任务能否提交"） | +| `submit()` | 限定时，任务必须落在自己的桶里，否则 `ConfigurationError` | +| `reconcile()` | 限定时，只清理自己桶里的僵尸任务 | + +**两条判据上的细节：** + +* **覆盖全部桶 = 不构成划分**，此时无桶任务同样会且只会由本节点执行，因此不再强制 + `bucket_key`（否则生成出来的"负责 0-63"配置会带来纯摩擦）。 +* **提交时是硬错误而不是警告**：落在别人桶里的任务不会有任何节点去跑它，它会永远停在 + `queued`，而且**没有任何报错**。与其让它静默堆积，不如在提交那一刻就说清楚。 + +启动时校验越界桶号（`0 <= bucket < num_buckets`）：放过去的症状同样是"任务永远排队且 +没有报错"，那是最难查的一类问题。 + +--- + +## 9. 命令轮询：声明了归属就从启动开始（已实现） + +`Controller` 在**声明了 `owned_buckets`** 时，构造那一刻就启动指令轮询线程。 + +否则有一个延迟生效的坑：别人 `stop` 一个属于本节点、当前还排在队列里的任务时，指令已经 +写进存储了，但本节点要等到**第一次提交**才会启动轮询——对一个定时任务节点来说，那可能是 +几分钟甚至几小时之后。 + +没有声明归属的实例（纯查询客户端）**仍然不轮询**，避免抢走别人的指令——这是早先就定下的 +规则，两条一起成立的判据是"你是不是一个真 worker"。 + +--- + +## 10. 运维视图（CLI） + +```bash +# 声明本节点负责的桶与集群桶数（两者都必须与部署一致） +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15,20 \ + ps --all --mine + +# 每个桶有多少任务；星号标出本节点负责的桶 +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 buckets + +# 重启恢复：只清理自己桶里的僵尸 +chongming-task --db tasks.db --owned-buckets 0-15 reconcile --dry-run +``` + +`--num-buckets` 必须显式给出（或确认默认值 64 与部署一致）：桶数是**部署范围的约定**， +各节点不一致会让同一个键落到不同的桶，而且不会有任何报错。 + +**桶归属的校验在做任何事之前完成。** 有些命令（`ps` / `stats` / `buckets`）直接查存储、 +不构造 `Controller`；如果把校验只放在 `Controller` 里，越界的桶号在这些命令上会被**静默 +接受**——筛选条件匹配不到任何东西，看起来就像"这个桶没有任务"。 + +--- + +## 11. 这个架构新增的风险（诚实） + +"各司其职"用配置换掉了协调，代价是**没有东西替你检查配置**。 + +### 11.1 配置漂移 + +| 情况 | 后果 | 今天能挡住吗 | +|---|---|---| +| 两个节点配了同一个桶 | 两个都可能执行同一个任务 | ❌ 挡不住。提交校验只能挡"写进别人的桶" | +| 某个桶没人配 | 它的任务永远 `queued`，**静默堆积** | ❌ 挡不住 | + +缓解手段： + +1. **提交时校验**（已做）能挡住"写进别人桶"这一类； +2. **监控"长时间未变化的 `queued` 任务"**——这是最有效的兜底，因为它直接盯住症状； +3. 把桶归属做成**一处生成、多处使用**的配置（配置即代码），从源头避免手写漂移。 + +### 11.2 孤儿任务 + +拥有某桶的节点下线后，它的任务没人跑，也**没人报警**——这是"各司其职"的固有代价。 + +**统一存储恰好能发现它**，而且只有统一存储能： + +``` +非终态任务 + 超过 N 分钟没有状态变化 + 该桶没有活着的节点 → 告警 +``` + +最后那个条件需要知道"谁负责哪个桶"（见 §11.3）。 + +### 11.3 桶归属需要一个**可审计的声明**（建议，未实现） + +今天 `owned_buckets` 只存在于各节点自己的配置里，中心看不到，因此无法检测重复与空洞。 +建议加一张**静态声明表**： + +```sql +CREATE TABLE node_buckets ( + bucket INTEGER PRIMARY KEY, + node TEXT NOT NULL, + updated_at TIMESTAMPTZ NOT NULL +); +``` + +节点启动时把自己的桶写进这张表（`ON CONFLICT (bucket) DO UPDATE`），并在冲突时**告警** +（"桶 7 同时被 node-a 与 node-b 声明"）。 + +> **⚠️ 注意它与被否决方案里的"桶租约表"是两件完全不同的事**（见 §17）： +> 那是**动态抢占**的运行时状态（在本架构里多余）；这是**静态归属的声明**（用于审计与 +> 告警，不参与抢占，也不决定谁能执行）。同名不同物，别混淆。 + +### 11.4 跨节点停止的送达 + +`stop` 一个属于别的节点、当前没在跑的任务会写一条指令，等那个节点取走。所以**节点必须 +持续轮询**——这一条已经在 §9 落地。 + +指令的语义仍然是"请求已受理"，不是"已停止"：要确认结果得查状态。 + +### 11.5 不能跨节点重跑 + +存储里没有参数（见 §13），所以谁也不能重跑别人的任务——**本来就是这样**，现在更明确： +连"将来做跨节点重跑"都不必考虑了。重跑只能由**拥有该任务定义的节点**自己做。 + +--- + +## 12. 做的时候不能破的不变量 + +这些是已经用测试和文档钉死的性质。破了任何一条，故障都是**静默**的。 + +| 不变量 | 破了会怎样 | +|---|---| +| **每个任务只有一个写入者** | 乐观并发冲突变成常态；`reconcile` 会不断"放手" | +| **`save` 不写租约列** | 一次状态迁移把租约覆盖回旧值 → 活着的任务被误判失联并杀掉 | +| **心跳必须先同步进内存快照**（`refresh_activity`） | `save` 会写心跳列，而快照不知道 `touch` 写了什么 → 看板上的心跳被覆盖回空 | +| **`release_command` 不标记完成** | 静默丢指令，真正的持有者永远收不到 | +| **`touch` 不递增版本号** | 高频写入与状态写入持续冲突，版本号变噪声 | +| **`save` 的版本号单调但不连续** | 写穿缓冲恢复后要能一次跳进；要求"只接受 +1"等于重新引入永久冻结 | +| **`CANCELLED` 不是 fail_fast 的触发条件** | 用户主动取消被记成失败；连坐再触发连坐 | +| **判活靠租约，不靠任务心跳** | 长时间不 checkpoint 的健康任务被误杀 | +| **`blake2b`，不用内置 `hash()`** | 跨进程分桶静默错位 | +| **桶数全局一致** | 同一个键落到不同的桶，无任何报错 | +| **返回值为 JSON 可序列化**（见 §14） | 存储里停在 running，重启后被判 `LOST` | + +### 关于时钟 + +租约判定用的是**本地时钟**（`snapshot.lease_until > self._clock()`），而 `lease_until` +由另一个进程用自己的时钟算出。同机或 NTP 良好时不是问题（TTL 默认 30 秒，NTP 偏差通常 +毫秒级）。**迁移到 Postgres 后建议改用存储侧 `now()` 做租约裁定**（见 §15），那样这个 +假设就彻底消失了。 + +--- + +## 13. 语义上的诚实：它是 at-least-once + +必须说清楚的事实： + +* 被 `stop` / `cancel` 的任务**可能已经产生了副作用**。系统把它记为 `stopped` / + `cancelled`，而不是假装什么都没发生。 +* 进程崩溃时，任务可能**仍在后台运行**。此时状态是 `LOST`——"结果未知"，**不是** + "没跑过"。把"未知"说成"已停止"会让人放心地重跑，而副作用已经发生两次。 +* `reconcile()` **不会重新执行**任何任务（存储里只有状态，没有参数）。重跑要靠调用方 + 自己 `submit`，而那需要调用方知道参数。 +* 因为节点不跑对方的任务，**跨节点的重复执行只可能来自"两个节点配了同一个桶"** + （§11.1），而不是来自租约竞争。 + +一句话：**它报告真相，不提供幻觉。** + +--- + +## 14. 已修复的缺陷（记录） + +这两处曾是真实的数据损坏，现在都有回归用例。记录在这里，因为它们解释了若干不变量的来历。 + +### 14.1 一次写穿失败会让任务在存储里**永久冻结** + +`_persist` 曾用 `expect_version = transition.version - 1`，这个式子**假设此前每一次迁移 +都成功落库**。一旦某次 `save` 失败（网络抖动、连接被断、返回值不可序列化……），存储停在 +k-1 而内存已到 k；此后**每一次**写入都因版本对不上而冲突——一次瞬时故障让该任务的存储 +状态永久冻结，最后被 `reconcile()` 判成 `LOST`。 + +修法：记录**上次确实写进存储的版本**（不是上次算出来的版本），未落库的迁移进**有界缓冲**， +下次迁移时一并补上；`close()` 时再尽力补一次。用"上次确实写进去的版本"**仍然**冲突时， +判定为**外部写入者**并**停止写穿**（`Controller.desynced_tasks`），而不是反复重试或覆盖 +对方。 + +``` +修复前 修复后 +进程内 succeeded 进程内 failed（result_invalid） +存储里 running version=3 存储里 failed version=4 +历史 [queued, starting, running] 历史 [queued, starting, running, failed] + +（一次网络抖动） +存储里 starting version=2（永久） 存储里 succeeded version=4 +历史 [queued, starting] 历史 [queued, starting, running, succeeded] +``` + +### 14.2 非法返回值会静默损坏存储状态 + +返回值会写入存储（`outcome` 列），因此必须 JSON 可序列化。不校验的话，失败发生在**写穿 +路径**上——而那条路径的失败只被记录、不会让任务失败，结果是进程内显示 `succeeded`、 +存储停在 `running`，随后 `reconcile()` 把一个**真正成功过**的任务判成 `LOST`。 + +修法：`TaskRun._settle()` 在 `mark_succeeded` **之前**校验，不可序列化就让任务**失败** +（失败类型 `result_invalid`），并且不重试（同一个值重试多少次都一样）。 + +> 顺带修掉的另一个独立缺陷：`CancellationToken.checkpoint()` 只在*进入*暂停分支时设置 +> `_ack`，于是 `resume()` 之后紧接着的 `request_pause()` 会因为任务线程仍在等待循环里 +> 而**永远得不到确认**——表现是"暂停 → 恢复 → 再暂停"里第二次暂停永久失效。 + +--- + +## 15. PostgreSQL 迁移建议 + +替换默认持久层（SQLite 文件）是跨机部署的**第 0 步**：SQLite 的网络文件系统锁不可靠， +不要跨机共享。 + +> **已实现**：`chongming_task.stores.postgres.PostgresTaskStore` +> （`pip install "chongming-task[postgres]"`；核心包仍是零依赖——没装驱动时它惰性报出安装 +> 命令）。CLI 同样认连接串：`--db 'postgresql://…'`。§15.2 的硬要求全部满足（`StoreContract` +> 在真库上全绿），§15.4 的三条并发用例已固化在 `tests/integration/test_postgres_store.py` +> 里（不给 DSN 就跳过，不会默默变绿）。 +> +> 本节其余部分保留为**设计依据**：它解释了为什么是这些要求，以及实现里**刻意偏离**了 +> 哪些建议——偏离处都记在 §15.1 与 §15.3 的注里，不留"文档说 A、代码做 B"的暗角。 + +### 15.1 Postgres 解锁了什么 + +| 能力 | 对现有设计的直接影响 | +|---|---| +| **服务端 `now()`** | 租约裁定不再依赖"各节点时钟有界漂移"（§12 的隐患直接消失） | +| **`SELECT … FOR UPDATE SKIP LOCKED`** | 认领不再需要"先查后写"，多个 worker 互不阻塞 | +| **`JSONB` + GIN** | 标签精确匹配不必再用 `LIKE '%"io"%'` 这种权宜之计 | +| **事务 + 行锁** | 乐观并发从"唯一防线"降级为"第二道保险" | + +**时间处理建议**：把"业务时间"与"租约裁定时间"分开——`created_at` / `started_at` / +`ended_at` / `last_heartbeat_at` 仍用**应用时钟**（`Clock` 可注入 → 可测试）；租约的 +"过期了吗"一律用**存储侧 `now()`**。 + +> **实现时的偏离（重要，也是上表里唯一没兑现的一行）**：租约裁定用的仍是**调用方传入的 +> `now`**，没有换成存储侧 `now()`。原因在契约：`claim_commands(*, now, lease_seconds, …)` +> 的时间由调用方给出，`StoreContract` 正是靠注入一个固定时钟才能可重复地测"租约到期 / +> 未到期"；后端若自作主张改用服务端时钟，契约测试与"时钟可注入"这条设计会同时失效。 +> 所以 **§12"关于时钟"的假设依旧成立**（各节点时钟需有界漂移，租约 TTL 30 秒足够宽容）， +> 要真正兑现这一行得先改协议——让契约接受"时间由存储决定"，那是独立的一次变更。 +> +> 其余三项都已用上：`SKIP LOCKED` 认领、`JSONB` + GIN 精确标签、`TIMESTAMPTZ` 行锁事务。 + +### 15.2 契约上的硬要求 + +| 要求 | 为什么 | +|---|---| +| `save` 的"校验 + 写入"必须原子 | 否则跨进程下版本检查形同虚设 | +| **版本号单调但不连续**：只要求 `expect_version == 行里的版本` | 写穿缓冲恢复后会一次跳进；若后端要求"只接受 +1"就等于重新引入 §14.1 的冻结 | +| 不要自动重试冲突 | 冲突意味着存在第二个写入者，是设计缺陷 | +| `claim_commands` 的"挑选 + 打租约"必须原子 | 否则同一次停止被下发两次 | +| `append` 幂等（`task_id` + `version`） | 崩溃恢复与写穿补写都会重放 | +| **`save` 不写租约列** | 否则活着的任务被误判失联并杀掉 | + +### 15.3 Schema 与索引建议 + +- 时间列一律 `TIMESTAMPTZ`，**不要** `timestamp`（后者无时区，跨机必然出错）。 +- `tags` / `progress` / `failure` / `outcome` 改 `JSONB`：标签查询变成 + `WHERE tags @> '["io"]'::jsonb` + GIN 索引。 +- **未完成行用部分索引**（认领类查询只关心未完成的，而表在持续变大）： + +```sql +CREATE INDEX ON commands (created_at) WHERE finished_at IS NULL; +CREATE INDEX ON tasks (bucket, created_at) WHERE status = 'queued'; +CREATE INDEX ON tasks (lease_until) WHERE status IN ('starting','running','pausing','stopping'); +``` + +> **两处没照做的索引（有意为之）**：`(lease_until) WHERE status IN (…)` 与 +> `(bucket, created_at) WHERE status = 'queued'` 都**没有**建。不是省事，而是当前代码路径 +> 用不上它们，建了就是纯写入开销： +> +> * 租约过期判定发生在 **Python 里**——`Controller.reconcile()` 用 `store.query()` 取回 +> 非终态任务后逐条对照 `lease_until`，SQL 里没有 `lease_until` 谓词，索引无从命中； +> * `query()` 是通用查询，拼出来的是 `status = ANY($1)`——即使只传一个值，规划器也无法 +> 证明它等价于 `status = 'queued'`，部分索引不会被选中。 +> +> 什么时候该加：把租约判定下推成 `WHERE lease_until < …`（那是一次查询接口变更），或给 +> `query()` 加"已知单一状态"的快路径。在那之前，建成的 `idx_tasks_status` / +> `idx_tasks_bucket` / `idx_tasks_created_at` 覆盖的是现有查询形状。 + +- `transitions` 只增不减：按月分区或定期归档。`prune()` 在规模上不够用。 +- **连接池**：`TaskStore` 会被多线程共享（SQLite 实现用 `check_same_thread=False` + 自己的 + 锁）。Postgres 实现应当用真的连接池，并把 `close()` 的语义写清楚。 + +### 15.4 契约测试的盲区：并发测试必须自建 + +`StoreContract` 只测**单连接**语义——设计使然（它不依赖任何外部环境，所以能随包发布）。 +Postgres 后端必须**额外**自证： + +| 要验证的 | 期望 | 已固化的用例 | +|---|---|---| +| N 个连接并发 `save` 同一任务 | 恰好一个成功，其余 `ConcurrencyConflict` | ✅ 8 线程 + 屏障，断言"1 个 ok + 7 个 conflict" | +| N 个连接并发 `claim_commands` | 同一条指令不会被两个连接取到 | ✅ 8 个消费者抢 20 条指令，断言无重复且全部认领 | +| `pg_terminate_backend` 杀掉持有者 | 租约仍能正常过期并被接管 | ✅ 杀连接后：期限内取不到、过期后可接管、池自愈 | + +用例在 `tests/integration/test_postgres_store.py`（`CHONGMING_TEST_POSTGRES` 给 DSN 才跑）。 +**不给 DSN 是 skip 而不是 pass**——"跨进程没测"这件事必须显式可见。 + +> 另一个已经踩过的坑：**`InMemoryTaskStore` 直接持有对象、从不序列化**，所以"内存后端 +> 能跑"不等于"真后端能跑"。类型/序列化类的契约必须在会序列化的后端上测。 + +--- + +## 16. 落地清单 + +| 阶段 | 内容 | 状态 | +|---|---|---| +| 0 | 修掉写穿永久冻结与非法返回值（§14） | ✅ 已完成 | +| 1 | 节点桶归属：`owned_buckets` + 提交校验 + `reconcile` 范围 | ✅ 已完成 | +| 2 | CLI：`--owned-buckets` / `--num-buckets` / `ps --mine` / `buckets` 标记 | ✅ 已完成 | +| 3 | 声明归属的节点从启动就轮询指令（§9） | ✅ 已完成 | +| 4 | 统一存储换成 Postgres（§15） | ✅ 已完成：`PostgresTaskStore`（可选 extra，契约 + 并发用例全绿） | +| 5 | 桶归属的中心声明 `node_buckets` + 重复/空洞告警（§11.3） | 待做：运维需求 | +| 6 | 孤儿任务告警（§11.2） | 待做：运维需求 | + +阶段 4 已完成，多节点可以跨机了（共用一套 Postgres）：各节点仍是自治的，靠 `owned_buckets` +划界，跨节点控制走指令通道。**阶段 1–3 在单机上也有价值**：提交校验能挡住配错桶， +`reconcile` 的范围更准，CLI 能看"我这一份"。 + +跨机之后仍未解决的是**运维可见性**（阶段 5 / 6）：两个节点配了同一个桶、或某个桶没人管， +现在依然不会自动报错——那两条是纯运维需求，与存储后端无关。 + +--- + +## 17. 附：被否决的方案——"任意节点可跑任意任务" + +早先设计过一个真正的集群方案：提交者把任务写进共享存储，**任意**节点都能认领并执行它， +节点通过成员表 + rendezvous 哈希分配桶，桶带租约，支持弹性伸缩与抢占接管。 + +**它被否决，因为它解决的问题不是这里的问题。** + +| 能力 | 集群方案 | 自治方案 | +|---|---|---| +| 需要持久化"配方"（函数名 + 参数） | ✅ 必须——别的节点要能重建这次执行 | ❌ 不需要——任务只在定义它的节点上跑 | +| 需要跨节点原子认领（`FOR UPDATE SKIP LOCKED` + 在事务里递增 attempt） | ✅ 必须（防重复执行） | ❌ 不需要（没人认领别人的任务） | +| 需要成员表、心跳、租约、epoch fencing | ✅ 必须 | ❌ 不需要 | +| 需要 rendezvous/一致性哈希分配 + 再均衡 | ✅ 必须 | ❌ 不需要（归属是配置） | +| 需要处理"接管时的在途任务归属" | ✅ 必须 | ❌ 不需要 | +| 弹性伸缩 / 算力池化 | ✅ 有 | ❌ 没有 | +| 跨机部署的复杂度 | 高 | 低（只需换存储后端） | + +**根因**：定时/业务任务**是有归属的**——谁的数据谁处理，谁的定时谁触发。它不是一个 +无差别的算力池，不需要"随便找台机器跑"。为它建集群，等于用一整套分布式协调去换一个 +用不到的弹性。 + +**从那个方案里仍然成立、值得记住的结论**（它们与架构无关，是分布式本身的性质）： + +1. **正确性应来自"认领任务的原子 CAS"，而不是"桶的归属"**——归属只是优化。自治方案里 + 这条依然成立：跨节点重复执行来自配置冲突，而不来自租约竞争。 +2. **`attempt` 的递增必须在认领事务里**，否则"认领后就崩溃"的任务会被无限重试。 +3. **执行任务期间绝不能持有事务或行锁**——任务可能跑几十分钟。 +4. **不要用 `LISTEN/NOTIFY` 当投递机制**（不持久、payload 有 8KB 上限、同事务内相同通知 + 会去重）；它只能当延迟优化，轮询必须保留为真相来源。 +5. **不要用 advisory lock 表达持久所有权**——它是会话级的，连接池会把它泄漏给后来的请求。 +6. **at-least-once 是这类系统的常态**（§13），任务作者要保证幂等。 + +--- + +## 18. 明确不做的事 + +* **不做跨节点的任务执行**（节点只跑自己的任务）。 +* **不做配方落库 / 跨节点认领 / 成员表 / rendezvous / 再均衡 / fencing**——见 §17。 +* **不做推送式调度器**（没有中心调度进程）。 +* **不做分布式信号量**：桶亲和性已经给了近似全局的每桶并发上限。 +* **不做资源感知调度**：不感知 CPU/内存/GPU，不做装箱。 +* **不做 exactly-once**：见 §13。 +* **不做跨节点 DAG 协调**：管道结构不落库，分布式 DAG 是另一个数量级的问题。 +* **不做多租户安全隔离**：分桶给的是**公平性**与**亲和性**，不是安全边界。 diff --git a/docs/store-guide.md b/docs/store-guide.md new file mode 100644 index 0000000..cb5852c --- /dev/null +++ b/docs/store-guide.md @@ -0,0 +1,438 @@ +# 持久层实现指南 + +**换掉默认持久层需要实现哪些接口**，以及每个接口的语义要求。 + +本文是**规格**，不是介绍。照着做即可替换内置的 SQLite 实现；有可执行的范例和可执行的 +契约测试，两者都在仓库里。 + +--- + +## 一句话版 + +实现 `TaskStore` 协议里的 **14 个方法**，然后用 +`chongming_task.testing.store_contract.StoreContract` 自证。不需要继承、不需要注册。 + +```python +from chongming_task import Controller +from chongming_task.stores.protocol import TaskStore # 结构化协议，无需继承 + + +class MyStore: # 只要方法齐备、语义正确 + ... + + +controller = Controller(store=MyStore(...)) # 直接塞进去 +``` + +`Controller` 只做**结构化检查**（`TaskStore` 是 `runtime_checkable` 的 `Protocol`）， +所以任何满足契约的对象都能直接用。继承它、注册它都不必要。 + +--- + +## 1. 方法清单 + +| 分组 | 方法 | 一句话职责 | +|---|---|---| +| **任务快照**（参与乐观并发） | `create(snapshot)` | 写入新任务 | +| | `save(snapshot, *, expect_version)` | 按版本更新，冲突即抛错 | +| | `get(task_id)` | 读单个 | +| | `query(query=None)` | 按条件查询 | +| | `counts()` | 各状态计数 | +| **审计轨迹**（只追加） | `append(transition)` | 追加一条迁移，幂等 | +| | `history(task_id, *, since_version=0)` | 读迁移历史 | +| **高频旁路**（不碰版本号） | `touch(task_id, *, progress=None, heartbeat_at=None)` | 只更新进度与心跳 | +| **控制指令**（跨进程） | `submit_command(command)` | 写入一条待执行指令 | +| | `claim_commands(*, now, lease_seconds, limit=50)` | 取走一批并打租约 | +| | `complete_command(command_id, *, now, outcome=None)` | 标记完成 | +| | `release_command(command_id, *, now)` | **交还**（取错了或没认领） | +| **运维** | `prune(*, keep_terminal)` | 丢弃最旧的终态任务及其历史 | +| | `close()` | 释放资源 | + +代码即真相：`src/chongming_task/stores/protocol.py`。本文与它的方法集合由测试 +（`tests/integration/test_custom_store.py`）强制保持一致——协议加了方法却没写进本文， +或者本文写了一个不存在的名字，测试都会失败。 + +--- + +## 2. 逐方法语义 + +### 2.1 任务快照 + +#### `create(snapshot)` + +写入一个新任务。**`task_id` 已存在必须抛错**（`StoreError`），不要静默覆盖。 + +> 为什么静默覆盖很危险：任务已经在跑，第二次 `create` 会把它的状态重置回 +> `PENDING`，而正在运行的执行载体不会因此停下——内存与存储从此永久分叉。 + +#### `save(snapshot, *, expect_version)` + +按乐观并发更新。`expect_version` 是调用方写入前读到的 `status_version`；不一致必须抛 +`ConcurrencyConflict`。 + +**"校验 + 写入"必须是原子的一步。** 关系库里它应当是一条 +`UPDATE ... WHERE task_id = ? AND status_version = ?`；拆成"先查后写"必然在跨进程下 +产生竞态。KV 后端用 `WATCH/MULTI`、Lua 脚本或 CAS。 + +**不要自动重试冲突。** 每个任务的写入者唯一（它的持有进程），冲突意味着存在第二个 +写入者——那是设计缺陷，掩盖它比报出来危险得多。 + +#### ⚠️ 版本号**单调但不连续** + +`expect_version` 是调用方认为存储里当前的版本。它**不保证**等于 `实际版本 - 1`： + +写穿失败会把迁移攒进缓冲，恢复后**一次跳进**（比如存储从 version 2 直接变成 4，中间的 +3 只在迁移历史里出现）。所以后端**不得**假设： + +* 版本号连续（每次 +1）； +* `expect_version == 当前行版本`（这是对的）但 `写入的快照版本 == expect_version + 1`（这 + 是**不**成立的）。 + +后端唯一要保证的是：`expect_version` 与行里的 `status_version` 相等时才允许写入。 +把它改写成"只接受 +1 的推进"会让写穿缓冲永远补不上，等于重新引入"一次抖动永久冻结"。 + +#### `get(task_id)` + +读单个任务，不存在返回 `None`（**不要抛异常**：查询不存在的任务是正常操作）。 + +#### `query(query=None)` + +按条件查询，按 `created_at` 升序（`newest_first=True` 则降序）。`None` 表示不限制。 + +**过滤要下推到后端**，不要取回全部再筛——`buckets` 那条尤其重要，它是分布式下 +"取我拥有的那组桶"的高频查询。SQLite 侧为它建了索引。 + +需要支持的过滤条件：`statuses` / `name` / `name_prefix` / `tag` / `buckets` / +`pipeline_run_id` / `limit`。其中 `pipeline_run_id` 是"把这条管道的节点整条列出来" +的入口——管道恰恰是最需要按整体查看的东西，没有它就只能全表拉回来自己筛。 +`tag` 必须**精确**匹配（`io` 不能命中 `ios`）。 + +#### `counts()` + +返回 `{TaskStatus: 数量}`。后端能用 `GROUP BY` 就别把全部记录读出来数。 + +### 2.2 审计轨迹 + +#### `append(transition)` + +追加一条迁移记录。**必须按 `task_id` + `version` 幂等**：崩溃恢复会重放，重复写入 +不应产生重复行。关系库用 `INSERT OR IGNORE` / `ON CONFLICT DO NOTHING`。 + +#### `history(task_id, *, since_version=0)` + +返回 `version > since_version` 的记录，**按版本升序**。`since_version` 是增量拉取的 +入口（看板轮询靠它）。 + +### 2.3 高频旁路 + +#### `touch(task_id, *, progress=None, heartbeat_at=None, lease_owner=None, lease_until=None)` + +只更新这几个字段，**不校验也不递增 `status_version`**。 + +> 这是整个契约里最容易做错的一条。进度可能每秒几十次；它若参与乐观并发，会与状态 +> 写入持续冲突，把版本号变成噪声，最终表现是"状态老是写不进去"，但没有任何报错指向 +> 真正的原因。 + +任务不存在时**静默忽略**（心跳晚到一步不该报错）。参数全为 `None` 时可以什么都不做。 +每个参数独立可选，不要要求调用方凑齐。 + +**两个信号必须分开，不能互相推导：** + +| 参数 | 谁在写 | 回答的问题 | +|---|---|---| +| `heartbeat_at` | 任务自己的 `checkpoint()` | **任务在推进吗**（停滞 = 卡死） | +| `lease_owner` / `lease_until` | 监督进程定期续租 | **还有人在管它吗**（过期 = 监督者没了） | + +拿任务心跳去判断"进程是否还活着"是错的：一个长时间不调 `checkpoint()` 的健康任务 +（纯 CPU 计算）心跳同样会停滞，于是重启恢复会把它**误判为丢失并杀掉**。反过来拿租约 +判断卡死也不对。实现必须原样存这两个信号，不要在中间做任何换算。 + +#### ⚠️ `save` **不得**写这两个租约列 + +这条单列出来，因为它是本契约里第二个"同一行两条写入路径"的坑（第一个是心跳）。 + +`touch` 写入的租约值**不在**调用方的内存快照里。若 `save` 把整份快照写回去，就会把 +租约覆盖成快照里的旧值（通常是 `None`），于是： + +* 一个活得好好的任务看起来"没有监督者"； +* 下一次 `reconcile()` 把它判成 `LOST`——**把一个正常任务杀掉**； +* 而且只在重启恢复时才暴露，平时完全看不出来。 + +所以 `save` 必须**显式跳过**租约列： + +```python +# SQLite 实现：save 的列集合 = 全部列 - 租约列 +_SAVE_COLUMNS = tuple(c for c in _TASK_COLUMNS if not c.startswith("lease_")) + +# 内存实现：整体替换时把租约从旧值带过来 +replace(snapshot, lease_owner=current.lease_owner, lease_until=current.lease_until) +``` + +`StoreContract` 里有一条用例（`test_save_does_not_clobber_the_lease`）专门盯这件事， +三个内置实现都跑它。**它抓出过真实缺陷**：内存实现最初就是整体替换，契约测试当场 +失败。 + +### 2.4 控制指令 + +#### `submit_command(command)` + +写入一条指令。**存储层只传递，不解释、不执行**——执行者是持有该任务的进程。 + +#### `claim_commands(*, now, lease_seconds, limit=50)` + +取走一批可执行指令并打上租约。可被取走的条件：**未完成**，且**无租约或租约已过期**。 + +**"挑选 + 打租约"必须是原子的。** 否则两个进程会同时取到同一条指令，于是同一次停止 +被下发两次（对停止还算无害，对将来的"重试"就是灾难）。SQLite 用 +`BEGIN IMMEDIATE` + `COMMIT`；Postgres 下标准做法是 +`SELECT ... FOR UPDATE SKIP LOCKED` 再 `UPDATE`（内置 `PostgresTaskStore` 就是这么做的， +见 §6）。 + +按 `created_at` 升序取，并遵守 `limit`。 + +#### `complete_command(command_id, *, now, outcome=None)` + +标记完成，此后不再被取走。`outcome` 是执行结果的摘要（如 `"stopped"` / +`"already-terminal"`），用于事后追溯。 + +#### `release_command(command_id, *, now)` + +**交还**一条已取走但无人认领的指令——清除租约，但不标记完成。 + +这个方法看似多余，实则不可省。多进程下任何能取指令的进程都可能取到**不属于自己**的 +任务（它手上没有那个任务）。此时只有三种选择，而前两种都是错的： + +| 做法 | 后果 | +|---|---| +| 标记完成 | 真正的持有者**永远收不到**这次请求——一个静默的丢指令 | +| 什么都不做，等租约过期 | 只要该进程持续轮询就会不断续租，真正的持有者被**无限期饿死** | +| **`release_command`** | 立即交还，真正的持有者马上能取到 ✓ | + +### 2.5 运维 + +#### `prune(*, keep_terminal)` + +丢弃最旧的终态任务及其历史，返回丢弃数量。**绝不能碰非终态任务**——删掉一个正在跑的 +任务等于把它的状态机扔了。按 `created_at` 升序决定丢谁。 + +#### `close()` + +释放资源。内存实现是空操作；**文件/连接型实现必须真的关**——没关的 SQLite 连接会 +泄漏文件句柄，在 Windows 上还会让数据库文件无法删除。 + +必须**幂等**：调用两次不报错。关闭之后再调用其它方法应当抛 `StoreError`（而不是 +`AttributeError` 或段错误）——把它当成一个明确的状态，不是未定义行为。 + +--- + +## 3. 并发要求汇总 + +| 要求 | 为什么 | +|---|---| +| `save` 的"校验 + 写入"原子 | 否则跨进程下版本检查形同虚设 | +| **版本号单调但不连续**：只要求 `expect_version == 行里的版本` | 写穿缓冲恢复后会一次跳进；若后端要求"只接受 +1"就等于重新引入"一次抖动永久冻结" | +| 不要自动重试冲突 | 冲突意味着存在第二个写入者，是设计缺陷 | +| `claim_commands` 的"挑选 + 打租约"原子 | 否则同一指令被下发两次 | +| 租约到期可重取 | 取走指令的进程崩溃后，指令不能永久卡住 | +| `append` 幂等 | 崩溃恢复会重放 | +| `touch` 不参与乐观并发 | 高频写入会与状态写入持续冲突 | +| **`save` 写了租约列** | 租约被覆盖回旧值，重启恢复把活着的任务判成 `LOST` | + +**契约测试不测跨进程。** 它只保证单连接语义正确；跨进程原子性必须由后端自己保证， +并在你的环境里单独压测。本文的范例实现（JSON 文件）就**只在单进程内正确**——它的 +模块文档里写明了这一点，不要照抄成多进程方案。 + +--- + +## 4. 序列化：别自己重写 + +`TaskSnapshot` 里有 `datetime`、`frozenset`、枚举和嵌套的 `Progress` / `Failure`。 +这些细节收敛在 `chongming_task.stores.codec`，**可以直接复用**： + +```python +from chongming_task.stores.codec import ( + snapshot_to_row, + row_to_snapshot, # 快照 <-> 扁平原语映射 + transition_to_row, + row_to_transition, # 迁移记录 + command_to_row, + row_to_command, # 控制指令 + encode_dt, + decode_dt, + encode_tags, + decode_tags, +) +``` + +映射的形状是**扁平的一层字典**（列名 → 原语）——关系库直接就是一行，文档库存成子 +对象，KV 直接写键值对。 + +三条硬要求： + +1. **时间一律先转 UTC 再存。** 把本地时间原样写进去，跨时区部署时排序与比较都会错， + 而且不报错。 +2. **往返必须保真**：`row_to_snapshot(snapshot_to_row(s)) == s`。契约测试用一条 + "每个字段都填满"的快照验证——只测几个字段会漏掉时区、`frozenset`、嵌套结构这类 + 静默丢失。 +3. **枚举写 `.value`**，不要指望 `json.dumps` 对枚举的行为。 + +--- + +## 5. 接线 + +```python +from chongming_task import Controller + +controller = Controller( + store=MyStore(...), + owner="worker-1", # 写进它下发的指令里，便于追溯 + command_poll_interval=0.5, # 指令轮询间隔 + command_lease=30.0, # 取走指令后的独占时间 +) +``` + +接上之后： + +| 方向 | 行为 | +|---|---| +| 写 | 每次状态迁移**写穿**到存储（挂在状态机的 `_apply` 上，因此不会漏） | +| 读 | `get` / `list_tasks` / `stats` / `bucket_counts` 走存储 | +| 控制 | 不属于本进程的任务 → 写一条指令，由持有者轮询取走 | +| 高频 | 进度与心跳由轮询线程定期经 `touch` 回写 | + +**两个必须知道的边界**： + +* **纯查询客户端不轮询指令。** 轮询线程只在提交过任务之后才启动——否则一个只读看板 + 会把别人的停止请求抢走。这是靠"提交过才启动"实现的，后端不需要配合。 +* **`BucketPolicy.num_buckets` 是部署范围内的约定。** 多节点各配不同的桶数会让同一个 + 键落到不同的桶。后端不感知这件事，但你的部署脚本应当断言各处一致。 + +--- + +## 6. 自证：契约测试 + +```python +from chongming_task.testing.store_contract import StoreContract + + +class TestMyStore(StoreContract): + def make_store(self): + """每次调用返回一个全新的、隔离的存储。""" + return MyStore(...) +``` + +套件**随包发布**，且**不依赖 pytest**（只用普通断言），因为它是给后端作者的规格而 +不是本仓库的内部测试。pytest 仍会自动收集子类里的用例。 + +它覆盖什么： + +* 协议的结构性要件（`isinstance(store, TaskStore)`）； +* **每个字段**的往返保真； +* 乐观并发的接受与拒绝，以及拒绝后状态未被改动； +* 查询的每一种过滤组合、排序、`limit`、空存储； +* 标签的**精确**匹配（`io` 不能命中 `ios`）； +* `since_version` 增量、幂等重放； +* `touch` 不改版本号、任务不存在时静默； +* 指令的取走 / 完成 / 租约过期重取 / **交还**； +* `prune` 只丢终态、历史一并清理。 + +**内置的实现跑的是同一套用例**——`InMemoryTaskStore`、`SqliteTaskStore`（文件库）、 +`SqliteTaskStore(":memory:")`（内存库），以及 `PostgresTaskStore` +（`import chongming_task` 后惰性取用，需要 `pip install "chongming-task[postgres]"`） +——所以"通过"这件事是有基准的。 + +**契约测试不测跨进程。** 它只保证单连接语义正确；跨进程原子性必须由后端自己保证， +并在你的环境里单独压测。内置 Postgres 实现把这份证据留在了仓库里 +（`tests/integration/test_postgres_store.py`，需要一台可连的库）： + +| 用例 | 钉住的承诺 | +|---|---| +| 8 个线程并发 `save` 同一版本 | 恰好一个成功，其余全部 `ConcurrencyConflict` | +| 8 个消费者并发 `claim_commands` | 每条指令恰好被取走一次（`SKIP LOCKED` 真的在起作用） | +| 持有者被 `pg_terminate_backend` 强杀 | 租约不提前释放也不永锁，到期可接管，连接池自愈 | +| 权限受限的角色 | 只用已有 schema，不做 DDL——只读运维控制台可用 | + +要跑它们，给一个 DSN 即可（每个用例自建独立 schema，跑完即删）： + +```bash +CHONGMING_TEST_POSTGRES='postgresql://user@host:5432/db' uv run pytest tests/integration +``` + +不给 DSN 时整个模块干净跳过——**跨进程证据缺席会显式表现为 skip，而不是默默变绿**。 +自己实现后端时，照这个模式补你自己的并发用例：契约测试全绿只说明单连接语义对， +不说明多进程下对。 + +--- + +## 7. 可执行范例 + +`tests/integration/test_custom_store.py` 里有一个完整的第三方后端 +（`JsonFileTaskStore`，把全部状态放在一个 JSON 文件里）。它： + +* 只实现契约，不继承任何东西； +* 复用 `stores.codec` 做序列化； +* 跑通完整的 `StoreContract`； +* 另有三条用例证明它真的能**接进 `Controller`**、并支持**跨进程控制**。 + +它是可以照抄的骨架。**但它演示的是接口形状，不是跨进程方案。** 无锁的文件型后端在 +Windows 上**无法**支撑"两个实例共用一个文件"：`Path.replace` 的原子替换与并发打开会 +互相冲突，实测抛 `PermissionError`；直接 `write_text` 更糟，读方会拿到空文件或半截 +JSON（`JSONDecodeError: Expecting value: line 1 column 1`）。 + +范例真实踩到的两个坑，都已固化进代码注释与用例： + +1. **不能缓存。** 两个实例各持一份内存副本时彼此看不到对方的写入，跨进程控制会 + **静默失效**（控制端把指令写进自己的副本，worker 压根不知道）。本实现因此每次 + 操作都重新读盘。 +2. **写回必须原子替换。** 见上。但即便如此，Windows 上仍不足以支撑共享。 + +要跨进程就用内置的 `SqliteTaskStore`——文件锁、WAL、`BEGIN IMMEDIATE` 都由 SQLite +负责，这正是它作为默认实现的理由。范例里有一条用例（ +`test_the_naive_file_store_is_not_cross_process_safe`）把这个局限**钉在测试里**， +而不是写在注释里自欺欺人。 + +--- + +## 8. 检查清单 + +替换默认持久层之前，逐条确认： + +- [ ] 14 个方法齐备，签名与 `protocol.py` 一致 +- [ ] `create` 对已存在的 `task_id` 抛错，不覆盖 +- [ ] `save` 的"校验 + 写入"是原子的，冲突抛 `ConcurrencyConflict`，且不自动重试 +- [ ] `save` 允许版本**跳号**（不要求连续），只按 `expect_version` 相等判定 +- [ ] `append` 幂等（`task_id` + `version`） +- [ ] `history` 按版本升序，支持 `since_version` +- [ ] `touch` **不**改 `status_version`，任务不存在时静默 +- [ ] `touch` 支持 `lease_owner` / `lease_until`，且每个参数独立可选 +- [ ] **`save` 不写租约列**（否则重启恢复会误杀活着的任务） +- [ ] `claim_commands` 的"挑选 + 打租约"原子（关系库：`SELECT ... FOR UPDATE SKIP LOCKED`），租约到期可重取 +- [ ] 实现了 `release_command`，且它不标记完成 +- [ ] `prune` 只丢终态，且一并清理历史 +- [ ] 时间以 UTC 存储，往返保真 +- [ ] 过滤（尤其 `buckets`）下推到后端，不是取回全部再筛 +- [ ] `close()` 真的释放连接/文件句柄，**幂等**，关闭后再用抛 `StoreError` +- [ ] `StoreContract` 全绿 +- [ ] 若要在多实例间共享：有真正的跨进程锁（无锁文件方案在 Windows 上不可行） +- [ ] 跨进程原子性在你的环境里单独验证过（契约测试不覆盖它） + +--- + +## 9. 常见陷阱 + +| 陷阱 | 症状 | 正确做法 | +|---|---|---| +| `save` 拆成"先查后写" | 跨进程下版本检查形同虚设，偶发丢更新 | 一条带 `WHERE version = ?` 的原子写 | +| `claim` 不加租约 | 取走指令的进程崩溃后，指令永久卡住 | 租约 + 到期可重取 | +| `claim` 不原子 | 同一次停止被下发两次 | `BEGIN IMMEDIATE` / Lua / CAS | +| 把 `release` 实现成 `complete` | 静默丢指令，持有者永远收不到 | 只清租约，不设 `finished_at` | +| `touch` 递增版本号 | 状态写入频繁冲突，但报错指向别处 | 高频旁路绝不碰版本号 | +| `append` 用普通 INSERT | 崩溃恢复产生重复历史行 | 幂等写 | +| 时间存本地时区 | 跨时区排序错，且不报错 | 先转 UTC | +| 标签用子串匹配 | `io` 命中 `ios`，筛选结果悄悄错 | 精确匹配（JSON 数组 + 带引号匹配） | +| `prune` 删掉非终态任务 | 正在跑的任务状态机被扔掉 | 只丢终态 | +| 忘记 `close()` | 文件句柄泄漏；Windows 下数据库文件删不掉 | 真的关 | +| `create` 静默覆盖 | 任务重跑但旧执行载体还在跑，内存与存储分叉 | 已存在就抛错 | +| **在内存里缓存整份数据** | 多实例各持副本，**静默**看不到对方写入 | 每次操作重新读，或配文件锁 + 失效判断 | +| **`write_text` 直接覆盖** | 读方拿到空文件/半截内容 | `Path.replace` 原子替换（含 pid 临时名） | +| **无锁文件当共享存储** | Windows 上并发读写直接 `PermissionError` | 需要共享就用带锁的存储（如 SQLite） | diff --git a/docs/task-control-design.md b/docs/task-control-design.md new file mode 100644 index 0000000..3f4e542 --- /dev/null +++ b/docs/task-control-design.md @@ -0,0 +1,1157 @@ +# chongming-task：Python 管道任务控制工具 — API 规划与架构方案 + +> 状态：设计草案 v1 +> 目标读者：本项目的实现者与使用方 +> 起点：仓库当前仅有 `src/chongming-task/enum.py`（6 个状态雏形） + +--- + +## 0. 先决问题（必须先解决，否则无法进入实现） + +| # | 问题 | 现状 | 结论 | +|---|---|---|---| +| 1 | 包目录名含连字符 | `src/chongming-task` | **必须改为 `src/chongming_task`**。`import chongming-task` 是语法错误，任何 `from chongming_task import ...` 都不可用。`pyproject.toml` 的 `packages = ["src/chongming-task"]` 同步修改（project name 可保留连字符，那是发行名，不是导入名）。 | +| 2 | 状态枚举语义重叠 | `FINISHED` / `STOPPED` 含义模糊 | `FINISHED` 无法区分"成功结束"与"任意终止"，需拆为 `SUCCEEDED` / `FAILED` / `STOPPED` / `CANCELLED`。详见 §2。 | +| 3 | Python 版本下限 | `requires-python = ">=3.14"` | 3.14 过于激进：可选依赖（数据库驱动、HTTP 框架、观测后端）的 wheel 覆盖不确定。**建议放宽到 `>=3.11`**，除非明确无第三方依赖。 | +| 4 | `enum.py` 文件名 | 与标准库同名 | 虽因绝对导入而可用，但 `chongming_task.enum` 易误读。建议改名为 `states.py`，`enum.py` 保留为兼容 re-export。 | + +--- + +## 1. 设计原则 + +1. **状态是唯一的真相来源，且只有一个写者。** 所有状态变更必须经过状态机（`StateMachine.apply(event)`），禁止业务代码直接改 `snapshot.status`。每次变更携带 `trigger` + `actor` + `reason`，天然可审计。 +2. **"请求的动作"与"当前状态"分离。** `stop()` / `pause()` 是**请求**，不是瞬时赋值。运行中的任务需要时间响应，因此快照同时暴露 `status`（现在在哪）和 `requested_action`（想让它去哪），UI 才能显示"停止中/暂停中"。 +3. **一个枚举只表达一件事。** 生命周期（`status`）、重试计数（`attempt`）、进度（`progress`）、健康（`last_heartbeat_at`）、结果（`failure`/`outcome`）是**正交维度**，绝不塞进同一个枚举。这是同类工具最常见的建模错误，也是"精细化控制"能否落地的前提。 +4. **取消是协作式的，停止是有层级的。** 停止 = 请求 + 优雅窗口 + 如实收场；线程**不可强杀**，所以窗口用尽只能放弃并记 `LOST`，绝不谎报已停止（见 §5.4、§13.6）。 +5. **内核无 I/O。** 领域层不 import `asyncio`、不碰数据库、不认识日志库。这样状态机可以用属性测试穷举验证。 +6. **可插拔优于可配置。** 存储、执行器、事件出口都用 `Protocol` 定义，v1 只提供最小实现。 +7. **v1 单进程。** 分布式调度是明确的非目标（§9），但接口预留（store / executor 抽象足以支撑后续远程化）。 + +--- + +## 2. 状态模型 + +### 2.1 `TaskStatus` 枚举 + +```python +from enum import StrEnum + + +class TaskStatus(StrEnum): + # ---- 非终态 ---- + PENDING = "pending" # 已创建，依赖尚未解析 + BLOCKED = "blocked" # 依赖未满足，等待上游 + QUEUED = "queued" # 就绪，等待执行槽 / 暂停后待恢复 + STARTING = "starting" # 执行器启动中（spawn / 建连 / 加载） + RUNNING = "running" + PAUSING = "pausing" # 已请求暂停，等待静默点 + PAUSED = "paused" # 已静止，不再消耗资源 + RETRY_WAIT = "retry_wait" # 本次尝试失败，退避等待中 + STOPPING = "stopping" # 已请求停止，优雅窗口内 + # ---- 终态 ---- + SUCCEEDED = "succeeded" # 正常完成 + FAILED = "failed" # 出错且重试耗尽 / 不可重试 + STOPPED = "stopped" # 人工主动停止（不是错误） + CANCELLED = "cancelled" # 编排层级联取消（上游失败 / 管道取消） + SKIPPED = "skipped" # 条件为假 / 依策略跳过 + LOST = "lost" # 执行载体失联（进程消失、心跳超时） +``` + +设计说明： + +- **`STOPPED` 与 `CANCELLED` 分开**：两者都"没跑完"，但责任方不同。`STOPPED` = 人对单个任务显式干预；`CANCELLED` = 编排层因上游失败/整体取消而级联触发。这对排障和统计（"这次跑批失败是人工干预还是真故障"）价值很高。 +- **不设 `TIMED_OUT` 状态**。超时是**失败原因**而非生命周期阶段，放在 `Failure.kind = TIMEOUT` 里。否则 `TIMEOUT → RETRY_WAIT` 这种组合会逼出"超时但正在重试"这类无法用单一枚举表达的状态，导致枚举爆炸。 +- **`LOST` 是终态但可被 `retry` 复活**：进程崩溃、心跳丢失时，系统无法确知结果，标记 `LOST` 而非 `FAILED` 是诚实的做法。 +- 兼容旧枚举：`FINISHED` / `STOPPED` 旧名保留为别名，`FINISHED = "succeeded"`。 + +### 2.2 终止态与"可复活" + +```python +TERMINAL = frozenset( + { + TaskStatus.SUCCEEDED, + TaskStatus.FAILED, + TaskStatus.STOPPED, + TaskStatus.CANCELLED, + TaskStatus.SKIPPED, + TaskStatus.LOST, + } +) +# 允许通过 retry 重新进入执行流程的终态 +REVIVABLE = frozenset( + { + TaskStatus.FAILED, + TaskStatus.STOPPED, + TaskStatus.CANCELLED, + TaskStatus.LOST, + } +) +``` + +复活**不修改历史**：`retry()` 递增 `attempt`，写入新的 `AttemptRecord`，状态回到 `QUEUED`。历史尝试永远只追加、不覆盖。 + +### 2.3 合法转移表（"精细控制"的核心资产） + +以**数据**而非 `if/else` 表达，可测试、可内省、可生成文档： + +```python +TRANSITIONS: Mapping[TaskStatus, frozenset[TaskStatus]] = { + PENDING: {BLOCKED, QUEUED, SKIPPED, CANCELLED, STOPPED}, + BLOCKED: {QUEUED, SKIPPED, CANCELLED, STOPPED}, + QUEUED: {STARTING, PAUSED, SKIPPED, STOPPED, CANCELLED}, + STARTING: {RUNNING, RETRY_WAIT, PAUSING, STOPPING, FAILED, LOST}, + RUNNING: {SUCCEEDED, FAILED, RETRY_WAIT, PAUSING, STOPPING, LOST}, + PAUSING: {PAUSED, RUNNING, FAILED, STOPPING, LOST}, # RUNNING = 暂停被撤销 + PAUSED: {QUEUED, RUNNING, STOPPED, CANCELLED}, # 恢复目标见下方 v1.3 说明 + RETRY_WAIT: {STARTING, QUEUED, PAUSED, STOPPED, FAILED, CANCELLED}, + STOPPING: {STOPPED, SUCCEEDED, FAILED, LOST}, # SUCCEEDED = 恰好跑完 + # 终态：出边为空集。TERMINAL 由本表推导（出边为空者）得出，避免两处维护而不一致。 + SUCCEEDED: frozenset(), + FAILED: frozenset(), + STOPPED: frozenset(), + CANCELLED: frozenset(), + SKIPPED: frozenset(), + LOST: frozenset(), +} +``` + +> **统一规则（实现期修正 v1.2）**：控制动作是否需要中间态，取决于任务是否 +> **已占用执行资源**。 +> +> - 未占用（`PENDING` / `BLOCKED` / `QUEUED` / `PAUSED` / `RETRY_WAIT`）→ 立即生效： +> `stop()` 直接落 `STOPPED`，`pause()` 直接落 `PAUSED`。 +> - 已占用（`STARTING` / `RUNNING` / `PAUSING`）→ 经中间态缓冲： +> `stop()` 先 `STOPPING`，`pause()` 先 `PAUSING`。 +> 这条规则替代了 v1.0 的做法。v1.0 只允许经 `CANCELLED` 离开未启动状态，导致 +> "停止一个排队中的任务"无合法路径——而 `stop()` 是人为主动停止，语义上应落 +> `STOPPED` 而非级联取消的 `CANCELLED`。同时 v1.0 允许 `QUEUED → PAUSING`， +> 会让暂停中的排队任务经 `abort_pause` 变成 `RUNNING`，从而产出**没有 +> `started_at` 的 RUNNING 快照**（`validate()` 能抓到这个不一致）。 +> +> 让未启动的任务显示"停止中/暂停中"是虚假信息，会让运维误以为需要等待；因此 +> 宁可让表更精确，也不保留语义空洞的边。`started_at` 改由 `mark_starting()` +> 写入，使任何到达 `RUNNING` 的路径都必然带有开始时间。 + +> **暂停恢复语义修正（v1.3）**：`PAUSED` 增加了到 `RUNNING` 的边，恢复目标由 +> `StateMachine.resume()` 按**暂停发生在哪一层**判定： +> +> | 暂停路径 | 恢复目标 | 理由 | +> |---|---|---| +> | `STARTING`/`RUNNING → PAUSING → PAUSED` | `RUNNING` | 就地暂停，执行槽从未释放，调用栈还在内存里 | +> | `QUEUED`/`RETRY_WAIT → PAUSED` | `QUEUED` | 从未启动过，需要重新排队 | +> +> v1.0–v1.2 主张"恢复必须重新争取执行槽，否则暂停会成为绕过并发限制的后门"。 +> 这个理由**建立在一个不成立的假设上**：它假定暂停会释放槽位。而我们的实现是 +> 就地协作暂停（任务阻塞在 `checkpoint()` 里），槽位从未释放，也不存在可用于 +> 换出/换回调用栈的用户态检查点。因此强制回到 `QUEUED` 只会让状态与实际执行 +> 情况脱节——任务仍在运行，状态却显示在排队，而且 `QUEUED → SUCCEEDED` 根本 +> 不在转移表里。 +> +> 这条修正连带暴露了两个必须用**守卫**（而非转移表）表达的区别： +> +> - `mark_running()` 要求已有 `started_at`。`PAUSED → RUNNING` 对就地恢复合法， +> 对"排队期间被暂停、从未启动"却毫无意义。 +> - `abort_pause()` 只允许从 `PAUSING` 回滚。否则它会借 `PAUSED → RUNNING` 这条 +> 边产出没有 `started_at` 的 `RUNNING` 快照。 +> +> 后者是**属性测试**跑出来的：随机事件序列 `mark_queued → request_pause → +> abort_pause` 精确命中了这个组合，手写用例时完全没想到。 + +规则： + +- 终态**没有**出边。任何"TERMINAL → 非终态"的写入都会被状态机拒绝，唯一的合法路径是 `retry()` / `reset()`（前者保留尝试历史，后者清空）。 +- 非法转移抛 `InvalidTransition(current, requested, allowed)`，而不是静默忽略或返回 `False`。同时提供只读预检 `StateMachine.can()/can_stop/can_pause/can_resume/can_retry`，让 UI 能在发指令前就置灰按钮。 +- 幂等性（对已处于目标态的任务重复 `stop()` 不报错）由**控制面**（P4 的 `Controller`）负责，不在状态机内处理——状态机只做"本次转移是否合法"的判断。 +- 有些规则**无法**用"状态 → 状态"表达，它们是**守卫**：`mark_running()` 要求已有 `started_at`；`abort_pause()` 要求当前处于 `PAUSING`；`mark_starting()` 要求 `attempt + 1 <= max_attempts`（否则抛 `AttemptsExhausted`）；`force_transition()` 要求非空 `actor` 与 `reason`。 + +### 2.4 快照模型（正交维度） + +```python +@dataclass(frozen=True, slots=True) +class TaskSnapshot: + # 身份 + task_id: str + pipeline_run_id: str | None + name: str + tags: frozenset[str] + # 生命周期 + status: TaskStatus + status_version: int # 乐观并发版本号，每次转移 +1 + requested_action: RequestedAction | None # STOP / PAUSE / None + # 尝试 + attempt: int + max_attempts: int + # 时间 + created_at: datetime + queued_at: datetime | None + started_at: datetime | None + ended_at: datetime | None + last_heartbeat_at: datetime | None + # 进度 + progress: Progress | None # done/total/unit/message/updated_at + # 结果 + failure: Failure | None # kind/message/traceback/retryable + outcome: Mapping[str, Any] | None + # 执行载体 + executor: str | None # "inline" | "thread" +``` + +`Failure.kind` 枚举：`EXCEPTION` / `NONZERO_EXIT` / `TIMEOUT_START` / `TIMEOUT_ATTEMPT` / `TIMEOUT_IDLE` / `CANCELLED` / `LOST` / `DEPENDENCY_FAILED` / `CONDITION_FALSE`。 + +--- + +## 3. 控制面 API + +API 分三层，职责不混：**声明层**（描述要做什么）、**控制层**（运行期干预）、**查询层**（观察）。 + +### 3.1 控制动词语义表 + +这是"精细化控制"的契约，实现前必须先冻结： + +| 动词 | 合法前置状态 | 瞬时效果 | 最终状态 | 幂等 | 强制语义 | +|---|---|---|---|---|---| +| `start()` | PENDING, BLOCKED | 解析依赖 → QUEUED | QUEUED | 是 | — | +| `stop(graceful, timeout)` | 任意非终态 | `requested_action=STOP`，status→STOPPING；向取消令牌发信号 | STOPPED / LOST | 是 | 超 `timeout` 未退 → 线程不可强杀：放弃等待并记 `LOST`（结果未知） | +| `pause(timeout)` | RUNNING, QUEUED | status→PAUSING；令牌置暂停位 | PAUSED | 是 | 超时未静默 → 回滚为 RUNNING 并告警 | +| `resume()` | PAUSED | status→QUEUED | RUNNING | 是 | — | +| `cancel(reason)` | 任意非终态 | 编排层级联 | CANCELLED | 是 | 上游失败时的自动行为 | +| `retry(force)` | FAILED, STOPPED, CANCELLED, LOST | `attempt += 1`，新 AttemptRecord | QUEUED | 是 | `force=True` 可越过 `max_attempts` | +| `skip(reason)` | PENDING, BLOCKED, FAILED | — | SKIPPED | 是 | — | +| `reset()` | 任意 | 清空 attempts，`attempt=0` | PENDING | 是 | 丢失全部历史（危险，需二次确认） | +| `wait(timeout)` | 任意 | 阻塞至终态 | — | 是 | 超时抛 `TimeoutError`，**不改变任务状态** | +| `force_transition(to, reason, actor)` | 任意 | 绕过转移表，写入 `AdminOverride` 审计事件 | 指定状态 | 否 | 唯一逃生舱，必须带 `actor` 与 `reason` | + +要点： + +- **`wait()` 必须有超时且不产生副作用**。这是最常见的误用点（无限阻塞的 `wait()` 会拖死调用方）。 +- **`force_transition` 是必需的，不是妥协**。现实运维需要"这个任务卡住了，人工标记为成功/失败"的能力。但必须：写审计事件、要求非空 `reason`、在快照上打 `overridden` 标记。 +- 停止语义分三层：`requested_action` 记录意图 → `STOPPING` 表示优雅窗口 → 窗口用尽后**如实收场**：配合的任务落 `STOPPED`，不配合的落 `LOST`（`Failure.kind = LOST`，结果未知），不留任何假装已停止的空档。 + +### 3.2 控制层代码示例 + +```python +from chongming_task import RetryPolicy, Timeout + +handle = controller.submit( + my_task, + args=(...), + retry=RetryPolicy(max_attempts=3), + timeout=Timeout(per_attempt=300, idle=60), +) + +handle.status # TaskStatus.RUNNING +handle.progress # Progress(done=120, total=500, unit="rows") +controller.can(handle.id, "pause") # (True, None) + +handle.pause(timeout=30) # -> PAUSED +handle.resume() # -> RUNNING +handle.stop(graceful=True, timeout=10) # -> STOPPED +controller.retry(handle.id) # -> QUEUED (attempt=2) +snap = controller.get(handle.id) +controller.force_transition(handle.id, TaskStatus.SUCCEEDED, reason="已人工核验", actor="ops@x") +``` + +异步版本完全对应：`await controller.submit(...)`、`await handle.wait()`。同步外观只是 async 内核对 `asyncio.run`/thread-local loop 的薄封装，**不是两套实现**。 + +### 3.3 声明层：`@task` 与 `Pipeline` + +```python +from chongming_task import task, TaskContext + + +@task( + retry=RetryPolicy(max_attempts=3, backoff="exponential"), + timeout=Timeout(start=30, per_attempt=600), + tags={"io"}, +) +def extract(ctx: TaskContext, source: str) -> list[Row]: + ctx.log.info("开始抽取", source=source) + rows = read(source) + ctx.progress(total=len(rows)) + for i, row in enumerate(rows): + ctx.checkpoint() # 协作取消/暂停点；被请求停止时抛 TaskCancelled + ctx.progress(done=i + 1) + ctx.emit("row", row) # 可选：中途产出（流式结果） + return rows +``` + +`TaskContext` 是运行期唯一被注入的对象，职责明确： + +| 成员 | 作用 | +|---|---| +| `task_id` / `pipeline_run_id` / `attempt` / `name` | 身份与重试可见性 | +| `log` | 已绑定 task 上下文字段的结构化 logger | +| `progress(done=None, total=None, unit=None, message=None)` | 进度上报（自动节流） | +| `checkpoint()` | 协作式取消/暂停点。收到停止请求时抛 `TaskCancelled`；请求暂停时阻塞至恢复 | +| `upstream(name)` | 读取上游步骤结果（类型标注用 `cast`） | +| `artifacts` | 该次尝试的产物目录（`Path`），随 attempt 隔离，重试不污染 | +| `emit(kind, payload)` | 发出自定义事件（进事件总线） | +| `deadline` | 本次尝试的绝对截止时间，便于长循环内部自行收敛 | + +**约定：长耗时循环必须调用 `checkpoint()`。** 不调用 = 无法优雅停止，只能等超时（`idle`）并被判 `LOST`——线程没有强杀这条后路。这条要写进文档并在 `--strict` 模式下用静态检查辅助（检测 `@task` 函数体内是否存在无 `checkpoint` 的循环）。 + +管道（DAG）： + +```python +pipe = Pipeline("nightly-etl", on_failure=AggregationPolicy.fail_fast) + +a = pipe.add(extract, args=("s3://bucket/x",)) +b = pipe.add(transform, deps=[a], retry=RetryPolicy(max_attempts=5)) +c = pipe.add(load, deps=[b], when=lambda up: up[a].row_count > 0) # 条件执行 -> 否则 SKIPPED +d = pipe.map(transform_one, items=up[a].rows, deps=[a], max_parallel=8) # 扇出 + +run = pipe.submit(controller) # -> PipelineRun +run.status # PipelineRunStatus，由子任务状态按策略聚合 +run.wait(timeout=3600) +run.summary() # 各状态计数 + 首个失败链 +``` + +管道级补充控制：`run.pause()` / `run.stop()` 级联到所有非终态子任务；`on_failure` 策略决定上游失败时下游是 `CANCELLED` 还是 `SKIPPED`。 + +### 3.4 查询层 + +```python +controller.get(task_id) -> TaskSnapshot +controller.list_tasks(status={RUNNING, RETRY_WAIT}, tag="io", name_prefix=..., limit=100) +controller.stats(pipeline_run_id=...) -> StatusCounts +controller.can(task_id, action) -> tuple[bool, str | None] +controller.events(task_id, since=..., types={...}) -> Iterator[Event] +controller.attempts(task_id) -> list[AttemptRecord] +controller.watch(filter, callback) / async for ev in controller.stream(filter) # 实时订阅 +``` + +--- + +## 4. 事件与可观测性 + +### 4.1 事件模型 + +```python +@dataclass(frozen=True, slots=True) +class TaskStatusChanged(Event): + task_id: str + from_status: TaskStatus + to_status: TaskStatus + trigger: str # "user:stop" | "scheduler:dispatch" | "retry:exhausted" | ... + actor: str + reason: str | None + at: datetime +``` + +事件类型全集：`TaskCreated`、`TaskStatusChanged`、`AttemptStarted`、`AttemptFailed`、`ProgressUpdated`、`TaskHeartbeat`、`TaskOutput`、`TaskLogLine`、`PipelineRunStatusChanged`、`AdminOverride`。 + +三条工程约束： + +1. **进度事件默认节流**（≥500ms 或 ≥1% 变化才发）。不加节流，10 万行数据的管道会用进度事件把存储打爆。 +2. **`TaskStatusChanged` 先落存储、后发总线**（write-ahead）。跨进程场景用 outbox 表保证至少一次投递；消费者需幂等（事件带 `event_id`）。 +3. 总线只做分发，**订阅者异常不得影响任务执行**（捕获并降级记录），否则一个坏订阅者能搞挂整个管道。 + +### 4.2 观测挂点 + +内核只定义 `Protocol`，不绑定任何后端： + +- 日志：标准 `logging` + 结构化字段注入（`task_id`/`run_id`/`attempt`）。JSON formatter 作为可选实现。 +- 指标：`MetricsSink.incr(name, tags)` / `observe(name, value, tags)`。默认 `NullSink`，可选 Prometheus/OTel 适配器（独立模块，不进核心依赖）。 +- 追踪：`Tracer.span(name, attrs)`，每个 attempt 一个 span。 + +**核心包零第三方运行时依赖**是可维护性的关键。 + +--- + +## 5. 架构方案 + +### 5.1 分层 + +``` +┌──────────────────────────────────────────────────────┐ +│ cli/ run | ps | status | stop | pause | │ 可选 +│ resume | retry | logs | events │ +├──────────────────────────────────────────────────────┤ +│ api/ 装饰器 · Pipeline · Handle · │ +│ Controller(门面) · TaskContext │ +├──────────────────────────────────────────────────────┤ +│ runtime/ 调度器 · 状态机 · 取消令牌 · │ +│ 心跳 · 执行器适配 · 事件总线 │ +├──────────────────────────────────────────────────────┤ +│ stores/ memory | sqlite | (redis) │ +├──────────────────────────────────────────────────────┤ +│ core/ 状态枚举 · 转移表 · 快照 · 策略 · │ 零 I/O +│ 错误体系 · 事件类型 · Protocol │ +└──────────────────────────────────────────────────────┘ +``` + +依赖方向严格单向 `core ← stores/runtime ← api ← cli`。用 `import-linter` 在 CI 中强制，防止内核被 I/O 污染。 + +### 5.2 数据流（一次任务的生命周期） + +``` +submit() ──► Store.create(PENDING) ──► Event(TaskCreated) + │ + ▼ +Scheduler.tick: 依赖就绪? ──► BLOCKED/QUEUED ──► 取并发槽 + │ + ▼ +状态机: QUEUED → STARTING ──► Executor.launch(ctx, token) ──► RUNNING + │ │ + │ Heartbeat(周期) ──► last_heartbeat_at + │ │ + │ ctx.progress / ctx.emit ──► 事件(节流) + ▼ +返回 / 抛异常 / 被取消 / 超时 / 心跳丢失 + │ + ▼ +状态机: RUNNING → SUCCEEDED | FAILED | RETRY_WAIT | STOPPED | LOST + │ + ▼ +Store 写终态 ──► Event ──► 唤醒下游依赖 / 触发 Pipeline 聚合 +``` + +### 5.3 并发模型 + +> **本节记录最初的设计意图，其中的 asyncio 内核没有被采纳。** 最终落地的是 +> **线程执行器 + 同步调度器**（§13）：调度器用一个专职派发线程与一把锁管理就绪队列， +> 执行器每个任务起一个守护线程。要监督的本来就是我方**阻塞式业务代码**（数据库、 +> HTTP、文件），先搬进事件循环再 `to_thread` 卸载只会多一层；而取消、暂停、超时 +> 在"每个任务一个线程"下已经足够干净。 +> +> 下面的分析与选型作为**决策记录**保留。最终落地的执行器有三个（`Inline` / `Thread` / +> `Process`）：`SubprocessExecutor` 从未实现，`AsyncExecutor` 不做（异步能力落在执行器里： +> 协程在本线程跑完，见 §13.5）。`ProcessExecutor` 的取舍与决策史见 §13.10。 + +**asyncio 单事件循环为内核**（未采纳），理由：DAG 调度、超时、优雅取消、心跳在 asyncio 下都远比线程池干净；线程池方案无法取消、无法精确超时。 + +- 同步阻塞型任务：`AsyncExecutor` 内用 `asyncio.to_thread` 卸载，配**独立的有界线程池**（默认 `min(32, cpu*4)`），避免耗尽默认执行器影响整个进程。（未实现） +- 调度器**事件驱动**，不做忙轮询：依赖满足、槽释放、心跳、超时都通过事件/定时器唤醒。（部分成立：现在的调度器是事件唤醒 + 固定间隔兜底重扫，见 `Controller._dispatch_loop`） + +`ConcurrencyPolicy` 支持：全局 `max_parallel`、按 `tags` 的具名信号量（如 `db=4`、`gpu=1`）、优先级（高优先进就绪队列）——**已交付**，见 §13.5。 + +### 5.4 取消与停止（最容易被低估的部分） + +三层机制，逐层兜底： + +1. **协作式**：`CancellationToken` → `ctx.checkpoint()` 抛 `TaskCancelled`。干净、可回滚。 +2. **优雅窗口**：`stop(graceful=True, timeout=T)`，给任务 T 秒收尾（写检查点、关连接）。 +3. **强制兜底**：超时后强杀。**这里必须诚实**： + + - 子进程 → `terminate()` → 宽限 → `kill()`（`ProcessExecutor`，见 §13.10）。 + - 线程 → **Python 无法安全强杀线程**。因此 `ThreadExecutor` 的强制停止**只能**放弃等待，并返回 `abandoned=True`（结果未知，落 `LOST`）。这是架构级限制，不是实现偷懒。 + - 结论：**硬性强杀是执行器的可选能力，不是默认前提**。默认走线程 + 协作式取消即可停止；只有确实需要"停止保证"或需要绕开 GIL 的任务才升级到进程。选型矩阵见 §13.2。 + +**暂停的诚实定义**：`pause()` = ① 不再向该任务的下游派发新工作；② 运行中的任务在下一个 `checkpoint()` 处挂起。对于**不调用 `checkpoint()` 的不可中断同步代码，暂停无法生效**，超时后回滚为 `RUNNING` 并告警。我们**不做**抢占式暂停（POSIX 的 `SIGSTOP` 不跨平台，且会冻结持锁线程）。这个限制必须明说，否则用户会误以为 `pause()` 是万能的。 + +### 5.5 重试、超时与恢复 + +- `RetryPolicy(max_attempts, backoff=Exponential(base=1, factor=2, max=300, jitter="full"), retry_on=(TransientError,), give_up_on=(ValueError,))`。默认 `jitter="full"`，避免重试风暴同步化。 +- `Timeout(start=, per_attempt=, idle=)`：`idle` 基于心跳判定"卡死"，是唯一能发现死锁的机制，三者独立可组合。 +- 恢复：进程启动时 `controller.reconcile()` 扫描非终态任务，心跳超 `idle_timeout` 者按 `on_orphan` 策略处理：`fail`（默认，保守）| `requeue` | `manual`。**不做静默续跑**——无法确知上次副作用是否已提交时，猜错比报错更贵。 + +### 5.6 持久化 + +契约见 [stores/protocol.py](src/chongming_task/stores/protocol.py)。**四类数据刻意 +分开**，因为它们的写入频次与一致性要求完全不同： + +```python +class TaskStore(Protocol): + # 1. 任务快照——查询面，参与乐观并发 + def create(self, snapshot: TaskSnapshot) -> None: ... + def save( + self, snapshot: TaskSnapshot, *, expect_version: int + ) -> None: ... # 冲突抛 ConcurrencyConflict + def get(self, task_id: str) -> TaskSnapshot | None: ... + def query(self, query: TaskQuery | None = None) -> list[TaskSnapshot]: ... + def counts(self) -> dict[TaskStatus, int]: ... + + # 2. 审计轨迹——只追加，重复写入幂等 + def append(self, transition: Transition) -> None: ... + def history(self, task_id: str, *, since_version: int = 0) -> list[Transition]: ... + + # 3. 高频旁路——不校验也不递增 status_version + def touch(self, task_id, *, progress=None, heartbeat_at=None) -> None: ... + + # 4. 控制指令——跨进程控制的媒介，带租约 + def submit_command(self, command: Command) -> None: ... + def claim_commands(self, *, now, lease_seconds, limit=50) -> list[Command]: ... + def complete_command(self, command_id, *, now, outcome=None) -> None: ... + + # 运维 + def prune(self, *, keep_terminal: int) -> int: ... + def close(self) -> None: ... +``` + +三个取舍值得记下来： + +* **`touch` 不碰版本号。** 进度与心跳可能每秒几十次；若参与乐观并发，会与状态写入 + 持续冲突，把 `status_version` 变成噪声。 +* **指令带租约而不是用 IPC 推送。** 租约到期即可重取：既不会重复下发，也不会因为 + 取走指令的进程崩溃而永久卡住。存储层**只传递、不解释、不执行**——执行者是持有该 + 任务的进程。 +* **查询与写入分成两套方法，而不是一个通用 `update`。** 状态写入必须严格按版本， + 高频旁路必须绕过版本；混在一个方法里，迟早会有人在错误的路径上用了错误的语义。 + +实现：`InMemoryTaskStore`（测试 / 单进程 / 其他实现的参照物）、`SqliteTaskStore` +（标准库 `sqlite3`，WAL + `BEGIN IMMEDIATE` 实现跨进程原子取指令）。并发正确性靠 +`status_version` 乐观锁；每个任务的写入者唯一，冲突即代表 bug，抛 +`ConcurrencyConflict` 而非重试掩盖。 + +**契约测试随包发布**：`chongming_task.testing.store_contract.StoreContract`。第三方 +后端只要通过这一套用例就可以替换内置实现——这是"持久层可扩展"能成立的前提。 + +面向后端作者的完整规格（14 个方法的逐条语义、并发与原子性要求、序列化保真、可执行 +范例、陷阱清单）见 **[持久层实现指南](store-guide.md)**。 + +分布式部分的现状、边界与落地清单见 **[分布式设计说明](distributed-design.md)**。选定架构是 +**多个自治的定时任务节点 + 一套统一的存储与控制平面**：任务定义与触发方式各节点自有， +靠 `owned_buckets` 区分"哪一份是我的"；节点之间**不互相执行对方的任务**（被否决的集群方案 +及其原因见该文 §17）。 + +### 5.7 目录结构 + +``` +src/chongming_task/ + __init__.py # 收敛公开 API（__all__ 显式声明） + clock.py # Clock / utcnow：可注入时钟 + errors.py # 错误体系 + states.py # TaskStatus / TRANSITIONS / TERMINAL（终态由表推导） + models.py # TaskSnapshot / Progress / Failure / AttemptRecord + policies.py # RetryPolicy / Timeout / ConcurrencyPolicy / AggregationPolicy + controller.py # TaskRegistry / Controller / StatusCounts / render_table（已可用） + run.py # run_task() / TaskRun：单任务受控执行（已可用） + core/ + state_machine.py # 唯一写者：apply(event) -> Transition + graph.py # DAG 构建、环检测、拓扑就绪判定（P3） + runtime/ + cancellation.py # CancellationToken：协作式停止/暂停 + context.py # RuntimeContext：checkpoint / 心跳的唯一入口 + heartbeat.py # Heartbeat + timeouts.py # 三档超时的纯计算判定 + scheduler.py # 调度器（P3） + reconciler.py # 重启恢复（P5） + executors/ + base.py # ExecutionRequest / Outcome / Handle / LocalExecutionHandle + thread.py # 线程执行器（默认） + inline.py # 内联执行器 + events.py # 事件类型 + EventBus（P4） + stores/ # 持久层（已可用） + protocol.py # TaskStore 契约 / TaskQuery / Command / CommandKind + _codec.py # SQLite 行编解码（快照 / 迁移 / 指令） + memory.py # InMemoryTaskStore + sqlite.py # SqliteTaskStore（标准库 sqlite3） + api/ # decorators / pipeline / handles / controller（P4） + observability/ # logging / metrics / tracing（P6） + testing/ + clock.py # FakeClock，随包发布，使用者的测试也能用 + store_contract.py # TaskStore 契约套件，第三方后端据此自证（零 pytest 依赖） + cli/ # main.py（P6） +tests/ + unit/ property/ integration/ e2e/ contract/ +docs/ + task-control-design.md # 本文件：设计动机、状态模型、架构、里程碑 + user-guide.md # 面向使用者的完整说明 + store-guide.md # 面向持久层实现者的接口规格与陷阱清单 +examples/ + 01_quickstart.py # 跑一个任务并观察状态轨迹 + 02_control.py # 暂停 / 恢复 / 停止，以及"停不掉时如实上报" + 03_retry_and_timeout.py # 重试、预算耗尽、超时 + 04_controller.py # 注册 / 提交 / 并发排队 / 查询 / 控制 +``` + +--- + +## 6. 错误体系 + +```python +class TaskError(Exception): ... +class InvalidTransition(TaskError): # current / requested / allowed +class DependencyFailed(TaskError): # upstream_ids, their Failure +class CircularDependency(TaskError): # cycle path +class TaskCancelled(TaskError): # 协作取消信号（在任务内部抛出） +class TaskTimeout(TaskError): # scope: start | attempt | idle +class TaskExecutionError(TaskError): # 包装用户异常，保留 __cause__ +class ConcurrencyConflict(TaskError): # expected_version / actual_version +class StoreError(TaskError): ... +class ConfigurationError(TaskError): ... # 如循环依赖、无效策略组合 +``` + +约定：**用户代码的异常必须被包装并保留 `__cause__`**，绝不吞掉 traceback（任务失败后 `Failure.traceback` 必须完整可读）。 + +--- + +## 7. 实现计划 + +### 7.1 里程碑 + +| 里程碑 | 阶段 | 交付物 | 验收标准 | 估时 | +|---|---|---|---|---| +| **M1 行走骨架** | P0 | 包重命名、ruff/mypy/pytest/hypothesis 配置、CI | `uv run pytest` 绿；`import chongming_task` 成功 | 0.5d | +| | P1 | `states.py` `models.py` `policies.py` `errors.py` + 状态机 | 转移表 100% 分支覆盖；hypothesis 属性测试通过（无终态逃逸） | 1.5d | +| | P2 | 取消令牌、超时、心跳、`Inline`/`Thread`/`Process` 执行器 | 可跑通"带重试+超时+取消"的单个任务；不配合的线程记 `LOST` 且不阻塞退出，不配合的子进程被真的杀掉 | 2d | +| **M2 控制面** | P3 | 调度器、DAG、依赖、重试退避、并发限流、管道聚合 | 100 节点随机 DAG 压力测试无死锁；`max_parallel` 与具名信号量被遵守 | 2.5d | +| | P4 | `Controller` 门面、句柄、事件总线、钩子、`force_transition` + 审计 | §3.1 语义表逐条有测试；每个动词都有幂等性与非法前置状态用例 | 1.5d | +| **M3 可运维** | P5 | `SqliteStore`、版本冲突、`reconcile()` 恢复 | 杀进程后重启，非终态任务被正确判定为 LOST 并按策略处理 | 2d | +| | P6 | CLI（`run/ps/status/stop/pause/resume/retry/logs/events`）、日志与指标 sink | CLI 端到端演示：启动 → 观察 → 暂停 → 恢复 → 停止 → 重试 | 1.5d | +| **M4 服务化（可选）** | P7 | HTTP/ASGI 控制面、远程执行器草案 | OpenAPI 覆盖全部控制动词 | 待评估 | + +到 P6 合计约 **11.5 人日**。 + +### 7.2 分阶段任务细目 + +**P1 领域内核**：`TaskStatus` + 别名 → `TRANSITIONS` 表 + `TERMINAL`/`REVIVABLE` → `Transition` 对象（携带 trigger/actor/reason/version）→ `StateMachine.apply()` → 快照与策略数据类 → 错误体系。 +*验收*：`tests/property/test_state_machine.py` 用 hypothesis 随机生成事件序列，断言：终态不可离开、`status_version` 单调递增、任何被接受的转移都在表中、被拒绝的转移状态零变化。 + +**P2 执行运行时**：`CancellationToken`（支持 stop/pause 两种信号，`checkpoint()` 语义）→ `Deadline`/`Timeout` 监视器 → `HeartbeatReporter` → `Executor` Protocol（`launch` / `signal_graceful` / `force_kill` / `join`）→ 实现（**最终只有两个**：`ThreadExecutor` 与 `InlineExecutor`）。 +*验收*：任务在协作路径上能被停止/暂停；不配合的线程被**如实**记 `LOST`（`abandoned=True`）且不阻塞进程退出。原计划的 `ProcessExecutor` 验收项已作废（§13.10）。 + +**P3 调度与 DAG**：`Graph`（建图、环检测、就绪队列）→ `Scheduler`（事件驱动 + 并发槽 + 优先级）→ 退避重试 → `AggregationPolicy`（`fail_fast` / `all_settled` / `quorum(n)`）→ 扇出 `map`。 +*验收*：随机 DAG（含菱形、长链、宽扇出）压力测试无死锁、无重复执行；`fail_fast` 下上游失败后下游正确进入 `CANCELLED`/`SKIPPED`。 + +**P4 控制面 API**：`TaskContext` → `@task` 装饰器（签名解析、参数校验）→ `Pipeline` DSL → `TaskHandle`/`PipelineRun` → `Controller` 门面 → `EventBus` → 生命周期钩子（`on_start/on_success/on_failure/on_retry/on_cancel`）。 +*验收*：§3.1 每个动词 × 每个前置状态 的参数化测试矩阵全绿。 + +**P5 持久化**：`TaskStore` Protocol → `InMemoryStore` → `SqliteStore`（WAL + 索引：`status`、`pipeline_run_id`、`created_at`）→ `AttemptRecord` 追加 → `Reconciler`。 +*验收*：契约测试套件对两个 store 实现同样通过；进程被 `kill -9` 后重启，孤儿任务按 `on_orphan` 策略正确落终态。 + +**P6 CLI 与观测**：`typer`（或 argparse，保持零依赖则用 argparse）CLI → 结构化日志 formatter → `MetricsSink` 挂点 → `NullSink` 默认。 +*验收*：演示脚本完整走一遍控制流，`--json` 输出可被脚本消费。 + +### 7.3 测试策略 + +| 层次 | 手段 | 目标 | +|---|---|---| +| 属性测试 | hypothesis 状态机测试 | 转移表是唯一真相，穷举非法序列不可破坏不变量 | +| 时间相关 | `FakeClock` 注入 | 重试退避/超时/心跳测试在毫秒内完成，不 `sleep` | +| 故障注入 | 会抛异常 / 会挂起 / 会丢心跳的假执行器 | 失败路径覆盖，而非只测 happy path | +| 契约测试 | 同一套 suite 跑所有 `TaskStore` / `Executor` 实现 | 新实现不偏离语义 | +| 集成 | 真实子进程的 `SIGTERM`/`kill`→Windows `CTRL_BREAK_EVENT`/`taskkill` | 强杀路径真的能工作 | +| 端到端 | 小规模真实管道（抽取→转换→加载，含一次失败重试） | 用户视角可用性 | + +CI 门槛：ruff（lint+format）、mypy strict（`core/` 必须零豁免）、pytest + 覆盖率 ≥85%（`core/` ≥95%）。 + +### 7.4 建议的起步动作（本周可做） + +1. 改包目录 `src/chongming-task` → `src/chongming_task`，同步 `pyproject.toml`；确认 `uv sync` 与 `import chongming_task` 通过。 +2. 冻结 §3.1 的动词语义表与 §2.3 的转移表（**这是最该先评审的产物**，改它的成本随时间指数上升）。 +3. 完成 P1 + P2 的最小切片，跑通一个带重试与取消的 `asyncio` 任务，作为可演示骨架。 + +--- + +## 8. 风险与对策 + +| # | 风险 | 影响 | 对策 | +|---|---|---|---| +| 8.1 | 用户同步代码不调 `checkpoint()` | `stop()`/`pause()` 只能等超时（`idle`）并落 `LOST`，体验落差 | 文档强约定（README §已知限制、用户指南 §2.2、`ctx.checkpoint()` 的 docstring）；后续可做 `--strict` 静态提示 | +| 8.2 | 线程不可强杀 | 无法保证停止 | **接受**（架构级限制）：如实标记 `LOST`；用 `timeout` + 幂等设计兜底；不再提供进程执行器（§13.10） | +| 8.3 | Windows 进程终止语义不同 | 优雅停止可能拿不到信号处理机会 | 平台分支：POSIX `SIGTERM`；Windows `CTRL_BREAK_EVENT`（需 console group）或直接 `taskkill /F`，并把 `forced=True` 如实记录 | +| 8.4 | Python 3.14 过新 | 可选依赖 wheel 缺失，阻塞 P5/P6 | 放宽到 `>=3.11`；核心包零依赖，把风险限制在可选适配器 | +| 8.5 | 状态并发写 | 状态错乱、事件重复 | 单写者 + `status_version` 乐观锁；冲突即抛错 | +| 8.6 | 进度/日志事件洪水 | 存储与总线被压垮 | 默认节流 + 采样 + 有界队列（满则丢弃并计数，不阻塞任务） | +| 8.7 | 范围膨胀（想做分布式/Web UI） | 里程碑失控 | §9 非目标写死；P7 之前不引入网络与多进程调度 | +| 8.8 | 暂停语义被误解为抢占式 | 用户预期落空 | 文档显式声明"边界暂停 + 协作挂起，非抢占" | + +--- + +## 9. 非目标（v1 明确不做） + +- 分布式调度、跨机器 worker、调度高可用 +- 抢占式暂停（含 POSIX `SIGSTOP`） +- Web UI / 图形化编排 +- 任意 Python 代码的自动检查点与图重算（Dask/Ray 式） +- 数据血缘、列级血缘 +- 多租户隔离与权限体系 + +这些留作 v2+ 的扩展点：`Executor` 与 `TaskStore` 抽象已为其预留位置。 + +--- + +## 10. 待确认的决策点 + +| # | 问题 | 结论 | +|---|---|---| +| D1 | 主场景是什么？ | **(c) 两者都要。** 执行器与 DAG 在 v1 即解耦：`Executor` 抽象同时覆盖进程内函数与外部长驻命令，调度层不认识执行器差异。 | +| D2 | v1 是否需要重启恢复（持久化）？ | **需要。** `TaskStore` 语义在 M1 就冻结，P5（SQLite + `reconcile()`）留在 M3 但不可砍。工期维持 ~11.5d。 | +| D3 | Python 版本下限与第三方依赖策略？ | **`requires-python = ">=3.11"`**（已改）。核心包零第三方运行时依赖，观测/存储适配器独立。已设 `ruff`/`mypy` 的 `target-version = py311`，防止误用 3.12+ 语法。 | +| D4 | 是否同意重命名包目录为 `chongming_task`？ | **已同意且已生效**，`pyproject.toml` 的 `packages` 同步为 `["src/chongming_task"]`。 | +| D5 | 控制面形态？ | **纯库 + CLI。** v1 不做 HTTP 服务；P6 落地 CLI，P7（服务化）保持可选。 | +| D6 | 暂停能力期望？ | **已定案：协作式暂停（边界暂停 + 协作挂起），不做抢占式。** 详细语义与取舍见 §11。P2 的 `CancellationToken` 已按此实现。 | + +--- + +## 11. 暂停语义详解（对应 D6） + +> **结论（已定案）**：采用**协作式暂停**，不做抢占式。以下解释保留，用于说明 +> 为什么这样选，以及使用者需要遵守哪些约束。 + +### 11.1 两种截然不同的"暂停" + +`pause()` 有两种可能实现，能力边界差别很大，必须先分清： + +| | 抢占式暂停 | 协作式暂停 | +|---|---|---| +| 实现手段 | POSIX `SIGSTOP` 冻结进程 | 任务在检查点主动检查暂停标志并挂起 | +| 生效时机 | 立即（任意指令处） | 下一个 `ctx.checkpoint()` | +| 跨平台 | **否**（Windows 无 `SIGSTOP` 等价物） | 是 | +| 持锁风险 | **高**：冻结时若持有锁/事务，会阻塞其他任务 | 无（任务自己挑安全点） | +| 对不检查标志的同步代码 | 有效 | **无效** | +| 实现复杂度 | 低（发信号） | 中（令牌 + 检查点 + 状态往返） | + +### 11.2 本方案的选择：协作式 + 边界暂停 + +`controller.pause(task_id)` 的实际语义是**两件事**： + +1. **不再派发新工作**——该任务的下游不会获得新的执行槽；管道级 `pause()` 停止向所有未开始的任务派发。 +2. **运行中的任务在下一个检查点挂起**——`ctx.checkpoint()` 检测到暂停请求后阻塞，直到 `resume()` 或 `stop()`。 + +状态往返：`RUNNING → PAUSING`（已请求、尚未静默）→ `PAUSED`（真正停住）。 + +恢复目标取决于**暂停发生在哪一层**（详见 §2.3 的 v1.3 修正）：就地暂停的任务回到 +`RUNNING`——它从未释放执行槽，调用栈还在内存里；从未启动就被暂停的任务回到 +`QUEUED`。**暂停不释放并发槽位**，因此暂停大量任务可能耗尽并发额度。 + +若任务在暂停窗口内没到达检查点（例如一段不调用 `checkpoint()` 的紧密循环），状态**回滚为 `RUNNING`** 并记录告警。这是有意为之：假装已暂停会让用户以为资源已释放，比如实报"没能暂停"更危险。 + +### 11.3 为什么不做抢占式 + +- **跨平台**：Windows 没有可用的 `SIGSTOP` 等价物（`SuspendThread` 需注入且不安全）。D3 已把支持范围定为含 Windows 的 `>=3.11`，抢占式会在 Windows 上直接失效。 +- **持锁冻结**：冻结一个持有数据库连接或锁的任务，会让其他任务一起卡死——把"暂停一个任务"变成"暂停整条管道"。 +- **线程本就不能安全强杀**（见 §8.2）：抢占式在线程执行器上本就无法保证。 + +### 11.4 由此产生的三条使用约束（必须写进用户文档） + +1. **长耗时循环必须调用 `ctx.checkpoint()`。** 不调用 = 只能靠超时强杀，`pause()` 与优雅 `stop()` 都会失效。 +2. **需要"停止保证"的任务才升级到进程执行器。** 线程的优雅停止完全可用（走 `checkpoint()`），只是优雅窗口超时后无法强杀，只能放弃并记 `LOST`。选型见 §13。 +3. **`pause()` 不是资源隔离手段。** 它不保证立即释放 CPU 或连接。 + +### 11.5 如果你确实需要抢占式 + +可行但受限：只有走进程/子进程的方案才能做抢占式暂停（POSIX `SIGSTOP`/`SIGCONT`；Windows 无对应能力，只能降级为协作式），而进程执行器已被删除（§13.10）。**结论：不提供**——若将来真的需要，它与 §13.10 的取舍要一起重新评估，因为抢占式暂停正是进程方案唯一不可替代的能力。 + +--- + +## 12. 实现进度 + +| 阶段 | 状态 | 产物 | +|---|---|---| +| P0 项目基线 | ✅ 完成 | 包目录 `src/chongming_task`；`requires-python >= 3.11`；`[dependency-groups] dev`；ruff / mypy strict / pytest + hypothesis 配置 | +| P1 领域内核 | ✅ 完成 | [states.py](src/chongming_task/states.py)（15 态 + 转移表 + 派生终态集）、[models.py](src/chongming_task/models.py)（快照与正交维度）、[errors.py](src/chongming_task/errors.py)、[policies.py](src/chongming_task/policies.py)、[state_machine.py](src/chongming_task/core/state_machine.py)（唯一写者，含审计与版本号） | +| P1 测试 | ✅ 完成 | `tests/unit/` + `tests/property/`：转移表结构性不变量、状态机全生命周期、停止/暂停/恢复语义、重试预算与 `force` 路径、人工改写审计、快照一致性；hypothesis 跑 500 组随机事件序列验证五条不变量 | +| P1 待补 | ✅ 完成 | `FakeClock` 已提升为随包发布的 [testing/clock.py](src/chongming_task/testing/clock.py)，测试不再各写一份 | +| P2 控制原语 | ✅ 完成 | [clock.py](src/chongming_task/clock.py)（可注入时钟）、[cancellation.py](src/chongming_task/runtime/cancellation.py)（协作式取消/暂停令牌，含异步路径）、[heartbeat.py](src/chongming_task/runtime/heartbeat.py)、[timeouts.py](src/chongming_task/runtime/timeouts.py)（三档超时的纯计算判定）、[context.py](src/chongming_task/runtime/context.py)（`RuntimeContext`，协作点与心跳的唯一入口） | +| P2 执行器 | ✅ 完成 | [base.py](src/chongming_task/runtime/executors/base.py)（请求/结果/句柄协议 + `LocalExecutionHandle` 三段式停止 + `run_target`：`async def` 任务在本线程把协程跑完）、[thread.py](src/chongming_task/runtime/executors/thread.py)、[inline.py](src/chongming_task/runtime/executors/inline.py)、[process.py](src/chongming_task/runtime/executors/process.py)（子进程：硬性停止、绕开 GIL、崩溃隔离；决策史见 §13.10）。`subprocess` 与 `AsyncExecutor` 不做 | +| P2 可用入口 | ✅ 完成 | [run.py](src/chongming_task/run.py)：`run_task()` 与 `TaskRun`，把状态机 / 令牌 / 心跳 / 超时 / 执行器串成可用的单任务受控执行 | +| 管理门面 | ✅ 完成 | [controller.py](src/chongming_task/controller.py)：`TaskRegistry`（定义白名单）、`Controller`（提交 / 查询 / 控制 / 并发额度排队）、`StatusCounts`、`render_table`（CLI 与看板共用） | +| 并发策略 | ✅ 完成 | `ConcurrencyPolicy` 由调度器执行：全局 `max_parallel`、按标签的**具名资源信号量**、**优先级**（`submit(priority=...)`，默认值来自策略）。三条都在 `tests/unit/test_controller.py` 有用例；`concurrency` 与 `max_parallel` 同时给出会报错，避免"谁说了算"靠猜 | +| 持久层 | ✅ 完成 | [stores/](src/chongming_task/stores/)：`TaskStore` 契约、`_codec`（SQLite 行编解码）、`InMemoryTaskStore`、`SqliteTaskStore`（标准库 `sqlite3`，WAL + 乐观并发 + 指令租约 + 交还）；[store_contract.py](src/chongming_task/testing/store_contract.py) 随包发布，第三方后端据此自证 | +| 存储接线 | ✅ 完成 | `Controller(store=...)`：状态迁移**写穿**（乐观并发 + 迁移历史）、查询**读穿**（跨进程可见）、控制**指令化**、进度心跳经 `touch` 高频旁路回写；存储故障记入 `persist_failures` 而不让任务失败 | +| 公开 API | ✅ 完成 | [__init__.py](src/chongming_task/__init__.py) 收敛全部受支持的表面，便于嵌入其他应用 | +| DAG 管道 | ✅ 完成 | [pipeline.py](src/chongming_task/pipeline.py)：静态定义（`Pipeline` / `Node`）+ 运行时推进（`PipelineRun`）。依赖复用**状态机里已有的** `blocked` 状态，不另造标记；等待中的节点不占并发额度。逐节点 `JoinMode` 与整条管道 `AggregationPolicy` 是两个独立的轴。`CANCELLED` 不作 fail_fast 触发条件（它是结果不是原因） | +| 任务编写 API | ✅ 完成 | [decorators.py](src/chongming_task/decorators.py) 的 `@task`（**不产生全局注册表**，规格挂在函数上）+ [runtime/context.py](src/chongming_task/runtime/context.py) 的 `TaskContext`：`checkpoint` / `progress` / `log` / `input`。进度走 `touch` 旁路（进快照与存储，不碰版本号）；上游数据在放行下游前注入 | +| 重启恢复 | ✅ 完成 | **监督者租约**（`lease_owner` / `lease_until`，经 `touch` 续租）+ `Controller.reconcile()`：判据是"还有人在管它吗"而非"它还在推进吗"，因此不会误杀心跳停滞的健康任务。只如实标记，**不重新执行**（存储里没有参数） | +| CLI | ✅ 完成 | [cli/main.py](src/chongming_task/cli/main.py)：纯标准库 `argparse`，11 个子命令。**只传任务名、绝不接受代码**；输出契约与退出码见 [cli.md](cli.md) | +| 文档 | ✅ 完成 | [README.md](../README.md)（快速上手 + 真实输出）、[user-guide.md](user-guide.md)（面向使用者）、[cli.md](cli.md)（命令行）、[store-guide.md](store-guide.md)（持久层实现者）、[examples/](../examples/)（十二个可直接运行的脚本 + 两个 CLI 任务模块，输出均为实跑结果） | + +当前质量门禁（均通过）： + +``` +uv run pytest -q → 563 passed, 44 skipped （跳过的是需要真实 Postgres 的那批） +CHONGMING_TEST_POSTGRES=... uv run pytest -q → 607 passed （一台可连的库，零跳过） +uv run ruff check . → All checks passed +uv run ruff format . → unchanged +uv run mypy → Success: no issues found in 34 source files +``` + +> 属性测试在首次运行时就抓到两处真实缺陷：一是 `_PAUSE_IS_DEFERRED` 包含 `STARTING` +> 但转移表缺少 `STARTING → PAUSING` 边；二是 `QUEUED → PAUSING → abort_pause` +> 会产出没有 `started_at` 的 `RUNNING` 快照。这两处都不是靠读代码发现的，而是 +> 随机事件序列的组合跑出来的——这正是把不变量写成可执行断言的价值。 +> +> P2 期间同类问题又出现一次：`idle` 超时原本不要求任务已开始执行，于是一个 +> **在队列里排了很久但从未启动**的任务会被判定为"卡死"。已修正为 `idle` 仅在 +> `started_at` 存在时参与竞争——"卡住"是运行中才有的概念。 +> +> 同一阶段还修掉一处**契约级**缺陷：执行请求原本携带捕获了上下文的闭包，而令牌 +> 不可 pickle，导致这类任务永远无法迁移到进程执行器。修正为"上下文由执行器注入" +> 之后，同一份任务定义才能自由迁移（详见 §13.3）。这类问题在只写线程执行器时 +> 完全看不出来，是**为了写进程执行器**才被迫暴露的——那份实现后来被删掉了 +> （§13.10），但它逼出来的这条契约留了下来，而且理由比当初更硬。 +> +> 进程执行器落地时又抓到一处（该模块现已删除，见 §13.10，但这条原则留下了）： +> 强杀之后 `handle.running` 一度仍返回 `True`—— +> 因为判据是"读取线程是否收工"，而从子进程被杀死到读取线程观察到 EOF 之间存在 +> 窗口。改为直接判断"载体是否还活着"后消失。这个 bug 只在断言 `running is False` +> 的用例里才会暴露，而写那条断言的理由是**上一轮刚定下的诚实性原则**。 +> +> **收口线程模式时的同一类清理**：进程执行器一度连模块带测试一起删掉，只服务于它的 +> 字段（`Failure.exit_code`、快照上的 `pid`）也一并删掉——没有任何代码路径会写它们， +> 留着只会让"存了但永远是 NULL"看起来像有意义的字段。同时把 `ConcurrencyPolicy` 真正 +> 接进调度器：`limits`（按标签的具名信号量）与 `priority` 此前只是声明、校验、写进 +> 文档，调度器从不读它们，设了等于没设且**不报任何错**；现在由 `Controller` 执行， +> 用例钉在 `tests/unit/test_controller.py`（§13.5）。 +> +> 后来进程执行器被**重新实现并纳入支持范围**（理由与决策史见 §13.10），那两个字段也 +> 随之回来——这次它们有生产者、有测试：`pid` 由 `ProcessExecutionHandle` 写入快照 +> （`ps` 对得上号的那个进程号），`exit_code` 由子进程的非零退出/崩溃路径写入 `Failure`。 + +--- + +## 13. 执行器选型（线程优先） + +### 13.1 为什么线程是细粒度管道的默认选择 + +一个常见误解是"任务控制工具应该用进程，因为进程能被杀掉"。**强杀是一项可选能力， +不是默认前提。** 默认应当用线程，理由： + +| 维度 | 线程 | 进程 | +|---|---|---| +| 创建开销 | 几十微秒 | 毫秒~几十毫秒 | +| 参数/返回值 | 直接传递对象 | **必须 pickle** | +| 可用的任务写法 | 闭包、lambda、局部函数、装饰器产物 | 上述**全部不可用** | +| 共享状态 | 连接池、缓存、已加载模型可直接复用 | 各自一份，内存翻倍 | +| 优雅停止（协作式） | ✅ 完全可用 | ✅ 可用 | +| **强制停止** | ❌ 只能放弃 | ✅ `terminate` → `kill` | +| CPU 并行 | ❌ 受 GIL 限制 | ✅ 真并行 | +| 崩溃隔离 | ❌ 段错误会带走整个进程 | ✅ | + +对一条由 `@task` 装饰函数组成的管道来说，"必须可 pickle"是实实在在的负担： +用装饰器、闭包或偏函数包装过的任务在进程执行器下直接不可用。而绝大多数管道步骤 +是 I/O 密集（数据库、HTTP、文件），线程完全够用。 + +### 13.2 选型矩阵 + +| 场景 | 选择 | 状态 | +|---|---|---| +| 阻塞式 I/O（DB / HTTP / 文件）、普通业务函数 | **`ThreadExecutor`（默认）** | ✅ 支持范围内 | +| 微秒级、确定性的纯计算步骤（同步运行路径 / 测试） | `InlineExecutor` | ✅ 支持范围内 | +| 原生 `async` 任务（`async def` + `await ctx.acheckpoint()`） | **线程 / 内联执行器在本线程把协程跑完** | ✅ 支持范围内 | +| CPU 密集计算、需要绕开 GIL | **`ProcessExecutor`** | ✅ 支持范围内 | +| 需要"停止保证"（优雅窗口用尽后必须真的停住） | **`ProcessExecutor`** | ✅ 支持范围内 | +| 运行不可信代码 / 依赖原生扩展可能段错误（崩溃隔离） | **`ProcessExecutor`** | ✅ 支持范围内 | +| 长驻外部命令、需要明确的退出码与信号控制 | 在任务里用 `subprocess` 自己管 | ℹ️ 不做执行器 | + +> **`AsyncExecutor` 不做**：多余。执行器只需要在本线程把协程跑完（`asyncio.run`）， +> 再包一层执行器只会让"取消令牌该由谁持有"多出一个需要同步的副本。 +> +> **`ProcessExecutor` 的取舍**：它服务上面三种场景，代价是三条结构性约束（可 pickle、 +> 模块级目标 + `__main__` 守卫、停止/暂停有轮询延迟）。默认仍是线程执行器—— +> 强杀是**可选能力，不是默认前提**（§5.4）。完整决策史见 §13.10。 + +### 13.3 令牌必须由执行器**注入**，不能被任务捕获 + +这是 P2 实现期发现并修正的一处契约缺陷，值得单独记录。 + +最初的设计里，执行请求携带一个**无参闭包**： + +```python +# 修正前的写法：闭包捕获 ctx，而 ctx 里装着令牌 +ExecutionRequest(task_id=..., call=lambda: user_func(ctx, source)) +``` + +闭包天然会捕获 `ctx`，而 `ctx` 里装着 `CancellationToken`——它的内部是 +`threading.Condition`，**根本不可 pickle**。当时的后果是：凡是用了闭包、`lambda` 或 +`functools.partial` 的任务，**永远无法迁移到进程执行器**——而进程执行器现在是**在用的** +（§13.10），这条约束也就照旧生效，`ProcessExecutor` 在 `launch()` 时就会把这类目标 +挡在门外并给出提示。 + +修正后的契约是：请求携带 `(target, args, kwargs)`，上下文由执行器构造后**注入** +为第一个位置参数： + +```python +def extract(ctx, source: str) -> list[Row]: ... +``` + +**这个约定在只剩线程执行器之后依然成立**，理由换了一个但同样硬：上下文属于**某一次 +尝试**，在启动那一刻才由执行器造出来。若靠闭包捕获，任务定义里握住的只可能是上一次 +尝试留下的令牌——停止指令会发给一个已经结束的执行，而且完全静默。 + +代价是要写成 `def f(ctx, ...)`；收益是任务定义不持有任何执行期状态，因此既能被线程 +执行器驱动，也能被内联执行器同步跑完，还能被测试直接调用。 + +### 13.4 控制能力长在句柄上 + +`ExecutionHandle` 同时持有令牌、心跳与执行载体，因此 `stop` / `pause` / `resume` +都是句柄方法，控制面不必自己拼装这三者，也不会出现"令牌与载体对不上号"。 + +语义完全一致的三件事收敛在基类 `LocalExecutionHandle`，子类只需回答两个问题： +还在跑吗（`running`），以及优雅窗口用尽后怎么办（`_on_stop_timeout`）。 +`stop()` 的三段式因此只写一遍： + +``` +请求停止 → 等待优雅窗口 → 兜底（线程不可强杀：放弃并如实报告） +``` + +兜底结论一旦产生就是**粘性**的：已记录的 `LOST` 不会被后台载体日后返回的结果 +悄悄改写，否则审计记录会自相矛盾。 + +### 13.5 线程执行器的两条实现约束 + +**一、必须用守护线程，不能用 `ThreadPoolExecutor`。** `ThreadPoolExecutor` 的工作 +线程会在解释器退出时被 `join`，于是一个不响应取消的任务会让**整个进程无法退出**。 +对一个专门用来监督"可能不配合"的任务的工具来说，这是致命缺陷；此外池子也无法 +放弃卡住的线程。因此 `ThreadExecutor` 每个任务起一个守护线程。 + +代价是：被放弃的线程会**泄漏**，并且它可能仍在后台产生副作用。所以放弃的结果记为 +`LOST`（未知），绝不记成 `STOPPED`。 + +**二、并发上限由调度器统一负责。** 执行器内部不再叠线程池，否则两层限制互相掩盖。 +`ConcurrencyPolicy.max_parallel` 与具名资源信号量都在调度层实现——**这一条已交付**： +`Controller(concurrency=ConcurrencyPolicy(...))` 的全局额度、按标签的具名信号量与 +优先级都由调度器执行（`Controller._can_start_locked` / `_take_runnable_locked`）， +用例钉在 `tests/unit/test_controller.py` 的"具名资源信号量与优先级"一节。 +执行器仍然一层池都没有：需要限流时改策略，不要改执行器。 + +顺带把 async 也收在这一层：`async def` 任务由执行器在**本线程**把协程跑完 +（`run_target`），因此不需要 `AsyncExecutor`，`await ctx.acheckpoint()` 照常工作。 + +### 13.6 放弃语义的诚实性要求 + +线程执行器的 `stop(graceful_timeout=T)` 有两种结局，绝不能混淆： + +- 任务在 T 秒内响应停止 → `abandoned=False`，`Failure.kind = CANCELLED`， + `retryable=False`（取消不是失败，重试它等于无视停止指令）。 +- 超时未响应 → `abandoned=True`，`Failure.kind = LOST`，且 `handle.running` + **仍然返回 `True`**——它确实还在跑。谎报为 `False` 会让上层以为资源已释放。 + +同一个诚实性原则也用在暂停上：`handle.pause(timeout=T)` 在任务未到达检查点时 +返回 `False`。控制面据此把状态回滚为 `RUNNING` 并告警，而不是记成 `PAUSED`。 +`InlineExecutor` 同样如此——它无法被暂停，就直接返回 `False`，不假装成功。 + +### 13.7 线程与进程的停止结局完全不同 + +这是选型时最该看清的一点：两种载体的"停了"含义不同——一个是**未知**，一个是**已知**。 + +| | `ThreadExecutor`（任务不配合） | `ProcessExecutor`（任务不配合） | +|---|---|---| +| 优雅窗口用尽后 | 放弃等待 | `terminate()` → 宽限 → `kill()` | +| `abandoned` | `True` | `False` | +| `Failure.kind` | `LOST`（**结果未知**） | `CANCELLED`（**结果已知**：任务未完成） | +| `handle.running` | 仍为 `True`（它确实还在跑） | `False`（确已死） | +| 后台副作用 | 可能继续产生 | 已终止 | + +也就是说：**线程的"停不掉"是一个未知状态，进程的"杀掉"是一个已知状态。** +把两者都简化成"已停止"是不诚实的，所以 `LOST` 与 `CANCELLED` 必须分开。 + +### 13.8 进程执行器的四处跨进程困难 + +实现进程执行器时逐一撞上的问题，都记在这里以免后人重踩（任何跨进程方案都会撞上 +这四条；`ProcessExecutor` 的每一处实现都对应其中一条）： + +1. **令牌传不过去。** `CancellationToken` 内部是 `threading.Condition`，不可 + pickle。子进程自建本地令牌，由轮询线程把 `multiprocessing.Event` 映射到它上面。 + 代价：**进程的协作式停止/暂停有轮询延迟**（当前 50ms），不像线程那样即时。 +2. **心跳传不过来。** 子进程的 `Heartbeat` 对象父进程看不见。若不管它，`idle` + 超时对进程执行器就完全失效——而它是唯一能发现"卡住但没报错"的机制。做法是让 + 每次 `checkpoint()` 顺带经管道回传一次心跳（节流），暂停期间由镜像线程代为回传 + （被暂停的任务不是卡死的）。 +3. **管道不是线程安全的。** 任务线程（心跳）与镜像线程（暂停期心跳）会并发写入 + 同一条管道，必须加锁，否则内容交错损坏。 +4. **父进程必须持续排空管道。** 若只在 `wait()` 里读，心跳就只在调用方主动轮询时 + 推进，第 2 点的努力又白费。因此父进程起一个读取线程，与是否有人 `wait` 无关。 + +### 13.9 进程执行器的两条使用限制 + +> 这两条限制**现在就生效**（`ProcessExecutor` 已纳入支持范围，见 §13.10）。它们不是 +> 实现缺陷，而是进程方案的固有成本——当年正是因为它们，这个执行器被搁置过一次。 + +**一、target / args / kwargs 必须可 pickle。** 即不能是 `lambda`、闭包或局部函数。 +这条限制在 `launch()` 时就会以 `ConfigurationError` 抛出，并附带"这类任务请改用 +线程执行器"的提示——而不是等子进程里神秘失败。 + +**二、子进程是守护进程，不能再创建自己的子进程。** 这是为了解释器退出安全： +`multiprocessing` 在退出时会先 `terminate()` 守护子进程再 `join()`，因此不响应 +停止的任务不会让整个进程卡住。代价是 `daemon=True` 的进程不允许再有子进程。 +若任务本身需要多进程，请用 `ThreadExecutor` 或把并行放在任务内部自行管理。 + +### 13.10 进程执行器：删过一次，又请回来了 + +**当前状态：已实现并纳入支持范围**（`runtime/executors/process.py`），公开 API 导出 +`ProcessExecutor`，用户指南 §7.4 有使用说明，测试在 `tests/unit/test_process_executor.py` +与 `tests/integration/test_process_execution.py`。 + +这段决策史值得完整留着，因为它是一个**反复动摇过**的决定，而每次动摇的理由都成立： + +| 阶段 | 决定 | 当时的理由 | +|---|---|---| +| 最初 | 实现、测试，但**搁置**：不导出、不宣传 | 线程更贴合以 `@task` + 闭包 + 共享连接池为主的写法，而进程要求可 pickle、参数可传输 | +| 收口线程模式时 | **删除**（模块、测试、只服务于它的 `Failure.exit_code` 与快照 `pid` 一起删） | "不再投入"的代码留着就是负债：没人维护的实现比没有实现更危险 | +| 现在 | **重新实现并纳入支持范围** | 三种场景只有它能覆盖：**硬性停止保证**（线程只能放弃并记 `lost`）、**绕开 GIL**、**崩溃隔离**（段错误只带走子进程） | + +重新引入时按这份文档里记着的约束逐条落地，没有重蹈覆辙： + +* **可 pickle 的失败要当场报**（§13.9 第一条）：`launch()` 先试 pickling，失败抛 + `ConfigurationError` 并附"这类任务请改用线程执行器"——不留"子进程里神秘失败"； +* **不给"取不到结论就杀"留下含糊空间**：子进程没给结论就消失时，按"是否报告过 + started"分成两种失败：从未开始（`nonzero_exit`，重试是干净的）与跑一半没了 + （`nonzero_exit` + "结果未知，可能有部分副作用"）； +* **父进程必须持续排空管道**（§13.8 第四条）：心跳就在那条管道上，只在 `wait()` 里读 + 会让 `idle` 超时在任务健康时误触发，所以句柄起一个独立读取线程； +* **判活以 `Process.is_alive()` 为准，不靠管道 EOF**：子进程若在反序列化参数时就死掉， + 父进程那侧的写端句柄从未交出去，管道**永远不给 EOF**（实测），只等 EOF 会一直阻塞； +* **停止/暂停的语义一条都没有重写**：父进程的 `Event` 由子进程里的轮询线程映射到 + 本地令牌上，于是"暂停中的任务也要停得掉"这类语义跨进程与同进程共用同一份实现。 + +三条代价依旧成立，并且**写在用户能看见的地方**（README 状态表、用户指南 §7.4、 +`ProcessExecutor` 的模块文档）：可 pickle 约束、模块级目标 + `__main__` 守卫、 +约 20ms 的停止/暂停轮询延迟。默认执行器仍然是线程。 + +**这段历史给出的判据**：进程执行器不是"更好的执行器"，而是一组**用约束换能力**的 +取舍。需要那三种能力时它不可替代；不需要时它的约束纯是负担。所以选型看场景 +（§13.2 的矩阵），不要看"哪个更强大"。 + +--- + +## 14. 管理后台：CLI 形态与跨进程约束 + +原计划（D5）是"纯库 + CLI"，HTTP 服务放到 P7；非目标里还写着"Web UI / 图形化 +编排"。后来提出的"统一管理后台（启动 + 查询运行情况）"确认了同样的方向并去掉了网络层： +**只做 CLI，不引入网络层**。 + +### 14.1 一个绕不过去的约束 + +CLI 的 `ps` / `status` / `stop` 是**另一个进程**，它看不到任务进程的内存。所以 +"跨进程查询"必须有共享媒介： + +| 方案 | 引入网络层 | 说明 | +|---|---|---| +| 共享存储（`sqlite3`） | 否 | 标准库自带，无 IPC，天然跨平台，**顺带就是持久化** | +| 本地 IPC（Unix socket / 命名管道） | 否 | 仍需写 IPC 与心跳，且平台分叉 | +| 不跨进程 | 否 | CLI 只能管自己启动的任务，不成其为"后台" | + +结论：**`sqlite3` 属于标准库**，因此第一条路既满足"不引入网络层"，也满足"核心包 +零第三方依赖"，还一并交付了持久化。于是"持久化随后"实际上是**提前**了——它不是 +额外工作，它**就是 CLI 的媒介本身**。 + +### 14.2 分层 + +``` +CLI（P6） ← 薄适配：解析参数、渲染 render_table + │ +共享存储 sqlite3（下一步） ← 跨进程媒介 + 持久化 + │ +Controller / TaskRegistry ← 已完成：提交、查询、控制、并发额度 + │ +TaskRun（run.py） ← 已完成：状态机 + 令牌 + 心跳 + 超时 + 执行器 +``` + +CLI **不直接操作 `TaskRun`**，只经 `Controller`。这样内存实现与存储实现可以互换， +CLI 不必知道数据从哪来。 + +### 14.3 安全底线（无论形态如何都不能破） + +控制面**只接受任务名字，绝不接受代码**。`Controller.submit()` 要求名字先在 +`TaskRegistry` 注册过，未注册即抛 `ConfigurationError`。 + +一个能启动任意可调用对象的入口等于**远程代码执行**。现在只有 CLI 时风险尚可控 +（本机用户本来就能跑代码），但这条规则必须从第一天就立好——将来任何形态的控制面 +（本地 socket、HTTP、看板）都建立在它之上。 + +### 14.4 已知限制 + +* `Controller` 只管**当前进程内**的任务；跨进程查询要等共享存储。 +* `prune()` 丢弃终态记录以防内存增长；当前**丢弃即永久丢失**。存储落地后会改为 + "从内存淘汰、存储仍可查"。 +* 控制器级的 `retry`（复活一个已失败的任务）**尚未实现**：它需要跨运行保留尝试 + 历史，而那依赖存储。任务内部的 `RetryPolicy` 已覆盖常见的重试场景。 + +--- + +## 15. 任务分桶与分布式预留 + +### 15.1 为什么现在就要做 + +分桶涉及的两件事里，**一件便宜、一件昂贵**： + +| 事项 | 事后补做的代价 | +|---|---| +| 给任务加上桶号与业务键 | 低——但**已跑过的任务补不回来**：没有记录原始键就无法重新分桶 | +| 桶数（`num_buckets`） | **高**——它一改，几乎所有键改换桶位，局部性与隔离假设同时失效，必须迁移 | + +所以策略是：**现在就把桶变成任务的一等属性并记下原始键**，而把真正的分布式分配 +（所有权、租约、再均衡）留到真有第二个进程时再做。前者不做，后者将来会卡在 +"历史数据没有桶"上。 + +### 15.2 桶的两种用途 + +1. **本地并发隔离**——每个桶各自限流，一个租户的任务积压不会饿死其他租户。 + 这是**今天就有价值**的部分，不需要任何分布式机制。 +2. **分布式分配单位**——一台机器拥有一组桶，只处理自己那组。桶数应明显大于机器 + 数量（经验上 4~16 倍），这样某台机器退出时它的桶能较均匀地分给其余机器。 + +### 15.3 两条硬约束（都有测试钉住） + +**一、桶数一旦开始使用就不能随便改。** `bucket = hash(key) % N`。因此 `N` 是创建时 +确定的常量、部署范围内的约定；要改就得迁移——这也正是任务会同时记下 `bucket_key` +的原因：迁移需要原始键。 + +**二、绝不能用内置 `hash()`。** CPython 对 `str` 的 `hash()` 带**进程级随机盐** +（`PYTHONHASHSEED`），同一个键在不同进程里会落到不同的桶。这不只是"可能不一致"， +而是确实不一致——测试里留了一条反例直接演示： + +```python +# 两段脚本分别在 PYTHONHASHSEED=0 / 1 下运行 +hash("k0") % 64 ... hash("k19") % 64 # → 两次输出不同 +BucketPolicy(64).bucket_for("k0") ... # → 两次输出完全相同 +``` + +这条性质如果搞错，"同桶任务在同一台机器上"会**静默**失效：本地性、顺序性、缓存 +复用全部不成立，却没有任何报错。因此 `BucketPolicy` 用 `blake2b`，并有跨进程用例 +（真实子进程 + 不同 `PYTHONHASHSEED`）守着它。 + +### 15.4 已实现 vs 预留 + +**已实现**（今天就能用）： + +| 能力 | 位置 | +|---|---| +| 稳定的分桶策略（`blake2b`，跨进程/跨机器一致） | `policies.BucketPolicy` | +| 桶号与业务键作为任务的一等属性（创建时即记录） | `TaskSnapshot.bucket` / `.bucket_key` | +| 每桶独立限流，且**不引入队头阻塞** | `Controller(max_parallel_per_bucket=...)` | +| 按桶查询**下推到存储层**，SQLite 侧有索引 | `TaskQuery.buckets` / `idx_tasks_bucket` | +| 每桶任务数统计 | `Controller.bucket_counts()` | + +调度器取下一个可跑任务时是**扫描**而不是取队首：某个桶满时，队首可能正好被它挡住， +而后面属于其他桶的任务本可以先跑。直接取队首会让一个桶的积压阻塞所有桶——那正是 +分桶要解决的问题，不能在调度器里又原样引入一次。 + +**刻意预留、尚未实现**（没有消费者之前不写死代码）： + +```python +# 将来的 TaskStore 扩展：桶的所有权与租约 +def claim_buckets( + self, *, owner: str, buckets: Iterable[int], now: datetime, lease_seconds: float +) -> list[int]: ... +def renew_buckets(self, *, owner: str, now: datetime, lease_seconds: float) -> list[int]: ... +def owned_buckets(self, *, now: datetime) -> dict[int, str]: ... +def release_buckets(self, *, owner: str) -> int: ... +``` + +为什么是这个形状——它复用了**已经存在且已被验证**的四块： + +| 将来需要的 | 现在已有的 | +|---|---| +| "谁拥有哪些桶"的租约 | `Command` 的租约机制（同一套语义：到期可重取，既防重复也不怕持有者崩溃） | +| 只取自己那组桶的任务 | `TaskQuery.buckets`，已下推到存储层并在 SQLite 建了索引 | +| 每桶公平性 | `Controller` 已按桶记账（`_active: task_id -> bucket`），远端公平用同一套账 | +| 改桶数时的迁移 | `bucket_key` 已经留在每条任务上 | + +所有权表将来大致长这样： + +```sql +CREATE TABLE bucket_owners ( + bucket INTEGER PRIMARY KEY, -- 主键本身即保证"一个桶同时只有一个所有者" + owner TEXT NOT NULL, + lease_until TEXT NOT NULL, + updated_at TEXT NOT NULL +); +``` + +**不需要额外的分布式锁**：`bucket` 作主键 + 租约，就足以表达"这个桶此刻归谁"。 + +### 15.5 明确**不**预留的部分 + +避免把"预留"做成"过度设计"： + +* **没有跨节点派发/传输层。** 远程执行器是另一套设计（序列化、网络、结果回收）， + 不该借分桶顺手带出来。 +* **没有再均衡算法。** 分摊策略取决于部署形态（一致性哈希？按负载？人工指定？）， + 现在拍板只会拍错。 +* **桶数不是运行时协商出来的**，而是部署范围内的配置常量。多节点各配不同的桶数 + 是最危险的一类错误——所以 `BucketPolicy` 有值语义、可比较，便于启动时断言各处 + 一致。 diff --git a/docs/user-guide.md b/docs/user-guide.md new file mode 100644 index 0000000..f8bf393 --- /dev/null +++ b/docs/user-guide.md @@ -0,0 +1,1313 @@ +# 用户指南 + +面向使用者的完整说明。设计动机与取舍见 [设计文档](task-control-design.md)，可直接 +运行的代码在 [examples/](../examples/)。 + +> **适用范围**：单任务执行、DAG 管道、并发额度、持久化与重启恢复、CLI 均已可用 +> （状态表见 [README](../README.md#当前状态务必先读)）。真正的边界是语义上的，不是功能 +> 缺失：不调 `checkpoint()` 停不下来、暂停不释放槽位、任务不跨节点迁移——见 §11。 + +--- + +## 目录 + +1. [核心概念](#1-核心概念) +2. [写一个任务](#2-写一个任务) +3. [跑一个任务](#3-跑一个任务) +4. [控制任务](#4-控制任务) +5. [重试](#5-重试) +6. [超时](#6-超时) +7. [执行器](#7-执行器) +8. [管理一批任务（Controller）](#8-管理一批任务controller) +9. [状态模型参考](#9-状态模型参考) +10. [错误与异常](#10-错误与异常) +11. [限制与 FAQ](#11-限制与-faq) +12. [测试自己的任务](#12-测试自己的任务) + +--- + +## 1. 核心概念 + +### 1.1 任务是一条轨迹，不是一个布尔值 + +大多数"任务控制"只是把任务记成"跑着 / 成功 / 失败"。这里不是。每个任务是一条 +**只追加的状态轨迹**，每个中间态都可表达、可观察、可干预： + +``` +pending → queued → starting → running → succeeded + ↓ ↓ + retry_wait pausing → paused → running + ↓ ↓ + starting stopping → stopped +``` + +每次状态变更都会产生一条 `Transition` 记录，带**触发源**、**操作者**、**原因**和 +**版本号**： + +```python +for record in run.transitions: + print(record.from_status, "->", record.to_status, "by", record.trigger) +``` + +### 1.2 状态只表达"生命周期阶段" + +`status` 是一个枚举，只回答"它现在处于哪个阶段"。其余信息属于**正交维度**，绝不 +塞进同一个枚举： + +| 维度 | 字段 | 为什么必须分开 | +|---|---|---| +| 生命周期 | `status` | 单一枚举，便于穷举校验 | +| 重试计数 | `attempt` / `max_attempts` | "第 3 次尝试"与"正在运行"是两件事 | +| 进度 | `progress` | 运行中的任务可能 0% 也可能 99% | +| 健康 | `last_heartbeat_at` | 用于发现"没报错但也不动" | +| 结果 | `failure` / `outcome` | 失败原因是技术细节，不是生命周期 | + +最典型的反例是**把超时做成状态**。那样会立刻逼出"已超时但正在重试"这种单一枚举 +表达不了的情况。因此超时是 `failure.kind = timeout_attempt`。 + +### 1.3 状态机是唯一写者 + +只有 `StateMachine` 能改 `status`。业务代码、执行器、控制面都不能直接赋值。这条 +纪律换来三件事：非法转移会被立刻拒绝、状态历史必然完整、并发写入可检测 +（`status_version` 乐观锁）。 + +```python +from chongming_task.core.state_machine import StateMachine +from chongming_task.states import TaskStatus + +machine = StateMachine.create("t1", "抽取", max_attempts=3) +machine.mark_queued() +machine.mark_starting(executor="thread") +machine.mark_running() +machine.mark_succeeded(outcome={"rows": 42}) + +# 非法转移会抛异常，而不是静默忽略 +machine.mark_running() +# InvalidTransition: 非法状态转移：succeeded -> running；动作 'executor:launched'；当前允许：无（终态） +``` + +非法转移**抛异常**而不返回 `False`：静默失败会让"停止指令没生效"这类问题极难排查。 +需要预检时用 `machine.can(target)` / `can_stop` / `can_pause` / `can_resume` / `can_retry`。 + +### 1.4 终态与复活 + +终态**没有出边**，这是转移表里显式声明的空集。复活只能走两个显式 API： + +- `retry()`：从 `failed` / `stopped` / `cancelled` / `lost` 复活，**保留**尝试历史。 +- `reset()`：回到 `pending`，**丢弃**全部尝试历史（危险，需二次确认）。 + +--- + +## 2. 写一个任务 + +### 2.1 签名 + +任务的第一个位置参数由执行器**注入**为执行期上下文： + +```python +def my_task(ctx, source: str, *, batch_size: int = 100) -> list[Row]: ... +``` + +调用时**不要自己传 `ctx`**： + +```python +run_task(my_task, "s3://bucket/x", batch_size=500) +``` + +> **为什么 `ctx` 必须注入而不是闭包捕获？** 因为上下文属于**某一次尝试**：取消令牌 +> 在启动那一刻才由执行器造出来，任务不可能提前攥着一个。若靠闭包捕获（`lambda: f(ctx, arg)`）， +> 任务定义里握住的只可能是上一次尝试留下的令牌——停止指令会发给一个已经结束的执行， +> 而且完全静默。注入式签名让同一份任务定义既能被线程执行器驱动，也能被内联执行器 +> 同步跑完，语义只有一份。 + +### 2.2 为什么必须调用 `checkpoint()` + +```python +def my_task(ctx): + for row in rows: + ctx.checkpoint() # ← 就这一行 + process(row) +``` + +`ctx.checkpoint()` 同时做三件事： + +1. 若已请求**停止** → 抛 `TaskCancelled`； +2. 若已请求**暂停** → 在此挂起，直到恢复或停止； +3. **刷新心跳**（`idle` 超时的依据）。 + +**不调用它的任务既停不掉也暂停不了。** Python 不能安全地终止线程，也没有跨平台的 +抢占式暂停手段，所以停止与暂停只能是协作式的。这是硬约束，不是可以绕过的实现细节。 + +长循环里每轮都调用即可，开销可以忽略（无请求时它只是两次布尔判断）。 + +### 2.3 清理逻辑里的 `is_stop_requested` + +在 `except` / `finally` 块里不要再调用 `checkpoint()`——它会阻塞，导致清理永远 +完不成。改用只读属性： + +```python +def my_task(ctx): + conn = connect() + try: + for row in rows: + ctx.checkpoint() + write(conn, row) + finally: + conn.commit() # 无论如何都要收尾 + if ctx.is_stop_requested: + print("任务被中断，已提交部分结果") # log/progress 属于 P4，此处暂用 print + conn.close() +``` + +### 2.4 产物目录 + +```python +def my_task(ctx): + target = ctx.artifact_dir / "part-0.parquet" + ... +``` + +`ctx.artifact_dir` 按 `attempt` 隔离，重试不会污染上一次的产物。传 `artifact_dir` +给 `TaskRun` / `run_task` 即可启用。 + +--- + +## 3. 跑一个任务 + +### 3.1 一行版：`run_task` + +```python +from chongming_task.run import run_task + +outcome = run_task(my_task, "s3://bucket/x", task_id="job-1", name="抽取") + +if outcome.ok: + print(outcome.value) +else: + print(outcome.status.value, outcome.failure.kind.value) +``` + +`run_task` 会阻塞到任务落终态。**不要在需要中途干预时用它**。 + +### 3.2 完整版：`TaskRun` + +```python +from chongming_task.run import TaskRun + +run = TaskRun(my_task, ("s3://bucket/x",), task_id="job-1", name="抽取") +run.start() # 立即返回 +... # 这里可以做任何事：暂停、停止、轮询状态 +outcome = run.wait(timeout=60) +``` + +| 成员 | 说明 | +|---|---| +| `start()` | 启动受监督执行，立即返回；重复调用报错 | +| `wait(timeout=None)` | 等待终态；超时抛 `TimeoutError` 且**不改变任务状态** | +| `stop(reason=...)` | 请求停止，不阻塞 | +| `pause(timeout=...)` | 请求暂停并等待确认，返回 `bool` | +| `resume()` | 撤销暂停 | +| `snapshot` | 当前 `TaskSnapshot`，运行期可安全读取 | +| `status` | 当前 `TaskStatus` | +| `transitions` | 完整迁移历史 | + +`TaskRun` 是线程安全的：上述方法可从不同线程调用。所有状态机写入都在同一把锁下 +进行，因此"唯一写者"这条不变量仍然成立。 + +### 3.3 结果对象 + +```python +outcome.ok # bool：是否 succeeded +outcome.status # TaskStatus +outcome.value # 任务返回值（未成功时为 None） +outcome.failure # Failure | None +outcome.snapshot # 终态快照：attempt、时间戳、progress、failure 等 +``` + +--- + +## 4. 控制任务 + +### 4.1 停止的三段式 + +``` +请求停止 → 优雅窗口（graceful_stop 秒）→ 兜底 +``` + +- 任务在窗口内响应 `checkpoint()` → 正常收尾，落 `stopped`。 +- 窗口用尽 → 线程无法强杀，**放弃等待**并落 `lost`。 + +```python +run.stop(reason="发布窗口") # 不阻塞 +outcome = run.wait(timeout=30) +``` + +**停不掉时不会说"已停止"**： + +| | 任务配合 | 任务不配合（线程） | +|---|---|---| +| `outcome.status` | `stopped` | **`lost`** | +| `outcome.failure.kind` | `cancelled` | `lost` | +| `failure.retryable` | `False` | `True` | +| 后台副作用 | 已收尾 | **可能仍在产生** | + +把两者都简化成"已停止"不只是不准确，而是会让运维误判资源已释放。 + +### 4.2 暂停与恢复 + +```python +run.pause() # → True 表示已确认挂起，状态变为 paused +run.resume() # → running +``` + +**`pause()` 返回 `False` 不是错误，是"暂停没生效"**：任务在窗口内没到达检查点。 +此时状态**回滚为 `running`**，并留下一条 `runtime:pause_aborted` 记录，绝不假装 +已暂停。 + +两条必须知道的性质： + +1. **暂停不释放并发槽位。** 就地挂起——任务的调用栈还在内存里。我们没有办法把 + 任意的 Python 调用栈换出再换回，所以暂停大量任务可能耗尽并发额度。 +2. **暂停期间不判超时。** 被暂停的任务本来就不会推进心跳，若照常判 `idle` 超时， + 暂停会被误判成卡死。 + +### 4.3 恢复的目标状态取决于暂停层级 + +这是实现期修正过的一处语义，容易踩： + +| 暂停发生在 | 恢复后回到 | 理由 | +|---|---|---| +| 运行中（经 `pausing`） | `running` | 就地暂停，槽位从未释放，调用栈还在 | +| 排队/重试等待中（未启动） | `queued` | 从未启动过，重新排队 | + +`TaskRun.resume()` 会自动判定，无需你关心。 + +### 4.4 停止请求发出后任务恰好跑完 + +这是合法结果：状态走 `stopping → succeeded`，不判为失败。停止是请求，不是判决。 + +### 4.5 无限循环（常驻）任务：怎么启、怎么停 + +一个"永远跑下去"的任务，任务体每轮**必须**调 `ctx.checkpoint()`： + +```python +@task +def worker(ctx, source: str) -> int: + processed = 0 + while not ctx.is_stop_requested: # ← 优雅收尾的写法 + ctx.checkpoint() # ← 不调用它 = 停不掉 + processed += 1 + ctx.progress(processed, unit="batch") + time.sleep(0.02) + ctx.log(f"收到停止请求，本次共处理 {processed} 批") + return processed # 正常返回 → 终态 succeeded，返回值保留 +``` + +#### 停止请求的三种收尾方式，终态不同 + +| 任务体的写法 | 终态 | 返回值 | 含义 | +|---|---|---|---| +| `while True: ctx.checkpoint()`（让它抛 `TaskCancelled`） | `stopped` | 无 | **立刻中止** | +| 检查 `ctx.is_stop_requested` 后正常 `return` | `succeeded` | ✅ 有 | **请收工**——把已处理的结果留下来 | +| 完全不调 `checkpoint()` | `lost` | 无 | 载体被放弃，**线程仍在后台跑** | + +第二行不是笔误：状态机允许 `stopping → succeeded`，因为"停止"是请求而不是判决。对"跑到 +下一个安全点再退出"这类需求，它比 `stopped` 更有用——已经做好的结果不会丢。 + +#### 日常"先停一下"应该用 `pause` / `resume`，不是 `stop` + +`stop` 是**终态且不可逆**：当前没有 `retry`，停止之后想再跑必须换一个 `task_id` 重新提交 +（`store.create` 会拒绝重复 id）。所以： + +```python +controller.pause("orders-worker") # → True：已确认挂起；False：未确认（状态回滚 running） +controller.resume("orders-worker") # 继续跑 +controller.stop("orders-worker", reason="发布窗口") # 彻底结束 +``` + +> `pause` 是**就地挂起**：任务阻塞在 `checkpoint()` 里，**继续占用执行槽**。暂停大量常驻 +> 任务可能耗尽并发额度。 + +#### 启停的两种方式 + +```python +# 方式一：同进程（需要这个进程一直活着） +controller = Controller(store=store, owner="node-a") +run = controller.submit("worker", "kafka://orders", task_id="orders-worker") +controller.stop("orders-worker", reason="发布窗口") +outcome = controller.wait("orders-worker", timeout=10) # → succeeded，value=已处理批数 +``` + +```bash +# 方式二：命令行（前台执行，交给 systemd / nohup；停止从另一个终端发起） +# --bucket-key 在限定了 --owned-buckets 时是**必需**的：没有归属桶的任务不会有任何 +# 节点认领它，所以 CLI 会在提交时报错，而不是留下一个永远排队的任务。 +chongming-task --db tasks.db --num-buckets 64 --owned-buckets 0-15 \ + run --tasks myapp.tasks worker --arg source=kafka://orders \ + --task-id orders-worker --bucket-key orders + +chongming-task --db tasks.db --owned-buckets 0-15 status orders-worker +chongming-task --db tasks.db --owned-buckets 0-15 stop orders-worker --reason 发布窗口 +``` + +**`run` 是前台进程**：它退出时任务就没人管了（监督线程是守护线程）。要长期跑请交给 +`systemd` / `nohup`。`Ctrl-C` 是安全的——CLI 会走 `finally` 里的 `controller.close()`， +把在跑的任务停干净。 + +#### 如果任务不调 `checkpoint()` + +它不是"停得慢"，而是**停不掉**：Python 无法安全终止线程，所以系统只能如实报 `lost` +（结果未知），而任务体**仍在后台继续执行**——真正终止它只能靠结束进程。这正是"诚实优先于 +便利"的那条设计原则：与其假装已停止，不如报"结果未知"。 + +完整可运行演示见 `examples/09_long_running.py`。 + +--- + +## 5. 重试 + +```python +from chongming_task.policies import ExponentialBackoff, RetryPolicy + +RetryPolicy( + max_attempts=5, # 总尝试次数（含首次） + backoff=ExponentialBackoff(base=1, factor=2, max_delay=300), + retry_on=(Exception,), # 哪些异常值得重试 + give_up_on=(ValueError, KeyError), # 哪些立刻放弃 +) +``` + +| 字段 | 说明 | +|---|---| +| `max_attempts` | **总**尝试次数，因此 `1` 表示不重试 | +| `backoff` | `FixedBackoff(delay)` 或 `ExponentialBackoff(base, factor, max_delay, jitter)` | +| `retry_on` | 值得重试的异常类型 | +| `give_up_on` | 命中即放弃，优先级高于 `retry_on` | + +退避默认带**抖动**（`Jitter.FULL`，在 `[0, delay)` 内均匀取值）。这不是花哨：没有 +抖动时，一整批同时失败的任务会在同一时刻集体重试，形成风暴。 + +### 5.1 取消永远不会被重试 + +`TaskCancelled` 继承自 `BaseException`，而 `retry_on` 默认是 `(Exception,)`，因此 +**取消信号天然不匹配任何重试条件**。同时 `Failure.retryable` 对取消置为 `False`。 +重试一个已被取消的任务，等于无视停止指令。 + +### 5.2 失败分类 + +每次失败都会带一个 `FailureKind`： + +| kind | 含义 | +|---|---| +| `exception` | 任务抛出了异常 | +| `nonzero_exit` | 外部命令返回非零退出码 | +| `timeout_start` / `timeout_attempt` / `timeout_idle` | 三档超时 | +| `cancelled` | 被取消 | +| `lost` | 结果未知 | +| `dependency_failed` | 上游失败 | +| `condition_false` | 条件不满足（配合 `skipped`） | + +--- + +## 6. 超时 + +```python +from chongming_task.policies import Timeout + +Timeout(start=30, per_attempt=600, idle=60) +``` + +三档相互独立，取**最早**到期者： + +| 作用域 | 含义 | 生效条件 | +|---|---|---| +| `start` | 从提交到真正开始执行 | 一旦开始执行即失效 | +| `per_attempt` | 单次尝试的总时长 | 本次尝试已开始 | +| `idle` | 距最近一次 `checkpoint()` 的静默时长 | 本次尝试已开始 | + +超时到期后系统会**请求停止**该任务，并把结果记成对应的 `timeout_*` 失败，而不是 +一个"超时状态"。超时默认**可重试**（多半是暂时性的）。 + +### 6.1 `idle` 的语义与陷阱 + +`idle` 衡量的是 **`checkpoint()` 间隔**，不是"CPU 是否在干活"。这是唯一能发现 +"卡住但没报错"（死锁、连接挂起）的机制，但它有个前提： + +> 如果你有一段几分钟不调用 `checkpoint()` 的重计算，它会被判定为疑似卡死。 + +所以 `idle` 阈值必须与任务里 `checkpoint()` 的频率相匹配。如果某段计算确实无法 +中途打断，要么把它调大，要么在进入该段之前手动 `ctx.heartbeat.beat()`。 + +另外两条边界：`idle` **只在本次尝试已经开始后才参与判定**，因此排队等待的时长不会 +被误判成静默超时；**暂停期间同样不判 `idle`**，因为被暂停的任务本来就不会推进心跳。 + +--- + +## 7. 执行器 + +### 7.1 线程执行器（默认） + +```python +from chongming_task.runtime.executors.thread import ThreadExecutor + +run = TaskRun(my_task, executor=ThreadExecutor()) +``` + +适合阻塞式 I/O（数据库、HTTP、文件）与普通业务函数。特点： + +- 创建开销几十微秒，共享内存，可复用连接池与缓存； +- 无参数序列化要求，闭包、lambda、局部函数都能用； +- `async def` 任务同样能跑：执行器在本线程内把协程跑完，`await ctx.acheckpoint()` + 照常响应停止与暂停； +- **无法被强制停止**（Python 不能安全终止线程）。 + +### 7.2 内联执行器 + +```python +from chongming_task.runtime.executors.inline import InlineExecutor + +run_task(trivial_step, executor=InlineExecutor()) +``` + +在调用线程内同步跑完，零调度开销，适合微秒级的纯计算步骤与确定性测试。 + +它**刻意违反**"`launch` 不得阻塞"的约定（这就是它存在的意义），因此**异步调度器 +不得使用它**。它也无法被暂停——`pause()` 如实返回 `False`，不假装成功。 + +### 7.3 为什么不用 `ThreadPoolExecutor` + +`ThreadPoolExecutor` 的工作线程会在**解释器退出时被 `join`**。一个不响应取消的 +任务会让整个进程无法退出——对一个专门监督"可能不配合"的任务的工具来说这是致命 +缺陷；此外池子也无法放弃卡住的线程。 + +因此 `ThreadExecutor` 每个任务起一个**守护线程**，代价是被放弃的线程会泄漏（所以 +结果记 `lost`）。并发上限由调度层统一负责，执行器不再叠一层池。 + +### 7.4 进程执行器：硬性停止 / 绕开 GIL / 崩溃隔离 + +```python +from chongming_task import ProcessExecutor, run_task + +outcome = run_task(fetch_all, executor=ProcessExecutor()) # 单个任务 +controller = Controller(executor=ProcessExecutor()) # 或整批任务 +``` + +任务跑在**独立子进程**里。它换来三件线程给不了的事： + +| 能力 | 线程执行器 | 进程执行器 | +|---|---|---| +| 优雅窗口用尽后 | 只能放弃等待，记 `lost`（结果未知） | `terminate()` → `kill()`，**真的停掉** | +| CPU 密集 | 受 GIL 限制 | 真并行 | +| 原生扩展段错误 | 带走**整个监督进程** | 只带走子进程，任务如实记一次失败 | + +代码里的体现是：超时或停止不下来的任务会被真的终止，最终状态是 `failed` +（超时/崩溃）或 `stopped`（停止请求），而不是 `lost`。快照里的 `pid` 就是那个子进程的 +进程号，`ps` 能直接对上号。 + +**代价是三条结构性的约束**，不是实现偷懒： + +1. **参数、返回值、上游数据必须可 pickle**：`lambda`、闭包、局部函数一概不行——它们 + 在子进程里找不到。违反时提交那一刻就报 `ConfigurationError` 并告诉你改用什么。 + 返回值的约束也比线程执行器严（那边只要求可 JSON 序列化）。 +2. **目标函数必须在可导入的模块里**，且调用方脚本要有 `if __name__ == "__main__":` + 守卫（`multiprocessing` 的 spawn 会重新 import 主模块）。用 `@task` 装饰过的 + **模块级**函数没问题；写在函数体里的临时函数不行。 +3. **停止与暂停有约 20ms 的轮询延迟**，信号要过进程边界；线程执行器是即时的。 + 另外子进程是守护进程，**不能再创建自己的子进程**——任务内部要并行请开线程。 + +```python +@task +def resize(ctx, path: str) -> str: # ✅ 模块级、参数可 pickle + ... + + +async def _inline(): # ❌ 闭包 / 局部函数 + ... +``` + +**什么时候该用它**：需要"停得住"的保证（例：任务里跑着别人写的、可能不响应取消的 +代码）、CPU 密集、或跑第三方原生扩展（段错误隔离）。**什么时候不该**：绝大多数 +阻塞式 I/O 任务——线程更轻、约束更少，而且能直接复用连接池与缓存。 + +--- + +## 8. 管理一批任务（Controller） + +`Controller` 是**统一的管理入口**：它持有任务定义的**白名单**、**运行中的任务** +与**并发额度**。将来的 CLI 与任何看板都只是这一层的薄适配。 + +### 8.1 基本用法 + +```python +from chongming_task.controller import Controller, render_table + +controller = Controller(max_parallel=2) # 最多同时跑 2 个 + +controller.register("extract", extract, tags={"io"}) # 先登记，之后只传名字 +controller.register("transform", transform) + +run = controller.submit("transform", 100, task_id="t0", label="转换-1") +controller.submit("transform", 200, task_id="t1", label="转换-2") + +print(render_table(controller.list_tasks())) # 查询 +controller.stop("t0", reason="手工干预") # 控制 +controller.wait_all(timeout=60) # 等全部落终态 +``` + +### 8.2 查询接口 + +| 方法 | 作用 | +|---|---| +| `get(task_id)` | 单个快照；不存在返回 `None` | +| `list_tasks(status=, name_prefix=, tag=, limit=)` | 按条件查询 | +| `stats()` | 各状态计数 | +| `runs()` | 全部 `TaskRun` 句柄（可按 id 继续控制） | +| `active_count` / `queued_count` | 真的在占用执行额度 / 正在排队 | +| `render_table(snapshots)` | 渲染等宽文本表，CLI 与看板共用同一份数据 | + +> 方法名是 `list_tasks` 而不是 `list`：后者会在类作用域里遮蔽内置的 `list`， +> 使类内所有 `list[...]` 类型注解失效（mypy 会报 "Function is not valid as a type"）。 + +### 8.3 控制接口 + +```python +controller.stop(task_id, reason=...) # → bool（是否找到该任务） +controller.pause(task_id, timeout=...) # → bool（是否**确认**已挂起） +controller.resume(task_id) # → bool +controller.stop_all(reason=...) # → int（本轮**发出**的请求数） +controller.close(reason=..., timeout=...) # 停止一切并收尾 +``` + +两个容易误读的返回值： + +- `stop_all()` 返回的是**已发出请求**的数量，不是"已停止"。请求发出后任务先进入 + `stopping`，落终态是异步的；要等就接着 `wait_all()`。 +- `pause()` 返回 `False` 表示暂停**没生效**（状态已回滚为 `running`）。 + +`close()` 会停止一切，且**排队中的任务不会真的执行任务体**——它们被"启动后立刻 +停止"，监督循环第一轮就落 `stopped`。 + +### 8.4 并发额度 + +`max_parallel` 限制**同时执行**的任务数；超出部分排队，状态为 `queued`： + +``` +TASK_ID NAME STATUS ATTEMPT ELAPSED PROGRESS FAILURE +------- ------ ------- ------- ------- -------- ------- +t0 转换-1 running 1/1 0.4s - - +t1 转换-2 running 1/1 0.4s - - +t2 转换-3 queued 0/1 - - - +t3 转换-4 queued 0/1 - - - +``` + +`queued` 是"已被受理、等待执行槽"，与 `pending`（尚未受理）不同。排队中的任务 +`ELAPSED` 显示 `-`——它还没执行过，显示一个时长会被误读成"已经跑了这么久"。 + +注意 `stats().unfinished` 是**非终态**数量，含排队与暂停；要知道"真的在占用执行 +资源"的有几个，用 `controller.active_count`。 + +#### 具名资源信号量 + +保护外部稀缺资源（连接数、配额、GPU）用 `ConcurrencyPolicy.limits`——**按标签** +记账，与全局额度叠加： + +```python +from chongming_task import ConcurrencyPolicy, Controller + +controller = Controller(concurrency=ConcurrencyPolicy(max_parallel=20, limits={"db": 4, "gpu": 1})) +controller.register("sync", sync_tenant) +controller.register("train", train_model) + +controller.submit("sync", "tenant-A", tags=["db"]) # 占一个 db 名额 +controller.submit("sync", "tenant-B", tags=["db"]) # 再占一个 +controller.submit("train", "v3", tags=["gpu"]) # 占那个唯一的 gpu 名额 +``` + +规则只有三条，都好记： + +- **没写进 `limits` 的标签不受限**，打标签不会自动限流； +- 一个任务带多个受限期标签时，**每个资源都要有余量**才放行； +- 名额算的是**活跃**任务（已占执行槽的），**含被暂停的**——暂停不释放执行槽， + 也就不该释放资源名额。 + +额度是**本进程**的账，不做跨进程协调：多节点各自配自己那一份（例如每个节点到库里 +的连接数）。 + +#### 优先级 + +`priority` 决定**排队顺序**：数值越大越先被执行，同优先级仍按提交顺序（先进先出）。 + +```python +controller.submit("sync", "tenant-A", priority=10) # 用户等着的那次同步 +controller.submit("sync", "tenant-B") # 默认优先级 0：批量回填 +``` + +默认值来自 `ConcurrencyPolicy.priority`，`submit(priority=...)` 可逐条覆盖。它只在 +**本进程的就绪队列**里比较——优先让"同一批排队任务里的急件"先走，不做跨节点抢占。 + +### 8.5 安全底线 + +**只接受名字，绝不接受代码。** `submit("extract", ...)` 里的 `extract` 必须在启动时 +`register` 过，否则抛 `ConfigurationError`。一个能启动任意可调用对象的入口等于 +**远程代码执行**——将来接 CLI 之后这条同样不能破。 + +重名注册也会抛错，不静默覆盖：静默覆盖会让"我明明改了实现，跑的还是旧的"这类问题 +极难发现。 + +### 8.6 跨进程限制（重要） + +`Controller` 管理的是**当前进程内**的任务。另一个进程里执行 `chongming-task ps` +**看不到它们**——内存不共享。 + +跨进程查询与持久化需要**共享媒介**： + +| 方案 | 引入网络层 | 说明 | +|---|---|---| +| 共享存储（`sqlite3`） | 否 | 标准库自带，无 IPC，天然跨平台，顺带就是持久化 | +| 本地 IPC（Unix socket / 命名管道） | 否 | 仍需写 IPC 与心跳，且平台分叉 | +| 不跨进程 | 否 | CLI 只能管自己启动的任务，不成其为"后台" | + +已选定第一条。`prune(keep_terminal=N)` 会丢弃最旧的终态记录以防内存无限增长。 + +### 8.7 持久层抽象 + +持久层契约是 `TaskStore`，内置三个实现： + +| 实现 | 适用 | 依赖 | +|---|---|---| +| `InMemoryTaskStore` | 单进程嵌入、单元测试、作为其他实现的**参照物** | 标准库 | +| `SqliteTaskStore` | 跨进程可见 + 持久化（**默认**） | 标准库 `sqlite3` | +| `PostgresTaskStore` | **跨机器**共用一套存储与指令通道 | 可选 `chongming-task[postgres]` | + +```python +from chongming_task import Progress, SqliteTaskStore, TaskQuery, TaskStatus + +store = SqliteTaskStore("tasks.db") +store.create(snapshot) +store.save(next_snapshot, expect_version=snapshot.status_version) # 乐观并发 +store.query(TaskQuery(statuses=frozenset({TaskStatus.RUNNING}), tag="io")) +store.touch("job-1", progress=Progress(done=120, total=500)) +``` + +同机跨进程用 SQLite 就够；**跨机器**换成连接串即可，语义与 API 完全一致： + +```python +from chongming_task import PostgresTaskStore # pip install "chongming-task[postgres]" + +store = PostgresTaskStore("postgresql://user:pw@db.internal:5432/chongming") +``` + +核心包不因此多出任何依赖：`PostgresTaskStore` 是**惰性取用**的，没装驱动时 +`import chongming_task` 照常工作（只有真正取它才会给出安装命令）。建表在首次连接时 +自动完成；角色没有 DDL 权限时，它检测到表已存在就直接用，所以只读运维账号也能连。 + +四类数据的写入频次与一致性要求不同，因此刻意分开： + +| 类别 | 方法 | 特点 | +|---|---|---| +| 任务快照 | `create` / `save` / `get` / `query` | 参与乐观并发（`status_version`） | +| 审计轨迹 | `append` / `history` | **只追加**，重复写入幂等 | +| 高频旁路 | `touch` | 进度与心跳，**不碰版本号**（否则会与状态写入持续冲突） | +| 控制指令 | `submit_command` / `claim_commands` / `complete_command` | 跨进程控制的媒介 | + +`claim_commands` 的**租约**解决的是"取走指令的进程崩了怎么办"：租约到期后指令可被 +重新取走，因此既不会重复下发，也不会因为一次崩溃而永久卡住。 + +#### 写自己的后端 + +别靠读文档猜语义——跑契约套件： + +```python +from chongming_task.testing.store_contract import StoreContract + + +class TestMyStore(StoreContract): + def make_store(self): + return MyStore(...) +``` + +套件**随包发布**且不依赖 pytest；内置实现跑的正是同一套用例（内存 / SQLite 文件 / +SQLite 内存 / Postgres——最后一份需要 `CHONGMING_TEST_POSTGRES` 给出 DSN，没有就跳过）。 + +**要换掉默认持久层，请看 [持久层实现指南](store-guide.md)**——那里有 14 个方法的逐条 +语义、并发与原子性要求、序列化保真要求，以及一个可执行的第三方后端范例和陷阱清单。 +`PostgresTaskStore` 本身就是照这份规格写出来的第二个真实后端：它的模块文档记了 +"与 SQLite 的三点差异"（连接池、`SKIP LOCKED`、原生类型）与跨进程并发用例， +可以当成比文件后端更接近生产的参照。 + +> **已经接线**：`Controller(store=...)` 可用——每次状态迁移写穿到存储，查询走存储 +> （因此能看到别的进程的任务），对不属于本进程的任务会把控制动作写成指令。详见 §8.9。 + +### 8.8 任务分桶 + +桶既是**本地并发隔离**的单位，也是将来**分布式分配**的单位（一台机器拥有一组桶）。 + +```python +controller = Controller(max_parallel=10, max_parallel_per_bucket=1) +controller.submit("sync", "租户A", task_id="a0", bucket_key="tenant-A") +controller.submit("sync", "租户B", task_id="b0", bucket_key="tenant-B") +``` + +`bucket_key` 是业务键（租户 id、数据源、日期分区……）。同一个键在**任何进程、任何 +机器**上都落到同一个桶。 + +| 成员 | 作用 | +|---|---| +| `submit(..., bucket_key=...)` | 提交时给出业务键 | +| `snapshot.bucket` / `.bucket_key` | 桶号与原始键都记在任务上 | +| `max_parallel_per_bucket=` | 每桶独立限流；一个桶积压不会饿死其他桶 | +| `list_tasks(bucket=...)` / `(buckets=[...])` | 按桶查询（`buckets` 是为分布式预留的：只取自己拥有的那组） | +| `bucket_counts()` | 每桶任务数，用于决定把桶分给谁 | +| `active_in_bucket(bucket)` | 某桶当前占用的执行额度 | +| `controller.bucket_policy` | 分桶策略（`num_buckets` 等） | + +`render_table` 也带上了 `BUCKET` 列。 + +**两条硬约束**（详见 [设计文档 §15](task-control-design.md)）： + +1. **桶数一旦开始用就不能随便改。** `bucket = hash(key) % N`，改 N 会让几乎所有键 + 改换桶位。所以任务会同时记下 `bucket_key`——将来真要改桶数，只有拿得到原始键 + 才能重新分桶。 +2. **绝不能用内置 `hash()`。** CPython 对 `str` 的 `hash()` 带进程级随机盐 + （`PYTHONHASHSEED`）：同一个键在不同进程里会落到不同的桶，于是"同桶任务在同一台 + 机器上"会**静默**失效。`BucketPolicy` 用 `blake2b`，跨进程一致。 + +调度器取下一个可跑任务时是**扫描**而不是取队首：某个桶满时队首可能正好被它挡住， +而后面属于其他桶的任务本可以先跑。 + +### 8.9 跨进程：一个进程跑，另一个进程看和管 + +```python +from chongming_task import Controller, SqliteTaskStore + +worker = Controller(store=SqliteTaskStore("tasks.db"), max_parallel=2, owner="worker-1") +worker.register("sync", sync_tenant) +worker.submit("sync", "租户A", task_id="job-1", bucket_key="tenant-A") + +# 另一个进程（同一个 SQLite 文件，各自一条连接） +console = Controller(store=SqliteTaskStore("tasks.db"), owner="ops-console") +console.list_tasks() # 看得到 job-1，尽管它并不持有 +console.stop("job-1", reason="运维停止") # 写成一条指令 +``` + +| 方向 | 行为 | +|---|---| +| 写 | 每次状态迁移**写穿**到存储（乐观并发，附迁移历史） | +| 读 | `get` / `list_tasks` / `stats` / `bucket_counts` 走存储 | +| 控制 | 不属于本进程的任务 → 写一条指令，由持有者轮询取走 | + +#### 三个必须知道的行为 + +1. **跨进程的 `pause()` 返回 `False`。** 那不是失败，是**未确认**——跨进程没有回执 + 通道。要观测结果请轮询 `get()`。`stop()` / `resume()` 返回的是"请求已受理"。 +2. **纯查询客户端不会抢指令。** 轮询线程只在提交过任务之后才启动；即便取到了不属于 + 自己的指令，也会通过 `release_command` 立即交还，而不是标记完成（那会让真正的 + 持有者永远收不到）。 +3. **存储故障不让任务失败**，但会记进 `controller.persist_failures`（最多 20 条）。 + 任务本身没问题，坏的只是可观测性——但绝不静默。 + +#### 两个使用注意 + +* **用完一定要关存储**：`controller.close(close_store=True)`，或自己 `store.close()`。 + 没关的 SQLite 连接会泄漏文件句柄，在 Windows 上还会让数据库文件无法删除。 +* `stop_all()` 只作用于**本进程持有**的任务。"全部停止"跨进程需要一份全局任务清单， + 那是分布式调度该决定的事。 + +`controller.owner` 是写进指令里的发起者标识（默认 `主机名:pid`），便于事后追溯是谁 +按的停止。 + +### 8.10 命令行控制台 + +`chongming-task` 是 `Controller` 之上的一层**薄适配**：解析参数、调用库、渲染输出。 +它不引入任何库没有的概念，也不绕过任何不变量。 + +```bash +chongming-task --db tasks.db ps +chongming-task --db tasks.db status demo-1 +chongming-task --db tasks.db stop demo-1 --reason "发布窗口" +``` + +| 命令 | 作用 | +|---|---| +| `tasks --tasks MODULE` | 列出模块里注册的任务定义 | +| `run TASK --tasks MODULE` | 提交并**前台执行**到落终态 | +| `ps` / `status` / `history` / `stats` / `buckets` | 查询 | +| `stop` / `pause` / `resume` | 控制 | + +退出码分三档：`0` 成功、`1` 命令失败、`3` **命令成功但任务没成功**——最后一档是给 +脚本判分支用的。 + +三点与库行为一致、但在命令行里更容易踩到的地方： + +* **`run` 是前台进程**，退出时任务就没人管了；长期运行请交给 `nohup` / `systemd`。 +* **`pause` 跨进程只报"未确认"**（退出码仍是 0），要看结果得 `status`。 +* **任务定义必须来自模块的 `register(controller)`**，CLI 只传名字——一个能启动任意 + 可调用对象的入口等于任意代码执行。 + +完整走查（含真实输出）见 **[命令行控制台](cli.md)**。 + +### 8.11 重启恢复（reconcile） + +进程意外退出后，存储里会留下一批永远停在 `running` 的任务——它们看着像在跑，其实 +载体早就随进程消失了。**进程启动时先跑一次 `reconcile()` 把它们清理掉**： + +```python +controller = Controller(store=SqliteTaskStore("tasks.db"), owner="worker-2") +report = controller.reconcile() +if report.lost: + log.warning("上次有 %d 个任务失去监督者", len(report.lost)) +``` + +#### 判据是"还有人在管它吗"，不是"它还在推进吗" + +每个监督进程都会给自己负责的任务**续租**（`lease_owner` / `lease_until`，由 `touch` +写入）。于是： + +| 情况 | 处置 | +|---|---| +| 租约未过期 | 有活着的进程在管，**一动不动** | +| 租约过期 + 任务**占着执行资源**（`starting`/`running`/`pausing`/`stopping`） | 载体已消失，判 `LOST` | +| 租约过期 + 任务**不占资源**（`queued`/`paused`/`retry_wait`） | 只是没人接手，**不改状态**，报告为 `adoptable` | +| 原本的监督者又活过来抢先写入 | 检测到版本冲突，**放手**（报 `skipped`） | + +**绝不能用任务心跳当判据。** `last_heartbeat_at` 由任务自己的 `checkpoint()` 驱动， +一个长时间不调 checkpoint 的健康任务（纯 CPU 计算）心跳同样停滞——拿它判断进程存活， +会把一个正常任务判成 `LOST` 并**杀掉**。这正是租约与心跳必须分成两个字段的原因。 + +```python +report = controller.reconcile(dry_run=True) # 先看一眼，不写任何状态 +report.lost # 会被（或已被）判为丢失的 +report.adoptable # 需要重新提交的 +report.skipped # 有人正在写、被跳过的 +``` + +#### 它**不会**重新执行任务 + +这是能力边界，不是遗漏：**存储里记的是状态，不是配方**。任务的函数与参数从未落库 +（序列化任意参数会带来不可反序列化、版本漂移等一堆问题），所以恢复一个失去载体的 +任务在原理上不可能。 + +`reconcile()` 能做的、也是它应当做的，是**如实标记**——让看板不再永远显示一个假的 +"运行中"。真正"重跑"要靠调用方自己重新 `submit`，而那需要调用方知道参数。 + +`lease_ttl`（默认 30 秒，可调）决定崩溃多久之后会被发现。 + +### 8.12 DAG 管道 + +把多个任务按依赖关系串起来： + +```python +from chongming_task import Pipeline +from chongming_task.policies import AggregationPolicy, JoinMode + +pipeline = Pipeline("nightly-etl", aggregation=AggregationPolicy.all_settled()) + +fetch = pipeline.node("fetch", "2026-10-02", node_id="fetch") +shards = [ + pipeline.node("transform", index, node_id=f"shard{index}", depends_on=[fetch]) + for index in range(3) +] +load = pipeline.node("load", node_id="load", depends_on=shards) +pipeline.node("audit", node_id="audit", depends_on=[load], join=JoinMode.ALL_SETTLED) + +run = controller.submit_pipeline(pipeline) +outcome = run.wait(timeout=60) +print(outcome.status, outcome.nodes) +``` + +> **第一个位置参数是已注册的任务名**，其后才是传给任务的位置参数。节点 id 由 +> `node_id=` 指定；省略时默认取任务名，同名多次则自动加 `#2` / `#3` 后缀。 +> `p.node("mark", "after")` 里的 `"after"` 是**传给 mark 的参数**，不是节点名—— +> 这正是那个参数叫 `node_id` 而不是 `name` 的原因。 + +#### 三种汇合语义（逐节点） + +| `JoinMode` | 何时放行 | +|---|---| +| `ALL_SUCCEEDED`（默认） | 全部上游成功——最常见的屏障 | +| `ALL_SETTLED` | 全部上游落终态即可，**不管成败**——用于清理、通知、汇总 | +| `ANY_SUCCEEDED` | 任一上游成功即可——多源冗余取数的先到先得 | + +#### 两种结算方式（整条管道） + +| `AggregationPolicy` | 行为 | +|---|---| +| `fail_fast()`（默认） | 首个真失败即**取消其余已经放行的节点** | +| `all_settled()` | 等所有节点落终态，再统一判定 | + +`quorum` 会被**明确拒绝**：它描述的是"同一件事的 N 份副本成功 k 份"，属于扇出分组而不是 +一条节点各异的 DAG。与其发明一个没人能预测的语义，不如说清楚。 + +#### 等待上游的节点不占并发额度 + +依赖由一个**真正的任务状态** `blocked` 表达，而且等待中的节点不进待执行队列： + +```python +assert controller.active_count == 1 # 只有根节点占额度 +assert controller.queued_count == 0 # 下游还在 blocked，不是 queued +``` + +一条 20 个节点的串行管道不会因为"等待"而把额度全锁死。 + +#### 三种"没成功"必须分得清 + +| 状态 | 含义 | +|---|---| +| `skipped` | 上游没成功，它**本来就不该跑** | +| `cancelled` | 本来要跑，被 fail_fast **连坐**了 | +| `stopped` | 值班同学**手动停的** | + +`CANCELLED` 刻意不作为 fail_fast 的触发条件，否则用户主动取消整条管道会被记成"失败"， +被连坐的节点也会再触发一轮连坐，因果就搅成一团了。 + +#### 限制 + +管道**结构在内存里**，不落库（任务的函数与参数从未持久化）。进程重启后一条跑到一半的 +管道无法续跑——节点会被 `reconcile()` 如实标记为丢失。 + +### 8.13 写一个任务：`@task` 与 `TaskContext` + +#### `@task` 装饰器 + +```python +from chongming_task import RetryPolicy, Timeout, task + + +@task(tags={"io"}, retry=RetryPolicy(max_attempts=3), timeout=Timeout(per_attempt=60)) +def extract(ctx, source: str) -> list[Row]: + """从 source 读取数据。""" + ... +``` + +描述默认取 **docstring 第一行**，名字默认取函数名。裸 `@task`（不带括号）也支持。 + +一次登记整个模块： + +```python +controller.register_module(myapp.tasks) # 模块里所有 @task 都被登记 +controller.register_all(extract, load) # 或者逐个登记 +``` + +**它不产生全局注册表。** 规格只挂在函数对象上（`__chongming_task__`），谁注册、注册到 +哪个控制器由调用方决定。全局注册表意味着 **import 即注册**，而 import 可能发生在任何 +地方（包括测试收集阶段），那种隐式行为极难排查。 + +#### `TaskContext` + +任务作者面对的就是这一个对象（公开名字 `TaskContext`，与内部的 `RuntimeContext` 是 +**同一个类**）： + +| 成员 | 作用 | +|---|---| +| `ctx.checkpoint()` | 协作点：响应停止与暂停 + 刷新心跳。**不调用它就不响应停止**，也会被 `idle` 超时判为疑似卡死 | +| `ctx.acheckpoint()` | 异步版本，`async def` 任务里 `await` 它（执行器会把协程真的跑完） | +| `ctx.progress(done, *, total=, unit=, message=)` | 上报进度 | +| `ctx.log(msg, *, level=)` / `.warning` / `.error` | 记日志 | +| `ctx.logs` | 本任务最近 200 条日志（内存中） | +| `ctx.input(节点id, 默认值)` / `ctx.upstream` | 取上游节点的返回值 | +| `ctx.artifact_dir` | 本次尝试的产物目录（按 attempt 隔离） | +| `ctx.is_stop_requested` / `ctx.is_pause_requested` | 只想探一下、不想抛异常时用 | + +#### 进度：进快照、进存储、不碰版本号 + +```python +for index, part in enumerate(partitions, start=1): + ctx.checkpoint() + ctx.progress(index, total=len(partitions), unit="partitions", message=part) +``` + +进度走**高频旁路**：不参与状态机、不递增 `status_version`，由 `Controller` 经 `touch` +定期回写到存储。所以它出现在 `ps` 的 `PROGRESS` 列，而不会把版本号变成噪声。 + +> **`progress()` 刻意不刷新心跳。** 心跳是"我还在推进"的整体信号，进度是"推进到哪了"。 +> 把心跳藏进进度里，会让一个只报进度、不做实际工作的任务看起来永远健康——那正是 +> `idle` 超时要抓的东西。 + +#### 日志：走标准库 + +```python +ctx.log("开始处理") # INFO +ctx.warning("分区为空，跳过") +ctx.error("写入失败") +``` + +logger 名是 `chongming_task.task.`，并且 `task_id` / `task_name` / `attempt` +会作为 `extra` 附在 `LogRecord` 上，因此可以按任务维度配 handler、过滤或加字段： + +```python +logging.getLogger("chongming_task.task").setLevel(logging.DEBUG) +``` + +`ctx.logs` 保留最近 200 条（内存中，进程退出即消失）。**它不是日志存储方案**——那属于 +应用自己的事，挂个 handler 就能落盘或入库。 + +#### 上游数据 + +管道的下游节点可以直接取上游的返回值： + +```python +@task +def transform(ctx, day: str) -> int: + partitions = ctx.input("extract", []) # extract 节点的返回值 + ... +``` + +只有**成功**的上游才会出现在 `ctx.upstream` 里。失败或跳过的上游没有值可传——塞一个 +`None` 会让下游以为"上游返回了 None"，而不是"上游没有值"。 + +#### 返回值必须可 JSON 序列化 + +返回值会**写入存储**（`outcome` 列），所以它必须是 JSON 可序列化的： + +```python +@task +def good(ctx) -> dict[str, Any]: + return {"rows": 3, "path": "s3://bucket/x.parquet"} # ✅ + + +@task +def bad(ctx) -> set[int]: + return {1, 2, 3} # ❌ set 不可序列化 +``` + +违反时任务会**失败**，失败类型是 `result_invalid`，消息写明原因。大数据请传**指针** +（对象存储 key、表名 + 主键、文件路径）而不是内容。 + +> 为什么这么严：返回值不落库的话，进程内会显示 `succeeded` 而存储里停在 `running` +> ——随后重启恢复会把一个**真正跑完**的任务判成 `LOST`。把一个成功过的任务报成"结果 +> 未知"会让人放心地重跑，而副作用已经发生过一次。所以宁可让它在返回时就响亮地失败。 + +标签同理：`tags` 必须是字符串（类型契约是 `frozenset[str]`），否则在提交时就会被拒绝。 + +**参数**暂时不强制可序列化——参数今天不落库，所以 `Path`、连接对象这类本地专用参数 +仍然合法。（节点之间不互相执行对方的任务，所以也不存在"参数要跨机器传输"的问题。） + +### 8.14 节点桶归属：多节点部署 + +多个节点各跑各的任务时，靠桶区分"哪一份是我的"： + +```python +controller = Controller( + store=SqliteTaskStore("tasks.db"), + owner="node-a", + owned_buckets=range(0, 16), # 本节点负责 0..15 号桶（按业务划分，可多于一个） +) +``` + +| 成员 | 说明 | +|---|---| +| `owned_buckets` | 本节点负责的桶；`None`（默认）表示不限定 | +| `owns_all_buckets` | 是否**不构成划分**：不限定，或显式覆盖了全部桶 | +| `owns_bucket(bucket)` | 某个桶是否归本节点 | + +限定时有两条行为： + +1. **提交时校验**——任务必须落在自己的桶里： + + ```python + controller.submit("sync", bucket_key="tenant-B") + # ConfigurationError: 桶 42（来自 bucket_key='tenant-B'）不属于本节点； + # 本节点负责 [0, 1, ..., 15]。任务若写进别人的桶，不会有任何节点执行它。 + ``` + + 这是**硬错误**而不是警告：落在别人桶里的任务不会有任何节点去跑它，它会永远停在 + `queued`，而且**没有任何报错**。与其让它静默堆积，不如在提交那一刻就说清楚。 + +2. **`reconcile()` 只清理自己桶里的僵尸任务**，不去动别的节点的任务。 + +**覆盖全部桶不算划分。** 如果显式列举了全部桶（例如生成出来的"负责 0-63"配置），此时没有 +任何"别的节点"，无桶任务同样会且只会由本节点执行——因此不再强制 `bucket_key`。 + +**启动时会校验桶号范围**，越界立刻报错：放过去的症状同样是"任务永远排队且没有报错"。 + +> 声明了归属的节点会在**构造那一刻**就启动指令轮询，而不是等第一次提交。否则别人 `stop` +> 一个还排在队列里的任务时，指令要等到本节点第一次提交才被取走——对定时任务节点来说可能 +> 是几小时之后。没有声明归属的纯查询客户端仍然不轮询，避免抢走别人的指令。 + +多节点部署的完整说明（能做什么、不能做什么、新增的风险、跨机部署）见 +**[分布式设计说明](distributed-design.md)**。跨机只需换存储后端：把 +`SqliteTaskStore("tasks.db")` 换成 `PostgresTaskStore("postgresql://...")`， +`owner` / `owned_buckets` / `reconcile()` / 跨节点控制全部照旧（见 §8.7）。 + +一份**可以直接照抄的完整模板**（部署参数与桶划分 `buckets_for_node()`、任务定义、 +`build_node()` 接线、启动恢复、按桶提交、跨节点停止、收尾校验，全流程带断言）见 +**[examples/11_best_practice.py](../examples/11_best_practice.py)**。 + +--- + +## 9. 状态模型参考 + +### 9.1 状态一览 + +**非终态**（9 个）： + +| 状态 | 含义 | +|---|---| +| `pending` | 已创建，依赖尚未解析 | +| `blocked` | 依赖未满足，等待上游 | +| `queued` | 就绪，等待执行槽；未启动的暂停也在恢复时回到这里 | +| `starting` | 执行器启动中 | +| `running` | 正在执行 | +| `pausing` | 已请求暂停，等待到达静默点 | +| `paused` | 已就地挂起 | +| `retry_wait` | 本次尝试失败，正在退避等待 | +| `stopping` | 已请求停止，处于优雅窗口内 | + +**终态**（6 个，无出边）： + +| 状态 | 含义 | +|---|---| +| `succeeded` | 正常完成 | +| `failed` | 出错且重试耗尽，或遇到不可重试的错误 | +| `stopped` | **人工主动停止**（不是错误） | +| `cancelled` | **编排层级联取消**（如兄弟任务失败） | +| `skipped` | 条件为假或依策略跳过 | +| `lost` | 载体失联或被放弃，**结果未知** | + +`stopped` 与 `cancelled` 分开，是因为两者责任方不同。排障时"这次跑批失败是人工 +干预还是真故障"是个必须能回答的问题。 + +### 9.2 完整转移表 + +```python +PENDING: {BLOCKED, QUEUED, SKIPPED, CANCELLED, STOPPED} +BLOCKED: {QUEUED, SKIPPED, CANCELLED, STOPPED} +QUEUED: {STARTING, PAUSED, SKIPPED, STOPPED, CANCELLED} +STARTING: {RUNNING, RETRY_WAIT, PAUSING, STOPPING, FAILED, CANCELLED, LOST} +RUNNING: {SUCCEEDED, FAILED, RETRY_WAIT, PAUSING, STOPPING, CANCELLED, LOST} +PAUSING: {PAUSED, RUNNING, STOPPING, FAILED, CANCELLED, LOST} +PAUSED: {QUEUED, RUNNING, STOPPED, CANCELLED} +RETRY_WAIT: {STARTING, QUEUED, PAUSED, STOPPED, FAILED, CANCELLED} +STOPPING: {STOPPED, SUCCEEDED, FAILED, LOST} +SUCCEEDED / FAILED / STOPPED / CANCELLED / SKIPPED / LOST: {} +``` + +一条贯穿全表的规则：**控制动作是否需要中间态，取决于任务是否已占用执行资源**。 +未占用（`pending`/`blocked`/`queued`/`paused`/`retry_wait`）→ 立即生效；已占用 +（`starting`/`running`/`pausing`）→ 经中间态缓冲。让未启动的任务显示"停止中/暂停中" +是虚假信息，会让运维误以为需要等待。 + +### 9.3 转移表之外的守卫 + +有些规则无法用"状态 → 状态"表达，它们是**守卫**： + +| 动作 | 守卫 | 理由 | +|---|---|---| +| `mark_running()` | 必须已有 `started_at` | `paused → running` 对就地恢复合法，对"排队期间被暂停、从未启动"却毫无意义 | +| `abort_pause()` | 必须处于 `pausing` | `paused → running` 是给*恢复*用的；把"回滚"也放进来会产出没有开始时间的 `running` 快照 | +| `mark_starting()` | `attempt + 1 <= max_attempts` | 否则抛 `AttemptsExhausted`（预算问题，不是状态表问题） | +| `force_transition()` | 必须给出非空 `actor` 与 `reason` | 这是唯一能任意跳转的入口，没有审计信息就无法事后追溯 | +| `mark_lost()` | 建议带 `failure` | `lost` 的全部意义是"不知道结果"，不记录原因就无从判断该重跑还是人工核对 | + +--- + +## 10. 错误与异常 + +``` +TaskError +├─ InvalidTransition 非法状态转移（含守卫不通过） +├─ DependencyFailed 上游未成功 +├─ CircularDependency 依赖成环 +├─ TaskTimeout 超时 +├─ TaskExecutionError 用户代码异常的统一包装（保留 __cause__） +├─ AttemptsExhausted 重试预算耗尽 +├─ ConcurrencyConflict 乐观并发写冲突 +├─ StoreError 存储层故障 +└─ ConfigurationError 配置错误 + +TaskCancelled ← 独立：继承 BaseException，不是 TaskError +``` + +### 9.1 `TaskCancelled` 为什么继承 `BaseException` + +因为它可能出现在用户代码的任何一行，而下面这种写法极其常见： + +```python +try: + while True: + ctx.checkpoint() + work() +except Exception as exc: # 想捕获业务异常 + log(exc) # 但会把停止信号一起吃掉 +``` + +若 `TaskCancelled` 继承 `Exception`，任务将**永远无法被停止**。因此它和 +`asyncio.CancelledError` 一样继承 `BaseException`： + +```python +try: + ... +except TaskCancelled: + raise # 取消要放行 +except Exception as exc: + ... # 只处理真正的业务异常 +``` + +测试里有一条用例专门用 `except Exception` 包裹循环来验证取消不会被吞掉。 + +--- + +## 11. 限制与 FAQ + +**Q：任务停不下来？** +检查它是否调用了 `ctx.checkpoint()`。不调用的任务只能等超时，然后落 `lost`。 + +**Q：为什么我的任务被判"疑似卡死"（`timeout_idle`）？** +`idle` 衡量的是 `checkpoint()` 间隔。长段无 `checkpoint()` 的计算会被误判——调大 +`idle` 阈值，或提高 `checkpoint()` 频率。 + +**Q：`pause()` 返回 `False` 是什么情况？** +任务在窗口内没到达检查点，暂停**没生效**，状态已回滚为 `running`。这是如实告知， +不是 bug。 + +**Q：暂停会不会释放并发额度？** +不会。暂停是就地挂起，槽位仍被占用。 + +**Q：任务被判 `lost` 之后怎么办？** +`lost` 表示结果未知。系统不会替你猜——你可以 `state_machine.retry()` 重跑，也可以 +人工核对副作用后 `force_transition()` 标记终态。 + +**Q：`wait(timeout=...)` 超时了，任务会被杀掉吗？** +不会。它只抛 `TimeoutError`，任务状态完全不变。要停止请显式 `stop()`。 + +**Q：能让任务跑到一半重启而不重跑吗？** +不能。没有用户态检查点机制（那属于 Dask/Ray 式的图重算，是明确的非目标）。重试 +总是从头开始，请把任务设计成幂等或可断点续传。 + +--- + +## 12. 测试自己的任务 + +确定性时钟随包发布，你的测试不必 `sleep`： + +```python +from chongming_task.testing import FakeClock +from chongming_task.run import run_task +from chongming_task.policies import Timeout + + +def test_idle_timeout_fires_without_waiting(): + clock = FakeClock() + # 通过 clock 注入控制时间，配合 handle.heartbeat 即可在毫秒内验证超时逻辑 +``` + +常用做法： + +- `FakeClock` 注入 `TaskRun(..., clock=clock)`，用 `clock.advance(n)` 推进时间； +- `InlineExecutor` 让执行同步化，测试不必等线程； +- `RetryPolicy(backoff=FixedBackoff(0))` 去掉重试等待； +- 断言 `run.transitions`，而不是只断言最终状态——中间态才是本项目的价值所在。 + +--- + +## 下一步 + +已实现的能力以 [README 状态表](../README.md#当前状态务必先读)为准：单任务与 DAG 管道执行、 +并发额度与排队、SQLite / Postgres 持久化与重启恢复、`@task` 装饰器与 `TaskContext` +（`log` / `progress` / `upstream`）、CLI 控制台、节点桶归属。 + +仍在计划中的部分：跨机的**运维告警**（权威桶声明、桶重复/空洞、孤儿任务，见 +[分布式设计说明](distributed-design.md) §11.2–11.3）与 `transitions` 的归档策略 +（规模上 `prune()` 不够用，见 [持久层实现指南](store-guide.md)）。 + +设计细节与理由见 [设计文档](task-control-design.md)。 diff --git a/examples/01_quickstart.py b/examples/01_quickstart.py new file mode 100644 index 0000000..94b8d89 --- /dev/null +++ b/examples/01_quickstart.py @@ -0,0 +1,49 @@ +"""示例 1：跑一个任务，并观察它在生命周期中经过的状态。 + +运行:: + + uv run python examples/01_quickstart.py +""" + +from __future__ import annotations + +import time + +from chongming_task.run import TaskRun +from chongming_task.runtime.cancellation import CancellationToken + + +def extract(ctx: CancellationToken, source: str) -> list[int]: + """模拟一次数据抽取。 + + 要点：长耗时循环里必须周期性地调用 ``ctx.checkpoint()``。它同时做三件事： + 响应停止请求、响应暂停请求、刷新心跳。 + """ + rows: list[int] = [] + for index in range(5): + ctx.checkpoint() + time.sleep(0.05) + rows.append(index) + print(f" 抽取完成：{source} -> {len(rows)} 行") + return rows + + +def main() -> None: + run = TaskRun(extract, ("s3://demo/rows",), task_id="demo-1", name="抽取") + run.start() + outcome = run.wait(timeout=30) + + print("\n状态迁移历史（只追加、可审计）：") + for record in run.transitions: + print( + f" {record.from_status.value:>10} -> {record.to_status.value:<10} by {record.trigger}" + ) + + print(f"\n最终状态：{outcome.status.value}") + print(f"返回值 ：{outcome.value}") + elapsed = (outcome.snapshot.ended_at - outcome.snapshot.started_at).total_seconds() + print(f"耗时 ：{elapsed:.2f}s") + + +if __name__ == "__main__": + main() diff --git a/examples/02_control.py b/examples/02_control.py new file mode 100644 index 0000000..240937d --- /dev/null +++ b/examples/02_control.py @@ -0,0 +1,74 @@ +"""示例 2：运行期干预 —— 暂停、恢复、停止。 + +演示协作式暂停（D6 定案的语义）与"停不掉时会如实上报"。 + +运行:: + + uv run python examples/02_control.py +""" + +from __future__ import annotations + +import time + +from chongming_task.run import TaskRun + + +def long_running(ctx) -> str: + """持续做小步工作，每一步都经过协作点。""" + for _ in range(10_000): + ctx.checkpoint() + time.sleep(0.01) + return "跑完了" + + +def uncooperative(ctx) -> str: + """刻意**不**调用 checkpoint：这样的任务既停不掉也暂停不了。 + + 它约 4 秒后自行结束，以免在示例里留下后台线程。 + """ + for _ in range(400): + time.sleep(0.01) + return "无视了停止请求" + + +def demo_pause_resume() -> None: + print("== 暂停 / 恢复 ==") + run = TaskRun(long_running, task_id="demo-pause") + run.start() + time.sleep(0.2) + + print(f" 暂停请求已受理：{run.pause(timeout=5)}；当前状态 {run.status.value}") + time.sleep(0.2) + print(f" 暂停期间状态保持：{run.status.value}") + + run.resume() + print(f" 恢复后状态 ：{run.status.value}") + + run.stop(reason="演示主动停止") + outcome = run.wait(timeout=10) + print(f" 停止后终态 ：{outcome.status.value}") + print(f" 失败归类 ：{outcome.failure.kind.value if outcome.failure else '-'}") + print(f" 可重试 ：{outcome.failure.retryable if outcome.failure else '-'}") + + +def demo_unstoppable() -> None: + print("\n== 停不掉的任务：如实上报 ==") + run = TaskRun(uncooperative, task_id="demo-stubborn", graceful_stop=0.3) + run.start() + time.sleep(0.2) + run.stop(reason="试一下") + + outcome = run.wait(timeout=10) + print(f" 终态 ：{outcome.status.value}") + print(f" 失败归类 ：{outcome.failure.kind.value if outcome.failure else '-'}") + print(" 含义 ：线程无法强杀，任务可能仍在后台产生副作用，因此不敢说'已停止'") + + +def main() -> None: + demo_pause_resume() + demo_unstoppable() + + +if __name__ == "__main__": + main() diff --git a/examples/03_retry_and_timeout.py b/examples/03_retry_and_timeout.py new file mode 100644 index 0000000..a0aaa42 --- /dev/null +++ b/examples/03_retry_and_timeout.py @@ -0,0 +1,86 @@ +"""示例 3：重试与超时。 + +运行:: + + uv run python examples/03_retry_and_timeout.py +""" + +from __future__ import annotations + +import time + +from chongming_task.policies import ExponentialBackoff, RetryPolicy, Timeout +from chongming_task.run import run_task + +_attempts = 0 + + +def flaky(ctx, succeed_on_attempt: int) -> str: + """前几次故意失败，用来演示重试与退避。""" + global _attempts + _attempts += 1 + if _attempts < succeed_on_attempt: + raise RuntimeError(f"第 {_attempts} 次尝试遇到暂时性故障") + return f"第 {_attempts} 次尝试成功" + + +def slow(ctx) -> str: + """跑得很慢，用来演示超时后会被请求停止。""" + for _ in range(10_000): + ctx.checkpoint() + time.sleep(0.01) + return "跑完了" + + +def demo_retry() -> None: + print("== 重试：失败两次后成功 ==") + _attempts = 0 + outcome = run_task( + flaky, + 3, + retry=RetryPolicy(max_attempts=5, backoff=ExponentialBackoff(base=0.05, factor=2)), + task_id="demo-retry", + name="会抖的任务", + ) + print(f" 终态 ：{outcome.status.value}") + print(f" 返回值 ：{outcome.value}") + print(f" 尝试次数 ：{outcome.snapshot.attempt}/{outcome.snapshot.max_attempts}") + + +def demo_retry_exhausted() -> None: + print("\n== 重试：预算耗尽 ==") + global _attempts + _attempts = 0 + outcome = run_task( + flaky, + 99, # 永远不会成功 + retry=RetryPolicy(max_attempts=2, backoff=ExponentialBackoff(base=0.05, factor=1)), + task_id="demo-exhausted", + name="必定失败", + ) + print(f" 终态 ：{outcome.status.value}") + print(f" 尝试次数 ：{outcome.snapshot.attempt}/{outcome.snapshot.max_attempts}") + print(f" 失败归类 ：{outcome.failure.kind.value if outcome.failure else '-'}") + + +def demo_timeout() -> None: + print("\n== 超时：单次尝试不超过 0.5 秒 ==") + outcome = run_task( + slow, + timeout=Timeout(per_attempt=0.5), + task_id="demo-timeout", + name="慢任务", + ) + print(f" 终态 ：{outcome.status.value}") + print(f" 失败归类 ：{outcome.failure.kind.value if outcome.failure else '-'}") + print(f" 可重试 ：{outcome.failure.retryable if outcome.failure else '-'}") + + +def main() -> None: + demo_retry() + demo_retry_exhausted() + demo_timeout() + + +if __name__ == "__main__": + main() diff --git a/examples/04_controller.py b/examples/04_controller.py new file mode 100644 index 0000000..6d7a929 --- /dev/null +++ b/examples/04_controller.py @@ -0,0 +1,82 @@ +"""示例 4：用 Controller 管理一批任务（注册 / 提交 / 排队 / 查询 / 控制）。 + +这是管理后台的核心：**一个统一入口**持有任务定义、运行中的任务与并发额度。 +将来的 CLI 或看板都只是这一层的薄适配。 + +运行:: + + uv run python examples/04_controller.py +""" + +from __future__ import annotations + +import time + +from chongming_task.controller import Controller, render_table + + +def transform(ctx, rows: int) -> int: + """转换：逐行推进。""" + for _ in range(30): + ctx.checkpoint() + time.sleep(0.02) + return rows * 2 + + +def extract(ctx, source: str) -> str: + """抽取：比转换慢一倍。""" + for _ in range(60): + ctx.checkpoint() + time.sleep(0.02) + return f"{source} -> 1200 行" + + +def main() -> None: + controller = Controller(max_parallel=2) + controller.register("transform", transform, tags={"cpu"}) + controller.register("extract", extract, tags={"io"}) + controller.register("load", transform, description="写回目标库") + + print("已注册任务：", "、".join(controller.registry.names())) + print("并发上限：2\n") + + for index in range(4): + controller.submit( + "transform", 100 * (index + 1), task_id=f"t{index}", label=f"转换-{index + 1}" + ) + print(f"提交了 4 个任务；此刻排队 {controller.queued_count} 个（超出并发上限的部分）") + + time.sleep(0.4) + print("\n--- 查询：运行中的视图 ---") + print(render_table(controller.list_tasks())) + stats = controller.stats() + print( + f" 未结束 {stats.unfinished}" + f"（其中排队 {controller.queued_count}，占用执行额度 {controller.active_count}）" + ) + + controller.wait_all(timeout=30) + print("\n--- 查询：全部结束 ---") + print(render_table(controller.list_tasks())) + + print("\n>>> 提交一个长任务，然后中途停止它") + controller.submit("extract", "s3://demo/big", task_id="t-long", label="抽取(大)") + time.sleep(0.3) + print(f" 当前状态：{controller.get('t-long').status.value}") + controller.stop("t-long", reason="手工干预") + outcome = controller.wait("t-long", timeout=10) + print( + f" 终态：{outcome.status.value}" + f"（失败归类 {outcome.failure.kind.value}，可重试 {outcome.failure.retryable}）" + ) + + print("\n--- 最终统计 ---") + final = controller.stats() + for status, count in sorted(final.by_status.items(), key=lambda item: item[0].value): + print(f" {status.value:<10} {count}") + print("\n提示：另一个进程里的 chongming-task ps 看不到这些任务——内存不共享，") + print(" 跨进程查询需要共享存储（stdlib sqlite3），那是下一步。") + + +if __name__ == "__main__": + main() diff --git a/examples/05_buckets.py b/examples/05_buckets.py new file mode 100644 index 0000000..dcfdaae --- /dev/null +++ b/examples/05_buckets.py @@ -0,0 +1,70 @@ +"""示例 5：任务分桶 —— 本地并发隔离，以及为分布式预留的分配单位。 + +运行:: + + uv run python examples/05_buckets.py +""" + +from __future__ import annotations + +import time + +from chongming_task.controller import Controller, render_table + + +def sync_tenant(ctx, tenant: str) -> str: + """同步一个租户的数据。""" + for _ in range(20): + ctx.checkpoint() + time.sleep(0.02) + return f"{tenant} 同步完成" + + +def main() -> None: + controller = Controller(max_parallel=10, max_parallel_per_bucket=1) + controller.register("sync", sync_tenant) + policy = controller.bucket_policy + + print(f"分桶策略：{policy.num_buckets} 个桶") + print("并发限制：全局 10，但每个桶最多 1 个\n") + + for index in range(4): + controller.submit( + "sync", "租户A", task_id=f"a{index}", label="A-同步", bucket_key="tenant-A" + ) + for index in range(2): + controller.submit( + "sync", "租户B", task_id=f"b{index}", label="B-同步", bucket_key="tenant-B" + ) + + print("业务键 -> 桶号（同一个键在任何进程、任何机器上都会落到同一个桶）：") + for key in ("tenant-A", "tenant-B"): + print(f" {key:<10} -> bucket {policy.bucket_for(key)}") + + time.sleep(0.5) + print("\n--- 查询：A 桶积压，但不影响 B 桶 ---") + print(render_table(controller.list_tasks())) + print( + f" 占用额度 {controller.active_count}，排队 {controller.queued_count}，" + f"A 桶在用 {controller.active_in_bucket(policy.bucket_for('tenant-A'))}" + ) + + controller.wait_all(timeout=30) + + print("\n--- 查询：全部结束 ---") + print(render_table(controller.list_tasks())) + + print("\n各桶任务数（分布式调度要靠它决定把桶分给谁）：") + for bucket, count in sorted( + controller.bucket_counts().items(), key=lambda kv: (kv[0] is None, kv[0]) + ): + print(f" bucket {bucket}: {count}") + + bucket_a = policy.bucket_for("tenant-A") + print(f"\n按桶查询（只取 A 桶）：{[s.task_id for s in controller.list_tasks(bucket=bucket_a)]}") + + print("\n分布式预留：桶是分配单位——一台机器拥有一组桶，只取自己那组桶的任务即可。") + + +if __name__ == "__main__": + main() diff --git a/examples/06_cross_process.py b/examples/06_cross_process.py new file mode 100644 index 0000000..d4f5407 --- /dev/null +++ b/examples/06_cross_process.py @@ -0,0 +1,91 @@ +"""示例 6：跨进程 —— 一个进程跑任务，另一个进程查询与控制。 + +这里用**同一个 SQLite 文件上的两个独立连接**模拟两个进程。真正的两个进程行为完全 +一样：它们看不到彼此的内存，只能通过存储交换状态与指令。 + +运行:: + + uv run python examples/06_cross_process.py +""" + +from __future__ import annotations + +import tempfile +import time +from pathlib import Path + +from chongming_task import Controller, SqliteTaskStore, render_table + + +def sync_tenant(ctx, tenant: str) -> str: + """稳步推进直到被停止。""" + for _ in range(300): + ctx.checkpoint() + time.sleep(0.02) + return f"{tenant} 同步完成" + + +def _wait_for_status(client: Controller, task_id: str, wanted: str, timeout: float = 20.0) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + snapshot = client.get(task_id) + if snapshot is not None and snapshot.status.value == wanted: + return True + time.sleep(0.02) + return False + + +def main() -> None: + with tempfile.TemporaryDirectory(prefix="chongming-demo-") as directory: + path = Path(directory) / "tasks.db" + + # ---- 进程 A：真正跑任务的 worker ---- + worker = Controller( + store=SqliteTaskStore(path), + max_parallel=2, + command_poll_interval=0.1, + owner="worker-1", + ) + worker.register("sync", sync_tenant) + worker.submit("sync", "租户A", task_id="job-1", label="A-同步", bucket_key="tenant-A") + worker.submit("sync", "租户B", task_id="job-2", label="B-同步", bucket_key="tenant-B") + print("worker-1 已提交 2 个任务（并发上限 2）\n") + + time.sleep(0.5) + + # ---- 进程 B：管理客户端，只查询与控制，不执行任何任务 ---- + client = Controller(store=SqliteTaskStore(path), owner="ops-console") + print("--- ops-console 看到的状态（它并不持有这些任务）---") + print(render_table(client.list_tasks())) + print(f" 它本地的任务句柄数：{len(client.runs())}") + print(f" 统计：{ {s.value: n for s, n in client.stats().by_status.items()} }") + + print("\n>>> ops-console 跨进程停止 job-1") + print(f" 下发结果：{client.stop('job-1', reason='运维从后台停止')}") + print(" （客户端无法给另一个进程的线程发信号，只能写一条指令）") + + if _wait_for_status(client, "job-1", "stopped"): + print(" worker-1 已取走指令并停止该任务") + else: + print(" !! 超时：指令没有被取走") + + print("\n--- job-1 的迁移历史（从存储读，可审计）---") + for record in client.store.history("job-1"): # type: ignore[union-attr] + print( + f" {record.version:>2} {record.from_status.value:>10} -> " + f"{record.to_status.value:<10} by {record.trigger}" + ) + + print("\n--- 让 job-2 自己跑完，再看最终状态 ---") + worker.wait_all(timeout=30) + print(render_table(client.list_tasks())) + + # 收尾：两个 Controller 各自关掉自己的存储连接。没关的 SQLite 连接会泄漏 + # 文件句柄，在 Windows 上还会让数据库文件无法删除。 + worker.close(timeout=10, close_store=True) + client.close(timeout=5, close_store=True) + print("\n要点：worker 负责执行，console 负责查询与控制，两者只通过存储耦合。") + + +if __name__ == "__main__": + main() diff --git a/examples/07_pipeline.py b/examples/07_pipeline.py new file mode 100644 index 0000000..2cf51f6 --- /dev/null +++ b/examples/07_pipeline.py @@ -0,0 +1,112 @@ +"""示例 7：DAG 管道 —— 一个源 → 两个分支 → 汇合。 + +管道把多个任务按依赖关系串起来，由一个调度器推进：节点先停在 ``blocked`` 等上游， +上游满足后才放行；注定跑不了的下游会被 ``skipped``。 + +运行:: + + uv run python examples/07_pipeline.py +""" + +from __future__ import annotations + +import time +from typing import Any + +from chongming_task import Controller, Pipeline, render_pipeline +from chongming_task.policies import AggregationPolicy, JoinMode + + +def fetch(ctx: Any, day: str) -> str: + """取数：模拟一次耗时操作。""" + for _ in range(10): + ctx.checkpoint() + time.sleep(0.03) + return f"{day} 的数据" + + +def transform(ctx: Any, day: str, shard: int) -> str: + """分片转换。""" + for _ in range(8): + ctx.checkpoint() + time.sleep(0.02) + return f"{day}/shard{shard}" + + +def load(ctx: Any, day: str) -> str: + """入库：依赖全部分片完成。""" + del ctx + return f"{day} 已入库" + + +def audit(ctx: Any, day: str) -> str: + """审计：**不管上游成败都要跑**（ALL_SETTLED）。""" + del ctx + return f"{day} 审计完成" + + +def main() -> None: + controller = Controller(max_parallel=4) + + # ---- 定义 ---- + pipeline = Pipeline("nightly-etl", aggregation=AggregationPolicy.all_settled()) + + controller.register("fetch", fetch, tags={"io"}) + controller.register("transform", transform) + controller.register("load", load, tags={"io"}) + controller.register("audit", audit) + + fetch_node = pipeline.node("fetch", "2026-10-02", node_id="fetch") + shards = [ + pipeline.node( + "transform", + "2026-10-02", + index, + node_id=f"shard{index}", + depends_on=[fetch_node], + ) + for index in range(3) + ] + # 入库：等所有分片成功（默认 ALL_SUCCEEDED 屏障） + load_node = pipeline.node("load", "2026-10-02", node_id="load", depends_on=shards) + # 审计：等 load 落终态即可，不管成败 + pipeline.node( + "audit", + "2026-10-02", + node_id="audit", + depends_on=[load_node], + join=JoinMode.ALL_SETTLED, + ) + + print(f"管道 {pipeline.name!r} 共 {len(pipeline)} 个节点，拓扑序：") + print(" " + " -> ".join(node.node_id for node in pipeline.topological_order())) + print(" 注意：位置参数是传给任务的，节点名由 node_id= 指定\n") + + # ---- 跑起来 ---- + run = controller.submit_pipeline(pipeline) + print(f"已提交，pipeline_run_id = {run.run_id}\n") + + time.sleep(0.05) + print("--- 刚提交时：下游都停在 blocked ---") + print(render_pipeline(run)) + + print("\n--- 等待收场 ---") + outcome = run.wait(timeout=60) + + print(render_pipeline(run)) + print(f"\n管道结论：{outcome.status.value}") + for node_id, status in outcome.nodes.items(): + print(f" {node_id:<8} {status.value}") + + print(f"\n每个任务都带 pipeline_run_id：{_run_id_of(controller, run, 'load')}") + controller.close(timeout=10) + print("\n要点：依赖由状态机表达（blocked -> queued），调度器只负责放行。") + + +def _run_id_of(controller: Controller, run: Any, node_id: str) -> str: + snapshot = controller.get(run.task_id_of(node_id)) + return "?" if snapshot is None else str(snapshot.pipeline_run_id) + + +if __name__ == "__main__": + main() diff --git a/examples/08_task_context.py b/examples/08_task_context.py new file mode 100644 index 0000000..1e0229a --- /dev/null +++ b/examples/08_task_context.py @@ -0,0 +1,119 @@ +"""示例 8：``@task`` 装饰器与 ``TaskContext``。 + +演示四件事： + +* ``@task`` 把元数据写在函数旁边，``register_module`` 一次登记整个模块； +* ``ctx.progress(...)`` 上报进度——它会进快照、进存储，并出现在 ``ps`` 的 PROGRESS 列； +* ``ctx.log(...)`` 走标准库 ``logging``，logger 名字是 ``chongming_task.task.``； +* ``ctx.input(节点id)`` 取上游节点的返回值——管道之间真正能传值。 + +运行:: + + uv run python examples/08_task_context.py +""" + +from __future__ import annotations + +import logging +import sys +import time +from typing import Any + +from chongming_task import Controller, Pipeline, RetryPolicy, render_table, task + + +@task(tags={"io"}, retry=RetryPolicy(max_attempts=2)) +def extract(ctx: Any, day: str) -> list[str]: + """按天抽取分区列表。""" + ctx.log(f"开始抽取 {day}") + partitions = [f"{day}/part{i}" for i in range(6)] + for index, part in enumerate(partitions, start=1): + ctx.checkpoint() + ctx.progress(index, total=len(partitions), unit="partitions", message=part) + time.sleep(0.05) + return partitions + + +@task +def transform(ctx: Any, day: str) -> int: + """把上游抽出的分区逐个转换，返回处理条数。""" + partitions = ctx.input("extract", []) + ctx.log(f"待转换分区：{len(partitions)} 个") + rows = 0 + for index, part in enumerate(partitions, start=1): + ctx.checkpoint() + rows += 100 + ctx.progress(index, total=len(partitions), unit="partitions", message=part) + time.sleep(0.05) + return rows + + +@task +def load(ctx: Any, day: str) -> str: + """入库：条数来自上游的返回值，而不是自己再算一遍。""" + rows = ctx.input("transform", 0) + ctx.log(f"{day} 共 {rows} 行待入库") + for index in range(1, 5): + ctx.checkpoint() + ctx.progress(index, total=4, unit="batch") + time.sleep(0.04) + return f"{day}: {rows} 行已入库" + + +def main() -> None: + logging.basicConfig( + level=logging.INFO, + format="%(levelname)-7s %(name)s %(message)s", + stream=sys.stdout, + ) + + controller = Controller(max_parallel=2) + + # ---- 一次登记整个模块 ---- + registered = controller.register_module(sys.modules[__name__]) + print("从 @task 元数据登记的：") + for item in registered: + print(f" {item.name:<10} tags={sorted(item.tags) or '-'} {item.description}") + + # ---- 串成管道：数据沿依赖流动 ---- + pipeline = Pipeline("daily-etl") + extract_node = pipeline.node("extract", "2026-10-02", node_id="extract") + transform_node = pipeline.node( + "transform", "2026-10-02", node_id="transform", depends_on=[extract_node] + ) + pipeline.node("load", "2026-10-02", node_id="load", depends_on=[transform_node]) + + run = controller.submit_pipeline(pipeline) + print(f"\n已提交管道 {run.run_id}\n") + + # ---- 跑的过程中看进度 ---- + deadline = time.monotonic() + 30 + while time.monotonic() < deadline and not run.settled: + snapshots = controller.list_tasks(pipeline_run_id=run.run_id) + cells: list[str] = [] + for snapshot in snapshots: + node = snapshot.name.split("/", 1)[-1] + progress = "-" + if snapshot.progress is not None: + progress = f"{snapshot.progress.done}/{snapshot.progress.total}" + cells.append(f"{node:<10} {snapshot.status.value:<10} {progress:<8}") + sys.stdout.write("\r" + " | ".join(cells).ljust(120)) + sys.stdout.flush() + time.sleep(0.1) + + outcome = run.wait(timeout=30) + print("\n\n--- 结束 ---") + print(render_table(controller.list_tasks(pipeline_run_id=run.run_id))) + print(f"\n管道结论：{outcome.status.value}") + + final = controller.get(run.task_id_of("load")) + print(f"load 的返回值：{final.outcome if final else None}") # type: ignore[union-attr] + + controller.close(timeout=10) + print( + "\n要点：ctx 是任务与运行时之间唯一的协作入口——停止、暂停、进度、日志、上游数据都在这里。" + ) + + +if __name__ == "__main__": + main() diff --git a/examples/09_long_running.py b/examples/09_long_running.py new file mode 100644 index 0000000..83ec514 --- /dev/null +++ b/examples/09_long_running.py @@ -0,0 +1,180 @@ +"""示例 9：无限循环任务的**启与停**。 + +一个"永远跑下去"的任务该怎么写、怎么启动、怎么停，以及三种收尾方式的终态差异。 + +运行:: + + uv run python examples/09_long_running.py + +## 三条规则 + +1. **任务体每轮必须调 ``ctx.checkpoint()``。** 停止是**协作式**的：只有调用了它，任务 + 才会看到停止请求。不调用 = 停不掉（见文件最后的对照）。 +2. **停止请求有两种收尾方式，终态不同**： + + ================================================ =========== ========== + 写法 终态 返回值 + ================================================ =========== ========== + ``while True: ctx.checkpoint()``（让它抛） ``stopped`` 无 + 检查 ``ctx.is_stop_requested`` 后正常 ``return`` ``succeeded`` ✅ 有 + 完全不调 ``checkpoint()`` ``lost`` 无 + ================================================ =========== ========== + + "停止"在第一种写法里是"立刻中止"，在第二种里是"**请收工**"——后者会把已经处理好的 + 结果留下来，对"跑到下一个安全点再退出"这类需求更合适。 +3. **停止是终态。** 目前没有 ``retry``，停止/失败之后想再跑要换一个 ``task_id`` + 重新提交（``create`` 会拒绝重复 id）。所以日常"先停一下"应该用 ``pause`` / ``resume``， + ``stop`` 留给"我要它彻底结束"。 +""" + +from __future__ import annotations + +import sys +import threading +import time +from typing import Any + +from chongming_task import Controller, InMemoryTaskStore, render_table + + +def worker(ctx: Any, source: str) -> int: + """无限循环：持续处理，直到收到停止请求后**优雅收尾**。""" + processed = 0 + while not ctx.is_stop_requested: + ctx.checkpoint() + processed += 1 + ctx.progress(processed, unit="batch", message=f"{source} 第 {processed} 批") + if processed % 20 == 0: + ctx.log(f"已处理 {processed} 批") + time.sleep(0.02) + + # 走到这里说明收到停止请求了：做收尾，然后正常返回—— + # 返回值会被保留下来，终态是 succeeded 而不是 stopped。 + ctx.log(f"收到停止请求，收尾：本次共处理 {processed} 批") + return processed + + +def main() -> None: + store = InMemoryTaskStore() + controller = Controller(store=store, owner="node-a") + controller.register("worker", worker) + + # ---- 启动 ---- + run = controller.submit("worker", "kafka://orders", task_id="orders-worker") + print(f"已启动 {run.snapshot.task_id} 状态={run.status.value}") + time.sleep(0.5) + + print("\n--- 运行中 ---") + print(render_table(controller.list_tasks())) + + # ---- 日常先停一下：pause / resume（不是 stop） ---- + print("\n--- pause（就地挂起，仍占用执行槽）---") + print(f" pause 是否确认: {controller.pause('orders-worker', timeout=5)}") + print(f" 状态: {run.status.value}") + + print("\n--- resume ---") + controller.resume("orders-worker") + time.sleep(0.2) + print(f" 状态: {run.status.value}") + + # ---- 让它彻底结束：stop ---- + print("\n--- stop（终态，不可逆）---") + controller.stop("orders-worker", reason="发布窗口") + outcome = controller.wait("orders-worker", timeout=10) + + print(f" 终态: {outcome.status.value}") + print(f" 返回值: {outcome.value} ← 优雅收尾把已处理的数量带了出来") + print("\n--- 结束后 ---") + print(render_table(controller.list_tasks())) + + print("\n--- 审计轨迹（谁按的、为什么，都在这里）---") + for record in store.history("orders-worker"): + reason = f" 原因={record.reason}" if record.reason else "" + print( + f" {record.version:>2} {record.from_status.value:>9} -> " + f"{record.to_status.value:<9} {record.trigger}{reason}" + ) + + # ---- 停止是终态：同一个 task_id 不能复活 ---- + print("\n--- 想再跑一次？同一个 task_id 会被拒绝 ---") + try: + controller.submit("worker", "kafka://orders", task_id="orders-worker") + except Exception as exc: + print(f" {type(exc).__name__}: {exc}") + print(" 换一个 task_id 重新提交即可（或让进程一直活着，用 pause/resume 做日常启停）。") + + controller.close(timeout=5) + + _contrast_uncooperative() + _print_cli_usage() + + +def _contrast_uncooperative() -> None: + """对照：不调 ``checkpoint()`` 的任务**停不掉**。 + + 这不是缺陷，是诚实：线程无法被强杀，所以系统只能如实报 ``lost``（"结果未知"）， + 而任务体仍在后台继续跑。真正终止它只能靠结束进程。 + """ + print("\n" + "=" * 70) + print("对照：一个从不调 checkpoint() 的无限任务") + print("=" * 70) + + release = threading.Event() + ticks = {"n": 0} + + def uncooperative(ctx: Any) -> int: + while not release.is_set(): + ticks["n"] += 1 + time.sleep(0.02) + return ticks["n"] + + controller = Controller(store=InMemoryTaskStore(), graceful_stop=0.5) + controller.register("stubborn", uncooperative) + controller.submit("stubborn", task_id="stubborn") + time.sleep(0.3) + + print(" 停止请求已发出，等待载体响应……") + controller.stop("stubborn") + outcome = controller.wait("stubborn", timeout=8) + print(f" 终态: {outcome.status.value} ← 不是 stopped，而是 lost（结果未知）") + print(f" 失败类型: {outcome.failure.kind.value if outcome.failure else None}") + + before = ticks["n"] + time.sleep(0.4) + alive = [t.name for t in threading.enumerate() if t.is_alive() and t.name.startswith("task-")] + print(f" 任务体的线程还在吗: {alive}") + print(f" 它还在继续干活吗: {ticks['n'] > before}（{before} -> {ticks['n']}）") + print(" → 线程无法被强杀，只能靠结束进程来真正终止它。") + + release.set() + controller.close(timeout=3) + + +def _print_cli_usage() -> None: + """运维视角：命令行怎么启、怎么停。""" + print("\n" + "=" * 70) + print("命令行（另一个进程，通过共享存储启停）") + print("=" * 70) + print( + """ +# 启动：前台执行，交给 systemd / nohup 保持存活 +# 注意 --bucket-key：限定了 --owned-buckets 时它是**必需**的，否则任务没有归属桶、 +# 不会有任何节点认领它（CLI 会在提交时报错）。 +chongming-task --db /var/lib/tasks.db --num-buckets 64 --owned-buckets 0-15 \\ + run --tasks myapp.tasks worker --arg source=kafka://orders \\ + --task-id orders-worker --bucket-key orders + +# 另一个终端：看状态、停止 +chongming-task --db /var/lib/tasks.db --owned-buckets 0-15 ps --all --mine +chongming-task --db /var/lib/tasks.db --owned-buckets 0-15 status orders-worker +chongming-task --db /var/lib/tasks.db --owned-buckets 0-15 stop orders-worker --reason 发布窗口 + +# Ctrl-C 也可以：main() 会走 finally 里的 controller.close()，把在跑的任务停干净 +""".rstrip() + ) + print(" 注意：`run` 是前台进程——它退出时任务就没人管了，请交给进程管理器。") + sys.stdout.write("") + + +if __name__ == "__main__": + main() diff --git a/examples/10_worker_walkthrough.py b/examples/10_worker_walkthrough.py new file mode 100644 index 0000000..41f612b --- /dev/null +++ b/examples/10_worker_walkthrough.py @@ -0,0 +1,177 @@ +"""示例 10：常驻任务的完整启停流程 —— **一个脚本跑通，不需要第二个终端**。 + +前面几轮把桶归属、跨进程、多终端全搅在一起了，所以这个示例刻意只用一条主线： + + 启动 → 边跑边看（进度/心跳/租约）→ 暂停 → 恢复 → 停止 → 校验 + +每一步都打印"你该看什么"，最后**断言**进程内的结论与存储里的状态一致——不一致会 +失败退出，不会让你盯着两段输出自己猜。 + +运行:: + + uv run python examples/10_worker_walkthrough.py + +命令行版本（需要第二个终端，因为 ``run`` 是前台进程）见文件末尾的说明。 +""" + +from __future__ import annotations + +import sys +import time +from typing import Any + +from chongming_task import Controller, SqliteTaskStore, render_table + + +def consume(ctx: Any, source: str, batch: int = 1000) -> int: + """常驻消费：无限循环 + 批次进度 + 收到停止请求后优雅收尾。""" + total = 0 + while not ctx.is_stop_requested: + ctx.checkpoint() # ← 不调用它 = 停不掉 + time.sleep(0.05) + total += batch + ctx.progress(total, unit="msg", message=source) + return total + + +def main() -> int: + # 用**文件**存储：这样存储里的状态与进程内的结论是两份数据，能对得上才有意义。 + store = SqliteTaskStore(":memory:") # 想留下文件就把 ":memory:" 换成 "tasks.db" + controller = Controller(store=store, owner="demo-node") + controller.register("consume", consume) + + # ---------------------------------------------------------------- 1. 启动 + print("=" * 72) + print("1) 启动") + print("=" * 72) + run = controller.submit("consume", "kafka://orders", task_id="orders-worker") + print(" controller.submit(...) 立即返回，任务在后台跑：") + print(f" {run.snapshot.task_id} 状态={run.status.value}") + print("\n 注意：`submit` 之后**必须有人让进程活着**（这里的 main 就是那个人）。") + + # ------------------------------------------------- 2. 边跑边看（两个视角） + print("\n" + "=" * 72) + print("2) 边跑边看：进程内的句柄 vs 存储里的事实") + print("=" * 72) + first = _watch(store, seconds=1.0) + second = _watch(store, seconds=1.0) + assert second["progress"] > first["progress"], "进度没有推进——回写链路有问题" + print(f"\n ✅ 进度在推进：{first['progress']} → {second['progress']}") + print(' ✅ 租约新鲜 → 有监督者在管它（进程崩溃时这里会显示"已过期"）') + + # ---------------------------------------------------------------- 3. 暂停 + print("\n" + "=" * 72) + print('3) pause：日常"先停一下"用这个，不要用 stop') + print("=" * 72) + confirmed = controller.pause("orders-worker", timeout=5) + print(f" pause 返回 {confirmed}（True = 已确认挂起；False = 任务没到检查点，状态已回滚）") + print(f" 状态: {store.get('orders-worker').status.value}") # type: ignore[union-attr] + assert confirmed is True + + frozen = _watch(store, seconds=0.6, show_progress=False) + print(f"\n 挂起期间进度不再推进（{frozen['progress']}）—— 但注意：它**仍占着执行槽**") + + print("\n resume 后状态: ", end="") + controller.resume("orders-worker") + time.sleep(0.2) + print(store.get("orders-worker").status.value) # type: ignore[union-attr] + + # ---------------------------------------------------------------- 4. 停止 + print("\n" + "=" * 72) + print('4) stop：终态、不可逆。返回的是"请求已受理"，不是"已停止"') + print("=" * 72) + controller.stop("orders-worker", reason="发布窗口") + outcome = controller.wait("orders-worker", timeout=15) + print(f" 终态: {outcome.status.value} 返回值: {outcome.value}") + print(" ↑ succeeded 而不是 stopped：因为任务体看到 stop 请求后**正常返回**了，") + print(" 已经处理好的数量被完整保留。（让它抛 TaskCancelled 才是 stopped。）") + + # ---------------------------------------------------------------- 5. 校验 + print("\n" + "=" * 72) + print("5) 校验：进程内的结论 与 存储里的状态 必须一致") + print("=" * 72) + print(render_table(controller.list_tasks())) + + stored = store.get("orders-worker") + assert stored is not None + assert stored.status is outcome.status, ( + f"存储={stored.status.value} 与进程内={outcome.status.value} 不一致！" + ) + assert stored.status_version == outcome.snapshot.status_version, "版本号不一致" + assert stored.outcome == {"value": outcome.value}, "返回值没有落库" + print( + f"\n ✅ 一致：status={stored.status.value} " + f"version={stored.status_version} outcome={stored.outcome}" + ) + + print("\n 审计轨迹（终态为什么是 succeeded，看这里就清楚了）：") + for record in store.history("orders-worker"): + reason = f" 原因={record.reason}" if record.reason else "" + print( + f" {record.version} {record.from_status.value:>9} -> " + f"{record.to_status.value:<9} {record.trigger}{reason}" + ) + + failures = controller.persist_failures or "无" + print(f"\n 写穿失败（应当为空；非空说明存储落后于内存）：{failures}") + print(" 永久分叉的任务（应当为空）：", controller.desynced_tasks or "无") + assert controller.persist_failures == (), "有写穿失败——存储不可信" + assert controller.desynced_tasks == {}, "有任务永久分叉" + + controller.close(timeout=10, close_store=True) + + print("\n" + "=" * 72) + print("全部通过 ✅") + print("=" * 72) + _cli_recipe() + return 0 + + +def _watch(store: SqliteTaskStore, *, seconds: float, show_progress: bool = True) -> dict[str, Any]: + """从**存储**采一次样（模拟另一个终端跑 `status`），返回关键字段。""" + deadline = time.monotonic() + seconds + snapshot = None + while time.monotonic() < deadline: + snapshot = store.get("orders-worker") + time.sleep(0.1) + assert snapshot is not None + lease = "无" + if snapshot.lease_until is not None: + from chongming_task import utcnow + + remaining = (snapshot.lease_until - utcnow()).total_seconds() + lease = f"剩余 {remaining:.0f}s" if remaining > 0 else f"已过期 {-remaining:.0f}s" + progress = snapshot.progress.done if snapshot.progress is not None else 0 + if show_progress: + print( + f" 存储: status={snapshot.status.value:<9} v={snapshot.status_version} " + f"progress={progress:<7} 心跳={_hhmmss(snapshot)} 租约={lease}" + ) + return {"progress": progress, "status": snapshot.status.value} + + +def _hhmmss(snapshot: Any) -> str: + return "-" if snapshot.last_heartbeat_at is None else str(snapshot.last_heartbeat_at)[11:19] + + +def _cli_recipe() -> None: + print( + """ +命令行版本（`run` 是前台进程，所以要**两个终端**） +──────────────────────────────────────────────────────── +# 终端 1：启动（前台阻塞；生产上交给 systemd / nohup） +set PYTHONPATH=examples +uv run chongming-task --db ./tasks.db run --tasks worker_tasks consume ^ + --arg source=kafka://orders --task-id orders-worker + +# 终端 2：看状态 → 停止 → 再确认 +uv run chongming-task --db ./tasks.db status orders-worker +uv run chongming-task --db ./tasks.db stop orders-worker --reason 发布窗口 +uv run chongming-task --db ./tasks.db status orders-worker <- 这一步看到 succeeded 才算停干净 +""".rstrip() + ) + print(" 提示：Ctrl-C 也能停（CLI 会走 finally 里的 close()），但 status 确认仍然要做。\n") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/11_best_practice.py b/examples/11_best_practice.py new file mode 100644 index 0000000..8183122 --- /dev/null +++ b/examples/11_best_practice.py @@ -0,0 +1,492 @@ +"""示例 11：最佳实践 —— 一份可以直接照抄的多节点任务服务模板。 + +示例 01–10 各自演示一个机制；这一个把它们收拢成**一个完整、可运行、可运维的 +最小形态**，回答集成时的四个必答题： + + ① 桶怎么划（含节点划分）—— 业务键稳定映射到桶；桶按连续区间分给节点； + 一处计算，代码与 CLI 共用。 + ② 谁跑谁的任务 —— 每个节点只接受自己桶里的任务；配错桶在**提交那一刻** + 报错，而不是产出一个永远没有节点执行的任务。 + ③ 进程启停做什么 —— 启动先 ``reconcile()``（只清理自己桶里的僵尸）； + 退出用 ``close(close_store=True)`` 把连接关干净。 + ④ 怎么观测与控制 —— 所有节点共用一套存储：一个控制台看全部； + 跨节点停止走指令通道（写指令 -> 持有者轮询取走）。 + +文件按"照抄顺序"分四部分： + + # 第一部分 部署参数与桶划分 + # 第二部分 任务定义（业务代码） + # 第三部分 接线：build_node()（唯一集成点） + # 第四部分 演示：启动 -> 提交 -> 观测 -> 控制 -> 收尾 + +运行:: + + uv run python examples/11_best_practice.py + +本示例在一个进程里模拟"两个节点 + 一个运维控制台"，是为了**一条命令跑通全流程**； +生产上它们是各自独立的进程（同一份代码，只有 NODE_INDEX 不同），见文件末尾的部署说明。 +注意：SQLite 文件存储要求所有节点在同一台机器上；**跨机部署**把 ``build_node()`` 里的 +``SqliteTaskStore(deployment.db_path)`` 换成 ``PostgresTaskStore(dsn)`` +（``pip install "chongming-task[postgres]"``）即可，业务代码一行都不用改。 +""" + +from __future__ import annotations + +import logging +import os +import sys +import tempfile +import time +from collections.abc import Sequence +from dataclasses import dataclass +from datetime import timedelta +from pathlib import Path + +from chongming_task import ( + BucketPolicy, + ConcurrencyPolicy, + ConfigurationError, + Controller, + ExponentialBackoff, + RetryPolicy, + SqliteTaskStore, + TaskContext, + TaskSnapshot, + TaskStatus, + render_table, + task, + utcnow, +) + +_log = logging.getLogger("best_practice") + +# ============================================================================= +# 第一部分：部署参数与桶划分（照抄进你的配置模块） +# ============================================================================= + +#: 桶数。默认 64；这里取 16 让演示输出可读。 +#: 它是**部署范围的约定**，不是调优旋钮：所有节点与所有运维命令都必须用同一个数。 +#: 经验值：节点数的 4~16 倍（给节点增减留出重新划分的空间）。 +NUM_BUCKETS = 16 + +#: 本示例部署的节点数。生产上每个节点是一个独立进程。 +NODE_COUNT = 2 + + +def buckets_for_node(node_index: int, node_count: int, num_buckets: int = NUM_BUCKETS) -> range: + """把桶**连续**分给各节点：第 ``node_index`` 个节点负责 ``[start, stop)`` 区间。 + + 为什么连续划分，而不是取模交错（``bucket % node_count``）： + + * 人可读可核对：``--owned-buckets 0-7`` 与代码里的 ``range(0, 8)`` 是同一件事； + * 负载不偏：桶号本身就是 blake2b 的均匀散列，等长区间的期望负载相同； + * 迁移简单：节点增减时整段区间搬运，不用逐桶列举。 + + 除不尽时余数归靠前的节点，保证**不重不漏**——漏桶是静默积压、重桶是重复执行， + 两者都不报错，所以划分必须在一处算出来、处处引用。 + """ + if node_count < 1: + raise ValueError(f"node_count 至少为 1，收到 {node_count}") + if num_buckets < node_count: + raise ValueError( + f"桶数（{num_buckets}）必须不小于节点数（{node_count}）：" + "否则会有节点分不到桶。经验上桶数取节点数的 4~16 倍。" + ) + if not 0 <= node_index < node_count: + raise ValueError(f"node_index 必须在 [0, {node_count}) 内，收到 {node_index}") + size, remainder = divmod(num_buckets, node_count) + start = node_index * size + min(node_index, remainder) + length = size + (1 if node_index < remainder else 0) + return range(start, start + length) + + +@dataclass(frozen=True) +class Deployment: + """一个节点进程的部署参数。生产环境从环境变量 / 配置中心来，不要写死在代码里。""" + + node_index: int + node_count: int + db_path: Path + num_buckets: int = NUM_BUCKETS + + @classmethod + def from_env(cls) -> Deployment: + """从环境变量读取部署参数（systemd / 容器编排注入）。""" + return cls( + node_index=int(os.environ.get("TASK_NODE_INDEX", "0")), + node_count=int(os.environ.get("TASK_NODE_COUNT", "1")), + db_path=Path(os.environ.get("TASK_DB", "tasks.db")), + ) + + @property + def owner(self) -> str: + """写进存储的节点身份。排障时能回答"是谁在管这个任务"。""" + return f"node-{self.node_index}" + + @property + def owned_buckets(self) -> range: + """本节点负责的桶。三个地方必须用同一份结果：Controller 参数、CLI 参数、日志。""" + return buckets_for_node(self.node_index, self.node_count, self.num_buckets) + + @property + def cli_buckets(self) -> str: + """同一个划分的 CLI 写法（``--owned-buckets 0-7``），供运维直接复制。""" + owned = self.owned_buckets + return f"{owned.start}-{owned.stop - 1}" + + +def owner_of(bucket: int, deployments: Sequence[Deployment]) -> Deployment: + """桶号反查节点。查不到意味着划分有空洞或配置不一致——直接炸，不静默。""" + for deployment in deployments: + if bucket in deployment.owned_buckets: + return deployment + raise AssertionError( + f"没有任何节点负责桶 {bucket}：请检查 num_buckets / node_count 是否各处一致" + ) + + +# ============================================================================= +# 第二部分：任务定义（业务代码：只依赖 ctx 约定，照抄进你的 tasks 模块） +# ============================================================================= + + +@task( + tags={"sync", "db"}, + retry=RetryPolicy(max_attempts=2, backoff=ExponentialBackoff(base=0.05, factor=2)), +) +def sync_tenant(ctx: TaskContext, tenant: str, batches: int = 24) -> dict[str, object]: + """同步一个租户的数据。 + + 标签 ``db`` 不只是检索用的：它还是**具名资源信号量**的键——每个节点到库里的 + 并发连接数是有限的，靠它限流比调小全局并发更准（见 ``build_node``）。 + + 长耗时循环里必须周期性调用 ``ctx.checkpoint()``——停止、暂停、心跳都在那里生效； + 不调用它的任务既停不掉，也会被 idle 超时判定为疑似卡死。 + """ + ctx.log(f"{tenant}：开始同步（{batches} 批）") + done = 0 + for index in range(1, batches + 1): + if ctx.is_stop_requested: + # 优雅收尾：正常 return -> 终态 succeeded，返回值保留（"请收工"语义）。 + # 若希望"停止即中止"，去掉这个检查、让 checkpoint() 抛 TaskCancelled + # -> 终态 stopped、无返回值。两种都如实上报，见示例 09。 + break + ctx.checkpoint() + time.sleep(0.05) + done = index + ctx.progress(done, total=batches, unit="批", message=tenant) + ctx.log(f"{tenant}：本次同步 {done}/{batches} 批") + return {"tenant": tenant, "batches": done} + + +# ============================================================================= +# 第三部分：接线 —— build_node()（唯一集成点，应用启动时调用一次） +# ============================================================================= + + +def register(controller: Controller) -> None: + """把任务定义登记进白名单。 + + 两种用法共用同一份定义：应用嵌入时由 :func:`build_node` 调用； + CLI 用 ``--tasks 本模块`` 时，由 CLI 调用（``register(controller)`` + 正是 CLI 的约定函数名）。 + """ + controller.register_all(sync_tenant) + + +def build_node(deployment: Deployment) -> Controller: + """建好一个节点：存储 + 身份 + 桶归属 + 并发上限 + 任务定义 + 启动恢复。 + + 这是应用**唯一**需要接进本库的地方：你的服务（Web / 定时器 / 消费者）启动时 + 调用一次，把返回的 ``Controller`` 持有到进程结束；退出前 ``close(close_store=True)``。 + """ + controller = Controller( + store=SqliteTaskStore(deployment.db_path), + owner=deployment.owner, + bucket_policy=BucketPolicy(num_buckets=deployment.num_buckets), + # 桶归属 = 划分的落地处：本节点只接受、只清理这些桶里的任务。 + owned_buckets=deployment.owned_buckets, + # 并发与资源约束集中在这一处（ConcurrencyPolicy）： + # max_parallel=4 —— 本进程同时最多执行 4 个； + # limits={"db": 3} —— 其中最多 3 个可以同时压数据库（带 db 标签的任务）， + # 保护的是"本节点到库里的连接数"这份预算； + # priority=... —— 默认优先级，submit(priority=...) 可逐条覆盖。 + concurrency=ConcurrencyPolicy(max_parallel=4, limits={"db": 3}), + max_parallel_per_bucket=1, # 同一个租户（同桶）串行，避免自己挤自己 + command_poll_interval=0.1, # 仅为让演示输出得快；生产用默认 0.5s 即可 + ) + register(controller) + # 启动恢复：把上次进程留下的、占着资源却没人管的僵尸任务如实标为 lost。 + # 只看**自己负责的桶**——别的节点的任务由它自己的 reconcile 处理。 + report = controller.reconcile() + owned = deployment.owned_buckets + _log.info( + "[%s] 启动：负责桶 %d-%d；reconcile：%d 个判失联，%d 个待重新提交", + deployment.owner, + owned.start, + owned.stop - 1, + len(report.lost), + len(report.adoptable), + ) + return controller + + +# ============================================================================= +# 第四部分：演示（启动 -> 提交 -> 观测 -> 控制 -> 收尾；这一段不用照抄） +# ============================================================================= + +#: 演示用的租户（业务键）。生产上它来自你的业务清单。 +TENANTS = ("tenant-acme", "tenant-umbrella", "tenant-globex", "tenant-hooli") + +#: 给"跨节点停止"演示留足运行窗口：80 批 × 0.05s ≈ 4 秒。 +SLOW_BATCHES = {"tenant-globex": 80} + +#: 模拟"上一个进程崩溃"遗留的僵尸任务的业务键（bucket 7，归 node-0）。 +ZOMBIE_KEY = "tenant-vandelay" + + +def _plant_zombie(db_path: Path, policy: BucketPolicy, key: str) -> str: + """往存储里放一条"崩溃进程留下的"任务：RUNNING 且租约已过期。 + + 真实场景里这是崩溃的进程自己留下的（写穿 + 租约续不上）；这里手工造一条， + 让 reconcile 有东西可清、让"如实标记而不是假装恢复"看得见。 + """ + now = utcnow() + task_id = f"sync-{key}" + snapshot = TaskSnapshot( + task_id=task_id, + name=f"同步-{key}", + bucket=policy.bucket_for(key), + bucket_key=key, + status=TaskStatus.RUNNING, + status_version=1, + attempt=1, + max_attempts=1, + created_at=now - timedelta(minutes=12), + queued_at=now - timedelta(minutes=12), + started_at=now - timedelta(minutes=11), + last_heartbeat_at=now - timedelta(minutes=11), + executor="thread", + lease_owner="node-0@dead-process", + lease_until=now - timedelta(minutes=6), + ) + store = SqliteTaskStore(db_path) + try: + store.create(snapshot) + finally: + store.close() + return task_id + + +def _print_plan(deployments: Sequence[Deployment], policy: BucketPolicy, db_path: Path) -> None: + """打印部署计划、桶划分与路由表——集成前先把"谁负责什么"看清楚。""" + print("=" * 72) + print(f"部署：{NUM_BUCKETS} 个桶，{NODE_COUNT} 个节点，共享一套存储（{db_path.name}）") + print("=" * 72) + print(" 节点 负责桶 等价的 CLI 参数") + print(" -------- ---------- -----------------------------------") + for deployment in deployments: + owned = deployment.owned_buckets + spec = f"--owned-buckets {deployment.cli_buckets}" + print(f" {deployment.owner:<8} {owned.start:>2}-{owned.stop - 1:<8} {spec}") + print(" 注意：--num-buckets 也必须一致，否则同一个键会落到不同的桶，且无任何报错。") + + print("\n 路由表（blake2b 稳定哈希：同一个键在任何进程、任何机器上都落到同一个桶）：") + for tenant in TENANTS: + bucket = policy.bucket_for(tenant) + print(f" {tenant:<18} -> 桶 {bucket:>2} -> {owner_of(bucket, deployments).owner}") + + +def main() -> int: + # 输出到 stdout（默认是 stderr）：这样日志与 print 的顺序一致， + # 管道重定向时也不会看到"日志跑到前面去了"。 + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s", stream=sys.stdout) + + with tempfile.TemporaryDirectory(prefix="chongming-best-practice-") as directory: + db_path = Path(directory) / "tasks.db" + deployments = [ + Deployment(node_index=index, node_count=NODE_COUNT, db_path=db_path) + for index in range(NODE_COUNT) + ] + policy = BucketPolicy(num_buckets=NUM_BUCKETS) + + _print_plan(deployments, policy, db_path) + + zombie_id = _plant_zombie(db_path, policy, ZOMBIE_KEY) + zombie_bucket = policy.bucket_for(ZOMBIE_KEY) + assert zombie_bucket in deployments[0].owned_buckets, "僵尸任务应落在 node-0 的桶里" + print( + f"\n[准备] 模拟上一个进程崩溃：存储里留有一条 RUNNING 且租约过期的任务 " + f"{zombie_id}（桶 {zombie_bucket}，归 node-0）" + ) + + # ---------------------------------------------------------------- 启动 + print("\n" + "=" * 72) + print("一、启动：每个节点先 reconcile()，只清理**自己桶里**的僵尸任务") + print("=" * 72) + nodes = [build_node(deployment) for deployment in deployments] + # 运维控制台：一个不持有任务的 Controller，只查询与控制（不声明 owned_buckets）。 + console = Controller(store=SqliteTaskStore(db_path), owner="ops-console") + + zombie = console.get(zombie_id) + assert zombie is not None and zombie.status is TaskStatus.LOST + print( + f" 控制台看到 {zombie_id} 现在是 {zombie.status.value}" + "（被如实标记为丢失，而不是永远显示 running；它不会被自动重跑——" + "存储里只有状态、没有配方）" + ) + + # ---------------------------------------------------------------- 提交 + print("\n" + "=" * 72) + print("二、触发：按桶把任务交给拥有它的节点（配错桶 = 永远不会有人执行）") + print("=" * 72) + for tenant in TENANTS: + bucket = policy.bucket_for(tenant) + node = nodes[owner_of(bucket, deployments).node_index] + run = node.submit( + "sync_tenant", + tenant, + task_id=f"sync-{tenant}", + label=f"同步-{tenant}", + bucket_key=tenant, + batches=SLOW_BATCHES.get(tenant, 24), + ) + print(f" {tenant:<18} 桶 {bucket:>2} -> {node.owner} 提交 {run.snapshot.task_id}") + + foreign_tenant = next( + tenant + for tenant in TENANTS + if owner_of(policy.bucket_for(tenant), deployments).node_index != 0 + ) + foreign_bucket = policy.bucket_for(foreign_tenant) + foreign_owner = owner_of(foreign_bucket, deployments).owner + print( + f"\n >>> 反例：把 {foreign_tenant}（桶 {foreign_bucket}，归 {foreign_owner}）" + "提交给 node-0：" + ) + try: + nodes[0].submit( + "sync_tenant", + foreign_tenant, + task_id="sync-misrouted", + bucket_key=foreign_tenant, + ) + except ConfigurationError as exc: + print(f" ✅ 提交那一刻就拦住了：{exc}") + else: + raise AssertionError("提交给错误的节点居然成功了——桶归属校验失效") + + # ---------------------------------------------------------------- 观测 + time.sleep(0.3) + print("\n" + "=" * 72) + print("三、观测：一套存储 = 一个看板看全部；按桶过滤 = 只看我这一份") + print("=" * 72) + print("--- 全局视图（控制台不在任何节点上，也能看到全部任务）---") + print(render_table(console.list_tasks())) + + owned_0 = deployments[0].owned_buckets + print(f"\n--- node-0 的视角（list_tasks(buckets={owned_0.start}-{owned_0.stop - 1})）---") + print(render_table(nodes[0].list_tasks(buckets=owned_0))) + + for node in nodes: + print(f" {node.owner}：占用执行额度 {node.active_count}，排队 {node.queued_count}") + + # ---------------------------------------------------------------- 控制 + target_id = f"sync-{foreign_tenant}" + target = console.get(target_id) + assert target is not None and not target.is_terminal, "演示假设：停止目标仍在运行" + + print("\n" + "=" * 72) + print("四、控制：跨节点停止（不能跨进程发信号 -> 写一条指令，由持有者取走）") + print("=" * 72) + print( + f" ops-console 请求停止 {target_id}（持有者是 {foreign_owner}）：" + f"下发结果 {console.stop(target_id, reason='发布窗口')}" + ) + outcome = nodes[1].wait(target_id, timeout=15) + print(f" 终态：{outcome.status.value}；返回值：{outcome.value}") + print(" 说明：任务体看到停止请求后优雅收尾 -> succeeded 且数值保留；") + print(" 让 checkpoint() 抛 TaskCancelled 则为 stopped。两种都如实上报。") + print("\n 审计轨迹（从共享存储读；谁按的、为什么，都可追溯）：") + for record in console.store.history(target_id): # type: ignore[union-attr] + reason = f" reason={record.reason}" if record.reason else "" + print( + f" {record.version:>2} {record.from_status.value:>9} -> " + f"{record.to_status.value:<9} {record.trigger}{reason}" + ) + + # ---------------------------------------------------------------- 收尾 + for node in nodes: + node.wait_all(timeout=30) + + print("\n" + "=" * 72) + print("五、收尾：校验一致性 -> close(close_store=True)") + print("=" * 72) + print(render_table(console.list_tasks())) + stats = console.stats() + print(f" 统计：共 {stats.total}，未结束 {stats.unfinished}，已结束 {stats.finished}") + + for tenant in TENANTS: + snapshot = console.get(f"sync-{tenant}") + assert snapshot is not None and snapshot.status in ( + TaskStatus.SUCCEEDED, + TaskStatus.STOPPED, + ), f"{tenant} 的终态不对：{snapshot.status if snapshot else '不存在'}" + for node in nodes: + assert node.persist_failures == (), f"{node.owner} 有写穿失败——存储不可信" + assert node.desynced_tasks == {}, f"{node.owner} 有永久分叉的任务" + print(" ✅ 结论与存储一致：没有写穿失败，没有永久分叉，僵尸任务如实落 lost") + + for node in nodes: + node.close(timeout=30, close_store=True) + console.close(timeout=5, close_store=True) + + _print_ops_notes() + + print("\n全部通过 ✅") + return 0 + + +def _print_ops_notes() -> None: + """部署与运维说明。""" + print( + f""" +生产部署：把它变成一个真正的服务 +──────────────────────────────────────────────────────────────────────── +# 每台机器 / 每个容器跑一个节点进程：同一份代码，只有 NODE_INDEX 不同 +TASK_NODE_INDEX=0 TASK_NODE_COUNT={NODE_COUNT} TASK_DB=/srv/tasks/tasks.db \\ + python -m your_service.node +TASK_NODE_INDEX=1 TASK_NODE_COUNT={NODE_COUNT} TASK_DB=/srv/tasks/tasks.db \\ + python -m your_service.node + +# 运维（任意一台机器；--num-buckets 与代码里的 NUM_BUCKETS 必须一致） +uv run chongming-task --db /srv/tasks/tasks.db --num-buckets {NUM_BUCKETS} \\ + ps --all +uv run chongming-task --db /srv/tasks/tasks.db --num-buckets {NUM_BUCKETS} \\ + --owned-buckets 0-7 ps --all --mine +uv run chongming-task --db /srv/tasks/tasks.db --num-buckets {NUM_BUCKETS} \\ + --owned-buckets 0-7 reconcile --dry-run +uv run chongming-task --db /srv/tasks/tasks.db --num-buckets {NUM_BUCKETS} \\ + stop sync-tenant-globex --reason "发布窗口" + +# 本模块也能直接当 CLI 任务模块用（register(controller) 约定） +PYTHONPATH=examples uv run chongming-task --db ./demo.db --num-buckets {NUM_BUCKETS} \\ + --owned-buckets 0-7 run --tasks 11_best_practice sync_tenant \\ + --arg tenant=tenant-acme --task-id sync-tenant-acme --bucket-key tenant-acme + +集成三步（照抄本文件的三个"部分"） +──────────────────────────────────────────────────────────────────────── +# 1. 第一部分 + 第二部分 -> 你的配置模块与任务模块； +# 2. 应用启动时 build_node(Deployment.from_env())，退出前 close(close_store=True)； +# 3. 触发处 submit(任务名, ..., bucket_key=业务键)。 +# 两条硬约束：任务只能提交给拥有它那个桶的节点；bucket_key 必须给 +# （限定了 owned_buckets 时不给会被当场拒绝——没有归属桶的任务没人认领）。 +# +# 跨机部署：SQLite 文件不能跨机共享，把第三部分的 store 换成 +# PostgresTaskStore("postgresql://user:pw@db.internal:5432/chongming") +# （pip install "chongming-task[postgres]"）即可，其余部分不用改。""".rstrip() + ) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/12_process_executor.py b/examples/12_process_executor.py new file mode 100644 index 0000000..16811c9 --- /dev/null +++ b/examples/12_process_executor.py @@ -0,0 +1,201 @@ +"""示例 12：进程执行器 —— 硬性停止、绕开 GIL、崩溃隔离。 + + uv run python examples/12_process_executor.py + +绝大多数任务该用默认的**线程执行器**：共享内存、没有 pickle 约束、能直接复用连接池。 +但有三件事线程做不到，只有子进程能： + +1. **硬性停止保证**：任务不响应停止（不调 ``ctx.checkpoint()``）时，线程只能放弃等待 + 并把结果记成 ``lost``（未知）；子进程会被 ``terminate()`` → ``kill()`` 真的停掉； +2. **绕开 GIL**：CPU 密集的计算能真正并行； +3. **崩溃隔离**：任务里加载的原生扩展段错误，只会带走子进程。 + +本示例把 1 和 3 跑给你看，并把代价说清楚。 + +这个文件本身也是"目标函数必须是模块级"的演示：下面的任务都定义在模块顶层，因为 +spawn 起的子进程要通过"模块 + 函数名"把它们找回来。 +""" + +from __future__ import annotations + +import logging +import os +import sys +import time +from typing import Any + +from chongming_task import ( + ConfigurationError, + FailureKind, + ProcessExecutor, + RetryPolicy, + TaskStatus, + Timeout, + run_task, +) +from chongming_task.runtime.executors.base import ExecutionRequest + +# 子进程不继承父进程的 logging 配置（spawn 起的是全新解释器），日志想看就得自己配。 +logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout) + + +# ----------------------------------------------------------------------------- +# 任务定义（模块级：spawn 的子进程要靠"模块 + 函数名"找回来） +# ----------------------------------------------------------------------------- + + +def cpu_burn(ctx: Any, rounds: int = 6) -> dict[str, object]: + """CPU 密集的活：子进程里不受父进程 GIL 的影响。""" + total = 0 + for _ in range(rounds): + ctx.checkpoint() + for index in range(400_000): + total += index % 7 + return {"rounds": rounds, "checksum": total, "pid": os.getpid()} + + +def deaf_loop(ctx: Any) -> str: + """**不**调用 checkpoint 的任务：停止请求它根本看不见。 + + 这正是不配合的任务的样子——拿线程执行器它只会落 ``lost``；拿进程执行器它会被真的杀掉。 + """ + del ctx + end = time.monotonic() + 60 + while time.monotonic() < end: + pass + return "never" + + +def crash_hard(ctx: Any) -> str: + """模拟原生扩展段错误：直接消失，一句话都不留。""" + del ctx + os._exit(139) + return "never" # pragma: no cover - 走不到这里 + + +# ----------------------------------------------------------------------------- +# 演示 +# ----------------------------------------------------------------------------- + + +def demo_cpu_bound() -> None: + print("=" * 72) + print("一、CPU 密集：任务跑在子进程里（父进程的 GIL 管不着它）") + print("=" * 72) + outcome = run_task( + cpu_burn, + 6, + task_id="cpu-1", + executor=ProcessExecutor(), + timeout=Timeout(per_attempt=30), + ) + assert outcome.ok, outcome.failure + payload = outcome.value + print(f" 结果：{payload}") + print(f" 监督进程 pid={os.getpid()}，任务进程 pid={outcome.snapshot.pid}") + print(f" 快照里的 executor={outcome.snapshot.executor}（写进存储，事后可查）") + print() + + +def demo_hard_stop() -> None: + print("=" * 72) + print("二、硬性停止：不响应停止的任务被真的杀掉（线程执行器只能放弃等待）") + print("=" * 72) + started = time.monotonic() + outcome = run_task( + deaf_loop, + task_id="deaf-1", + executor=ProcessExecutor(), + timeout=Timeout(idle=1), # 一秒没有心跳就判卡死 + retry=RetryPolicy(max_attempts=1), + graceful_stop=0.5, # 优雅窗口：0.5 秒内不响应就强杀 + ) + elapsed = time.monotonic() - started + print(f" 任务自己的循环要跑 60 秒，实际用了 {elapsed:.1f} 秒") + print(f" 终态：{outcome.status.value}；失败类型：{outcome.failure.kind.value}") + print(f" 子进程 pid={outcome.snapshot.pid} —— 它已经不在了（ps 查不到）") + assert outcome.status is TaskStatus.FAILED + assert outcome.failure.kind is FailureKind.TIMEOUT_IDLE, "超时如实上报，不谎报'已停止'" + assert elapsed < 30, "不该等到任务自己结束" + print(" 注意：超时仍然记成超时（不是'已停止'）——但资源确实释放了。") + print() + + +def demo_crash_isolation() -> None: + print("=" * 72) + print("三、崩溃隔离：子进程崩了，监督进程活着并如实记一笔") + print("=" * 72) + outcome = run_task( + crash_hard, + task_id="crash-1", + executor=ProcessExecutor(), + timeout=Timeout(per_attempt=30), + ) + assert not outcome.ok + print(f" 监督进程还活着（pid={os.getpid()}），任务终态：{outcome.status.value}") + print(f" 失败类型：{outcome.failure.kind.value}；退出码：{outcome.failure.exit_code}") + print(f" 说明：{outcome.failure.message}") + print(" 换成线程执行器，这个崩溃会带走整个监督进程。") + print() + + +def demo_picklable_guard() -> None: + print("=" * 72) + print("四、代价：不可 pickle 的目标在**提交那一刻**就被拦下") + print("=" * 72) + + def local_closure(ctx: Any) -> str: # 局部函数：子进程里找不到它 + del ctx + return "nope" + + # 1) 直接交给执行器：当场抛错，并告诉你该改成什么。 + request = ExecutionRequest(task_id="bad-0", name="bad", target=local_closure, args=()) + try: + ProcessExecutor().launch(request) + except ConfigurationError as exc: + print(f" launch() 直接报错 {type(exc).__name__}：") + for line in str(exc).splitlines(): + print(f" {line}") + else: # pragma: no cover - 应当抛错 + raise AssertionError("不可 pickle 的目标必须被拦下") + + # 2) 走 run_task：任务**落 failed 并带上这条诊断**，调用方不会干等，也不会被 + # 说成"框架内部错误"——那是把任务侧的问题指向错误的排查方向。 + outcome = run_task(local_closure, task_id="bad-1", executor=ProcessExecutor()) + assert outcome.status is TaskStatus.FAILED + assert outcome.failure is not None + print(f"\n 走 run_task：终态 {outcome.status.value}，retryable={outcome.failure.retryable}") + print(f" 失败信息首行：{outcome.failure.message.splitlines()[0]}") + print(" 配置类错误不重试——重试多少次都是同一个结果，只会白烧重试预算。") + assert outcome.failure.retryable is False + print() + + +def main() -> int: + print() + demo_cpu_bound() + demo_hard_stop() + demo_crash_isolation() + demo_picklable_guard() + + print("=" * 72) + print("选型：默认线程，需要这三种能力时才换进程") + print("=" * 72) + print( + """ + 线程（默认） 进程（ProcessExecutor） + ------------------------ ------------------------------------------ + 共享内存、可复用连接池 参数与返回值必须可 pickle + 闭包 / lambda 都能用 目标必须是模块级函数，调用方要有 __main__ 守卫 + 停止/暂停即时生效 停止/暂停有约 20ms 轮询延迟 + 不配合的任务只能放弃（lost） 不配合的任务被真的杀掉 + 受 GIL 限制 真并行 + 原生扩展段错误带走整个进程 只带走子进程 +""" + ) + print("全部通过 ✅") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/tasks_demo.py b/examples/tasks_demo.py new file mode 100644 index 0000000..1382291 --- /dev/null +++ b/examples/tasks_demo.py @@ -0,0 +1,47 @@ +"""给 CLI 用的任务模块。 + +CLI **只传任务名**，定义必须来自这样一个模块——一个能启动任意可调用对象的命令行 +入口等于任意代码执行，所以"任务定义从哪里来"必须先有答案。约定就是本文件里的 +``register(controller)``。 + +用法（把本目录加进 ``PYTHONPATH``，或把模块放进你自己的包里）:: + + chongming-task --db /tmp/tasks.db tasks --tasks tasks_demo + chongming-task --db /tmp/tasks.db run --tasks tasks_demo add --arg left=2 --arg right=3 +""" + +from __future__ import annotations + +import time +from typing import Any + + +def sync_tenant(ctx: Any, tenant: str, batches: int = 120) -> str: + """同步一个租户的数据。""" + for index in range(batches): + ctx.checkpoint() + if index % 20 == 0: + time.sleep(0.05) + return f"{tenant} 同步完成（{batches} 批）" + + +def add(ctx: Any, left: int, right: int) -> int: + """两数相加。""" + del ctx + return left + right + + +def flaky(ctx: Any, fail_times: int = 2) -> str: + """前几次故意失败，用来演示重试。""" + state = getattr(flaky, "_attempts", 0) + 1 + flaky._attempts = state # type: ignore[attr-defined] + if state <= fail_times: + raise RuntimeError(f"第 {state} 次尝试遇到暂时性故障") + return f"第 {state} 次尝试成功" + + +def register(controller: Any) -> None: + """把任务登记进白名单。CLI 导入本模块后就会调用它。""" + controller.register("sync", sync_tenant, tags={"io"}) + controller.register("add", add, tags={"cpu"}) + controller.register("flaky", flaky) diff --git a/examples/worker_tasks.py b/examples/worker_tasks.py new file mode 100644 index 0000000..81f870d --- /dev/null +++ b/examples/worker_tasks.py @@ -0,0 +1,48 @@ +"""一个可以直接用的常驻任务模块（CLI 的 ``--tasks`` 就指向它）。 + +它是给命令行用的示例模块，也是你自己模块的模板。用法:: + + PYTHONPATH=examples uv run chongming-task --db ./tasks.db \ + run --tasks worker_tasks consume --arg source=kafka://orders --task-id orders-worker + +## 三件必须做对的事 + +1. **每轮调 ``ctx.checkpoint()``**——停止是协作式的，不调用它就停不掉（会落 ``lost``）。 +2. **用 ``ctx.is_stop_requested`` 收尾**——这样任务终态是 ``succeeded``，返回值保留； + 让 ``checkpoint()`` 抛则是 ``stopped``、无返回值。 +3. **进度按批次上报，不要每条一次**——``ctx.progress()`` 每次都要取运行锁，它是给 + "人看得懂"的频率设计的。 +""" + +from __future__ import annotations + +import time +from typing import Any + +from chongming_task import RetryPolicy, task + + +@task(tags={"kafka"}, retry=RetryPolicy(max_attempts=1)) +def consume(ctx: Any, source: str, batch: int = 1000) -> int: + """常驻消费 source；收到停止请求后收尾并返回本次共处理多少条。""" + ctx.log(f"开始消费 {source}") + total = 0 + while not ctx.is_stop_requested: + ctx.checkpoint() + + # 假装取了一批消息。真实实现里这里是一次 poll / 一次拉取。 + time.sleep(0.05) + total += batch + + # 按批次上报进度（不是每条一次）。 + ctx.progress(total, unit="msg", message=source) + + ctx.log(f"收到停止请求，收尾：本次共处理 {total} 条") + return total + + +@task +def once(ctx: Any, value: int) -> int: + """一个短任务，用来对照"有始有终"的普通形态。""" + del ctx + return value * 2 diff --git a/myapp/__init__.py b/myapp/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/myapp/tasks.py b/myapp/tasks.py new file mode 100644 index 0000000..5c0ae8e --- /dev/null +++ b/myapp/tasks.py @@ -0,0 +1,17 @@ +# myapp/tasks.py +from chongming_task import task + + +@task(tags={"io"}) +def worker(ctx, source: str) -> int: + """常驻消费。""" + processed = 0 + while not ctx.is_stop_requested: # ← 优雅收尾（终态 succeeded，返回值保留） + ctx.checkpoint() # ← 不调用它 = 停不掉 + processed += 1 + ctx.progress(processed, unit="batch") + return processed + + +def register(controller): # 或者不写它，只靠 @task 也行 + controller.register_module(__import__(__name__)) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..8a6db76 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,94 @@ +[build-system] +build-backend = "hatchling.build" +requires = ["hatchling"] + +[project] +dependencies = [] +description = "精细化状态控制的 Python 管道任务控制工具" +name = "chongming-task" +readme = "README.md" +requires-python = ">=3.11" +version = "0.1.0" +classifiers = [ + "Development Status :: 3 - Alpha", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", +] + +[project.scripts] +chongming-task = "chongming_task.cli.main:main" + +[project.optional-dependencies] +postgres = [ + "psycopg-pool>=3.2", + "psycopg[binary]>=3.2", +] + +# 核心包运行时零第三方依赖（CLI 也只用标准库 argparse）。 +# 观测后端等适配器在各自阶段追加为 [project.optional-dependencies]。 + +[dependency-groups] +dev = [ + "pytest>=8.0", + "hypothesis>=6.100", + "ruff>=0.6", + "mypy>=1.11", + "psycopg[binary]>=3.2", + "psycopg-pool>=3.2", +] + +[tool.hatch.build.targets.wheel] +packages = ["src/chongming_task"] + +# --------------------------------------------------------------------------- +# 工具链 +# --------------------------------------------------------------------------- + +[tool.ruff] +line-length = 100 +target-version = "py311" +src = ["src", "tests"] +# pytest 的临时目录默认落在项目根下，测试往里写的临时 .py 文件不该被 lint 扫到 +# （其中不乏**故意写坏**的样本，例如"导入一个不存在的依赖"）。 +extend-exclude = ["pytest-of-*"] + +[tool.ruff.lint] +select = ["E", "F", "I", "UP", "B", "SIM", "RUF", "C4", "PTH"] +ignore = [ + "RUF001", # 允许注释/字符串中的中文全角标点 + "RUF002", + "RUF003", +] + +[tool.ruff.lint.per-file-ignores] +"tests/**" = ["B011"] + +[tool.ruff.lint.isort] +known-first-party = ["chongming_task"] + +# 内核必须保持可被状态机属性测试穷举验证的纯粹性，故 src 全量 strict。 +[tool.mypy] +python_version = "3.11" +strict = true +warn_unreachable = true +warn_redundant_casts = true +show_error_codes = true +files = ["src"] + +[[tool.mypy.overrides]] +module = "tests.*" +disallow_untyped_defs = false +disallow_incomplete_defs = false + +[tool.pytest.ini_options] +testpaths = ["tests"] +addopts = "-ra --strict-markers --strict-config" +markers = [ + "slow: 耗时较长的集成测试", + "windows_only: 仅在 Windows 上有意义的用例", + "posix_only: 仅在 POSIX 上有意义的用例", +] +filterwarnings = ["error"] diff --git a/src/chongming_task/__init__.py b/src/chongming_task/__init__.py new file mode 100644 index 0000000..d9b4346 --- /dev/null +++ b/src/chongming_task/__init__.py @@ -0,0 +1,208 @@ +"""精细化状态控制的 Python 管道任务控制工具。 + +公开 API 收敛在这里，方便直接嵌进其他应用:: + + from chongming_task import Controller, ThreadExecutor + + controller = Controller(max_parallel=4) + controller.register("extract", extract) # 先登记，之后只传名字 + controller.submit("extract", "s3://bucket/x", task_id="job-1") + +只跑一个任务而不需要管理多个时:: + + from chongming_task import run_task + + outcome = run_task(extract, "s3://bucket/x") + +一切都以**名字**而不是代码提交，因此 :class:`Controller` 依赖 +:class:`~chongming_task.controller.TaskRegistry` 做白名单——一个能启动任意可调用 +对象的入口等于远程代码执行。 + +存储层 +------ +持久化契约是 :class:`~chongming_task.stores.protocol.TaskStore`，内置 +:class:`~chongming_task.stores.memory.InMemoryTaskStore` 与 +:class:`~chongming_task.stores.sqlite.SqliteTaskStore`（标准库 ``sqlite3``）。 +跨机部署用 :class:`~chongming_task.stores.postgres.PostgresTaskStore`——它需要 +可选的 psycopg（``pip install "chongming-task[postgres]"``），因此**惰性导入**： +``from chongming_task import PostgresTaskStore`` 能用，但它不在 ``__all__`` 里， +装没装驱动都不影响 ``import chongming_task`` 本身。 + +``Controller(store=...)`` 一旦给出存储，本实例就成为跨进程可见的一环：状态迁移 +写穿、查询读穿、对不属于自己的任务下发指令。第三方后端可用 +``chongming_task.testing.store_contract.StoreContract`` 验证自己。 +""" + +from __future__ import annotations + +from typing import Any + +from .clock import Clock, utcnow +from .controller import ( + Controller, + ReconcileEntry, + ReconcileOutcome, + ReconcileReport, + Registration, + StatusCounts, + TaskRegistry, + render_table, +) +from .core.state_machine import StateMachine, Transition, Trigger +from .decorators import TaskSpec, get_task_spec, task +from .errors import ( + AttemptsExhausted, + CircularDependency, + ConcurrencyConflict, + ConfigurationError, + DependencyFailed, + InvalidTransition, + StoreError, + TaskCancelled, + TaskError, + TaskExecutionError, + TaskTimeout, +) +from .models import ( + AttemptRecord, + Failure, + FailureKind, + Progress, + RequestedAction, + TaskSnapshot, + TimeoutScope, +) +from .pipeline import ( + Node, + Pipeline, + PipelineOutcome, + PipelineRun, + PipelineStatus, + render_pipeline, +) +from .policies import ( + AggregationMode, + AggregationPolicy, + Backoff, + BucketPolicy, + ConcurrencyPolicy, + ExponentialBackoff, + FixedBackoff, + Jitter, + JoinMode, + RetryPolicy, + Timeout, +) +from .run import TaskRun, TaskRunOutcome, run_task +from .runtime.cancellation import CancellationToken +from .runtime.context import RuntimeContext, TaskContext +from .runtime.executors.base import ExecutionHandle, ExecutionOutcome, Executor +from .runtime.executors.inline import InlineExecutor +from .runtime.executors.process import ProcessExecutor +from .runtime.executors.thread import ThreadExecutor +from .states import ( + NON_TERMINAL, + RESOURCE_HOLDING, + REVIVABLE, + TERMINAL, + TRANSITIONS, + TaskStatus, +) +from .stores.memory import InMemoryTaskStore +from .stores.protocol import Command, CommandKind, TaskQuery, TaskStore +from .stores.sqlite import SqliteTaskStore + +__version__ = "0.1.0" + + +def __getattr__(name: str) -> Any: + """惰性导出可选依赖的实现（``PostgresTaskStore``）。 + + 核心包**零第三方运行时依赖**，所以需要 psycopg 的实现不能放进 ``__all__``： + ``from chongming_task import *`` 不该迫使所有使用方安装数据库驱动。真正取到 + 这个名字时才导入；缺驱动时 psycopg 的 ImportError 会带上安装指引。 + """ + if name == "PostgresTaskStore": + from .stores.postgres import PostgresTaskStore + + return PostgresTaskStore + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + + +__all__ = [ + "NON_TERMINAL", + "RESOURCE_HOLDING", + "REVIVABLE", + "TERMINAL", + "TRANSITIONS", + "AggregationMode", + "AggregationPolicy", + "AttemptRecord", + "AttemptsExhausted", + "Backoff", + "BucketPolicy", + "CancellationToken", + "CircularDependency", + "Clock", + "Command", + "CommandKind", + "ConcurrencyConflict", + "ConcurrencyPolicy", + "ConfigurationError", + "Controller", + "DependencyFailed", + "ExecutionHandle", + "ExecutionOutcome", + "Executor", + "ExponentialBackoff", + "Failure", + "FailureKind", + "FixedBackoff", + "InMemoryTaskStore", + "InlineExecutor", + "InvalidTransition", + "Jitter", + "JoinMode", + "Node", + "Pipeline", + "PipelineOutcome", + "PipelineRun", + "PipelineStatus", + "ProcessExecutor", + "Progress", + "ReconcileEntry", + "ReconcileOutcome", + "ReconcileReport", + "Registration", + "RequestedAction", + "RetryPolicy", + "RuntimeContext", + "SqliteTaskStore", + "StateMachine", + "StatusCounts", + "StoreError", + "TaskCancelled", + "TaskContext", + "TaskError", + "TaskExecutionError", + "TaskQuery", + "TaskRegistry", + "TaskRun", + "TaskRunOutcome", + "TaskSnapshot", + "TaskSpec", + "TaskStatus", + "TaskStore", + "TaskTimeout", + "ThreadExecutor", + "Timeout", + "TimeoutScope", + "Transition", + "Trigger", + "get_task_spec", + "render_pipeline", + "render_table", + "run_task", + "task", + "utcnow", +] diff --git a/src/chongming_task/cli/__init__.py b/src/chongming_task/cli/__init__.py new file mode 100644 index 0000000..97b9d09 --- /dev/null +++ b/src/chongming_task/cli/__init__.py @@ -0,0 +1,13 @@ +"""命令行控制台。 + +``chongming-task`` 是**另一个进程**：它只能通过共享存储看到任务。除 ``run``（自身即 +worker，前台执行到结束）之外，所有子命令都需要 ``--db`` 或 ``CHONGMING_TASK_DB``。 + +用法与退出码见 :mod:`chongming_task.cli.main`。 +""" + +from __future__ import annotations + +from .main import build_parser, main + +__all__ = ["build_parser", "main"] diff --git a/src/chongming_task/cli/main.py b/src/chongming_task/cli/main.py new file mode 100644 index 0000000..741d638 --- /dev/null +++ b/src/chongming_task/cli/main.py @@ -0,0 +1,923 @@ +"""命令行控制台。 + +CLI 是**另一个进程**：它看不到任务进程的内存，只能通过共享存储查询与控制。这正是 +持久层存在的理由——没有它，``chongming-task ps`` 什么也看不到。 + +因此除 ``run``（它自己就是 worker，前台执行到结束）之外，所有子命令都要求给出存储 +位置，通过 ``--db`` 或环境变量 ``CHONGMING_TASK_DB``。 + +退出码： + +* ``0`` 命令成功； +* ``1`` 命令本身失败（参数错、任务不存在、存储打不开等）； +* ``3`` 命令成功，但被查询/被执行的任务**没有成功**（便于 shell 里判分支）。 +""" + +from __future__ import annotations + +import argparse +import importlib +import json +import os +import sys +from collections.abc import Sequence +from datetime import datetime +from typing import Any + +from ..clock import utcnow +from ..controller import Controller, render_table +from ..decorators import get_task_spec +from ..errors import ConfigurationError, TaskError +from ..models import TaskSnapshot +from ..policies import BucketPolicy, RetryPolicy, Timeout +from ..runtime.executors.base import Executor +from ..runtime.executors.process import ProcessExecutor +from ..states import TERMINAL, TaskStatus +from ..stores.codec import snapshot_to_row +from ..stores.protocol import CommandKind, TaskQuery, TaskStore +from ..stores.sqlite import SqliteTaskStore + +__all__ = ["main"] + +EXIT_OK = 0 +EXIT_ERROR = 1 +EXIT_TASK_NOT_SUCCESSFUL = 3 + +_ENV_DB = "CHONGMING_TASK_DB" + + +# ---------------------------------------------------------------------- +# 参数解析 +# ---------------------------------------------------------------------- + + +def build_parser() -> argparse.ArgumentParser: + """构造参数解析器。单独暴露是为了让测试能直接查帮助与解析结果。""" + parser = argparse.ArgumentParser( + prog="chongming-task", + description="精细化状态控制的管道任务控制台。", + epilog=( + "存储位置可用 --db 或环境变量 CHONGMING_TASK_DB 指定；" + "没有共享存储就看不到别的进程的任务。" + ), + ) + parser.add_argument( + "--db", + metavar="PATH", + help=( + "任务存储：SQLite 文件路径，或 postgresql:// 连接串（跨机部署，需要 " + f'"chongming-task[postgres]"). 未给出时读环境变量 {_ENV_DB}。' + ), + ) + parser.add_argument("--json", action="store_true", help="以 JSON 输出，便于脚本消费。") + parser.add_argument("--owner", default=None, help="本客户端的标识，写进它下发的指令里。") + parser.add_argument( + "--num-buckets", + type=int, + default=None, + metavar="N", + help=( + "集群的桶数。**必须与部署一致**——它决定 --owned-buckets 的合法性以及 buckets " + "视图的行数。不给出时按默认值（64）。" + ), + ) + parser.add_argument( + "--owned-buckets", + default=None, + metavar="SPEC", + help=( + "本节点负责的桶，如 '0-15,20,30'。给出后：reconcile 只清理这些桶里的僵尸任务，" + "ps --mine 只看这些桶。不给出表示不限定归属（单机语义）。" + ), + ) + + sub = parser.add_subparsers(dest="command", required=True, metavar="COMMAND") + + # --- 定义 --- + tasks = sub.add_parser("tasks", help="列出某个任务模块里注册的任务定义。") + tasks.add_argument("--tasks", required=True, metavar="MODULE", help="形如 myapp.tasks 的模块。") + tasks.set_defaults(handler=_cmd_tasks) + + # --- 提交 --- + run = sub.add_parser("run", help="提交并**前台执行**一个任务，直到它落终态。") + run.add_argument("task", metavar="TASK", help="已注册的任务名。") + run.add_argument("--tasks", required=True, metavar="MODULE", help="形如 myapp.tasks 的模块。") + run.add_argument("--task-id", default=None, help="指定 task_id；默认随机生成。") + run.add_argument("--label", default=None, help="展示名；默认用注册名。") + run.add_argument( + "--bucket-key", + default=None, + help="分桶的业务键（租户、数据源……）。同一个键在任何进程都落到同一个桶。", + ) + run.add_argument( + "--arg", + action="append", + default=[], + metavar="K=V", + help="任务参数，可重复。值若是合法 JSON 会按 JSON 解析，否则按字符串。", + ) + run.add_argument("--max-attempts", type=int, default=1, help="总尝试次数（含首次）。") + run.add_argument( + "--timeout", type=float, default=None, metavar="SECONDS", help="单次尝试超时。" + ) + run.add_argument( + "--executor", + choices=("thread", "process"), + default="thread", + help=( + "任务在哪跑。默认 thread；process 起独立子进程，因此**能被真的杀掉**" + "（优雅窗口用尽后 terminate/kill），代价是参数与返回值必须可 pickle、" + "目标函数必须在可导入的模块里。" + ), + ) + run.add_argument( + "--graceful-stop", type=float, default=5.0, metavar="SECONDS", help="优雅停止窗口。" + ) + run.set_defaults(handler=_cmd_run) + + # --- 查询 --- + ps = sub.add_parser("ps", help="列出任务。默认只显示未落终态的。") + ps.add_argument("--all", action="store_true", help="包含已落终态的任务。") + ps.add_argument( + "--status", action="append", default=[], metavar="STATUS", help="按状态过滤，可重复。" + ) + ps.add_argument("--bucket", type=int, default=None, metavar="N", help="只看某个桶。") + ps.add_argument( + "--pipeline", + default=None, + metavar="RUN_ID", + help="只看某次管道运行的任务（RUN_ID 即 pipeline_run_id）。", + ) + ps.add_argument( + "--mine", + action="store_true", + help="只看本节点负责的桶（需要 --owned-buckets）。", + ) + ps.add_argument("--limit", type=int, default=None, metavar="N", help="最多显示多少条。") + ps.set_defaults(handler=_cmd_ps) + + status = sub.add_parser("status", help="单个任务的详情与迁移历史。") + status.add_argument("task_id", metavar="TASK_ID") + status.set_defaults(handler=_cmd_status) + + history = sub.add_parser("history", help="单个任务的迁移历史（可审计轨迹）。") + history.add_argument("task_id", metavar="TASK_ID") + history.set_defaults(handler=_cmd_history) + + stats = sub.add_parser("stats", help="各状态计数。") + stats.set_defaults(handler=_cmd_stats) + + reconcile = sub.add_parser( + "reconcile", + help="清理上次进程留下的僵尸任务（重启恢复）。启动时先跑一次。", + ) + reconcile.add_argument( + "--stale-after", + type=float, + default=None, + metavar="SECONDS", + help="从未续过租的任务的宽限期；默认取租约时长的两倍。", + ) + reconcile.add_argument("--dry-run", action="store_true", help="只报告，不改任何状态。") + reconcile.add_argument("--reason", default=None, help="写进审计轨迹的原因。") + reconcile.set_defaults(handler=_cmd_reconcile) + + buckets = sub.add_parser("buckets", help="每个桶的任务数；标出本节点负责的桶。") + buckets.set_defaults(handler=_cmd_buckets) + + # --- 控制 --- + stop = sub.add_parser("stop", help="请求停止一个任务。") + stop.add_argument("task_id", metavar="TASK_ID") + stop.add_argument("--reason", default=None, help="停止原因，会写进审计轨迹。") + stop.set_defaults(handler=_cmd_stop) + + pause = sub.add_parser("pause", help="请求暂停一个任务。") + pause.add_argument("task_id", metavar="TASK_ID") + pause.add_argument( + "--timeout", type=float, default=5.0, metavar="SECONDS", help="等待确认的时长。" + ) + pause.set_defaults(handler=_cmd_pause) + + resume = sub.add_parser("resume", help="恢复一个被暂停的任务。") + resume.add_argument("task_id", metavar="TASK_ID") + resume.set_defaults(handler=_cmd_resume) + + return parser + + +# ---------------------------------------------------------------------- +# 公共设施 +# ---------------------------------------------------------------------- + + +def _emit(args: argparse.Namespace, *, text: str, payload: Any = None) -> None: + """按 ``--json`` 决定输出文本还是 JSON。""" + if args.json: + sys.stdout.write(json.dumps(payload, ensure_ascii=False, sort_keys=True) + "\n") + else: + sys.stdout.write(text + "\n") + + +def _open_store(args: argparse.Namespace) -> TaskStore: + """打开任务存储。 + + ``--db`` 接受两种形态，按前缀区分： + + * ``postgresql://`` / ``postgres://`` —— Postgres 连接串（跨机部署用； + 需要 ``pip install "chongming-task[postgres]"``）； + * 其余 —— SQLite 文件路径。 + + 两者走同一份 :class:`TaskStore` 契约，因此上面所有子命令对它们都成立。 + """ + target = args.db or os.environ.get(_ENV_DB) + if not target: + raise ConfigurationError( + f"未指定任务存储。请用 --db PATH（SQLite 文件或 postgresql:// 连接串），" + f"或设置环境变量 {_ENV_DB}。\n" + "CLI 是另一个进程，没有共享存储就看不到任何任务。" + ) + if target.startswith(("postgresql://", "postgres://")): + try: + from ..stores.postgres import PostgresTaskStore + except ImportError as exc: + raise ConfigurationError( + f"--db 给的是 Postgres 连接串，但可选依赖未安装：{exc}" + ) from exc + return PostgresTaskStore(target) + return SqliteTaskStore(target) + + +def _bucket_policy(args: argparse.Namespace) -> BucketPolicy: + """按 ``--num-buckets`` 构造分桶策略。 + + 桶数是**部署范围的约定**：各节点必须一致，否则同一个键会落到不同的桶，而且不会有 + 任何报错。CLI 默认按 64 假设，部署用了别的值就必须显式告知。 + """ + count = getattr(args, "num_buckets", None) + return BucketPolicy() if count is None else BucketPolicy(num_buckets=count) + + +def _controller(args: argparse.Namespace, store: Any) -> Controller: + """按命令行选项构造控制器。 + + **所有需要控制器的子命令都必须走这里**：曾经 ``run`` 自己 ``Controller(...)``， + 于是 ``--owned-buckets`` / ``--num-buckets`` 被静默忽略——参数接受了却不起作用， + 比报错更难查。 + """ + return Controller( + store=store, + owner=args.owner or _default_owner(), + bucket_policy=_bucket_policy(args), + owned_buckets=args.owned, + executor=_executor_for(args), + graceful_stop=getattr(args, "graceful_stop", 5.0), + ) + + +def _executor_for(args: argparse.Namespace) -> Executor | None: + """按 ``--executor`` 选执行器；``None`` 表示"交给 Controller 的默认值"。 + + 默认值只写在一个地方（``Controller``），这里不再抄一遍——两处默认值迟早分叉。 + """ + if getattr(args, "executor", None) == "process": + return ProcessExecutor() + return None + + +def _resolve_owned_buckets(args: argparse.Namespace) -> frozenset[int] | None: + """解析并**校验** ``--owned-buckets``，结果存回 ``args.owned``。 + + 在 ``main()`` 里对所有子命令统一做一次：有些命令（``ps`` / ``stats`` / ``buckets``） + 直接查存储、不构造 ``Controller``，如果只在 ``Controller`` 里校验，越界的桶号在这些 + 命令上会被**静默接受**——筛选条件匹配不到任何东西，看起来就像"这个桶没有任务"。 + """ + spec = getattr(args, "owned_buckets", None) + if spec is None: + args.owned = None + return None + resolved = _parse_bucket_spec(spec) + limit = _bucket_policy(args).num_buckets + for bucket in sorted(resolved): + if bucket < 0 or bucket >= limit: + raise ConfigurationError( + f"桶号越界：{bucket}；桶数是 {limit}（合法范围 0..{limit - 1}）。" + f"若部署的桶数不是 {limit}，请用 --num-buckets 指明。" + ) + args.owned = resolved + return resolved + + +def _parse_bucket_spec(text: str) -> frozenset[int]: + """解析 ``'0-15,20,30'`` 形式的桶集合。 + + 支持区间是为了让"这个节点负责 0 到 15 号桶"这类配置写起来不难受——手写 16 个数字 + 既啰嗦又容易漏。 + """ + buckets: set[int] = set() + for chunk in text.split(","): + piece = chunk.strip() + if not piece: + continue + if "-" in piece: + low_text, _, high_text = piece.partition("-") + try: + low, high = int(low_text), int(high_text) + except ValueError: + raise ConfigurationError(f"桶区间写法不正确：{piece!r}（应为 0-15）") from None + if low > high: + raise ConfigurationError(f"桶区间起止颠倒：{piece!r}") + buckets.update(range(low, high + 1)) + continue + try: + buckets.add(int(piece)) + except ValueError: + raise ConfigurationError(f"桶号必须是整数：{piece!r}") from None + if not buckets: + raise ConfigurationError(f"--owned-buckets 没有解析出任何桶：{text!r}") + return frozenset(buckets) + + +def _default_owner() -> str: + return f"cli:{os.getpid()}" + + +def _parse_statuses(raw: Sequence[str]) -> frozenset[TaskStatus] | None: + if not raw: + return None + parsed: set[TaskStatus] = set() + for item in raw: + try: + parsed.add(TaskStatus(item)) + except ValueError: + known = "、".join(status.value for status in TaskStatus) + raise ConfigurationError(f"未知状态 {item!r}；可用：{known}") from None + return frozenset(parsed) + + +def _parse_args_pairs(pairs: Sequence[str]) -> dict[str, Any]: + """把 ``--arg k=v`` 解析成关键字参数。 + + 值先尝试按 JSON 解析（于是 ``n=3`` 得到整数、``flag=true`` 得到布尔），失败则按 + 字符串——这样不必为"传个数字"发明额外语法。 + """ + result: dict[str, Any] = {} + for pair in pairs: + if "=" not in pair: + raise ConfigurationError(f"--arg 需要 K=V 形式，收到 {pair!r}") + key, _, raw = pair.partition("=") + if not key: + raise ConfigurationError(f"--arg 的键不能为空：{pair!r}") + try: + result[key] = json.loads(raw) + except json.JSONDecodeError: + result[key] = raw + return result + + +def _load_tasks_module(name: str) -> Any: + """导入任务模块。 + + 这是"任务定义从哪里来"的答案：CLI 只传名字，因此必须有一个模块把可调用对象登记 + 进注册表——一个能启动任意可调用对象的命令行入口等于任意代码执行。 + + 两种写法都接受： + + * 定义 ``register(controller)`` 自行登记（早期约定，最灵活）； + * 只用 ``@task`` 装饰函数，交给 :meth:`Controller.register_module` 扫描 + ——这是后来加的写法，CLI 也必须支持，否则 ``@task`` 在命令行下用不了。 + """ + try: + module = importlib.import_module(name) + except ImportError as exc: + missing = getattr(exc, "name", None) + if missing and (name == missing or name.startswith(f"{missing}.")): + # 模块自身没找到：最常见的一种，也是最不该让人猜的一种。 + raise ConfigurationError( + f"无法导入任务模块 {name!r}：{exc}\n" + "任务模块必须在 sys.path 上：要么是**已安装的包**，要么把它所在的目录加进 " + "PYTHONPATH。\n" + "注意：`chongming-task` 是入口脚本，**不会**把当前目录放进 sys.path——" + "放在当前目录的模块请用：\n" + f" PYTHONPATH=. chongming-task ... --tasks {name}\n" + f"或 uv run python -m chongming_task.cli.main ... --tasks {name}" + ) from exc + # 模块找到了，但模块**内部**导入失败——那是它自己的依赖问题，别把矛头指向路径。 + raise ConfigurationError(f"任务模块 {name!r} 找到了，但它导入失败：{exc}") from exc + + if not callable(getattr(module, "register", None)) and not _has_decorated_tasks(module): + raise ConfigurationError( + f"任务模块 {name!r} 里既没有 register(controller) 函数，也没有用 @task 装饰过的函数。" + ) + return module + + +def _has_decorated_tasks(module: Any) -> bool: + """模块里是否存在 ``@task`` 装饰过的函数。""" + return any(get_task_spec(member) is not None for member in vars(module).values()) + + +def _register_tasks(module: Any, controller: Controller) -> None: + """把模块里的任务登记进控制器：优先 ``register()``，否则扫描 ``@task``。""" + register = getattr(module, "register", None) + if callable(register): + register(controller) + return + controller.register_module(module) + + +def _format_elapsed(snapshot: TaskSnapshot, now: datetime) -> str: + if snapshot.started_at is None: + return "-" + end = snapshot.ended_at or now + return f"{(end - snapshot.started_at).total_seconds():.1f}s" + + +def _format_detail(snapshot: TaskSnapshot, *, now: datetime | None = None) -> str: + """``status`` 的文本视图。 + + 刻意把"正交维度"分开展示：状态是一件事，重试、进度、健康、失败原因是另外几件。 + """ + moment = now if now is not None else utcnow() + failure = snapshot.failure + rows: list[tuple[str, str]] = [ + ("task_id", snapshot.task_id), + ("name", snapshot.name), + ("status", snapshot.status.value), + ("bucket", "-" if snapshot.bucket is None else str(snapshot.bucket)), + ("bucket_key", snapshot.bucket_key or "-"), + ("attempt", f"{snapshot.attempt}/{snapshot.max_attempts}"), + ("status_version", str(snapshot.status_version)), + ("executor", snapshot.executor or "-"), + ("created_at", _iso(snapshot.created_at)), + ("started_at", _iso(snapshot.started_at)), + ("ended_at", _iso(snapshot.ended_at)), + ("last_heartbeat_at", _iso(snapshot.last_heartbeat_at)), + ("lease_owner", snapshot.lease_owner or "-"), + ("lease_until", _lease_text(snapshot, moment)), + ("elapsed", _format_elapsed(snapshot, moment)), + ("progress", "-" if snapshot.progress is None else _progress_text(snapshot)), + ("failure", "-" if failure is None else f"{failure.kind.value}: {failure.message}"), + ("retryable", "-" if failure is None else str(failure.retryable)), + ("overridden", str(snapshot.overridden)), + ] + if snapshot.tags: + rows.insert(4, ("tags", ", ".join(sorted(snapshot.tags)))) + width = max(len(key) for key, _ in rows) + return "\n".join(f"{key:>{width}} : {value}" for key, value in rows) + + +def _lease_text(snapshot: TaskSnapshot, now: datetime) -> str: + """监督者租约的**可判读**形式。 + + 为什么不能只显示 ``last_heartbeat_at``：一个"看起来在跑"的任务可能早就没有监督者了 + （进程崩了、被杀了），而心跳停在半路与"任务卡住"长得一模一样。租约过期才是**没有人在管 + 它**的判据。这正是本命令最容易被误读的地方，所以直接给出结论而不是原始时间。 + """ + if snapshot.lease_until is None: + return "-（从未续租）" if not snapshot.is_terminal else "-" + remaining = (snapshot.lease_until - now).total_seconds() + stamp = _iso(snapshot.lease_until) + if snapshot.is_terminal: + # 终态任务留着最后一任监督者，纯属审计信息。 + return f"{stamp}（任务已结束）" + if remaining > 0: + return f"{stamp}（剩余 {remaining:.0f}s）" + return f"{stamp} ⚠ 已过期 {-remaining:.0f}s —— 没有监督者在管它，reconcile 会判它 lost" + + +def _progress_text(snapshot: TaskSnapshot) -> str: + progress = snapshot.progress + if progress is None: # pragma: no cover - 调用方已判空 + return "-" + if progress.total is None: + return str(progress.done) + return f"{progress.done}/{progress.total}" + + +def _iso(value: datetime | None) -> str: + return "-" if value is None else value.isoformat(timespec="seconds") + + +def _render_history(records: Sequence[Any]) -> str: + """迁移历史的表格视图。""" + if not records: + return "(没有迁移记录)" + rows: list[tuple[str, ...]] = [("VER", "FROM", "TO", "AT", "TRIGGER", "ACTOR", "REASON")] + for record in records: + rows.append( + ( + str(record.version), + record.from_status.value, + record.to_status.value, + _iso(record.at), + record.trigger, + record.actor, + record.reason or "-", + ) + ) + widths = [max(len(row[index]) for row in rows) for index in range(len(rows[0]))] + lines = [ + " ".join(cell.ljust(width) for cell, width in zip(row, widths, strict=True)).rstrip() + for row in rows + ] + lines.insert(1, " ".join("-" * width for width in widths)) + return "\n".join(lines) + + +def _snapshot_json(snapshot: TaskSnapshot) -> dict[str, Any]: + """快照的 JSON 视图。 + + 直接用存储层的行编码：输出与存储同构、全部是原语，因此脚本消费它不需要了解 + ``datetime`` / 枚举的序列化规则。 + """ + return snapshot_to_row(snapshot) + + +# ---------------------------------------------------------------------- +# 子命令 +# ---------------------------------------------------------------------- + + +def _cmd_tasks(args: argparse.Namespace) -> int: + module = _load_tasks_module(args.tasks) + probe = Controller() + _register_tasks(module, probe) + registrations = probe.registry.all() + + payload = [ + { + "name": item.name, + "description": item.description, + "tags": sorted(item.tags), + "max_attempts": item.retry.max_attempts, + } + for item in registrations + ] + lines = ["NAME TAGS MAX_ATTEMPTS DESCRIPTION"] + lines.append("-" * 60) + for item in registrations: + tags = ",".join(sorted(item.tags)) or "-" + lines.append(f"{item.name} {tags} {item.retry.max_attempts} {item.description}") + if not registrations: + lines.append("(该模块没有注册任何任务)") + _emit(args, text="\n".join(lines), payload=payload) + return EXIT_OK + + +def _cmd_run(args: argparse.Namespace) -> int: + module = _load_tasks_module(args.tasks) + store = _open_store(args) + controller = _controller(args, store) + try: + _register_tasks(module, controller) + run = controller.submit( + args.task, + task_id=args.task_id, + label=args.label, + bucket_key=args.bucket_key, + retry=RetryPolicy(max_attempts=args.max_attempts), + timeout=Timeout(per_attempt=args.timeout), + **_parse_args_pairs(args.arg), + ) + task_id = run.snapshot.task_id + if not args.json: + sys.stderr.write(f"已提交 {args.task} -> {task_id}（前台执行，Ctrl-C 可中断）\n") + outcome = controller.wait(task_id, timeout=None) + finally: + controller.close(close_store=True) + + snapshot = outcome.snapshot + _emit( + args, + text=_format_detail(snapshot), + payload=_snapshot_json(snapshot), + ) + # 写穿失败意味着**存储里的状态与上面的结论不一致**。这必须喊出来：否则运维在 + # `status` 里看到的是"还在 running"，而这里的进程说 succeeded——两边都不算错， + # 却对不上，而且没有任何线索指向原因。 + _warn_about_persistence(args, controller) + return EXIT_OK if outcome.ok else EXIT_TASK_NOT_SUCCESSFUL + + +def _warn_about_persistence(args: argparse.Namespace, controller: Controller) -> None: + """把写穿失败与永久分叉报到 stderr。 + + 这些信息**只存在于写进程的内存里**，`status` / `ps` 在另一个进程里看不到它们—— + 所以必须由 `run` 自己说出来。 + """ + if args.json: + return + failures = controller.persist_failures + if failures: + sys.stderr.write( + f"\n警告：{len(failures)} 次写穿失败——**存储里的状态落后于上面的结论**。\n" + " 用 `status` 观察到的可能是旧状态；最近几次失败：\n" + ) + for item in failures[-5:]: + sys.stderr.write(f" {item}\n") + for task_id, reason in controller.desynced_tasks.items(): + sys.stderr.write( + f"\n警告：{task_id} 的存储状态已**永久分叉**（已停止写穿以免覆盖对方）：\n" + f" {reason}\n" + " 该任务的存储状态不会再更新，请人工核对。\n" + ) + + +def _cmd_ps(args: argparse.Namespace) -> int: + store = _open_store(args) + try: + statuses = _parse_statuses(args.status) + if statuses is None and not args.all: + # 「在跑的东西」通常指未落终态——包括排队与暂停。 + statuses = frozenset(status for status in TaskStatus if status not in TERMINAL) + + wanted_buckets: frozenset[int] | None = None + if args.mine: + wanted_buckets = args.owned + if wanted_buckets is None: + raise ConfigurationError("--mine 需要同时给出 --owned-buckets") + elif args.bucket is not None: + wanted_buckets = frozenset({args.bucket}) + + snapshots = store.query( + TaskQuery( + statuses=statuses, + buckets=wanted_buckets, + pipeline_run_id=args.pipeline, + limit=args.limit, + ) + ) + finally: + store.close() + + if not snapshots: + _emit(args, text="(没有匹配的任务)", payload=[]) + return EXIT_OK + _emit( + args, + text=render_table(snapshots), + payload=[_snapshot_json(item) for item in snapshots], + ) + return EXIT_OK + + +def _cmd_status(args: argparse.Namespace) -> int: + store = _open_store(args) + try: + snapshot = store.get(args.task_id) + if snapshot is None: + raise ConfigurationError(f"没有这个任务：{args.task_id}") + records = store.history(args.task_id) + finally: + store.close() + + detail = _format_detail(snapshot) + text = f"{detail}\n\n迁移历史：\n{_render_history(records)}" + payload = { + "task": _snapshot_json(snapshot), + "history": [ + { + "version": record.version, + "from": record.from_status.value, + "to": record.to_status.value, + "at": record.at.isoformat(), + "trigger": record.trigger, + "actor": record.actor, + "reason": record.reason, + } + for record in records + ], + } + _emit(args, text=text, payload=payload) + return EXIT_OK + + +def _cmd_history(args: argparse.Namespace) -> int: + store = _open_store(args) + try: + records = store.history(args.task_id) + finally: + store.close() + _emit( + args, + text=_render_history(records), + payload=[ + { + "version": record.version, + "from": record.from_status.value, + "to": record.to_status.value, + "at": record.at.isoformat(), + "trigger": record.trigger, + "actor": record.actor, + "reason": record.reason, + } + for record in records + ], + ) + return EXIT_OK + + +def _cmd_stats(args: argparse.Namespace) -> int: + store = _open_store(args) + try: + counts = store.counts() + finally: + store.close() + + lines = [ + f"{status.value:<12} {count}" + for status, count in sorted(counts.items(), key=lambda kv: kv[0].value) + ] + total = sum(counts.values()) + unfinished = sum(count for status, count in counts.items() if status not in TERMINAL) + lines.append("-" * 20) + lines.append(f"{'total':<12} {total}") + lines.append(f"{'unfinished':<12} {unfinished}") + _emit( + args, + text="\n".join(lines), + payload={ + "by_status": {status.value: count for status, count in counts.items()}, + "total": total, + "unfinished": unfinished, + }, + ) + return EXIT_OK + + +def _cmd_reconcile(args: argparse.Namespace) -> int: + store = _open_store(args) + try: + controller = _controller(args, store) + report = controller.reconcile( + stale_after=args.stale_after, + reason=args.reason, + dry_run=args.dry_run, + ) + finally: + store.close() + + payload = { + "dry_run": report.dry_run, + "entries": [ + { + "task_id": item.task_id, + "previous_status": item.previous_status.value, + "outcome": item.outcome.value, + "detail": item.detail, + } + for item in report.entries + ], + } + + if not report: + _emit(args, text="没有需要处理的任务。", payload=payload) + return EXIT_OK + + rows: list[tuple[str, ...]] = [("OUTCOME", "TASK_ID", "WAS", "DETAIL")] + for item in report.entries: + rows.append((item.outcome.value, item.task_id, item.previous_status.value, item.detail)) + widths = [max(len(row[index]) for row in rows) for index in range(len(rows[0]))] + lines = [ + " ".join(cell.ljust(width) for cell, width in zip(row, widths, strict=True)).rstrip() + for row in rows + ] + lines.insert(1, " ".join("-" * width for width in widths)) + + lines.append("") + if report.dry_run: + lines.append( + f"预演：有 {len(report.lost)} 个任务**会被**标记为 lost，" + f"{len(report.adoptable)} 个等待重新提交。未写入任何状态。" + ) + else: + lines.append( + f"已把 {len(report.lost)} 个僵尸任务标记为 lost，" + f"{len(report.adoptable)} 个任务无人接手（状态未改动，需要重新提交）。" + ) + if report.skipped: + lines.append(f"{len(report.skipped)} 个被跳过（有人正在写）。") + lines.append("注意：本命令不会、也无法重新执行任务——存储里记的是状态，不是参数。") + + _emit(args, text="\n".join(lines), payload=payload) + return EXIT_OK + + +def _cmd_buckets(args: argparse.Namespace) -> int: + store = _open_store(args) + try: + snapshots = store.query() + finally: + store.close() + + counts: dict[int | None, int] = {} + for snapshot in snapshots: + counts[snapshot.bucket] = counts.get(snapshot.bucket, 0) + 1 + + num_buckets = _bucket_policy(args).num_buckets + mine = args.owned + lines = [f"{'':<1} {'BUCKET':>6} {'TASKS':>5}"] + lines.append("-" * 16) + for bucket in range(num_buckets): + # 星号标出本节点负责的桶——"我这一份"是自治架构里最常看的视图。 + mark = "*" if mine is not None and bucket in mine else " " + lines.append(f"{mark} {bucket:>6} {counts.get(bucket, 0):>5}") + if None in counts: + lines.append(f"{' '} {'(none)':>6} {counts[None]:>5}") + _emit( + args, + text="\n".join(lines), + payload={ + "num_buckets": num_buckets, + "by_bucket": {str(bucket): count for bucket, count in counts.items()}, + "owned_buckets": None if mine is None else sorted(mine), + }, + ) + return EXIT_OK + + +def _route_control( + args: argparse.Namespace, kind: CommandKind, *, reason: str | None = None +) -> bool: + """把一个控制动作交给 Controller 路由（本地直下，远程写成指令）。""" + store = _open_store(args) + try: + controller = _controller(args, store) + if kind is CommandKind.STOP: + return controller.stop(args.task_id, reason=reason) + if kind is CommandKind.PAUSE: + confirmed = controller.pause(args.task_id, timeout=getattr(args, "timeout", 5.0)) + if not confirmed: + # 两种可能：任务不属于本进程（无法确认），或任务没到达检查点。 + sys.stderr.write( + "暂停**未确认**。若任务由别的进程持有，这是正常的——" + "跨进程没有回执通道，请用 status 观察结果。\n" + ) + return confirmed or controller.get(args.task_id) is not None + return controller.resume(args.task_id) + finally: + store.close() + + +def _cmd_stop(args: argparse.Namespace) -> int: + handled = _route_control(args, CommandKind.STOP, reason=args.reason) + if not handled: + raise ConfigurationError(f"没有找到任务 {args.task_id}；无法停止。") + _emit( + args, + text=f"已请求停止 {args.task_id}" + (f"（原因：{args.reason}）" if args.reason else ""), + payload={"task_id": args.task_id, "requested": "stop", "reason": args.reason}, + ) + return EXIT_OK + + +def _cmd_pause(args: argparse.Namespace) -> int: + handled = _route_control(args, CommandKind.PAUSE) + if not handled: + raise ConfigurationError(f"没有找到任务 {args.task_id}；无法暂停。") + _emit( + args, + text=f"已请求暂停 {args.task_id}（用 status 观察是否真的挂起）", + payload={"task_id": args.task_id, "requested": "pause"}, + ) + return EXIT_OK + + +def _cmd_resume(args: argparse.Namespace) -> int: + handled = _route_control(args, CommandKind.RESUME) + if not handled: + raise ConfigurationError(f"没有找到任务 {args.task_id}；无法恢复。") + _emit( + args, + text=f"已请求恢复 {args.task_id}", + payload={"task_id": args.task_id, "requested": "resume"}, + ) + return EXIT_OK + + +# ---------------------------------------------------------------------- +# 入口 +# ---------------------------------------------------------------------- + + +def main(argv: Sequence[str] | None = None) -> int: + """命令行入口。返回退出码（见模块文档）。""" + parser = build_parser() + args = parser.parse_args(list(argv) if argv is not None else None) + try: + # 无论哪个子命令，都先解析并校验一次桶归属：有些命令直接查存储、不构造 + # Controller，把校验放在 Controller 里会让越界桶号在这些命令上被静默接受。 + _resolve_owned_buckets(args) + return int(args.handler(args)) + except (ConfigurationError, TaskError) as exc: + sys.stderr.write(f"错误：{exc}\n") + return EXIT_ERROR + except KeyboardInterrupt: + sys.stderr.write("已中断\n") + return EXIT_ERROR + except BrokenPipeError: # pragma: no cover - 管道被下游关闭 + return EXIT_OK + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/src/chongming_task/clock.py b/src/chongming_task/clock.py new file mode 100644 index 0000000..8c66e91 --- /dev/null +++ b/src/chongming_task/clock.py @@ -0,0 +1,26 @@ +"""时钟抽象。 + +时间必须可注入：重试退避、超时判定、心跳陈旧度这些逻辑一旦依赖真实时间， +测试就只能靠 ``sleep``，既慢又不稳定。因此运行时的每一处取时都走 :data:`Clock`。 + +生产用 :func:`utcnow`；测试用 ``chongming_task.testing.clock.FakeClock``。 +""" + +from __future__ import annotations + +from collections.abc import Callable +from datetime import UTC, datetime + +__all__ = ["Clock", "utcnow"] + + +def utcnow() -> datetime: + """默认取时，返回带 UTC 时区的当前时间。""" + return datetime.now(UTC) + + +#: 时钟。返回带时区的当前时间。 +#: +#: 注意：这里取的是**墙钟**，可能因 NTP 校时发生跳变。超时判定基于两个时刻之差， +#: 若对跳变敏感，可注入基于 ``time.monotonic`` 的自定义实现（代价是失去可读时间戳）。 +Clock = Callable[[], datetime] diff --git a/src/chongming_task/controller.py b/src/chongming_task/controller.py new file mode 100644 index 0000000..5c118c7 --- /dev/null +++ b/src/chongming_task/controller.py @@ -0,0 +1,1682 @@ +"""任务管理门面：注册、提交、查询、控制。 + +这是管理后台（CLI，将来可能的看板）**唯一需要对接的层**。它统一持有三样东西： + +* **任务定义注册表**——启动时把可调用对象登记成名字，之后一切提交都只传名字； +* **运行中的任务**——``task_id -> TaskRun``，控制指令据此路由； +* **并发额度**——超出上限的提交会排队，状态为 ``queued`` 而不是立刻起线程。 + +安全底线：**只允许启动已注册的任务，绝不接受代码。** 一个能启动任意可调用对象的 +入口等于任意代码执行——将来无论接 CLI 还是 HTTP，这条都不能破。 + +跨进程说明 +---------- +本类管理的是**当前进程内**的任务。另一个进程里的 ``chongming-task ps`` 看不到它们， +因为内存不共享。跨进程查询与持久化要等共享存储（``sqlite3`` 属于标准库，不引入 +第三方依赖）落地——那正是 CLI 的媒介。 +""" + +from __future__ import annotations + +import os +import socket +import threading +import time +import unicodedata +import uuid +from collections import deque +from collections.abc import Callable, Iterable, Mapping, Sequence +from dataclasses import dataclass, field +from datetime import UTC, datetime, timedelta +from enum import StrEnum +from pathlib import Path +from typing import Any + +from .clock import Clock, utcnow +from .core.state_machine import StateMachine, Transition +from .decorators import get_task_spec +from .errors import ConcurrencyConflict, ConfigurationError, InvalidTransition, StoreError +from .models import Failure, FailureKind, Progress, TaskSnapshot +from .pipeline import Pipeline, PipelineRun, start_pipeline +from .policies import BucketPolicy, ConcurrencyPolicy, RetryPolicy, Timeout +from .run import TaskRun, TaskRunOutcome +from .runtime.executors.base import Executor +from .runtime.executors.thread import ThreadExecutor +from .states import NON_TERMINAL, RESOURCE_HOLDING, TERMINAL, TaskStatus +from .stores.protocol import Command, CommandKind, TaskQuery, TaskStore + +__all__ = [ + "Controller", + "ReconcileEntry", + "ReconcileOutcome", + "ReconcileReport", + "Registration", + "StatusCounts", + "TaskRegistry", + "render_table", +] + +#: 表格列之间的分隔。 +_COLUMN_SEPARATOR = " " + + +def _validated_tags(tags: Iterable[str]) -> frozenset[str]: + """把标签收成集合，并拒绝非字符串元素。 + + 标签会被写进存储（JSON 数组），所以非字符串元素既不能正确往返、又只会在**写穿** + 路径上才炸——而那条路径的失败不会让任务失败，问题会以"看板上的标签不对"的形式 + 出现。因此按真实类型契约（``frozenset[str]``）在入口拒绝。 + + 注册与提交两条路径都要过这里：``submit`` 会把注册时的标签与本次传入的合并， + 只校验其中一条会漏。 + """ + resolved = frozenset(tags) + bad = sorted(repr(tag) for tag in resolved if not isinstance(tag, str)) + if bad: + raise ConfigurationError(f"标签必须是字符串，收到：{', '.join(bad)}") + return resolved + + +#: 每个任务最多缓冲多少条未落库的迁移。 +#: +#: 必须有界：存储长期不可用时，内存不能跟着无限增长。超出后丢弃最旧的——审计轨迹会 +#: 出现缺口，所以丢弃这件事本身也会被记录（见 Controller._flush_pending）。 +_PENDING_TRANSITIONS = 100 + + +def _first_doc_line(target: Callable[..., Any]) -> str: + """取函数 docstring 的第一行，作为注册项的默认描述。""" + doc = (target.__doc__ or "").strip() + return doc.splitlines()[0] if doc else "" + + +@dataclass(frozen=True, slots=True) +class Registration: + """一个已注册的任务定义。""" + + name: str + target: Callable[..., Any] + retry: RetryPolicy + timeout: Timeout + tags: frozenset[str] = frozenset() + description: str = "" + + +class TaskRegistry: + """任务定义的白名单。 + + 只按**名字**提交任务，因此运行期的任何控制面都不需要、也无法传递代码。 + """ + + def __init__(self) -> None: + self._items: dict[str, Registration] = {} + self._lock = threading.RLock() + + def register( + self, + name: str, + target: Callable[..., Any], + *, + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, + tags: Iterable[str] = (), + description: str = "", + ) -> Registration: + """登记一个任务定义。 + + 重名会抛 :class:`ConfigurationError` 而不是静默覆盖——静默覆盖会让"我明明 + 改了实现，跑的还是旧的"这类问题变得极难发现。 + """ + if not name or not name.strip(): + raise ConfigurationError("任务名不能为空") + if not callable(target): + raise ConfigurationError(f"任务 {name!r} 的 target 必须是可调用对象") + resolved_tags = _validated_tags(tags) + with self._lock: + if name in self._items: + raise ConfigurationError( + f"任务名 {name!r} 已注册；如需替换请先注销（当前不支持静默覆盖）" + ) + registration = Registration( + name=name, + target=target, + retry=retry if retry is not None else RetryPolicy(), + timeout=timeout if timeout is not None else Timeout(), + tags=resolved_tags, + description=description or _first_doc_line(target), + ) + self._items[name] = registration + return registration + + def register_module(self, module: Any) -> tuple[Registration, ...]: + """把一个模块里所有被 ``@task`` 装饰过的函数登记进来。 + + 显式调用，而不是"import 时自动注册"：后者会让导入产生副作用，而 import 可能 + 发生在任何地方（包括测试收集阶段），那种隐式行为极难排查。 + """ + registered: list[Registration] = [] + seen: set[str] = set() + for member in vars(module).values(): + if not callable(member): + continue + spec = get_task_spec(member) + if spec is None: + continue + name = spec.name if spec.name is not None else getattr(member, "__name__", None) + if not name or name in seen: + continue + seen.add(name) + registered.append( + self.register( + name, + member, + retry=spec.retry, + timeout=spec.timeout, + tags=spec.tags, + description=spec.description, + ) + ) + if not registered: + module_name = getattr(module, "__name__", repr(module)) + raise ConfigurationError(f"模块 {module_name!r} 里没有用 @task 装饰过的函数") + return tuple(registered) + + def register_decorated(self, target: Callable[..., Any]) -> Registration: + """登记一个被 ``@task`` 装饰过的函数。""" + spec = get_task_spec(target) + if spec is None: + raise ConfigurationError(f"{getattr(target, '__name__', target)!r} 没有用 @task 装饰过") + name = spec.name if spec.name is not None else getattr(target, "__name__", None) + if not name: + raise ConfigurationError("无法确定任务名：请用 @task(name=...) 显式给出") + return self.register( + name, + target, + retry=spec.retry, + timeout=spec.timeout, + tags=spec.tags, + description=spec.description, + ) + + def get(self, name: str) -> Registration: + """取一个已注册的定义；不存在则抛 :class:`ConfigurationError`。""" + with self._lock: + try: + return self._items[name] + except KeyError: + known = ", ".join(sorted(self._items)) or "(空)" + raise ConfigurationError(f"未注册的任务 {name!r}；已注册：{known}") from None + + def names(self) -> tuple[str, ...]: + """已注册的任务名。""" + with self._lock: + return tuple(sorted(self._items)) + + def all(self) -> tuple[Registration, ...]: + """全部已注册定义。""" + with self._lock: + return tuple(self._items[name] for name in sorted(self._items)) + + def __contains__(self, name: object) -> bool: + with self._lock: + return name in self._items + + def __len__(self) -> int: + with self._lock: + return len(self._items) + + +@dataclass(frozen=True, slots=True) +class StatusCounts: + """各状态的计数，供看板首页使用。""" + + by_status: Mapping[TaskStatus, int] + + @property + def total(self) -> int: + """任务总数。""" + return sum(self.by_status.values()) + + def of(self, status: TaskStatus) -> int: + """某个状态的数量。""" + return self.by_status.get(status, 0) + + @property + def unfinished(self) -> int: + """尚未落终态的数量。 + + 包含排队（``queued``）、暂停（``paused``）与重试等待（``retry_wait``）—— + 它们都不等于"正在跑"。要区分"真的在占用执行资源"，请用 + :attr:`Controller.active_count`。 + """ + return sum(count for status, count in self.by_status.items() if status not in TERMINAL) + + @property + def finished(self) -> int: + """已落终态的数量。""" + return sum(count for status, count in self.by_status.items() if status in TERMINAL) + + +class ReconcileOutcome(StrEnum): + """重启恢复对单个任务的处置结果。""" + + MARKED_LOST = "marked-lost" + """占着执行资源却没有监督者——载体已随进程消失，判 ``LOST``。""" + + ADOPTABLE = "adoptable" + """没有执行载体（排队/暂停/退避中），只是没人接手。**不改状态**，需要重新提交。""" + + SKIPPED = "skipped" + """有人正在写这个任务，或状态机不允许该转移——不动它。""" + + +@dataclass(frozen=True, slots=True) +class ReconcileEntry: + """一个任务的重启恢复结论。""" + + task_id: str + previous_status: TaskStatus + outcome: ReconcileOutcome + detail: str + + +@dataclass(frozen=True, slots=True) +class ReconcileReport: + """一次 :meth:`Controller.reconcile` 的完整结论。""" + + entries: tuple[ReconcileEntry, ...] = () + dry_run: bool = False + + @property + def lost(self) -> tuple[ReconcileEntry, ...]: + """被判为丢失的任务。""" + return tuple(item for item in self.entries if item.outcome is ReconcileOutcome.MARKED_LOST) + + @property + def adoptable(self) -> tuple[ReconcileEntry, ...]: + """无人接手、但保住状态的任务（排队/暂停/退避）。""" + return tuple(item for item in self.entries if item.outcome is ReconcileOutcome.ADOPTABLE) + + @property + def skipped(self) -> tuple[ReconcileEntry, ...]: + """被跳过的任务。""" + return tuple(item for item in self.entries if item.outcome is ReconcileOutcome.SKIPPED) + + def __bool__(self) -> bool: + """是否有任何需要关注的结论。""" + return bool(self.entries) + + +@dataclass(frozen=True, slots=True) +class _ActiveSlot: + """一个正在占用并发额度的任务：它的桶与标签。 + + 记桶是为了算每桶额度；记标签是为了算具名资源额度（``ConcurrencyPolicy.limits``）。 + 两者都必须记在**占用当时**——任务的标签与桶不会变，但重新从 ``_runs`` 里查一遍 + 会让"额度记账"依赖另一个数据结构的状态，得不偿失。 + """ + + bucket: int | None + tags: frozenset[str] + + +@dataclass(slots=True) +class _PersistState: + """一个任务的写穿状态。 + + ``last_persisted_version`` 是**我们上次确实写进存储的版本**——注意不是"上次算出来的 + 版本"。二者一旦分叉（写穿失败），用后者会导致此后每一次写入都因版本对不上而失败， + 也就是一次瞬时故障把该任务的存储状态**永久冻结**。 + """ + + lock: threading.RLock = field(default_factory=threading.RLock) + last_persisted_version: int = 0 + pending: deque[tuple[TaskSnapshot, Transition]] = field(default_factory=deque) + desynced: str | None = None + dropped: int = 0 + + +@dataclass(slots=True) +class _Submission: + """Controller 内部记录：任务运行 + 它的提交参数。""" + + run: TaskRun + registration: Registration + args: tuple[Any, ...] = () + kwargs: Mapping[str, Any] = field(default_factory=dict) + + +class Controller: + """任务管理门面。""" + + def __init__( + self, + *, + registry: TaskRegistry | None = None, + executor: Executor | None = None, + concurrency: ConcurrencyPolicy | None = None, + max_parallel: int | None = None, + max_parallel_per_bucket: int | None = None, + bucket_policy: BucketPolicy | None = None, + clock: Clock | None = None, + graceful_stop: float = 5.0, + poll: float = 0.05, + store: TaskStore | None = None, + owner: str | None = None, + command_poll_interval: float = 0.5, + command_lease: float = 30.0, + lease_ttl: float = 30.0, + owned_buckets: Iterable[int] | None = None, + ) -> None: + """``concurrency`` 是并发与资源约束的**唯一配置口** + （见 :class:`~chongming_task.policies.ConcurrencyPolicy`）：全局 + ``max_parallel``、按标签的具名信号量、默认优先级。 + + ``max_parallel`` 是它的简写，给只调一个数字的场合用；两者同时给出会报错—— + 两份配置谁说了算必须明确，不能靠"后者覆盖前者"这种猜法。 + + ``max_parallel_per_bucket`` 额外给每个桶各自限流：一个桶把额度占满不会饿死 + 其他桶。这是分桶在**本地**就有价值的地方，不必等到分布式。 + + ``store`` 一旦给出，本实例同时成为**可跨进程的一环**：每次状态迁移写穿到 + 存储；查询走存储（因此能看到别的进程的任务）；停止/暂停/恢复对不属于本进程 + 的任务会写成一条指令，由持有者轮询取走。 + + ``owned_buckets`` 是**本节点负责的桶**，也是"自治节点 + 统一平面"架构里唯一的 + 划分单位：每个节点有自己的任务定义与触发方式，但共用同一套存储与控制层， + 靠桶区分"哪一份是我的"。 + + * ``None``（默认）= **不限定**，适合单机，或负责全部桶的节点； + * 给定集合 = 本节点只负责这些桶。此时： + + - :meth:`submit` 只接受落在这些桶里的任务（配错桶会立刻报错，而不是产出一个 + **永远没人认领**的任务）； + - :meth:`reconcile` 只清理这些桶里的僵尸任务，不去动别的节点的任务。 + + 它的作用不是"抢占"，而是**划界**：节点之间不互相执行对方的任务。 + """ + if concurrency is not None and max_parallel is not None: + raise ConfigurationError( + "concurrency 与 max_parallel 只能给一个：前者是完整的并发策略，" + "后者只是它的简写。两个都给会让'到底哪个生效'变成一个需要猜的问题。" + ) + if max_parallel is not None and max_parallel < 1: + raise ConfigurationError(f"max_parallel 至少为 1，收到 {max_parallel}") + if max_parallel_per_bucket is not None and max_parallel_per_bucket < 1: + raise ConfigurationError( + f"max_parallel_per_bucket 至少为 1，收到 {max_parallel_per_bucket}" + ) + self._registry = registry if registry is not None else TaskRegistry() + self._executor: Executor = executor if executor is not None else ThreadExecutor() + # 简写在这里被翻译成策略对象，之后调度器只认策略——不再有第二条读配置的路径。 + if concurrency is not None: + self._concurrency = concurrency + else: + self._concurrency = ConcurrencyPolicy(max_parallel=max_parallel) + self._max_parallel_per_bucket = max_parallel_per_bucket + self._bucket_policy = bucket_policy if bucket_policy is not None else BucketPolicy() + self._owned_buckets = self._validated_owned_buckets(owned_buckets) + self._clock: Clock = clock if clock is not None else utcnow + self._graceful_stop = graceful_stop + self._poll = poll + + self._runs: dict[str, _Submission] = {} + self._pending: deque[TaskRun] = deque() + self._active: dict[str, _ActiveSlot] = {} + """占用并发额度的任务：``task_id -> 桶与标签``。 + + 用映射而不是计数器：被 :meth:`close` 直接启动的排队任务从未计入额度， + 若用计数器，它们结束时会把别人的名额一并扣掉。记下桶号是为了算每桶额度， + 记下标签是为了算具名资源额度。 + """ + self._closed = False + self._lock = threading.RLock() + self._wake = threading.Event() + self._dispatcher: threading.Thread | None = None + + # ---- 跨进程（可选） ---- + self._store = store + self._owner = owner if owner is not None else f"{socket.gethostname()}:{os.getpid()}" + self._command_poll_interval = command_poll_interval + self._command_lease = command_lease + self._lease_ttl = lease_ttl + self._persist_failures: list[str] = [] + self._persist_states: dict[str, _PersistState] = {} + self._listener_failures: list[str] = [] + self._listeners: list[Callable[[TaskRunOutcome], None]] = [] + self._pipelines: dict[str, PipelineRun] = {} + self._worker: threading.Thread | None = None + + if self._store is not None and self._owned_buckets is not None: + # 声明了"我负责哪些桶"= 这是一个真 worker，从启动就轮询指令。 + # + # 否则会有一个延迟生效的坑：别人 `stop` 一个属于本节点、当前还排在队列里的 + # 任务时，指令已经写进存储了，但本节点要等到**第一次提交**才会启动轮询线程 + # ——对一个定时任务节点来说，那可能是几分钟甚至几小时之后。 + # + # 没有声明归属的实例（纯查询客户端）仍然不轮询，避免抢走别人的指令。 + self._ensure_worker() + + @property + def owner(self) -> str: + """本实例的身份，写进它下发的指令里，便于事后追溯是谁按的停止。""" + return self._owner + + @property + def store(self) -> TaskStore | None: + """绑定的存储；未接线时为 ``None``。""" + return self._store + + @property + def persist_failures(self) -> tuple[str, ...]: + """最近的持久化失败摘要（最多 20 条）。 + + 存储故障**不该让任务失败**——任务本身没问题，坏的只是可观测性。但也绝不能 + 静默：写穿失败会记在这里，让"看板上的状态是旧的"这件事有迹可循。 + """ + with self._lock: + return tuple(self._persist_failures) + + @property + def listener_failures(self) -> tuple[str, ...]: + """最近的监听器异常摘要（最多 20 条）。 + + 监听器是外部代码（管道调度器就是其中之一）。它抛异常不该影响任务本身的结论， + 但也绝不能静默——否则管道会停在半路而没有任何线索。 + """ + with self._lock: + return tuple(self._listener_failures) + + # ------------------------------------------------------------------ + # 注册 + # ------------------------------------------------------------------ + + @property + def registry(self) -> TaskRegistry: + """任务定义注册表。""" + return self._registry + + def register( + self, + name: str, + target: Callable[..., Any], + *, + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, + tags: Iterable[str] = (), + description: str = "", + ) -> Registration: + """登记任务定义。""" + return self._registry.register( + name, + target, + retry=retry, + timeout=timeout, + tags=tags, + description=description, + ) + + def register_all(self, *targets: Callable[..., Any]) -> tuple[Registration, ...]: + """登记若干被 ``@task`` 装饰过的函数，元数据从装饰器上取。""" + return tuple(self._registry.register_decorated(item) for item in targets) + + def register_module(self, module: Any) -> tuple[Registration, ...]: + """把一个模块里所有被 ``@task`` 装饰过的函数登记进来。 + + ``@task`` 只把规格挂在函数对象上，**不产生全局注册表**——谁注册、注册到哪个 + 控制器由你决定，导入模块也不会有副作用。 + """ + return self._registry.register_module(module) + + # ------------------------------------------------------------------ + # 提交 + # ------------------------------------------------------------------ + + def submit( + self, + task: str, + /, + *args: Any, + task_id: str | None = None, + label: str | None = None, + tags: Iterable[str] = (), + bucket_key: str | None = None, + priority: int | None = None, + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, + artifact_dir: Path | None = None, + pipeline_run_id: str | None = None, + deferred: bool = False, + **kwargs: Any, + ) -> TaskRun: + """提交一个**已注册**的任务。 + + 返回 :class:`~chongming_task.run.TaskRun`，它自身就带 ``stop`` / ``pause`` + / ``resume`` / ``wait`` / ``snapshot``，因此句柄不需要另造一个类型。 + + 超出并发额度（``ConcurrencyPolicy.max_parallel``）、桶额度 + （``max_parallel_per_bucket``）或某个具名资源额度（``limits``）时，任务进入 + 队列，状态为 ``queued``——它是"已被受理、等待执行槽"，与 ``pending`` + （尚未受理）不同。**排队不是失败**：额度一有富余，调度器就会把它取走。 + + ``tags`` 除了用于筛选查询，还决定具名资源：带 ``gpu`` 标签的任务会占用 + ``limits["gpu"]`` 里的一个名额。 + + ``priority`` 是排队顺序：数值越大越先被执行，同优先级按提交顺序。不给则用 + ``ConcurrencyPolicy.priority`` 的默认值。 + + ``bucket_key`` 是分桶的业务键（租户 id、数据源、日期分区……）。同一个键在 + 任何进程、任何机器上都落到同一个桶，因此它既是本地并发隔离的单位，也是将来 + 分布式分配的单位——一台机器拥有一组桶。 + + ``deferred=True`` 表示"先受理但**不要**排队执行"，任务停在 ``blocked``。 + 管道调度器用它创建下游节点：等上游满足后再 :meth:`_release`。 + """ + registration = self._registry.get(task) + resolved_id = task_id if task_id is not None else uuid.uuid4().hex[:12] + if priority is not None and not isinstance(priority, int): + # 在这里拦下：非整数优先级会在调度器比较时抛 TypeError，而那个异常发生 + # 在**派发线程**里——线程会直接死掉，之后所有排队任务都不再启动，且没有 + # 任何报错指向真正的原因。 + raise ConfigurationError(f"priority 必须是整数，收到 {priority!r}") + merged_tags = _validated_tags((*registration.tags, *tags)) + bucket = self._bucket_for(bucket_key) + self._check_bucket_ownership(bucket, task_id=resolved_id, bucket_key=bucket_key) + + with self._lock: + if self._closed: + raise ConfigurationError("Controller 已关闭，不能再提交任务") + if resolved_id in self._runs: + raise ConfigurationError(f"task_id 已存在：{resolved_id}") + + run = TaskRun( + registration.target, + args, + kwargs=kwargs, + task_id=resolved_id, + name=label if label is not None else registration.name, + tags=merged_tags, + bucket=bucket, + bucket_key=bucket_key, + pipeline_run_id=pipeline_run_id, + priority=self._concurrency.priority if priority is None else priority, + executor=self._executor, + retry=retry if retry is not None else registration.retry, + timeout=timeout if timeout is not None else registration.timeout, + artifact_dir=artifact_dir, + clock=self._clock, + graceful_stop=self._graceful_stop, + poll=self._poll, + on_finished=self._on_run_finished, + on_change=self._make_observer(resolved_id), + ) + if self._store is not None: + # 必须先建行：之后每一次迁移都是按版本更新的 UPDATE。 + self._store.create(run.snapshot) + # 基准：存储里现在是这个版本，写穿的 expect_version 以它为准。 + self._seed_persist_state(resolved_id, run.snapshot.status_version) + if deferred: + # 等待上游：停在 blocked，**不**进待执行队列，执行器取不到它。 + run.block(reason="等待上游任务") + else: + # 先落 queued：它表示"已被受理"，与 pending（尚未受理）不同。 + run.enqueue() + self._runs[resolved_id] = _Submission( + run=run, + registration=registration, + args=args, + kwargs=dict(kwargs), + ) + if deferred: + start_now = False + else: + start_now = self._can_start_locked(run.snapshot) + if start_now: + self._active[resolved_id] = _ActiveSlot(bucket, run.snapshot.tags) + else: + self._pending.append(run) + + if start_now: + run.start() + elif not deferred: + self._ensure_dispatcher() + # 只有真正提交过任务的实例才去轮询指令：否则一个纯查询客户端会把别人的 + # 停止请求抢走（见 TaskStore.release_command 的说明）。 + self._ensure_worker() + return run + + def _bucket_for(self, bucket_key: str | None) -> int | None: + """把业务键映射成桶号；未给键则返回 ``None``（该任务不参与分桶）。""" + if bucket_key is None: + return None + return self._bucket_policy.bucket_for(bucket_key) + + def _release(self, run: TaskRun) -> None: + """把一个等待上游的任务放进待执行队列（``blocked`` → ``queued``）。 + + 由管道调度器在上游满足后调用。若任务已经落终态（例如刚被连坐取消），什么也 + 不做——把它塞进队列只会让调度器取到一个不该启动的任务。 + """ + with self._lock: + if self._closed or run.snapshot.is_terminal: + return + run.unblock() + self._pending.append(run) + self._ensure_dispatcher() + self._wake.set() + + # ------------------------------------------------------------------ + # 管道（DAG） + # ------------------------------------------------------------------ + + def submit_pipeline( + self, + pipeline: Pipeline, + *, + pipeline_run_id: str | None = None, + ) -> PipelineRun: + """提交一条管道，返回可等待的运行句柄。 + + 定义会在**创建任何任务之前**全部校验完（任务名已注册、无环、依赖存在、结算 + 方式可用）：否则一个写错的 DAG 会在存储里留下一堆半成品任务，而它们看起来 + 与正常提交的没有区别。 + """ + run = start_pipeline(self, pipeline, pipeline_run_id=pipeline_run_id) + with self._lock: + self._pipelines[run.run_id] = run + return run + + def pipelines(self) -> tuple[PipelineRun, ...]: + """本进程提交过的管道运行。""" + with self._lock: + return tuple(self._pipelines.values()) + + def get_pipeline(self, run_id: str) -> PipelineRun | None: + """按 id 取管道运行；不存在返回 ``None``。""" + with self._lock: + return self._pipelines.get(run_id) + + def _watch_pipeline(self, run: PipelineRun) -> None: + """让管道运行订阅任务落终态事件，然后立刻评估一次。 + + 立刻评估是必须的：根节点在上面的循环里已经被提交，它们可能在订阅之前就跑完 + 了，那样监听器永远收不到通知，管道会一直停在 ``running``。 + """ + run_id = run.run_id + + def listener(outcome: TaskRunOutcome) -> None: + if outcome.snapshot.pipeline_run_id != run_id: + return + run._on_task_finished(outcome.snapshot.task_id) + + self.add_listener(listener) + run._evaluate() + + # ------------------------------------------------------------------ + # 写穿到存储 + # ------------------------------------------------------------------ + + def _make_observer(self, task_id: str) -> Callable[[TaskSnapshot, Transition], None]: + """构造挂在状态机上的观察者。 + + 挂在状态机的 ``_apply`` 上而不是散落在各业务方法里，是为了保证**任何**一次 + 迁移都不会漏写——漏一次就意味着存储与内存悄悄分叉，而且不会有任何报错。 + """ + + def observe(snapshot: TaskSnapshot, transition: Transition) -> None: + self._persist(snapshot, transition) + + del task_id + return observe + + def _persist_state(self, task_id: str) -> _PersistState: + """取（或创建）一个任务的写穿状态。""" + with self._lock: + state = self._persist_states.get(task_id) + if state is None: + state = _PersistState() + self._persist_states[task_id] = state + return state + + def _seed_persist_state(self, task_id: str, version: int) -> None: + """记录"存储里当前是哪个版本"，作为后续写穿的 ``expect_version`` 基准。""" + self._persist_state(task_id).last_persisted_version = version + + def _persist(self, snapshot: TaskSnapshot, transition: Transition) -> None: + """把一次状态迁移写穿到存储。 + + ## 为什么 ``expect_version`` 用"上次**确实写进去**的版本" + + 曾经的写法是 ``transition.version - 1``，它**假设此前每一次迁移都成功落库**。 + 一旦某次 ``save`` 失败（返回值不可序列化、网络抖动、连接被断……），存储就停在 + k-1 而内存已到 k；此后的每一次写入都会因为 ``expect_version`` 与存储对不上而 + 冲突——**一次瞬时故障让该任务的存储状态永久冻结**，最后被 ``reconcile()`` 判成 + ``LOST``，把一个真正跑完的任务报成"结果未知"。 + + 现在记的是"我上次确实写进去的版本"：漏写一笔不会污染后续写入，下一次迁移会把 + 存储一次追上（版本跳号是允许的，``status_version`` 只需要单调）。 + + ## 存储故障不让任务失败 + + 任务本身没问题，坏的只是可观测性。所以失败会进缓冲区、等下次再试，并记进 + :attr:`persist_failures`。但绝不静默——持续分叉会出现在 :attr:`desynced_tasks`。 + """ + store = self._store + if store is None: + return + state = self._persist_state(snapshot.task_id) + with state.lock: + if state.desynced is not None: + # 已经放弃这个任务的写入（存在外部写入者），不再刷屏。 + return + state.pending.append((snapshot, transition)) + if len(state.pending) > _PENDING_TRANSITIONS: + excess = len(state.pending) - _PENDING_TRANSITIONS + for _ in range(excess): + state.pending.popleft() + state.dropped += excess + self._flush_pending(snapshot.task_id, state) + + def _flush_pending(self, task_id: str, state: _PersistState) -> None: + """尝试把缓冲的写穿补上。须持有 ``state.lock``。 + + 顺序是：**只写最新快照**（存储行只需要反映当前状态），再按序补写全部迁移 + （审计轨迹需要完整）。``append`` 按 ``(task_id, version)`` 幂等，所以中途失败 + 可以安全重来。 + """ + store = self._store + if store is None or not state.pending: + return + + latest = state.pending[-1][0] + try: + store.save(latest, expect_version=state.last_persisted_version) + except ConcurrencyConflict as exc: + # 用"上次确实写进去的版本"仍然冲突 → 存储里的版本是**别人**推进的。 + # 覆盖它会抹掉对方的状态，所以放手并明确报告，而不是反复重试。 + state.desynced = ( + f"检测到外部写入者（期望版本 {exc.expected_version}，" + f"实际 {exc.actual_version}），已停止写穿以免覆盖对方" + ) + state.pending.clear() + self._remember_failure(f"{task_id}: {state.desynced}") + return + except Exception as exc: + # 瞬时故障：保留缓冲，下一次迁移时再试。任务本身不受影响。 + self._remember_failure( + f"{task_id}@{latest.status_version} 写穿失败（已缓存待重试）：" + f"{type(exc).__name__}: {exc}" + ) + return + + state.last_persisted_version = latest.status_version + while state.pending: + _, record = state.pending[0] + try: + store.append(record) + except Exception as exc: + self._remember_failure( + f"{task_id}@{record.version} 迁移轨迹补写失败（已缓存待重试）：" + f"{type(exc).__name__}: {exc}" + ) + return + state.pending.popleft() + + if state.dropped: + self._remember_failure( + f"{task_id}: 写穿积压超过 {_PENDING_TRANSITIONS} 条上限，" + f"丢弃了 {state.dropped} 条迁移记录（轨迹有缺口）" + ) + state.dropped = 0 + + def _flush_all_pending(self) -> None: + """收尾前把积压的写穿再试一次（尽力而为）。""" + if self._store is None: + return + with self._lock: + items = list(self._persist_states.items()) + for task_id, state in items: + with state.lock: + if state.desynced is None: + self._flush_pending(task_id, state) + + @property + def desynced_tasks(self) -> Mapping[str, str]: + """存储状态与内存**永久分叉**、已放弃写入的任务：``task_id -> 原因``。 + + 唯一会让"内存显示成功、存储停在半路"变成永久的情形（通常是检测到外部写入者）。 + 必须可见——否则只能靠翻 :attr:`persist_failures` 才发现，而那里只留最近 20 条。 + """ + with self._lock: + return { + task_id: state.desynced + for task_id, state in self._persist_states.items() + if state.desynced is not None + } + + # ------------------------------------------------------------------ + # 查询 + # ------------------------------------------------------------------ + + def get(self, task_id: str) -> TaskSnapshot | None: + """单个任务快照；不存在返回 ``None``。 + + 绑定了存储时**优先读存储**——那才能看到别的进程提交的任务。本地回退是为了 + 容忍写穿失败：内存里的状态总比没有强。 + """ + store = self._store + if store is not None: + snapshot = store.get(task_id) + if snapshot is not None: + return snapshot + with self._lock: + submission = self._runs.get(task_id) + return submission.run.snapshot if submission is not None else None + + def runs(self) -> tuple[TaskRun, ...]: + """全部任务句柄，按提交时间排序。""" + with self._lock: + items = list(self._runs.values()) + items.sort( + key=lambda item: item.run.snapshot.created_at or datetime.min.replace(tzinfo=UTC) + ) + return tuple(item.run for item in items) + + def list_tasks( + self, + *, + status: TaskStatus | Iterable[TaskStatus] | None = None, + name_prefix: str | None = None, + tag: str | None = None, + bucket: int | None = None, + buckets: Iterable[int] | None = None, + pipeline_run_id: str | None = None, + limit: int | None = None, + ) -> list[TaskSnapshot]: + """按条件查询任务快照。 + + 绑定了存储时**走存储查询**——因此能看到别的进程提交的任务，且过滤是下推的 + （SQLite 侧有索引），不是取回全部再筛。 + + ``buckets`` 是为将来的分布式预留的：一台机器只关心自己拥有的那组桶。 + + 方法名刻意不叫 ``list``：那会在类作用域里遮蔽内置的 ``list``，使类内所有 + ``list[...]`` 类型注解（例如 :meth:`wait_all` 的返回值）失效——mypy 会报 + "Function is not valid as a type"。 + """ + wanted: frozenset[TaskStatus] | None = None + if status is not None: + wanted = frozenset({status}) if isinstance(status, TaskStatus) else frozenset(status) + wanted_buckets: frozenset[int] | None = None + if buckets is not None: + wanted_buckets = frozenset(buckets) + elif bucket is not None: + wanted_buckets = frozenset({bucket}) + + store = self._store + if store is not None: + return store.query( + TaskQuery( + statuses=wanted, + name_prefix=name_prefix, + tag=tag, + buckets=wanted_buckets, + pipeline_run_id=pipeline_run_id, + limit=limit, + ) + ) + + snapshots = [run.snapshot for run in self.runs()] + if wanted is not None: + snapshots = [s for s in snapshots if s.status in wanted] + if wanted_buckets is not None: + snapshots = [s for s in snapshots if s.bucket in wanted_buckets] + if pipeline_run_id is not None: + snapshots = [s for s in snapshots if s.pipeline_run_id == pipeline_run_id] + if name_prefix is not None: + snapshots = [s for s in snapshots if s.name.startswith(name_prefix)] + if tag is not None: + snapshots = [s for s in snapshots if tag in s.tags] + return snapshots[:limit] if limit is not None else snapshots + + def stats(self) -> StatusCounts: + """各状态计数。绑定了存储时统计的是**全部进程**的任务。""" + store = self._store + if store is not None: + return StatusCounts(by_status=store.counts()) + counts: dict[TaskStatus, int] = {} + for run in self.runs(): + counts[run.status] = counts.get(run.status, 0) + 1 + return StatusCounts(by_status=counts) + + def bucket_counts(self) -> dict[int | None, int]: + """每个桶的任务数（``None`` 键表示未分桶的任务）。 + + 分布式调度要靠它回答"哪些桶是空的、哪些桶积压了"，进而决定把桶分给谁。 + + 绑定了存储时是 O(全表)——查询协议还没有"按桶聚合计"的入口，等真的成为 + 瓶颈时再加；现在加就是无消费者的死代码。 + """ + if self._store is not None: + snapshots = self._store.query() + else: + snapshots = [run.snapshot for run in self.runs()] + counts: dict[int | None, int] = {} + for snapshot in snapshots: + counts[snapshot.bucket] = counts.get(snapshot.bucket, 0) + 1 + return counts + + @property + def bucket_policy(self) -> BucketPolicy: + """分桶策略。桶数是集群范围的约定，分布式部署时各处必须一致。""" + return self._bucket_policy + + def _validated_owned_buckets(self, buckets: Iterable[int] | None) -> frozenset[int] | None: + """校验并收下"本节点负责的桶"。 + + 越界的桶号在启动时就报错：如果放过去，症状是"某些任务永远停在 ``queued`` 且 + 没有任何报错"——那是最难查的一类问题。 + """ + if buckets is None: + return None + resolved = frozenset(buckets) + if not resolved: + raise ConfigurationError( + "owned_buckets 不能是空集：那意味着本节点什么都不负责。" + "若想表示'不限定归属'，请传 None（默认）。" + ) + limit = self._bucket_policy.num_buckets + # 先逐个检查类型，**再**排序：对混合类型排序会先抛 TypeError， + # 把一条清晰的配置错误变成一句莫名其妙的比较错误。 + for bucket in resolved: + if not isinstance(bucket, int) or isinstance(bucket, bool): + raise ConfigurationError(f"桶号必须是整数，收到 {bucket!r}") + if bucket < 0 or bucket >= limit: + raise ConfigurationError( + f"桶号越界：{bucket}；本集群的桶数是 {limit}（合法范围 0..{limit - 1}）" + ) + return resolved + + @property + def owned_buckets(self) -> frozenset[int] | None: + """本节点负责的桶；``None`` 表示不限定。 + + ``None`` 与"空集"是两件不同的事：前者是"我不管归属"，后者是"我一个桶都不负责"。 + 后者通常是配置写漏了，所以构造时会拒绝空集。 + """ + return self._owned_buckets + + @property + def owns_all_buckets(self) -> bool: + """是否**不构成划分**：不限定，或显式覆盖了全部桶。 + + 覆盖全部桶时没有任何"别的节点"，因此无桶任务同样会且只会由本节点执行—— + 此时再强制每个任务写 ``bucket_key`` 就只是纯摩擦。 + """ + owned = self._owned_buckets + if owned is None: + return True + return len(owned) == self._bucket_policy.num_buckets + + def owns_bucket(self, bucket: int | None) -> bool: + """某个桶是否归本节点负责（只回答桶的归属，不回答"无桶任务能否提交"）。 + + 未限定时恒为 ``True``；限定时 ``None`` 不算归我。 + """ + if self._owned_buckets is None: + return True + return bucket is not None and bucket in self._owned_buckets + + def _check_bucket_ownership( + self, bucket: int | None, *, task_id: str, bucket_key: str | None + ) -> None: + """提交时确认任务落在本节点负责的桶里。 + + 判据只有一条：**这个任务将来会不会有人执行**。 + + 为什么是**硬错误**而不是警告：一个落在别人桶里的任务不会有任何节点去跑它， + 它会永远停在 ``queued``，而且**没有任何报错**。与其让它静默堆积，不如在提交 + 那一刻就说清楚。 + """ + if self.owns_all_buckets: + return + owned = self._owned_buckets + assert owned is not None # owns_all_buckets 为 False 时必然有限定 + if bucket is None: + raise ConfigurationError( + f"本节点只负责桶 {sorted(owned)}，因此提交任务 {task_id!r} 时必须给出 " + "bucket_key——否则它没有归属桶，不会有任何节点认领它。" + ) + if bucket not in owned: + raise ConfigurationError( + f"桶 {bucket}（来自 bucket_key={bucket_key!r}）不属于本节点；" + f"本节点负责 {sorted(owned)}。任务若写进别人的桶，不会有任何节点执行它。" + ) + + @property + def active_count(self) -> int: + """当前占用并发额度的任务数。""" + with self._lock: + return len(self._active) + + def active_in_bucket(self, bucket: int | None) -> int: + """某个桶当前占用的执行额度。""" + with self._lock: + return sum(1 for slot in self._active.values() if slot.bucket == bucket) + + @property + def queued_count(self) -> int: + """正在排队等待执行槽的任务数。""" + with self._lock: + return len(self._pending) + + def _can_start_locked(self, snapshot: TaskSnapshot) -> bool: + """须在持有 ``_lock`` 时调用。三道闸都要有余量才放行。 + + 顺序由"最便宜且最可能挡住"到"最贵"：全局额度 → 桶额度 → 具名资源。不过 + 顺序只影响性能，不影响结论——三者是**与**关系，任何一道不过都不启动。 + """ + policy = self._concurrency + if policy.max_parallel is not None and len(self._active) >= policy.max_parallel: + return False + if self._max_parallel_per_bucket is not None: + in_bucket = sum(1 for slot in self._active.values() if slot.bucket == snapshot.bucket) + if in_bucket >= self._max_parallel_per_bucket: + return False + return self._resources_available_locked(snapshot.tags) + + def _resources_available_locked(self, tags: frozenset[str]) -> bool: + """具名信号量：任务需要的**每个**受限期资源都还有余量。须持有 ``_lock``。 + + 只统计候选任务自己关心的那几个标签：没写进 ``limits`` 的标签不受限，因此 + 数它们纯属浪费——标签集可能很大，而受限期通常只有一两个。 + """ + limits = self._concurrency.limits + if not limits or not tags: + return True + wanted = [tag for tag in tags if tag in limits] + if not wanted: + return True + in_use: dict[str, int] = dict.fromkeys(wanted, 0) + for slot in self._active.values(): + for tag in slot.tags: + if tag in in_use: + in_use[tag] += 1 + return all(in_use[tag] < limits[tag] for tag in wanted) + + # ------------------------------------------------------------------ + # 重启恢复 + # ------------------------------------------------------------------ + + def reconcile( + self, + *, + stale_after: float | None = None, + reason: str | None = None, + dry_run: bool = False, + ) -> ReconcileReport: + """清理上次进程留下的**僵尸任务**。进程启动时应当先跑一次。 + + ## 判据是"还有人在管它吗"，不是"它还在推进吗" + + 每个监督进程都会给自己负责的任务**续租**（见 :attr:`lease_owner`）。所以： + + * 租约未过期 → 有活着的进程在管，**不动它**； + * 租约已过期 / 从来没有租约（且已过宽限期）→ 没有监督者： + + - 任务**占着执行资源**（``starting`` / ``running`` / ``pausing`` / ``stopping``） + → 载体已随进程消失，谁也不会再推进它，判 ``LOST``； + - 任务**不占资源**（``queued`` / ``paused`` / ``retry_wait`` …） + → 只是没人接手，**不改状态**，报告为 :attr:`ReconcileReport.adoptable`。 + + 这里刻意**不用任务心跳**（``last_heartbeat_at``）当判据。心跳由任务自己的 + ``checkpoint()`` 驱动，一个长时间不调 checkpoint 的健康任务心跳同样会停滞—— + 拿它判断进程存活，会把正常任务杀掉。反过来用它做 idle 超时才是对的。 + + ## 它不做什么 + + **不会重新执行任何任务。** 存储里记的是状态，不是配方：任务的函数与参数从未 + 落库（序列化任意参数会引入不可反序列化、版本漂移等一堆问题）。所以恢复一个 + 失去载体的任务在原理上不可能，能做的只是把它如实标记为丢失，让看板不再永远 + 显示一个假的"运行中"。 + + 真正"重跑"要靠调用方重新 :meth:`submit`，而那需要调用方自己知道参数。 + + ``dry_run=True`` 只出报告、不写存储——这类操作会改状态，先看一眼是值得的。 + """ + store = self._store + if store is None: + raise ConfigurationError( + "reconcile 需要共享存储：没有存储就无从判断哪个任务还有监督者。" + ) + now = self._clock() + grace = self._lease_ttl * 2 if stale_after is None else stale_after + entries: list[ReconcileEntry] = [] + + # 只看**本节点负责的桶**。统一存储里可能有多个自治节点的任务；去动别人的任务 + # 既越界（那是它的责任）也没有意义（它的租约由它自己续）。 + for snapshot in store.query(TaskQuery(statuses=NON_TERMINAL, buckets=self._owned_buckets)): + if self._find(snapshot.task_id) is not None: + continue # 本进程正在管它，当然活着 + if self._has_supervisor(snapshot, now=now, grace=grace): + continue + if snapshot.status not in RESOURCE_HOLDING: + entries.append( + ReconcileEntry( + snapshot.task_id, + snapshot.status, + ReconcileOutcome.ADOPTABLE, + "没有执行载体，等待重新提交", + ) + ) + continue + entries.append(self._reconcile_lost(snapshot, reason=reason, dry_run=dry_run)) + + return ReconcileReport(entries=tuple(entries), dry_run=dry_run) + + @staticmethod + def _has_supervisor(snapshot: TaskSnapshot, *, now: datetime, grace: float) -> bool: + """是否还有活着的监督者。""" + if snapshot.lease_until is not None: + return snapshot.lease_until > now + # 从未续过租：可能是刚提交、工作线程还没轮到第一次续租，给一个宽限期。 + reference = snapshot.started_at or snapshot.created_at + if reference is None: + return False + return (now - reference).total_seconds() < grace + + def _reconcile_lost( + self, + snapshot: TaskSnapshot, + *, + reason: str | None, + dry_run: bool, + ) -> ReconcileEntry: + """把一个僵尸任务如实标记为 ``LOST``。""" + store = self._store + assert store is not None # 调用方已确保 + message = reason or "监督者已失联（进程重启后未再续租）" + if dry_run: + return ReconcileEntry( + snapshot.task_id, snapshot.status, ReconcileOutcome.MARKED_LOST, message + ) + + # 仍然经由状态机构造迁移：它是唯一写者，边界校验（终态集合、转移表）都在那里。 + # 直接拼一条 Transition 会绕开这些校验，也会让"非法转移"从不该发生的路径进来。 + machine = StateMachine(snapshot, clock=self._clock) + try: + record = machine.mark_lost( + reason=message, + failure=Failure(kind=FailureKind.LOST, message=message, retryable=False), + ) + store.save(machine.snapshot, expect_version=snapshot.status_version) + store.append(record) + except ConcurrencyConflict: + # 原本的监督者又活过来了（或别人抢先处理了）。这时候**必须放手**： + # 强行覆盖会把对方正在写的状态抹掉。 + return ReconcileEntry( + snapshot.task_id, + snapshot.status, + ReconcileOutcome.SKIPPED, + "有人正在写这个任务，已跳过", + ) + except (InvalidTransition, StoreError) as exc: + return ReconcileEntry( + snapshot.task_id, + snapshot.status, + ReconcileOutcome.SKIPPED, + f"{type(exc).__name__}: {exc}", + ) + return ReconcileEntry( + snapshot.task_id, snapshot.status, ReconcileOutcome.MARKED_LOST, message + ) + + # ------------------------------------------------------------------ + # 控制 + # ------------------------------------------------------------------ + + def stop(self, task_id: str, *, reason: str | None = None) -> bool: + """请求停止单个任务。返回是否找到了该任务。 + + 本进程持有的任务直接下发；属于**别的进程**的任务则往存储里写一条指令， + 由持有者轮询取走。两种情况下返回值都表示"请求已被受理"。 + """ + run = self._find(task_id) + if run is not None: + run.stop(reason=reason) + return True + return self.request(task_id, CommandKind.STOP, reason=reason) + + def pause(self, task_id: str, *, timeout: float = 5.0) -> bool: + """请求暂停单个任务；返回是否**确认**已挂起。 + + 对属于别的进程的任务，本方法会下发指令并返回 ``False``——**跨进程无法确认 + 暂停是否生效**（那需要回执通道）。要观测结果请轮询 :meth:`get`。 + 返回 ``False`` 在这里的含义是"未确认"，不是"失败"。 + """ + run = self._find(task_id) + if run is not None: + return run.pause(timeout=timeout) + if self.request(task_id, CommandKind.PAUSE): + return False + return False + + def resume(self, task_id: str) -> bool: + """恢复单个任务。返回请求是否已被受理。""" + run = self._find(task_id) + if run is not None: + run.resume() + return True + return self.request(task_id, CommandKind.RESUME) + + def request( + self, + task_id: str, + kind: CommandKind, + *, + reason: str | None = None, + ) -> bool: + """向**别的进程**持有的任务下发一条控制指令。 + + 这是跨进程控制的唯一通道：一个进程无法给另一个进程的执行载体发信号，只能 + 把意图写进共享存储，由持有者轮询取走。 + + 任务在本进程内时不要用它——直接调 :meth:`stop` / :meth:`pause` / + :meth:`resume`，那是同步的、能拿到确认的。 + """ + store = self._store + if store is None: + return False + # 先确认任务真实存在。否则一个拼错的 task_id 会写出一条**永远不会有人执行**的 + # 指令，一直躺在存储里；而调用方还会以为"停止已下发"。 + if store.get(task_id) is None: + return False + store.submit_command( + Command( + command_id=uuid.uuid4().hex[:12], + task_id=task_id, + kind=kind, + created_at=self._clock(), + requested_by=self._owner, + reason=reason, + ) + ) + self._ensure_worker() + self._wake.set() + return True + + def stop_all(self, *, reason: str | None = None) -> int: + """向所有未落终态的任务**发出**停止请求，返回本轮发出的请求数。 + + 是"已发出"而不是"已停止"：请求发出后任务先进入 ``stopping``，落终态是 + 异步的。要等它们真的结束，请接着调 :meth:`wait_all`。 + + 只作用于**本进程持有**的任务。别的进程的任务要靠共享存储里的指令，而 + "全部停止"跨进程需要一份全局任务清单——那是分布式调度该决定的事，不是 + 一个便利方法顺手能定义的。 + """ + count = 0 + for run in self.runs(): + if not run.snapshot.is_terminal: + run.stop(reason=reason) + count += 1 + return count + + # ------------------------------------------------------------------ + # 等待与收尾 + # ------------------------------------------------------------------ + + def wait(self, task_id: str, timeout: float | None = None) -> TaskRunOutcome: + """等待单个任务落终态。""" + run = self._find(task_id) + if run is None: + raise ConfigurationError(f"没有这个任务：{task_id}") + return run.wait(timeout=timeout) + + def wait_all(self, timeout: float | None = None) -> list[TaskRunOutcome]: + """等待**所有**任务落终态，返回已落终态的结果。 + + 包含仍在排队的任务——它们会随着并发额度释放而被自动启动。若 ``timeout`` + 到期仍有任务未结束，只返回已经落终态的那些（不会抛异常，也不会改变任何任务 + 的状态）。 + + 不能用"逐个 ``run.wait()``"实现：排队中的任务尚未启动，没有可等待的执行。 + """ + deadline = None if timeout is None else time.monotonic() + timeout + while True: + pending = [run for run in self.runs() if not run.snapshot.is_terminal] + if not pending: + break + if deadline is not None and time.monotonic() >= deadline: + break + time.sleep(self._poll) + + outcomes: list[TaskRunOutcome] = [] + for run in self.runs(): + if run.started and run.snapshot.is_terminal: + outcomes.append(run.wait()) + return outcomes + + def close( + self, + *, + reason: str | None = None, + timeout: float = 30.0, + close_store: bool = False, + ) -> list[TaskRunOutcome]: + """停止一切并收尾：排队的任务会被"启动后立刻停止"，不会真的执行任务体。 + + ``close_store`` 默认 ``False``：存储是调用方传进来的，通常由调用方决定它的 + 生命周期（多个 Controller 可能共用同一个存储）。但**用完一定要关**——一个 + 没关的 SQLite 连接会泄漏文件句柄，在 Windows 上还会让数据库文件无法删除。 + """ + # 先关闸，再停止：否则调度器可能抢在清空队列之前把排队任务跑起来。 + with self._lock: + self._closed = True + still_pending = list(self._pending) + self._pending.clear() + self.stop_all(reason=reason) + for run in still_pending: + # 监督循环第一轮就会看到停止请求并落 stopped，任务体不会被执行。 + run.start() + self._wake.set() + outcomes = self.wait_all(timeout=timeout) + for thread in (self._dispatcher, self._worker): + if thread is not None: + thread.join(timeout) + # 收尾前再试一次：把因瞬时故障积压的写穿补上。尽力而为——补不上也不会让这里 + # 抛异常，原因已经记在 persist_failures / desynced_tasks 里。 + self._flush_all_pending() + if close_store and self._store is not None: + self._store.close() + return outcomes + + def prune(self, *, keep_terminal: int = 1000) -> int: + """丢弃最旧的终态记录，防止内存无限增长；返回丢弃的数量。 + + 绑定了存储时**只清内存**：存储里的记录仍然可查。这正是持久化的意义—— + 内存可以丢，历史不该丢。 + """ + with self._lock: + terminal = [item for item in self._runs.values() if item.run.snapshot.is_terminal] + terminal.sort( + key=lambda item: item.run.snapshot.created_at or datetime.min.replace(tzinfo=UTC) + ) + excess = len(terminal) - max(0, keep_terminal) + if excess <= 0: + return 0 + for item in terminal[:excess]: + task_id = item.run.snapshot.task_id + del self._runs[task_id] + # 写穿状态也跟着丢：留着它只会让字典无限增长。 + self._persist_states.pop(task_id, None) + return excess + + # ------------------------------------------------------------------ + # 内部 + # ------------------------------------------------------------------ + + def _find(self, task_id: str) -> TaskRun | None: + with self._lock: + submission = self._runs.get(task_id) + return submission.run if submission is not None else None + + def _on_run_finished(self, outcome: TaskRunOutcome) -> None: + """任务落终态：归还并发额度并唤醒调度器。""" + with self._lock: + self._active.pop(outcome.snapshot.task_id, None) + listeners = tuple(self._listeners) + self._wake.set() + # 监听器在额度已归还、调度器已唤醒**之后**才被调用，因此它们可以安全地提交 + # 新任务。它们的异常绝不能影响任务本身的结论，但也不能静默——见 listener_failures。 + for listener in listeners: + try: + listener(outcome) + except Exception as exc: + with self._lock: + self._listener_failures.append( + f"{outcome.snapshot.task_id}: {type(exc).__name__}: {exc}" + ) + del self._listener_failures[:-20] + + def add_listener(self, listener: Callable[[TaskRunOutcome], None]) -> None: + """注册一个"任务落终态"的监听器，返回 ``None``。 + + 管道调度器靠它推进——这是它唯一需要知道的时机，不必轮询。监听器在并发额度 + 已归还之后被调用，因此里面可以安全地提交新任务。 + """ + with self._lock: + self._listeners.append(listener) + + def _ensure_dispatcher(self) -> None: + with self._lock: + if self._dispatcher is not None: + return + self._dispatcher = threading.Thread( + target=self._dispatch_loop, name="controller-dispatch", daemon=True + ) + self._dispatcher.start() + + def _take_runnable_locked(self) -> TaskRun | None: + """取下一个可启动的排队任务；没有则返回 ``None``。须持有 ``_lock``。 + + 必须**扫描**而不是直接取队首：某个桶或某个具名资源满的时候，队首可能正好被 + 它挡住，而后面不受它约束的任务本可以先跑。直接取队首会让一处积压阻塞所有 + 任务——那正是分桶与信号量要解决的问题，不能在调度器里又引入一次。 + + 在**都能启动**的任务里取优先级最高的；同优先级取先提交的——优先级机制不该 + 打乱同优先级的时间序。 + + 扫描途中顺手清掉**已落终态**的排队任务（被取消或跳过）：它们绝不能被启动， + 否则监督循环会撞上非法状态转移。 + """ + best_index = -1 + best_priority = 0 + index = 0 + while index < len(self._pending): + run = self._pending[index] + if run.snapshot.is_terminal: + del self._pending[index] + continue + if self._can_start_locked(run.snapshot) and ( + best_index < 0 or run.priority > best_priority + ): + best_index = index + best_priority = run.priority + index += 1 + if best_index < 0: + return None + run = self._pending[best_index] + del self._pending[best_index] + return run + + def _dispatch_loop(self) -> None: + while True: + self._wake.wait(0.2) + self._wake.clear() + while True: + with self._lock: + # 关闭后不再启动任何新工作：排队的任务由 close() 自己收尾， + # 否则它可能抢在 close() 清空队列之前把任务跑起来。 + if self._closed: + return + run = self._take_runnable_locked() + if run is None: + break + self._active[run.snapshot.task_id] = _ActiveSlot( + run.snapshot.bucket, run.snapshot.tags + ) + run.start() + + # ------------------------------------------------------------------ + # 跨进程：指令轮询与心跳回写 + # ------------------------------------------------------------------ + + def _ensure_worker(self) -> None: + """启动跨进程工作线程。只在真的有存储、且真的提交过任务时启动。 + + 一个纯查询客户端若也去取指令，就会把别人的停止请求抢走；即便它把指令交还， + 持续轮询也会让真正的持有者反复错过。所以入口设成"提交过任务才启动"。 + """ + if self._store is None: + return + with self._lock: + if self._worker is not None: + return + self._worker = threading.Thread( + target=self._worker_loop, name="controller-worker", daemon=True + ) + self._worker.start() + + def _worker_loop(self) -> None: + while True: + self._wake.wait(self._command_poll_interval) + self._wake.clear() + if self._closed: + return + self._drain_commands() + self._flush_activity() + + def _drain_commands(self) -> None: + store = self._store + if store is None: + return + try: + commands = store.claim_commands(now=self._clock(), lease_seconds=self._command_lease) + except Exception as exc: + self._remember_failure(f"claim_commands: {type(exc).__name__}: {exc}") + return + + for item in commands: + run = self._find(item.task_id) + if run is None: + # 不属于本进程：**交还**而不是标记完成，否则真正的持有者永远收不到 + # 这次停止请求（一个静默的丢指令）。 + store.release_command(item.command_id, now=self._clock()) + continue + outcome = self._apply_command(item, run) + store.complete_command(item.command_id, now=self._clock(), outcome=outcome) + + def _apply_command(self, item: Command, run: TaskRun) -> str: + """执行一条指令。返回写回存储的结果摘要，便于事后追溯。""" + if run.snapshot.is_terminal: + return "already-terminal" + if item.kind is CommandKind.STOP: + run.stop(reason=item.reason) + return "stopped" + if item.kind is CommandKind.PAUSE: + confirmed = run.pause(timeout=self._graceful_stop) + return "paused" if confirmed else "pause-not-confirmed" + run.resume() + return "resumed" + + def _flush_activity(self) -> None: + """把本进程任务的进度与心跳推给存储。 + + 走 ``touch``（高频旁路）而不是状态写入：进度可能每秒几十次，若参与乐观并发 + 会与状态写入持续冲突。 + """ + store = self._store + if store is None: + return + lease_until = self._clock() + timedelta(seconds=self._lease_ttl) + for run in self.runs(): + # 先让快照跟上载体的真实心跳，否则下一次全量 save 会把 touch 写进去的 + # 心跳覆盖回 None（同一行的两条写入路径互相打架）。 + snapshot = run.refresh_activity() + if snapshot.is_terminal: + continue + try: + store.touch( + snapshot.task_id, + progress=snapshot.progress, + heartbeat_at=snapshot.last_heartbeat_at, + # 续租回答的是"还有人在管它吗"，与心跳是两件事——用任务心跳判断 + # 监督者存活，会把一个长时间不调 checkpoint 的健康任务误判失联。 + lease_owner=self._owner, + lease_until=lease_until, + ) + except Exception as exc: + self._remember_failure(f"touch {snapshot.task_id}: {type(exc).__name__}: {exc}") + + def _remember_failure(self, summary: str) -> None: + with self._lock: + self._persist_failures.append(summary) + del self._persist_failures[:-20] + + +# ---------------------------------------------------------------------- +# 渲染：CLI 与看板共用 +# ---------------------------------------------------------------------- + + +def _display_width(text: str) -> int: + """按终端显示宽度计算：中日韩全角字符占两列。 + + 不这么做，中文任务名会把整张表挤歪——运维看板里这种事非常刺眼。 + """ + return sum(2 if unicodedata.east_asian_width(ch) in {"W", "F"} else 1 for ch in text) + + +def _pad(text: str, width: int) -> str: + return text + " " * max(0, width - _display_width(text)) + + +def _format_elapsed(snapshot: TaskSnapshot, now: datetime) -> str: + """已执行时长。 + + 尚未开始执行时返回 ``-`` 而不是"距创建多久"——排队中的任务显示一个时长会被 + 误读成"已经跑了这么久"。 + """ + if snapshot.started_at is None: + return "-" + end = snapshot.ended_at or now + return f"{(end - snapshot.started_at).total_seconds():.1f}s" + + +def _format_progress(progress: Progress | None) -> str: + if progress is None: + return "-" + if progress.total is None: + return str(progress.done) + return f"{progress.done}/{progress.total}" + + +def _format_bucket(bucket: int | None) -> str: + return "-" if bucket is None else str(bucket) + + +def render_table(snapshots: Sequence[TaskSnapshot], *, now: datetime | None = None) -> str: + """把任务快照渲染成等宽文本表。 + + CLI 直接打印它；将来的看板可以拿同样的数据渲染 HTML，两边不会出现口径差异。 + """ + moment = now if now is not None else utcnow() + rows: list[tuple[str, ...]] = [ + ("TASK_ID", "NAME", "BUCKET", "STATUS", "ATTEMPT", "ELAPSED", "PROGRESS", "FAILURE") + ] + for snapshot in snapshots: + failure: Failure | None = snapshot.failure + rows.append( + ( + snapshot.task_id, + snapshot.name, + _format_bucket(snapshot.bucket), + snapshot.status.value, + f"{snapshot.attempt}/{snapshot.max_attempts}", + _format_elapsed(snapshot, moment), + _format_progress(snapshot.progress), + failure.kind.value if failure is not None else "-", + ) + ) + + widths = [max(_display_width(row[index]) for row in rows) for index in range(len(rows[0]))] + lines = [] + for position, row in enumerate(rows): + line = _COLUMN_SEPARATOR.join( + _pad(cell, width) for cell, width in zip(row, widths, strict=True) + ).rstrip() + lines.append(line) + if position == 0: + lines.append(_COLUMN_SEPARATOR.join("-" * width for width in widths)) + return "\n".join(lines) diff --git a/src/chongming_task/core/__init__.py b/src/chongming_task/core/__init__.py new file mode 100644 index 0000000..e0c7b7f --- /dev/null +++ b/src/chongming_task/core/__init__.py @@ -0,0 +1,5 @@ +"""运行时内核：状态机、依赖图、调度器。 + +本层刻意**不**依赖 ``asyncio``、不访问数据库、不认识日志库——领域逻辑必须能被 +属性测试穷举验证，这要求它保持纯粹。 +""" diff --git a/src/chongming_task/core/state_machine.py b/src/chongming_task/core/state_machine.py new file mode 100644 index 0000000..41c3f8e --- /dev/null +++ b/src/chongming_task/core/state_machine.py @@ -0,0 +1,632 @@ +"""任务状态机——任务状态的**唯一写入者**。 + +不变量 +------ +1. 任何状态变更都经转移表校验，非法转移抛 :class:`~chongming_task.errors.InvalidTransition`， + 而不是静默忽略或返回 ``False``——"停止指令没生效"是最难排查的一类问题。 +2. 每次成功变更都递增 ``status_version`` 并追加一条 :class:`Transition`， + 因此状态历史只追加、可审计，并可作为存储层乐观并发的依据。 +3. 终态不可离开；复活只能走显式的 :meth:`StateMachine.retry` / :meth:`StateMachine.reset`。 +4. 时间全部取自注入的 ``clock``，使重试/超时相关测试无需真实等待。 + +不负责的事 +---------- +* **线程安全**。调用方（调度器）必须保证同一任务只有一个写入者。 +* 依赖解析、并发槽、执行器启动。这些由调度器决定，状态机只记录结果。 +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass, replace +from datetime import datetime +from enum import StrEnum +from typing import Any + +from ..clock import Clock, utcnow +from ..errors import AttemptsExhausted, InvalidTransition +from ..models import Failure, Progress, RequestedAction, TaskSnapshot +from ..states import REVIVABLE, TERMINAL, TRANSITIONS, TaskStatus + +__all__ = ["ChangeObserver", "Clock", "StateMachine", "Transition", "Trigger", "utcnow"] + + +class Trigger(StrEnum): + """状态变更的标准触发源。 + + 用枚举而非自由字符串，避免审计日志里出现 ``user:stop`` / ``stop`` / + ``user_stop`` 之类的拼写变体——一旦出现，"按触发源统计"就彻底失效了。 + """ + + DEPENDENCIES_UNMET = "orchestrator:dependencies_unmet" + DEPENDENCIES_RESOLVED = "orchestrator:dependencies_resolved" + DISPATCH = "scheduler:dispatch" + LAUNCHED = "executor:launched" + COMPLETED = "executor:completed" + FAILED = "executor:failed" + RETRY_SCHEDULED = "retry:scheduled" + RETRY_EXHAUSTED = "retry:exhausted" + RETRY_REQUESTED = "user:retry" + RESET_REQUESTED = "user:reset" + STOP_REQUESTED = "user:stop" + STOP_CONFIRMED = "runtime:stop_confirmed" + STOP_FORCED = "runtime:stop_forced" + PAUSE_REQUESTED = "user:pause" + PAUSE_REACHED = "runtime:pause_reached" + PAUSE_ABORTED = "runtime:pause_aborted" + RESUME_REQUESTED = "user:resume" + CANCEL_REQUESTED = "orchestrator:cancel" + SKIP_REQUESTED = "orchestrator:skip" + HEARTBEAT_LOST = "runtime:heartbeat_lost" + ADMIN_OVERRIDE = "admin:override" + + +#: **统一规则**：控制动作是否需要中间态，取决于任务是否**已占用执行资源**。 +#: 未占用（PENDING/BLOCKED/QUEUED/PAUSED/RETRY_WAIT）→ 立即生效； +#: 已占用（STARTING/RUNNING/PAUSING）→ 经中间态缓冲。 +#: 让未启动的任务显示"停止中/暂停中"是虚假信息，会让运维误以为需要等待。 +_STOP_IS_DEFERRED: frozenset[TaskStatus] = frozenset( + {TaskStatus.STARTING, TaskStatus.RUNNING, TaskStatus.PAUSING} +) +_PAUSE_IS_DEFERRED: frozenset[TaskStatus] = frozenset({TaskStatus.STARTING, TaskStatus.RUNNING}) + + +#: 状态变更观察者：``(变更后的快照, 本次迁移记录)``。 +#: +#: **契约：本回调不得抛异常。** 它在状态已经落定之后被调用；若它抛出去，调用方会 +#: 看到一个"状态已变但操作失败"的中间态。需要处理失败的实现（例如写存储失败）应当 +#: 自己捕获并记录下来。 +ChangeObserver = Callable[[TaskSnapshot, "Transition"], None] + + +@dataclass(frozen=True, slots=True) +class Transition: + """一次已生效的状态变更记录。只追加，不修改。""" + + task_id: str + from_status: TaskStatus + to_status: TaskStatus + at: datetime + trigger: str + actor: str = "system" + reason: str | None = None + version: int = 0 + """变更后的 ``status_version``，用于与存储层比对。""" + + @property + def is_override(self) -> bool: + """是否为人工强制改写。此类记录在审计视图中必须单独可见。""" + return self.trigger == Trigger.ADMIN_OVERRIDE + + +class StateMachine: + """持有单个任务的状态，并作为其唯一写入者。""" + + __slots__ = ("_clock", "_history", "_on_change", "_snapshot") + + def __init__( + self, + snapshot: TaskSnapshot, + *, + clock: Clock | None = None, + on_change: ChangeObserver | None = None, + ) -> None: + self._snapshot = snapshot + self._clock: Clock = clock if clock is not None else utcnow + self._history: list[Transition] = [] + self._on_change = on_change + + @classmethod + def create( + cls, + task_id: str, + name: str, + *, + max_attempts: int = 1, + pipeline_run_id: str | None = None, + tags: frozenset[str] = frozenset(), + bucket: int | None = None, + bucket_key: str | None = None, + clock: Clock | None = None, + on_change: ChangeObserver | None = None, + ) -> StateMachine: + """创建处于 ``PENDING`` 的新任务。""" + now = (clock if clock is not None else utcnow)() + snapshot = TaskSnapshot( + task_id=task_id, + name=name, + status=TaskStatus.PENDING, + created_at=now, + max_attempts=max_attempts, + pipeline_run_id=pipeline_run_id, + tags=tags, + bucket=bucket, + bucket_key=bucket_key, + ) + return cls(snapshot, clock=clock, on_change=on_change) + + # ------------------------------------------------------------------ + # 只读视图 + # ------------------------------------------------------------------ + + @property + def snapshot(self) -> TaskSnapshot: + """当前快照。不可变，可安全外传。""" + return self._snapshot + + @property + def status(self) -> TaskStatus: + """当前状态。""" + return self._snapshot.status + + @property + def history(self) -> tuple[Transition, ...]: + """本实例产生过的全部转移记录。""" + return tuple(self._history) + + def can(self, target: TaskStatus) -> bool: + """能否合法转移到 ``target``（仅依据转移表）。""" + return target in TRANSITIONS[self._snapshot.status] + + @property + def can_stop(self) -> bool: + """能否停止。供控制面预检，使 UI 能提前置灰按钮。""" + return self.can(TaskStatus.STOPPING) or self.can(TaskStatus.STOPPED) + + @property + def can_pause(self) -> bool: + """能否暂停。未启动的任务立即暂停，运行中的任务经 ``PAUSING`` 缓冲。""" + return self.can(TaskStatus.PAUSING) or self.can(TaskStatus.PAUSED) + + @property + def can_resume(self) -> bool: + """能否恢复。目标状态由 :meth:`resume` 根据暂停层级判定。""" + return self.status is TaskStatus.PAUSED + + @property + def can_retry(self) -> bool: + """能否经 :meth:`retry` 复活。""" + return self.status in REVIVABLE + + # ------------------------------------------------------------------ + # 写入核心 + # ------------------------------------------------------------------ + + def _apply( + self, + target: TaskStatus, + trigger: str, + *, + actor: str = "system", + reason: str | None = None, + enforce_table: bool = True, + **changes: Any, + ) -> Transition: + """写入路径的唯一实现。 + + ``enforce_table=False`` 仅供 :meth:`retry` / :meth:`reset` / + :meth:`force_transition` 使用——这三者会自行完成更严格的守卫与审计， + 而不受转移表约束。 + """ + current = self._snapshot + if enforce_table and target not in TRANSITIONS[current.status]: + raise InvalidTransition(current.status, target, action=str(trigger)) + + now = self._clock() + if target in TERMINAL: + changes.setdefault("ended_at", now) + changes.setdefault("requested_action", None) + + new = replace( + current, + status=target, + status_version=current.status_version + 1, + **changes, + ) + record = Transition( + task_id=new.task_id, + from_status=current.status, + to_status=target, + at=now, + trigger=str(trigger), + actor=actor, + reason=reason, + version=new.status_version, + ) + self._snapshot = new + self._history.append(record) + if self._on_change is not None: + # 契约：观察者不得抛异常（见 ChangeObserver）。挂在 _apply 上而不是各 + # 业务方法里，是为了保证**任何**一次迁移都不会漏掉——漏一次就意味着 + # 存储与内存悄悄分叉。 + self._on_change(new, record) + return record + + # ------------------------------------------------------------------ + # 调度推进（由调度器/执行器调用） + # ------------------------------------------------------------------ + + def mark_blocked(self, *, reason: str | None = None) -> Transition: + """依赖未满足，等待上游。""" + return self._apply(TaskStatus.BLOCKED, Trigger.DEPENDENCIES_UNMET, reason=reason) + + def mark_queued(self, *, reason: str | None = None) -> Transition: + """依赖已满足，进入就绪队列等待执行槽。""" + return self._apply( + TaskStatus.QUEUED, + Trigger.DEPENDENCIES_RESOLVED, + reason=reason, + queued_at=self._clock(), + ) + + def mark_starting(self, *, executor: str, reason: str | None = None) -> Transition: + """开始一次新的尝试，同时递增 ``attempt``。 + + 重试预算在此处统一校验，而不是分散到各调用点——预算只在"真正要跑"的 + 那一刻才应该被消费。 + """ + used = self._snapshot.attempt + if used + 1 > self._snapshot.max_attempts: + raise AttemptsExhausted(self._snapshot.task_id, used, self._snapshot.max_attempts) + return self._apply( + TaskStatus.STARTING, + Trigger.DISPATCH, + reason=reason, + attempt=used + 1, + executor=executor, + started_at=self._clock(), + ) + + def mark_running(self, *, pid: int | None = None) -> Transition: + """执行器已拉起，工作真正开始。 + + ``started_at`` 在 :meth:`mark_starting` 时即已写入——它标记**本次尝试**的 + 起点；这里只补执行载体信息（``pid``：进程执行器下就是子进程的进程号）。 + 这样任何到达 ``RUNNING`` 的路径（包括 ``STARTING → PAUSING → RUNNING`` 的 + 暂停撤销）都必然带有开始时间。 + + 带**守卫**：没有 ``started_at`` 就不允许进入 ``RUNNING``。转移表无法表达 + "必须在本次尝试中已经启动过"——``PAUSED → RUNNING`` 这条边对就地暂停是 + 合法的，对"排队期间被暂停、从未启动"的任务却毫无意义。守卫负责补上这个 + 区别，否则会产出 ``RUNNING`` 却没有开始时间的自相矛盾快照。 + """ + if self._snapshot.started_at is None: + raise InvalidTransition( + self._snapshot.status, + TaskStatus.RUNNING, + action="guard:尚未开始本次尝试", + ) + return self._apply(TaskStatus.RUNNING, Trigger.LAUNCHED, pid=pid) + + def mark_succeeded(self, *, outcome: Mapping[str, Any] | None = None) -> Transition: + """正常完成。""" + return self._apply(TaskStatus.SUCCEEDED, Trigger.COMPLETED, outcome=outcome) + + def mark_retry_wait(self, failure: Failure) -> Transition: + """本次尝试失败但仍有重试余额，进入退避等待。 + + 必须携带 ``failure``：否则界面无法回答"它为什么在等"。 + + 顺带清掉 ``pid``：那一次尝试的载体已经没了，留着一个死进程号只会让人去 `ps` + 一个不存在的进程——比没有 pid 更容易误导。 + """ + return self._apply( + TaskStatus.RETRY_WAIT, Trigger.RETRY_SCHEDULED, failure=failure, pid=None + ) + + def mark_retry_exhausted(self, failure: Failure) -> Transition: + """重试余额耗尽，落 ``FAILED``。""" + return self._apply(TaskStatus.FAILED, Trigger.RETRY_EXHAUSTED, failure=failure) + + def mark_failed(self, failure: Failure) -> Transition: + """遇到不可重试的错误，直接失败。""" + return self._apply(TaskStatus.FAILED, Trigger.FAILED, failure=failure) + + def mark_skipped(self, *, reason: str | None = None) -> Transition: + """条件为假或依策略跳过。""" + return self._apply(TaskStatus.SKIPPED, Trigger.SKIP_REQUESTED, reason=reason) + + def mark_cancelled( + self, *, reason: str | None = None, failure: Failure | None = None + ) -> Transition: + """编排层级联取消。""" + return self._apply( + TaskStatus.CANCELLED, + Trigger.CANCEL_REQUESTED, + reason=reason, + failure=failure, + ) + + def mark_lost(self, *, reason: str | None = None, failure: Failure | None = None) -> Transition: + """心跳丢失、载体被放弃，结果未知。 + + 应当带上 ``failure``：``LOST`` 的全部意义就是"不知道结果如何"， + 若不同时记录为什么不知道，事后根本无从判断该重跑还是该人工核对。 + """ + return self._apply( + TaskStatus.LOST, + Trigger.HEARTBEAT_LOST, + reason=reason, + failure=failure, + ) + + # ------------------------------------------------------------------ + # 控制面（由 Controller / 用户调用） + # ------------------------------------------------------------------ + + def request_stop(self, *, actor: str = "user", reason: str | None = None) -> Transition: + """请求停止。 + + 任务若正占用执行资源（STARTING/RUNNING/PAUSING），进入 ``STOPPING`` 等待 + 静默；否则立即落 ``STOPPED``。对尚未真正跑起来的任务显示"停止中"是虚假 + 信息，会让运维误以为需要等待。 + """ + if self._snapshot.status in _STOP_IS_DEFERRED: + return self._apply( + TaskStatus.STOPPING, + Trigger.STOP_REQUESTED, + actor=actor, + reason=reason, + requested_action=RequestedAction.STOP, + ) + return self._apply(TaskStatus.STOPPED, Trigger.STOP_REQUESTED, actor=actor, reason=reason) + + def mark_stopped( + self, + *, + actor: str = "system", + forced: bool = False, + reason: str | None = None, + failure: Failure | None = None, + ) -> Transition: + """运行期确认任务已停止。``forced`` 表示走的是强杀/放弃等待路径。""" + return self._apply( + TaskStatus.STOPPED, + Trigger.STOP_FORCED if forced else Trigger.STOP_CONFIRMED, + actor=actor, + reason=reason, + failure=failure, + ) + + def request_pause(self, *, actor: str = "user", reason: str | None = None) -> Transition: + """请求暂停。 + + 与 :meth:`request_stop` 同一原则：任务是否已占用执行资源，决定要不要经过 + ``PAUSING`` 缓冲。尚未跑起来的任务（QUEUED / RETRY_WAIT）本就静止，直接落 + ``PAUSED``——对它们显示"暂停中"是虚假信息。 + + 已在 ``PAUSING`` 的任务不会被"加急"成 ``PAUSED``：它还没到达检查点， + 谎称已暂停会让用户以为资源已释放。幂等处理由控制面负责。 + """ + status = self._snapshot.status + if status is TaskStatus.PAUSING: + raise InvalidTransition(status, TaskStatus.PAUSED, action=str(Trigger.PAUSE_REQUESTED)) + if status in _PAUSE_IS_DEFERRED: + return self._apply( + TaskStatus.PAUSING, + Trigger.PAUSE_REQUESTED, + actor=actor, + reason=reason, + requested_action=RequestedAction.PAUSE, + ) + if self.can(TaskStatus.PAUSED): + return self._apply( + TaskStatus.PAUSED, Trigger.PAUSE_REQUESTED, actor=actor, reason=reason + ) + raise InvalidTransition(status, TaskStatus.PAUSING, action=str(Trigger.PAUSE_REQUESTED)) + + def mark_paused(self, *, reason: str | None = None) -> Transition: + """任务已到达静默点并挂起。 + + 请求已经完成，因此清空 ``requested_action``——它承载的是"已受理但尚未 + 完成的请求"，留着一个已完成的请求会让 UI 永远显示"暂停中"。 + """ + return self._apply( + TaskStatus.PAUSED, + Trigger.PAUSE_REACHED, + reason=reason, + requested_action=None, + ) + + def abort_pause(self, *, reason: str | None = None) -> Transition: + """暂停超时仍未静默，回滚为 ``RUNNING``。 + + 如实告知"没能暂停"，而不是假装已暂停——后者会让用户以为资源已释放。 + + 带**守卫**：只允许从 ``PAUSING`` 回滚。``PAUSED → RUNNING`` 这条边是为 + *就地恢复* 准备的（见 :meth:`resume`），它要求本次尝试已经启动过；把 + "回滚"也放进来会产出一个没有 ``started_at`` 的 ``RUNNING`` 快照。 + """ + if self._snapshot.status is not TaskStatus.PAUSING: + raise InvalidTransition( + self._snapshot.status, + TaskStatus.RUNNING, + action=str(Trigger.PAUSE_ABORTED), + ) + return self._apply( + TaskStatus.RUNNING, + Trigger.PAUSE_ABORTED, + reason=reason, + requested_action=None, + ) + + def resume(self, *, actor: str = "user", reason: str | None = None) -> Transition: + """恢复被暂停的任务。 + + 目标状态取决于**暂停发生在哪一层**（见 :meth:`_resume_target`）： + + * 就地暂停——任务阻塞在 ``checkpoint()`` 里，始终占着执行槽，它的调用栈 + 还在内存中，因此恢复就是回到 ``RUNNING``。我们没有办法把任意的 Python + 调用栈换出内存再换回来，所以**暂停不释放并发槽位**。 + * 尚未开始就被暂停——从未启动过，恢复应当回到 ``QUEUED`` 重新排队。 + """ + target = self._resume_target() + changes: dict[str, Any] = {"requested_action": None} + if target is TaskStatus.QUEUED: + changes["queued_at"] = self._clock() + return self._apply( + target, + Trigger.RESUME_REQUESTED, + actor=actor, + reason=reason, + **changes, + ) + + def _resume_target(self) -> TaskStatus: + """判定恢复应该回到哪个状态。 + + 先看迁移历史：暂停若是从 ``PAUSING`` 进入的，就是就地暂停。 + 历史缺失时（例如快照直接从存储加载、尚未经历任何迁移）退化为按 + ``attempt`` 判断——已经开过尝试的任务只可能是就地暂停的。 + """ + for record in reversed(self._history): + if record.to_status is TaskStatus.PAUSED: + return ( + TaskStatus.RUNNING + if record.from_status is TaskStatus.PAUSING + else TaskStatus.QUEUED + ) + return TaskStatus.RUNNING if self._snapshot.attempt >= 1 else TaskStatus.QUEUED + + # ------------------------------------------------------------------ + # 复活与人工干预（绕过转移表，因此自带更严格的守卫与审计） + # ------------------------------------------------------------------ + + def retry( + self, + *, + actor: str = "user", + reason: str | None = None, + force: bool = False, + ) -> Transition: + """从终态复活，重新进入就绪队列。 + + 不修改历史：``attempt`` 只在 :meth:`mark_starting` 中递增，既有尝试记录 + 保持不变。重试预算耗尽时抛 :class:`AttemptsExhausted`；``force=True`` + 会抬高 ``max_attempts`` 以放行——这是"我知道它已经失败够多次了，再试一次"。 + """ + current = self._snapshot + if current.status not in REVIVABLE: + raise InvalidTransition( + current.status, + TaskStatus.QUEUED, + action=str(Trigger.RETRY_REQUESTED), + ) + changes: dict[str, Any] = { + "failure": None, + "ended_at": None, + "outcome": None, + "requested_action": None, + "overridden": False, + "queued_at": self._clock(), + # 执行载体与进度都属于已经结束的那次尝试，不应带入新尝试： + # pid 是**那一次**的进程号，重试会起一个全新的载体。 + # （逐次尝试的完整明细在迁移历史里：starting / retry_wait 自带版本号、 + # 时间戳与原因。AttemptRecord 目前没有写入方。） + "started_at": None, + "progress": None, + "pid": None, + } + if current.attempt >= current.max_attempts: + if not force: + raise AttemptsExhausted(current.task_id, current.attempt, current.max_attempts) + changes["max_attempts"] = current.attempt + 1 + return self._apply( + TaskStatus.QUEUED, + Trigger.RETRY_REQUESTED, + actor=actor, + reason=reason, + enforce_table=False, + **changes, + ) + + def reset(self, *, actor: str = "user", reason: str | None = None) -> Transition: + """重置为 ``PENDING``，丢弃全部尝试历史。 + + 危险操作：重试计数与失败原因都会消失。调用方应要求二次确认。 + """ + return self._apply( + TaskStatus.PENDING, + Trigger.RESET_REQUESTED, + actor=actor, + reason=reason, + enforce_table=False, + attempt=0, + failure=None, + outcome=None, + progress=None, + requested_action=None, + queued_at=None, + started_at=None, + ended_at=None, + last_heartbeat_at=None, + executor=None, + pid=None, + overridden=False, + ) + + def force_transition( + self, + target: TaskStatus, + *, + actor: str, + reason: str, + failure: Failure | None = None, + ) -> Transition: + """绕过转移表人工改写状态。 + + 现实运维需要"这个任务卡住了，我确认后把它标成失败"的能力；堵死它只会让人 + 绕过系统去改数据库。因此本方法存在，但要求非空 ``actor`` 与 ``reason``， + 并把 ``overridden=True`` 写进快照，使人工干预在报表中单独可见。 + """ + if not actor or not actor.strip(): + raise ValueError("force_transition 必须记录 actor") + if not reason or not reason.strip(): + raise ValueError("force_transition 必须记录 reason") + changes: dict[str, Any] = {"overridden": True, "requested_action": None} + if failure is not None: + changes["failure"] = failure + if target in TERMINAL: + # _apply 会补 ended_at。SUCCEEDED 语义上不可能带失败原因，顺手抹平， + # 否则会留下一个恒被 validate() 判为不一致的快照。 + if target is TaskStatus.SUCCEEDED: + changes["failure"] = None + else: + # 非终态带着结束时间是自相矛盾的。 + changes["ended_at"] = None + if target is TaskStatus.RUNNING and self._snapshot.started_at is None: + changes["started_at"] = self._clock() + if target is TaskStatus.RETRY_WAIT and failure is None: + raise ValueError( + "force_transition 到 RETRY_WAIT 必须给出 failure，否则无法解释为何在等待重试" + ) + return self._apply( + target, + Trigger.ADMIN_OVERRIDE, + actor=actor, + reason=reason, + enforce_table=False, + **changes, + ) + + # ------------------------------------------------------------------ + # 高频旁路更新 + # ------------------------------------------------------------------ + + def heartbeat(self, *, at: datetime | None = None) -> TaskSnapshot: + """刷新心跳时间。 + + ``at`` 允许调用方给出**真实的心跳时刻**（来自执行载体），而不是"现在"。 + 这个区别很关键：若一律填"现在"，一个卡死的任务也会显得心跳新鲜，``idle`` + 超时就永远不会触发——而那正是它存在的唯一理由。 + + 刻意**不**递增 ``status_version``：心跳是高频写入，若参与乐观并发，会和 + 状态写入持续冲突。存储层应将心跳与进度走各自的写入路径。 + """ + self._snapshot = replace(self._snapshot, last_heartbeat_at=at or self._clock()) + return self._snapshot + + def update_progress(self, progress: Progress) -> TaskSnapshot: + """更新进度。同样不改变状态，也不递增 ``status_version``。""" + self._snapshot = replace(self._snapshot, progress=progress) + return self._snapshot diff --git a/src/chongming_task/decorators.py b/src/chongming_task/decorators.py new file mode 100644 index 0000000..b94e8a0 --- /dev/null +++ b/src/chongming_task/decorators.py @@ -0,0 +1,97 @@ +"""``@task`` 装饰器：把任务元数据写在函数定义旁边。 + +```python +from chongming_task import RetryPolicy, task + +@task(tags={"io"}, retry=RetryPolicy(max_attempts=3), timeout=Timeout(per_attempt=60)) +def extract(ctx, source: str) -> list[Row]: + \"\"\"从 source 读取数据。\"\"\" + ... +``` + +然后一次性登记整个模块:: + + controller.register_module(myapp.tasks) + +## 为什么挂在函数上、而不是写进一个全局注册表 + +装饰器只把规格**附在函数对象上**（属性 ``__chongming_task__``），不产生任何全局状态。 +好处不只是"好测试"： + +* 谁注册、注册到哪个 :class:`~chongming_task.controller.TaskRegistry` 由调用方决定， + 同一个模块可以登记进多个互不干扰的控制器； +* 导入一个模块**不会**有副作用。全局注册表意味着 import 即注册，而 import 可能发生 + 在任何地方（包括测试收集阶段），那种隐式行为极难排查。 +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from dataclasses import dataclass, field +from typing import Any, TypeVar + +from .policies import RetryPolicy, Timeout + +__all__ = ["TASK_SPEC_ATTR", "TaskSpec", "get_task_spec", "task"] + +#: 规格挂在函数上的属性名。 +TASK_SPEC_ATTR = "__chongming_task__" + +_F = TypeVar("_F", bound=Callable[..., Any]) + + +@dataclass(frozen=True, slots=True) +class TaskSpec: + """``@task`` 声明的任务元数据。""" + + name: str | None = None + """注册名。省略时用函数名。""" + + description: str = "" + """描述。省略时取 docstring 的第一行。""" + + tags: frozenset[str] = field(default_factory=frozenset) + retry: RetryPolicy | None = None + timeout: Timeout | None = None + + +def task( + _func: _F | None = None, + *, + name: str | None = None, + description: str | None = None, + tags: Iterable[str] = (), + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, +) -> Any: + """声明一个任务。 + + 既可以带参数用（``@task(tags={"io"})``），也可以不带参数用（``@task``）——后者的 + 写法太自然，不支持会让人反复踩空。 + """ + + def decorate(func: _F) -> _F: + resolved_description = description + if resolved_description is None: + first_line = (func.__doc__ or "").strip().splitlines() + resolved_description = first_line[0].strip() if first_line else "" + spec = TaskSpec( + name=name if name is not None else getattr(func, "__name__", None), + description=resolved_description, + tags=frozenset(tags), + retry=retry, + timeout=timeout, + ) + setattr(func, TASK_SPEC_ATTR, spec) + return func + + if _func is not None: + # 裸 @task 的用法：参数直接就是被装饰的函数。 + return decorate(_func) + return decorate + + +def get_task_spec(target: Callable[..., Any]) -> TaskSpec | None: + """取函数上的任务规格；没被 ``@task`` 装饰过则返回 ``None``。""" + spec = getattr(target, TASK_SPEC_ATTR, None) + return spec if isinstance(spec, TaskSpec) else None diff --git a/src/chongming_task/enum.py b/src/chongming_task/enum.py new file mode 100644 index 0000000..7eda939 --- /dev/null +++ b/src/chongming_task/enum.py @@ -0,0 +1,34 @@ +"""兼容层（已弃用）。 + +早期版本在此定义 ``State``。状态模型已迁移至 :mod:`chongming_task.states`，并且必须 +使用语义明确的 ``TaskStatus``——``State`` / ``FINISHED`` 无法区分"成功结束"与 +"其它终止"，而这一区分是精细化控制的基础。 + +本模块仅为避免旧引用断裂而保留。新代码请直接:: + + from chongming_task.states import TaskStatus +""" + +from __future__ import annotations + +import warnings + +from .states import TaskStatus + +__all__ = ["TaskStatus"] + + +def __getattr__(name: str) -> object: + """在访问 ``State`` 时给出弃用提示（PEP 562）。 + + ``State`` 刻意不在此处静态定义：一旦写成模块级赋值，就会绕过本函数， + 弃用提示将永远不会触发。 + """ + if name == "State": + warnings.warn( + "chongming_task.enum.State 已弃用，请改用 chongming_task.states.TaskStatus", + DeprecationWarning, + stacklevel=2, + ) + return TaskStatus + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/src/chongming_task/errors.py b/src/chongming_task/errors.py new file mode 100644 index 0000000..67b37b8 --- /dev/null +++ b/src/chongming_task/errors.py @@ -0,0 +1,164 @@ +"""错误体系。 + +约定 +---- +* 用户代码与执行器抛出的异常必须被包装为 :class:`TaskExecutionError` 并保留 + ``__cause__``，绝不吞掉 traceback。 +* :class:`TaskCancelled` 是**控制流信号**而非错误，刻意继承 ``BaseException``。 +""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence + +from .models import TimeoutScope +from .states import TaskStatus, allowed_from + +__all__ = [ + "AttemptsExhausted", + "CircularDependency", + "ConcurrencyConflict", + "ConfigurationError", + "DependencyFailed", + "InvalidTransition", + "StoreError", + "TaskCancelled", + "TaskError", + "TaskExecutionError", + "TaskTimeout", +] + + +class TaskError(Exception): + """本项目所有错误的基类。 + + 注意它**不**包含 :class:`TaskCancelled`，后者是控制流信号。 + """ + + +class InvalidTransition(TaskError): + """请求了转移表中不允许的状态转移。 + + 刻意抛异常而不是返回 ``False``：静默失败会让"停止指令没生效"这类问题极难排查。 + 需要预检的调用方应使用 ``StateMachine.can_apply()``。 + """ + + def __init__( + self, + current: TaskStatus, + requested: TaskStatus, + *, + allowed: Iterable[TaskStatus] | None = None, + action: str | None = None, + ) -> None: + self.current = current + self.requested = requested + self.allowed = frozenset(allowed) if allowed is not None else allowed_from(current) + self.action = action + targets = "、".join(sorted(s.value for s in self.allowed)) or "无（终态）" + parts = [f"非法状态转移：{current.value} -> {requested.value}"] + if action is not None: + parts.append(f"动作 {action!r}") + parts.append(f"当前允许：{targets}") + super().__init__("；".join(parts)) + + +class DependencyFailed(TaskError): + """上游依赖未成功，本任务无法执行。""" + + def __init__(self, upstream_ids: Iterable[str], *, reason: str | None = None) -> None: + self.upstream_ids = tuple(upstream_ids) + self.reason = reason + detail = f"上游任务未成功：{', '.join(self.upstream_ids) or '(未提供)'}" + super().__init__(f"{detail}（{reason}）" if reason else detail) + + +class CircularDependency(TaskError): + """管道依赖图中存在环。""" + + def __init__(self, cycle: Sequence[str]) -> None: + self.cycle = tuple(cycle) + path = " -> ".join((*self.cycle, self.cycle[0])) if self.cycle else "(空环)" + super().__init__(f"管道中存在循环依赖：{path}") + + +class TaskCancelled(BaseException): + """协作式取消信号，由 ``TaskContext.checkpoint()`` 抛出。 + + 刻意继承 ``BaseException`` 而非 :class:`TaskError`：若继承 ``Exception``， + 用户代码里常见的 ``try/except Exception`` 会静默吞掉取消信号，任务将无法被 + 停止——这是协作式取消最典型的失效方式。``asyncio.CancelledError`` 采用同样的 + 处理方式。 + """ + + def __init__(self, reason: str | None = None, *, forced: bool = False) -> None: + self.reason = reason + self.forced = forced + """是否由强制手段（强杀/放弃等待）导致。""" + super().__init__(reason or "任务已被取消") + + +class TaskTimeout(TaskError): + """超出某个作用域的时限。""" + + def __init__(self, scope: TimeoutScope, limit: float, *, task_id: str | None = None) -> None: + self.scope = scope + self.limit = limit + self.task_id = task_id + where = f"任务 {task_id} " if task_id else "" + super().__init__(f"{where}超时（{scope.value}）：已超过 {limit} 秒") + + +class TaskExecutionError(TaskError): + """用户代码/执行器异常的统一包装。原异常保留在 ``__cause__``。""" + + def __init__( + self, + message: str, + *, + task_id: str | None = None, + attempt: int | None = None, + ) -> None: + self.task_id = task_id + self.attempt = attempt + super().__init__(message) + + +class AttemptsExhausted(TaskError): + """重试预算已耗尽，不允许再开始新的尝试。 + + 与 :class:`InvalidTransition` 区分：这不是"状态表不允许"，而是"状态表允许但 + 预算不够"。若混为一谈，运维看到报错会去查转移表，方向完全错了。 + """ + + def __init__(self, task_id: str, attempt: int, max_attempts: int) -> None: + self.task_id = task_id + self.attempt = attempt + self.max_attempts = max_attempts + super().__init__( + f"任务 {task_id} 重试预算已耗尽：已完成 {attempt} 次尝试，上限 {max_attempts}" + ) + + +class ConcurrencyConflict(TaskError): + """乐观并发校验失败：快照在本次写入前已被他处修改。 + + 每个任务的写入者应当唯一（调度器）。出现本错误通常意味着存在第二个写入者， + 是设计缺陷的信号，因此**不做自动重试掩盖**。 + """ + + def __init__(self, task_id: str, expected_version: int, actual_version: int) -> None: + self.task_id = task_id + self.expected_version = expected_version + self.actual_version = actual_version + super().__init__( + f"任务 {task_id} 并发写冲突：期望版本 {expected_version}，实际 {actual_version}" + ) + + +class StoreError(TaskError): + """存储层故障。""" + + +class ConfigurationError(TaskError): + """配置错误，如循环依赖、无效策略组合、执行器名未知。""" diff --git a/src/chongming_task/models.py b/src/chongming_task/models.py new file mode 100644 index 0000000..1e88837 --- /dev/null +++ b/src/chongming_task/models.py @@ -0,0 +1,316 @@ +"""任务快照与正交维度模型。 + +:class:`TaskSnapshot` 既是状态机的输入/输出，也是持久化的单元。它把生命周期 +(``status``) 与其余**正交维度**分开承载：把"重试中""已超时""进度 50%"塞进同一个 +枚举是同类工具最常见的建模错误，也是"精细化控制"无法落地的主因。 +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from dataclasses import dataclass, field +from datetime import datetime +from enum import StrEnum +from typing import Any, Final + +from .states import REVIVABLE, TERMINAL, TaskStatus + +__all__ = [ + "AttemptRecord", + "Failure", + "FailureKind", + "Progress", + "RequestedAction", + "TaskSnapshot", + "TimeoutScope", + "describe_serialization_problem", +] + + +class RequestedAction(StrEnum): + """已受理但尚未完成的控制请求。 + + 与 ``status`` 分开承载：``stop()`` / ``pause()`` 是**请求**而非瞬时赋值， + 运行中的任务需要时间响应，UI 依赖本字段渲染"停止中 / 暂停中"。 + """ + + STOP = "stop" + PAUSE = "pause" + + +class TimeoutScope(StrEnum): + """超时作用域。超时是**失败原因**，不是生命周期状态。""" + + START = "start" + """从提交到真正开始执行的等待时长。""" + ATTEMPT = "attempt" + """单次尝试的总时长。""" + IDLE = "idle" + """基于心跳的静默超时，用于发现死锁/卡死——这是唯一能发现"没有报错但不动了"的机制。""" + + +class FailureKind(StrEnum): + """失败或未正常完成的归类。""" + + EXCEPTION = "exception" + NONZERO_EXIT = "nonzero_exit" + TIMEOUT_START = "timeout_start" + TIMEOUT_ATTEMPT = "timeout_attempt" + TIMEOUT_IDLE = "timeout_idle" + CANCELLED = "cancelled" + LOST = "lost" + DEPENDENCY_FAILED = "dependency_failed" + CONDITION_FALSE = "condition_false" + RESULT_INVALID = "result_invalid" + """任务违反了框架契约，例如返回值不可持久化。 + + 与 ``EXCEPTION`` 分开是有意的：一个是"任务自己出错了"，一个是"任务的产出框架 + 接不住"。报表与告警需要能分开——它们的修法完全不同（前者查逻辑，后者改接口）。 + """ + + @classmethod + def for_timeout(cls, scope: TimeoutScope) -> FailureKind: + """由超时作用域得到对应归类，避免在两处重复书写映射。""" + return cls(f"timeout_{scope.value}") + + +#: 一次返回值编码尝试中使用的 JSON 选项。 +#: +#: 必须与存储层编码快照时**完全一致**，否则校验通过、写库仍可能失败。 +_JSON_OPTIONS: Final[dict[str, Any]] = {"ensure_ascii": False, "sort_keys": True} + + +def describe_serialization_problem(value: Any) -> str | None: + """检查一个值能否被持久化：可以返回 ``None``，否则返回原因。 + + 为什么需要**在写库之前**检查：持久化契约要求返回值 JSON 可序列化（存储层要把快照 + 编码成 JSON）。若不检查，失败会发生在**写穿路径**上——而那条路径的失败只被记录下来、 + 不会让任务失败。结果是进程内显示 ``succeeded``、存储里停在 ``running``，随后 + ``reconcile()`` 再把一个真正成功过的任务判成 ``LOST``。 + + 调用方应当把"不可序列化"当成**任务失败**：违反契约是任务的问题，不是存储的问题。 + """ + try: + # 与 codec 一样把值包在 outcome 里编码，这样键类型、排序等问题也能被发现。 + json.dumps({"value": value}, **_JSON_OPTIONS) + except (TypeError, ValueError) as exc: + return f"{type(value).__name__} 不可 JSON 序列化：{exc}" + return None + + +@dataclass(frozen=True, slots=True) +class Progress: + """进度。与 ``status`` 正交：``RUNNING`` 的任务可能进度为 0，也可能已达 99%。""" + + done: int = 0 + total: int | None = None + unit: str | None = None + message: str | None = None + updated_at: datetime | None = None + + def __post_init__(self) -> None: + if self.done < 0: + raise ValueError(f"progress.done 不能为负：{self.done}") + if self.total is not None and self.total < 0: + raise ValueError(f"progress.total 不能为负：{self.total}") + if self.total is not None and self.done > self.total: + raise ValueError(f"progress.done({self.done}) 不能超过 total({self.total})") + + @property + def fraction(self) -> float | None: + """完成比例。 + + ``total`` 未知或为 0 时返回 ``None`` 而非 0.0——0.0 会被渲染成"尚未开始"， + 而实际情况是"无法估计"，两者必须区分。 + """ + if self.total is None or self.total == 0: + return None + return self.done / self.total + + +@dataclass(frozen=True, slots=True) +class Failure: + """一次失败的结构化描述。""" + + kind: FailureKind + message: str + traceback: str | None = None + """完整 traceback。用户代码的异常必须保留 ``__cause__`` 并在此完整记录。""" + exit_code: int | None = None + """执行载体的退出码。**只有进程执行器会写它**（子进程非零退出、或被信号终止）。 + + 同进程执行器为 ``None``：任务出错走的是异常，没有"退出码"这回事。它存在的价值 + 是把"子进程为什么没了"从一句猜不出的话变成一个有据可查的事实——配合 + :attr:`TaskSnapshot.pid`，运维可以 `ps` 对上号。 + """ + retryable: bool = True + """是否值得重试。由策略与错误类型共同决定。""" + + +@dataclass(frozen=True, slots=True) +class AttemptRecord: + """单次尝试的不可变历史记录。 + + 注意"只追加"：重试不会修改既有记录。``TaskSnapshot.attempt`` 始终等于最新 + 尝试的序号。 + + .. note:: + **当前运行时还不产出它**（没有任何代码路径构造 ``AttemptRecord``）：一次尝试 + 的完整明细今天由迁移历史承载——``starting`` / ``retry_wait`` 这些迁移本身就带 + 版本号、时间戳、触发源与原因，看板与 ``history`` 都读它。这个类型留给"把逐次 + 尝试单独持久化"的将来（例如按尝试维度的耗时统计），**不要**在没有写入方之前 + 把它当成可依赖的接口。 + """ + + task_id: str + attempt: int + """从 1 开始的尝试序号。""" + started_at: datetime + ended_at: datetime | None = None + status: TaskStatus = TaskStatus.RUNNING + failure: Failure | None = None + executor: str | None = None + + def __post_init__(self) -> None: + if self.attempt < 1: + raise ValueError(f"attempt 从 1 开始，收到 {self.attempt}") + + +@dataclass(frozen=True, slots=True) +class TaskSnapshot: + """某一时刻任务的完整状态。 + + 不可变。任何状态变更都由状态机产生**新的**快照并递增 ``status_version``， + 因此快照可以安全地跨线程/协程传递，也可以直接作为持久化记录。 + """ + + # ---- 身份 ---- + task_id: str + name: str + pipeline_run_id: str | None = None + tags: frozenset[str] = field(default_factory=frozenset) + + # ---- 分桶 ---- + bucket: int | None = None + """所属桶号。由 :class:`~chongming_task.policies.BucketPolicy` 从 ``bucket_key`` 算出。 + + 它既是本地并发隔离的单位，也是将来分布式分配的单位（一台机器拥有一组桶）。 + 任务一旦创建就把桶**记下来**，而不是每次按需重算：万一哈希策略或桶数发生变化， + 重算会把一个正在跑的任务挪到别的桶，局部性与隔离假设随之悄悄失效。 + """ + bucket_key: str | None = None + """算出桶号所依据的业务键（如租户 id、数据源、日期分区）。 + + 留下它是为了**迁移**：将来若必须改动桶数，只有拿得到原始键才能重新分桶。 + """ + + # ---- 生命周期 ---- + status: TaskStatus = TaskStatus.PENDING + status_version: int = 0 + """乐观并发版本号。每次成功转移 +1；存储层用它检测并发写入冲突。""" + requested_action: RequestedAction | None = None + """已受理但尚未完成的请求，用于区分"正在停止"与"已停止"。""" + + # ---- 尝试 ---- + attempt: int = 0 + """已开始的尝试次数；0 表示尚未执行过。""" + max_attempts: int = 1 + + # ---- 时间 ---- + created_at: datetime | None = None + queued_at: datetime | None = None + started_at: datetime | None = None + ended_at: datetime | None = None + last_heartbeat_at: datetime | None = None + """最近一次心跳。超过 idle 超时即判定为卡死或失联。""" + + # ---- 结果 ---- + progress: Progress | None = None + failure: Failure | None = None + outcome: Mapping[str, Any] | None = None + + # ---- 执行载体 ---- + executor: str | None = None + """执行器标识：``inline``（同步、测试用）、``thread``（默认）或 ``process``。""" + pid: int | None = None + """执行载体的操作系统进程号。**只有进程执行器会写它**：线程与内联执行器下任务 + 就跑在监督进程里，再存一个 pid 只是重复（``lease_owner`` 已经带主机与进程信息）。 + + 有了它，事后排障能直接对上号：任务卡住时可以先 `ps` 看这个进程还在不在、在干什么。""" + + # ---- 监督者租约 ---- + lease_owner: str | None = None + """当前**监督**这个任务的进程标识。 + + 与 ``last_heartbeat_at`` 是两件不同的事，不能合并： + + * ``last_heartbeat_at`` 由任务自己的 ``checkpoint()`` 驱动，回答"**任务在推进吗**"。 + 它停滞意味着卡死——这是发现死锁的唯一信号。 + * ``lease_owner`` / ``lease_until`` 由监督进程定期续租，回答"**还有人在管它吗**"。 + + 若拿任务心跳去判断"进程是否还活着"，一个长时间不调 ``checkpoint()`` 的健康任务 + 会被误判为失联而被判 ``LOST``——那是把一个正常任务杀掉。反过来拿租约去判断卡死 + 也不行。所以两个信号必须分开。 + """ + lease_until: datetime | None = None + """租约到期时刻。过期即"没有监督者"，重启恢复据此判定僵尸任务。""" + + # ---- 人工干预 ---- + overridden: bool = False + """是否经 ``force_transition()`` 人工改写过。此类任务在报表中应被单独标出。""" + + def __post_init__(self) -> None: + if not self.task_id: + raise ValueError("task_id 不能为空") + if not self.name: + raise ValueError("name 不能为空") + if self.status_version < 0: + raise ValueError(f"status_version 不能为负：{self.status_version}") + if self.attempt < 0: + raise ValueError(f"attempt 不能为负：{self.attempt}") + if self.max_attempts < 1: + raise ValueError(f"max_attempts 至少为 1，收到 {self.max_attempts}") + if self.attempt > self.max_attempts: + raise ValueError(f"attempt({self.attempt}) 不能超过 max_attempts({self.max_attempts})") + if self.bucket is not None and self.bucket < 0: + raise ValueError(f"bucket 不能为负：{self.bucket}") + if self.bucket is not None and self.bucket_key is None: + raise ValueError("给出了 bucket 却没有 bucket_key，将无法在改桶数时重新分桶") + if self.lease_until is not None and self.lease_owner is None: + # 没有主人的租约无法追溯"是谁在管"，也无法在接管时判断该不该抢。 + raise ValueError("给出了 lease_until 却没有 lease_owner") + + @property + def is_terminal(self) -> bool: + """是否已进入终态。""" + return self.status in TERMINAL + + @property + def is_revivable(self) -> bool: + """是否还能经 ``retry()`` 重新执行。""" + return self.status in REVIVABLE + + def validate(self) -> tuple[str, ...]: + """返回跨字段一致性问题清单，空元组表示一致。 + + 刻意**不**在 ``__post_init__`` 中强制这些规则：它们涉及时间线完整性， + 在逐步构造快照时（例如先建终态再补 ``ended_at``）会造成无谓摩擦。 + 持久化层与恢复流程应在写入/加载时调用本方法。 + """ + problems: list[str] = [] + if self.is_terminal and self.ended_at is None: + problems.append(f"终态 {self.status.value} 必须带有 ended_at") + if not self.is_terminal and self.ended_at is not None: + problems.append(f"非终态 {self.status.value} 不应带有 ended_at") + if self.status is TaskStatus.RUNNING and self.started_at is None: + problems.append("RUNNING 状态必须带有 started_at") + if self.status is TaskStatus.SUCCEEDED and self.failure is not None: + problems.append("SUCCEEDED 不应带有 failure") + if self.status is TaskStatus.RETRY_WAIT and self.failure is None: + problems.append("RETRY_WAIT 必须带有上次失败原因，否则无法解释为何在等待重试") + if self.overridden and self.requested_action is not None: + problems.append("已被人工改写的任务不应残留 requested_action") + if self.requested_action is not None and self.is_terminal: + problems.append(f"终态 {self.status.value} 不应残留 requested_action") + return tuple(problems) diff --git a/src/chongming_task/pipeline.py b/src/chongming_task/pipeline.py new file mode 100644 index 0000000..e65b9a2 --- /dev/null +++ b/src/chongming_task/pipeline.py @@ -0,0 +1,589 @@ +"""DAG 管道：把多个任务按依赖关系串起来，由一个调度器推进。 + +状态模型早就为管道准备好了（``PENDING → BLOCKED → QUEUED``、``SKIPPED``、 +``CANCELLED``），这里补的是**谁来驱动**： + +* :class:`Pipeline` —— 静态定义：节点、依赖、汇合语义、参数； +* :class:`PipelineRun` —— 一次运行：创建节点任务，监听每个节点落终态，放行满足条件的 + 下游，跳过注定跑不了的下游，按结算策略收场。 + +## 两个不同的轴，别混 + +| 概念 | 回答的问题 | +|---|---| +| :class:`~chongming_task.policies.JoinMode`（逐节点） | **这一个节点何时该跑** | +| :class:`~chongming_task.policies.AggregationPolicy`（整条管道） | **管道何时结束、是否连坐** | + +## 一个刻意的限制 + +管道的**结构在内存里**，不落库：任务的函数与参数从未持久化（理由同 :meth:`Controller.reconcile` +的说明）。因此进程重启后，一条跑到一半的管道无法续跑——它的节点会被 ``reconcile`` +如实标记，而不是被假装恢复。 +""" + +from __future__ import annotations + +import threading +import uuid +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field +from enum import StrEnum +from typing import TYPE_CHECKING, Any + +from .errors import ConfigurationError +from .policies import AggregationMode, AggregationPolicy, JoinMode, RetryPolicy, Timeout +from .states import TERMINAL, TaskStatus + +if TYPE_CHECKING: # pragma: no cover - 仅为类型检查 + from .controller import Controller + from .run import TaskRun + +__all__ = [ + "Node", + "Pipeline", + "PipelineOutcome", + "PipelineRun", + "PipelineStatus", + "render_pipeline", +] + + +@dataclass(frozen=True, slots=True) +class Node: + """管道里的一个节点：一个任务 + 它的上游 + 汇合语义。""" + + node_id: str + """节点在本管道内的唯一标识，也是 ``task_id`` 的后半段。""" + + task: str + """在 :class:`~chongming_task.controller.TaskRegistry` 里注册过的任务名。""" + + args: tuple[Any, ...] = () + kwargs: Mapping[str, Any] = field(default_factory=dict) + depends_on: tuple[str, ...] = () + join: JoinMode = JoinMode.ALL_SUCCEEDED + label: str | None = None + bucket_key: str | None = None + tags: frozenset[str] = frozenset() + retry: RetryPolicy | None = None + timeout: Timeout | None = None + + +class Pipeline: + """一条管道的**静态定义**。可重复提交，每次运行互不影响。""" + + def __init__( + self, + name: str, + *, + aggregation: AggregationPolicy | None = None, + ) -> None: + if not name: + raise ConfigurationError("管道必须有名字") + self.name = name + self.aggregation = aggregation if aggregation is not None else AggregationPolicy.fail_fast() + self._nodes: list[Node] = [] + self._by_id: dict[str, Node] = {} + self._counts: dict[str, int] = {} + + # ------------------------------------------------------------------ + # 定义 + # ------------------------------------------------------------------ + + def node( + self, + task: str, + *args: Any, + depends_on: Iterable[Node | str] = (), + join: JoinMode = JoinMode.ALL_SUCCEEDED, + node_id: str | None = None, + label: str | None = None, + bucket_key: str | None = None, + tags: Iterable[str] = (), + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, + **kwargs: Any, + ) -> Node: + """追加一个节点并返回它，供下游用 ``depends_on=[它]`` 引用。 + + 第一个参数是**已注册的任务名**，其后是传给任务的位置参数——节点 id 由 + ``node_id`` 关键字指定，因此不会被误读成第二个位置参数（写成 ``name=`` 时 + 很容易误以为 ``p.node("mark", "after")`` 里的 ``"after"`` 是节点名，其实它是 + 传给 ``mark`` 的参数）。 + + ``node_id`` 省略时取任务名；同一任务名出现多次时自动加 ``#2`` / ``#3`` + 后缀——既不强迫调用方给每个节点起名，也不会撞车。 + """ + if not task: + raise ConfigurationError("节点必须给出已注册的任务名") + resolved_id = node_id if node_id is not None else self._next_id(task) + if resolved_id in self._by_id: + raise ConfigurationError(f"节点 id 重复：{resolved_id}") + + node = Node( + node_id=resolved_id, + task=task, + args=tuple(args), + kwargs=dict(kwargs), + depends_on=tuple(_as_node_id(item) for item in depends_on), + join=join, + label=label, + bucket_key=bucket_key, + tags=frozenset(tags), + retry=retry, + timeout=timeout, + ) + self._nodes.append(node) + self._by_id[resolved_id] = node + return node + + def _next_id(self, task: str) -> str: + self._counts[task] = self._counts.get(task, 0) + 1 + index = self._counts[task] + return task if index == 1 else f"{task}#{index}" + + @property + def nodes(self) -> tuple[Node, ...]: + """按定义顺序返回全部节点。""" + return tuple(self._nodes) + + def __len__(self) -> int: + return len(self._nodes) + + def node_by_id(self, node_id: str) -> Node: + """按 id 取节点；不存在即抛错。""" + try: + return self._by_id[node_id] + except KeyError: + raise ConfigurationError(f"管道 {self.name!r} 里没有节点 {node_id!r}") from None + + # ------------------------------------------------------------------ + # 校验 + # ------------------------------------------------------------------ + + def validate(self) -> None: + """定义期的静态校验。**在创建任何任务之前**调用，避免留下半条管道。""" + if not self._nodes: + raise ConfigurationError(f"管道 {self.name!r} 没有任何节点") + + if self.aggregation.mode is AggregationMode.QUORUM: + # 诚实拒绝而不是发明语义：quorum 是"同一件事跑 N 份、成功 k 份即可"， + # 它属于**一组同质节点**，而不是一条节点各异、依赖交错的 DAG。 + # 硬套到 DAG 上只能得到一个没人能预测含义的数字。 + raise ConfigurationError( + f"管道 {self.name!r} 不支持 quorum 结算：quorum 描述的是" + "「同一件事的 N 份副本成功 k 份」，它属于扇出分组而不是整条 DAG。" + "请改用 fail_fast 或 all_settled。" + ) + + for node in self._nodes: + for dependency in node.depends_on: + if dependency not in self._by_id: + raise ConfigurationError( + f"节点 {node.node_id!r} 依赖了不存在的节点 {dependency!r}" + ) + if node.node_id in node.depends_on: + raise ConfigurationError(f"节点 {node.node_id!r} 依赖了自己") + + self._check_acyclic() + + def _check_acyclic(self) -> None: + """DFS 找环。 + + 必须显式检测：有环时调度器只会安静地什么也不做（所有节点都在等对方）， + 表现为"管道卡住了"而不是"定义错了"——那种故障最难查。 + """ + state: dict[str, int] = {} # 0=未访问 1=访问中 2=已完成 + path: list[str] = [] + + def visit(node_id: str) -> None: + if state.get(node_id) == 2: + return + if state.get(node_id) == 1: + cycle = " -> ".join([*path[path.index(node_id) :], node_id]) + raise ConfigurationError(f"管道 {self.name!r} 存在环形依赖：{cycle}") + state[node_id] = 1 + path.append(node_id) + for dependency in self._by_id[node_id].depends_on: + visit(dependency) + path.pop() + state[node_id] = 2 + + for node in self._nodes: + visit(node.node_id) + + def topological_order(self) -> tuple[Node, ...]: + """拓扑序。仅用于展示与调试——真正的推进由运行时状态决定。""" + order: list[Node] = [] + seen: set[str] = set() + + def visit(node_id: str) -> None: + if node_id in seen: + return + seen.add(node_id) + for dependency in self._by_id[node_id].depends_on: + visit(dependency) + order.append(self._by_id[node_id]) + + for node in self._nodes: + visit(node.node_id) + return tuple(order) + + +class PipelineStatus(StrEnum): + """整条管道的状态。""" + + RUNNING = "running" + SUCCEEDED = "succeeded" + FAILED = "failed" + CANCELLED = "cancelled" + + +@dataclass(frozen=True, slots=True) +class PipelineOutcome: + """一次管道运行的结论。""" + + run_id: str + status: PipelineStatus + nodes: Mapping[str, TaskStatus] + failure: str | None = None + + @property + def ok(self) -> bool: + """是否成功。""" + return self.status is PipelineStatus.SUCCEEDED + + +#: 节点落终态后算"不成功"的状态。 +_BAD_TERMINALS = frozenset( + { + TaskStatus.FAILED, + TaskStatus.LOST, + TaskStatus.CANCELLED, + TaskStatus.STOPPED, + } +) + +#: 会触发 fail_fast 连坐的结局。 +#: +#: **``CANCELLED`` 刻意不在其中**：它是连坐的**结果**，不是原因。把它也算作触发条件 +#: 会同时造成两个错误——用户主动取消整条管道会被记成"失败"；被连坐取消的节点再触发 +#: 一轮连坐，因果搅成一团。 +_FAIL_FAST_TRIGGERS = frozenset({TaskStatus.FAILED, TaskStatus.LOST, TaskStatus.STOPPED}) + + +class PipelineRun: + """一次管道运行。由 :meth:`Controller.submit_pipeline` 创建。""" + + def __init__( + self, + controller: Controller, + pipeline: Pipeline, + run_id: str, + task_ids: Mapping[str, str], + ) -> None: + self._controller = controller + self.pipeline = pipeline + self.run_id = run_id + self._task_ids = dict(task_ids) + self._runs: dict[str, TaskRun] = {} + self._released: set[str] = set() + self._settled = threading.Event() + self._lock = threading.RLock() + self._failure: str | None = None + self._cancelled = False + + # ------------------------------------------------------------------ + # 只读视图 + # ------------------------------------------------------------------ + + def __repr__(self) -> str: + return f"" + + @property + def task_ids(self) -> Mapping[str, str]: + """节点 id → task_id。""" + return dict(self._task_ids) + + def task_id_of(self, node_id: str) -> str: + """某个节点对应的 task_id。""" + return self._task_ids[node_id] + + def node_status(self, node_id: str) -> TaskStatus: + """某个节点当前的执行状态。""" + run = self._runs.get(node_id) + if run is not None: + return run.status + snapshot = self._controller.get(self._task_ids[node_id]) + return snapshot.status if snapshot is not None else TaskStatus.PENDING + + @property + def status(self) -> PipelineStatus: + """整条管道的状态，由各节点状态与结算策略推导。 + + **先看是否还有节点没落终态。** 顺序反了会出真问题：fail_fast 触发时 ``_failure`` + 已经被记下，但被连坐的节点还在收尾；此时若直接返回 ``FAILED``，等待者会被提前 + 唤醒，看到一个"已经失败、但还有任务在跑"的矛盾状态。 + """ + with self._lock: + statuses = [self.node_status(node.node_id) for node in self.pipeline.nodes] + if any(status not in TERMINAL for status in statuses): + return PipelineStatus.RUNNING + if self._failure is not None: + return PipelineStatus.FAILED + if self._cancelled: + return PipelineStatus.CANCELLED + if any(status in _BAD_TERMINALS for status in statuses): + return PipelineStatus.FAILED + if all(status is TaskStatus.SKIPPED for status in statuses): + # 一个都没跑：既不是成功也不是失败，但管道确实结束了。 + return PipelineStatus.CANCELLED + return PipelineStatus.SUCCEEDED + + def outcome(self) -> PipelineOutcome: + """当前结论。""" + return PipelineOutcome( + run_id=self.run_id, + status=self.status, + nodes={node.node_id: self.node_status(node.node_id) for node in self.pipeline.nodes}, + failure=self._failure, + ) + + def wait(self, timeout: float | None = None) -> PipelineOutcome: + """等到整条管道收场（或超时），返回结论。 + + 超时**不改变任何任务状态**——"我没等到"不等于"它失败了"。 + """ + self._settled.wait(timeout) + return self.outcome() + + @property + def settled(self) -> bool: + """是否已有结论。""" + return self._settled.is_set() + + # ------------------------------------------------------------------ + # 控制 + # ------------------------------------------------------------------ + + def cancel(self, *, reason: str | None = None) -> int: + """取消整条管道：所有未落终态的节点都落 ``CANCELLED``。返回处理了几个。""" + with self._lock: + self._cancelled = True + count = 0 + for node in self.pipeline.nodes: + run = self._runs.get(node.node_id) + if run is None or run.snapshot.is_terminal: + continue + run.cancel(reason=reason or f"管道 {self.pipeline.name} 被取消") + count += 1 + self._settle() + return count + + # ------------------------------------------------------------------ + # 内部：由 Controller 驱动 + # ------------------------------------------------------------------ + + def _register(self, node_id: str, run: TaskRun) -> None: + self._runs[node_id] = run + if not self.pipeline.node_by_id(node_id).depends_on: + with self._lock: + self._released.add(node_id) + + def _on_task_finished(self, task_id: str) -> None: + """某个节点落终态：先看要不要连坐，再放行/跳过下游。""" + node_id = self._node_id_of(task_id) + if node_id is None: + return + + run = self._runs.get(node_id) + if run is not None and run.status in _FAIL_FAST_TRIGGERS: + self._maybe_fail_fast(node_id, run.status) + + self._evaluate() + + def _node_id_of(self, task_id: str) -> str | None: + for node_id, candidate in self._task_ids.items(): + if candidate == task_id: + return node_id + return None + + def _maybe_fail_fast(self, node_id: str, status: TaskStatus) -> None: + """首个真失败即取消其余**已经放行**的节点。 + + 只取消"已经放行"的（排队中 / 正在跑）：**等待上游的节点交给** :meth:`_evaluate` + 去 ``skip``。两者的语义不同——``cancelled`` 是"本来要跑，被连坐了"， + ``skipped`` 是"上游没成功，它本来就不该跑"。把后者也标成 cancelled，事后就 + 分不清哪些是真被中断的工作。 + """ + if self.pipeline.aggregation.mode is not AggregationMode.FAIL_FAST: + return + if status not in _FAIL_FAST_TRIGGERS: + return + with self._lock: + if self._failure is not None or self._cancelled: + return + released = set(self._released) + self._failure = f"节点 {node_id} 落 {status.value}，按 fail_fast 取消其余任务" + reason = self._failure + for node in self.pipeline.nodes: + if node.node_id not in released: + continue + run = self._runs.get(node.node_id) + if run is not None and not run.snapshot.is_terminal: + run.cancel(reason=reason) + + def _evaluate(self) -> None: + """反复放行可跑的下游、跳过注定跑不了的下游，直到没有变化。""" + while True: + statuses = { + node.node_id: self.node_status(node.node_id) for node in self.pipeline.nodes + } + with self._lock: + decisions: list[tuple[Node, bool, str]] = [] + for node in self.pipeline.nodes: + if node.node_id in self._released: + continue + verdict = _decide(node, statuses) + if verdict is None: + continue + self._released.add(node.node_id) + decisions.append((node, verdict[0], verdict[1])) + if not decisions: + break + for node, runnable, detail in decisions: + run = self._runs[node.node_id] + if runnable: + # 先注入上游返回值再放行——任务一旦开始执行就注入不了了。 + run.set_inputs(self._upstream_values(node)) + self._controller._release(run) + else: + run.skip(reason=detail) + + self._settle() + + def _upstream_values(self, node: Node) -> dict[str, Any]: + """收集某个节点全部**成功**上游的返回值。 + + 只有成功的上游才有值：失败或跳过的上游没有返回值，塞一个 ``None`` 进去会让 + 下游以为"上游返回了 None"，而不是"上游没有值"。下游用 + ``ctx.input(节点id, 默认值)`` 取即可。 + """ + values: dict[str, Any] = {} + for dependency in node.depends_on: + upstream = self._runs.get(dependency) + if upstream is None: + continue + result = upstream.result + if result is not None and result.ok: + values[dependency] = result.value + return values + + def _settle(self) -> None: + """如果已经有结论，唤醒等待者。""" + if self.status is not PipelineStatus.RUNNING: + self._settled.set() + + +def _decide(node: Node, statuses: Mapping[str, TaskStatus]) -> tuple[bool, str] | None: + """判断一个节点现在该放行、该跳过，还是继续等。 + + 返回 ``None`` 表示"还不到时候"。返回 ``(True, "")`` 放行，``(False, 原因)`` 跳过。 + """ + if not node.depends_on: + return (True, "") + + upstream = [statuses[dependency] for dependency in node.depends_on] + succeeded = sum(1 for status in upstream if status is TaskStatus.SUCCEEDED) + settled = all(status in TERMINAL for status in upstream) + + if node.join is JoinMode.ALL_SUCCEEDED: + if succeeded == len(upstream): + return (True, "") + if settled: + failed = [ + dependency + for dependency in node.depends_on + if statuses[dependency] is not TaskStatus.SUCCEEDED + ] + return (False, f"上游未成功：{', '.join(failed)}") + return None + + if node.join is JoinMode.ANY_SUCCEEDED: + if succeeded: + return (True, "") + if settled: + return (False, "全部上游都未成功") + return None + + # ALL_SETTLED：等所有上游落终态，不管成败。 + if settled: + return (True, "") + return None + + +def _as_node_id(item: Node | str) -> str: + return item.node_id if isinstance(item, Node) else item + + +def render_pipeline(run: PipelineRun) -> str: + """把一次管道运行渲染成表格（CLI 与示例共用）。""" + rows: list[tuple[str, ...]] = [("NODE", "TASK", "STATUS", "DEPENDS_ON", "JOIN")] + for node in run.pipeline.topological_order(): + upstream = ",".join(node.depends_on) if node.depends_on else "-" + rows.append( + ( + node.node_id, + node.task, + run.node_status(node.node_id).value, + upstream, + node.join.value, + ) + ) + widths = [max(len(row[index]) for row in rows) for index in range(len(rows[0]))] + lines = [ + " ".join(cell.ljust(width) for cell, width in zip(row, widths, strict=True)).rstrip() + for row in rows + ] + lines.insert(1, " ".join("-" * width for width in widths)) + return "\n".join(lines) + + +def start_pipeline( + controller: Controller, + pipeline: Pipeline, + *, + pipeline_run_id: str | None = None, +) -> PipelineRun: + """创建一条管道的全部节点任务，并返回可等待的运行句柄。 + + 先做**全部**校验再创建任务：否则一个写错的 DAG 会在存储里留下一堆半成品任务， + 而它们看起来就像正常提交的一样。 + """ + pipeline.validate() + for node in pipeline.nodes: + # 未注册即抛 ConfigurationError——控制面只接受任务名，绝不接受代码。 + controller.registry.get(node.task) + + run_id = pipeline_run_id if pipeline_run_id is not None else uuid.uuid4().hex[:12] + task_ids = {node.node_id: f"{run_id}:{node.node_id}" for node in pipeline.nodes} + run = PipelineRun(controller, pipeline, run_id, task_ids) + + for node in pipeline.topological_order(): + task_run = controller.submit( + node.task, + *node.args, + task_id=task_ids[node.node_id], + label=node.label if node.label is not None else f"{pipeline.name}/{node.node_id}", + tags=node.tags, + bucket_key=node.bucket_key, + retry=node.retry, + timeout=node.timeout, + pipeline_run_id=run_id, + deferred=bool(node.depends_on), + **dict(node.kwargs), + ) + run._register(node.node_id, task_run) + + controller._watch_pipeline(run) + return run diff --git a/src/chongming_task/policies.py b/src/chongming_task/policies.py new file mode 100644 index 0000000..8fedf31 --- /dev/null +++ b/src/chongming_task/policies.py @@ -0,0 +1,315 @@ +"""重试、超时、并发与聚合策略。 + +所有策略对象都不可变，可安全共享、复用并作为配置跨进程传递。 + +一个刻意的选择：``RetryPolicy.retry_on`` 默认为 ``(Exception,)``， +而 :class:`chongming_task.errors.TaskCancelled` 继承自 ``BaseException``， +因此**取消永远不会被当作可重试的失败**——这是协作式取消能否生效的关键。 +""" + +from __future__ import annotations + +import hashlib +import random +from collections.abc import Mapping +from dataclasses import dataclass, field +from enum import StrEnum +from typing import Protocol, runtime_checkable + +__all__ = [ + "AggregationMode", + "AggregationPolicy", + "Backoff", + "BucketPolicy", + "ConcurrencyPolicy", + "ExponentialBackoff", + "FixedBackoff", + "Jitter", + "JoinMode", + "RetryPolicy", + "Timeout", +] + + +class Jitter(StrEnum): + """退避抖动策略。默认加抖动，否则大量同时失败的任务会同步重试形成风暴。""" + + NONE = "none" + FULL = "full" + """在 ``[0, delay)`` 内均匀取值。""" + EQUAL = "equal" + """在 ``[delay/2, delay)`` 内均匀取值，保留下限。""" + + +@runtime_checkable +class Backoff(Protocol): + """退避算法。``attempt`` 从 1 开始。""" + + def delay_for(self, attempt: int, *, rng: random.Random | None = None) -> float: + """第 ``attempt`` 次尝试失败后应等待的秒数。""" + ... + + +@dataclass(frozen=True, slots=True) +class FixedBackoff: + """固定间隔退避。""" + + delay: float = 1.0 + + def __post_init__(self) -> None: + if self.delay < 0: + raise ValueError(f"delay 不能为负：{self.delay}") + + def delay_for(self, attempt: int, *, rng: random.Random | None = None) -> float: + del rng # 固定退避无需随机源 + return self.delay + + +@dataclass(frozen=True, slots=True) +class ExponentialBackoff: + """指数退避：``min(base * factor ** (attempt - 1), max_delay)``，再按需施加抖动。""" + + base: float = 1.0 + factor: float = 2.0 + max_delay: float = 300.0 + jitter: Jitter = Jitter.FULL + + def __post_init__(self) -> None: + if self.base < 0: + raise ValueError(f"base 不能为负：{self.base}") + if self.factor < 1: + raise ValueError(f"factor 至少为 1，收到 {self.factor}") + if self.max_delay < 0: + raise ValueError(f"max_delay 不能为负：{self.max_delay}") + + def delay_for(self, attempt: int, *, rng: random.Random | None = None) -> float: + if self.base == 0: + return 0.0 + raw = min(self.base * self.factor ** max(attempt - 1, 0), self.max_delay) + if self.jitter is Jitter.NONE: + return raw + source = rng if rng is not None else random + draw = source.random() + if self.jitter is Jitter.FULL: + return raw * draw + return raw * (0.5 + 0.5 * draw) + + +@dataclass(frozen=True, slots=True) +class RetryPolicy: + """重试策略。 + + ``max_attempts`` 是**总尝试次数**（含首次），因此 ``1`` 表示不重试。 + """ + + max_attempts: int = 1 + backoff: Backoff = field(default_factory=ExponentialBackoff) + retry_on: tuple[type[BaseException], ...] = (Exception,) + give_up_on: tuple[type[BaseException], ...] = () + + def __post_init__(self) -> None: + if self.max_attempts < 1: + raise ValueError(f"max_attempts 至少为 1，收到 {self.max_attempts}") + for exc_type in (*self.retry_on, *self.give_up_on): + if not (isinstance(exc_type, type) and issubclass(exc_type, BaseException)): + raise TypeError(f"重试条件必须是异常类型，收到 {exc_type!r}") + + def should_retry(self, attempt: int, exc: BaseException | None = None) -> bool: + """第 ``attempt`` 次尝试失败后是否应重试。 + + ``exc`` 为 ``None`` 表示失败原因不是异常（如非零退出码），此时只要还有 + 尝试余额就重试。 + """ + if attempt >= self.max_attempts: + return False + if exc is None: + return True + if self.give_up_on and isinstance(exc, self.give_up_on): + return False + return isinstance(exc, self.retry_on) + + def delay_for(self, attempt: int, *, rng: random.Random | None = None) -> float: + """第 ``attempt`` 次尝试失败后应等待的秒数。""" + return self.backoff.delay_for(attempt, rng=rng) + + +@dataclass(frozen=True, slots=True) +class Timeout: + """超时策略。三个作用域相互独立，可任意组合。 + + ``idle`` 基于心跳判定，用于发现"没有报错但也不动"的死锁——这是其余两者 + 无法覆盖的情形。 + """ + + start: float | None = None + """从提交到真正开始执行的等待上限。""" + per_attempt: float | None = None + """单次尝试的总时长上限。""" + idle: float | None = None + """两次心跳之间的最大间隔。""" + + def __post_init__(self) -> None: + for name in ("start", "per_attempt", "idle"): + value = getattr(self, name) + if value is not None and value <= 0: + raise ValueError(f"Timeout.{name} 必须为正数，收到 {value}") + + @property + def is_enabled(self) -> bool: + """是否设置了任何一档超时。""" + return any(v is not None for v in (self.start, self.per_attempt, self.idle)) + + +@dataclass(frozen=True, slots=True) +class ConcurrencyPolicy: + """并发与资源约束：调度器（``Controller``）真正执行的那一份配置。 + + 三个字段各管一件事，互不重叠： + + * ``max_parallel`` —— **同时执行**的任务总数上限。超出部分排队（状态 ``queued``）， + 不是拒绝，也不是多起线程。 + * ``limits`` —— 按**任务标签**的具名信号量，用来保护外部稀缺资源：一个带 ``gpu`` + 标签的任务，只有当该标签下当前活跃的任务数还没到 ``limits["gpu"]`` 时才会被 + 启动。没写进 ``limits`` 的标签不受限；一个任务带多个受限期标签时，每个都要有 + 余量才放行。 + * ``priority`` —— **默认优先级**：:meth:`~chongming_task.controller.Controller.submit` + 没给 ``priority`` 时用它；数值越大越先从就绪队列里被取走，同优先级仍按提交顺序。 + + 两个容易误解的地方，写在这里省得猜： + + * 名额与优先级都是**本进程**的账，不做跨进程协调。它保护的是本节点那一份资源 + 预算（例如本节点到某个库的连接数）；多节点各自配自己的份额。优先级也只在 + 本进程的就绪队列里比较——不做跨节点抢占。 + * 统计的是**活跃**任务（已占执行槽的），**包含被暂停的**——暂停不释放执行槽， + 也就不该释放资源名额。 + + 超限的任务**排队等待**而不是失败：调度器每次有槽位释放都会重新扫描就绪队列， + 因此资源一空出来它就会被取走，不需要调用方重试。 + """ + + max_parallel: int | None = None + limits: Mapping[str, int] = field(default_factory=dict, hash=False) + priority: int = 0 + """默认优先级；数值越大越先被调度。""" + + def __post_init__(self) -> None: + if self.max_parallel is not None and self.max_parallel < 1: + raise ValueError(f"max_parallel 至少为 1，收到 {self.max_parallel}") + for name, limit in self.limits.items(): + if limit < 1: + raise ValueError(f"资源 {name!r} 的并发上限至少为 1，收到 {limit}") + + +class JoinMode(StrEnum): + """一个节点在**什么条件下可以开始**——即它怎么"汇合"多个上游。 + + 注意它与 :class:`AggregationPolicy` 是**两个不同的轴**，不要混为一谈： + + * ``JoinMode`` 回答"这一个节点何时该跑"（逐节点的前置条件）； + * ``AggregationPolicy`` 回答"整条管道何时算结束、要不要连坐取消"（全局结算）。 + """ + + ALL_SUCCEEDED = "all_succeeded" + """全部上游成功才跑（默认）。最常见的屏障语义。""" + + ALL_SETTLED = "all_settled" + """全部上游落终态就跑，**不管成败**。用于清理、通知、汇总这类收尾节点。""" + + ANY_SUCCEEDED = "any_succeeded" + """任一上游成功就跑（先到先得）。用于多源冗余取数的场景。""" + + +class AggregationMode(StrEnum): + """整条管道的结算方式。""" + + FAIL_FAST = "fail_fast" + """首个失败即取消其余未完成任务。""" + ALL_SETTLED = "all_settled" + """等所有任务落终态，再统一判定。""" + QUORUM = "quorum" + """成功数达到阈值即视为通过。""" + + +@dataclass(frozen=True, slots=True) +class AggregationPolicy: + """管道层如何由子任务状态推导自身状态。 + + 工厂方法刻意命名为 ``fail_fast`` / ``all_settled`` / ``of_quorum``： + 若把 quorum 工厂也叫作 ``quorum``，它会与同名字段在类属性上冲突， + dataclass 的默认值会覆盖掉该方法，导致调用时报 ``NoneType is not callable``。 + """ + + mode: AggregationMode = AggregationMode.FAIL_FAST + quorum: int | None = None + + def __post_init__(self) -> None: + if self.mode is AggregationMode.QUORUM: + if self.quorum is None or self.quorum < 1: + raise ValueError("quorum 模式必须给出 >= 1 的 quorum") + elif self.quorum is not None: + raise ValueError("仅 quorum 模式可以设置 quorum") + + @classmethod + def fail_fast(cls) -> AggregationPolicy: + return cls(AggregationMode.FAIL_FAST) + + @classmethod + def all_settled(cls) -> AggregationPolicy: + return cls(AggregationMode.ALL_SETTLED) + + @classmethod + def of_quorum(cls, quorum: int) -> AggregationPolicy: + return cls(AggregationMode.QUORUM, quorum=quorum) + + +@dataclass(frozen=True, slots=True) +class BucketPolicy: + """分桶策略：把业务键**稳定地**映射到固定数量的桶。 + + 桶既是本地并发隔离的单位（每个桶各自限流，互不挤占），也是将来分布式分配 + 的单位（一台机器拥有一组桶）。两者都要求同一个键在任何进程、任何机器上都 + 落到同一个桶。 + + 为什么桶数必须固定 + ------------------ + ``bucket = hash(key) % num_buckets``。改动 ``num_buckets`` 会让**几乎所有** + 键改换桶位，局部性与隔离假设同时失效。因此桶数是创建时确定的常量；真要改 + 就得做迁移——这也是任务会同时记下 ``bucket_key`` 的原因：迁移需要原始的键。 + + 为什么不用内置 ``hash()`` + ------------------------- + CPython 对 ``str`` 的 ``hash()`` 带**进程级随机盐**（``PYTHONHASHSEED``）。 + 同一个键在不同进程里会落到不同的桶，而且是**静默**失效——"同桶任务应当跑在 + 同一台机器上"这条约定会莫名其妙地不成立，却没有任何报错。这里用 ``blake2b``， + 跨进程、跨机器、跨版本都稳定。 + + ``num_buckets`` 默认取 64：粒度过粗则隔离不足，过细则单桶任务太少、调度开销 + 占比升高。将来要做分布式时，它应当明显大于机器数量（经验上 4~16 倍）， + 以便某台机器退出时能把它的桶较均匀地分给其余机器。 + """ + + num_buckets: int = 64 + + def __post_init__(self) -> None: + if self.num_buckets < 1: + raise ValueError(f"num_buckets 至少为 1，收到 {self.num_buckets}") + + def bucket_for(self, key: str) -> int: + """把业务键映射到桶号。同一个键永远得到同一个桶。""" + if not key: + raise ValueError("bucket_key 不能为空字符串") + digest = hashlib.blake2b(key.encode("utf-8"), digest_size=8).digest() + return int.from_bytes(digest, "big") % self.num_buckets + + def is_valid(self, bucket: int) -> bool: + """``bucket`` 是否落在当前策略的范围内。 + + 跨策略校验很有用：读取用旧桶数写入的任务时，应当发现它越界，而不是 + 悄悄按错误的桶处理。 + """ + return 0 <= bucket < self.num_buckets + + @property + def buckets(self) -> range: + """全部桶号。""" + return range(self.num_buckets) diff --git a/src/chongming_task/run.py b/src/chongming_task/run.py new file mode 100644 index 0000000..a9a0334 --- /dev/null +++ b/src/chongming_task/run.py @@ -0,0 +1,750 @@ +"""单任务受控执行。 + +这是运行时零件（状态机 + 取消令牌 + 心跳 + 超时 + 执行器）的**第一个可用入口**。 +在调度器（DAG、并发上限、持久化恢复）落地之前，它是唯一能把它们串起来跑的东西； +落地之后它仍然是"跑一个任务"的最短路径，用于脚本、测试与简单场景。 + +它负责的全部事情： + +* 驱动状态机走过 ``PENDING → QUEUED → STARTING → RUNNING → 终态``； +* 按 :class:`~chongming_task.policies.RetryPolicy` 自动重试并退避； +* 按 :class:`~chongming_task.policies.Timeout` 判定超时，并在超时后请求停止； +* 把外部控制（停止 / 暂停 / 恢复）转成状态机迁移。 + +它**不**负责：DAG、并发上限、持久化与重启恢复、CLI。那些属于后续阶段。 +""" + +from __future__ import annotations + +import contextlib +import threading +import time +import traceback +import uuid +from collections.abc import Callable, Iterable, Mapping +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import Any + +from .clock import Clock, utcnow +from .core.state_machine import ChangeObserver, StateMachine, Transition +from .errors import InvalidTransition +from .models import ( + Failure, + FailureKind, + Progress, + TaskSnapshot, + TimeoutScope, + describe_serialization_problem, +) +from .policies import RetryPolicy, Timeout +from .runtime.executors.base import ( + ExecutionHandle, + ExecutionOutcome, + ExecutionRequest, + Executor, + classify_exception, +) +from .runtime.executors.thread import ThreadExecutor +from .runtime.timeouts import is_expired, next_deadline +from .states import TaskStatus + +__all__ = ["TaskRun", "TaskRunOutcome", "run_task"] + +#: 监督循环的轮询间隔。它同时决定了停止与超时的响应粒度。 +_DEFAULT_POLL_SECONDS = 0.05 + +#: 默认的优雅停止窗口。 +_DEFAULT_GRACEFUL_STOP_SECONDS = 5.0 + + +def _cancelled_failure(reason: str | None) -> Failure: + """编排层级联取消所对应的失败信息。""" + return Failure( + kind=FailureKind.CANCELLED, + message=reason or "编排层级联取消", + retryable=False, + ) + + +@dataclass(frozen=True, slots=True) +class TaskRunOutcome: + """一次受控执行的最终结果。""" + + snapshot: TaskSnapshot + value: Any = None + """任务返回值；未成功时为 ``None``。""" + + @property + def status(self) -> TaskStatus: + """最终状态。""" + return self.snapshot.status + + @property + def ok(self) -> bool: + """是否正常完成。""" + return self.snapshot.status is TaskStatus.SUCCEEDED + + @property + def failure(self) -> Failure | None: + """失败信息；正常完成时为 ``None``。""" + return self.snapshot.failure + + +class TaskRun: + """一次受监督的任务运行。 + + 典型用法:: + + run = TaskRun(extract, ("s3://bucket/x",)) + run.start() + run.pause() # 任务在下一个 ctx.checkpoint() 处挂起 + run.resume() + run.stop() # 请求停止；优雅窗口内自行收尾 + outcome = run.wait() + + 线程安全：``start`` / ``wait`` / ``stop`` / ``pause`` / ``resume`` 可从不同 + 线程调用。所有状态机写入都在同一把锁下进行，因此"状态机只有一个写者"这条 + 不变量仍然成立——后续调度器必须遵守同一条纪律。 + """ + + def __init__( + self, + target: Callable[..., Any], + args: tuple[Any, ...] = (), + *, + kwargs: dict[str, Any] | None = None, + task_id: str | None = None, + name: str | None = None, + tags: Iterable[str] = (), + bucket: int | None = None, + bucket_key: str | None = None, + pipeline_run_id: str | None = None, + priority: int = 0, + executor: Executor | None = None, + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, + artifact_dir: Path | None = None, + clock: Clock | None = None, + graceful_stop: float = _DEFAULT_GRACEFUL_STOP_SECONDS, + poll: float = _DEFAULT_POLL_SECONDS, + on_finished: Callable[[TaskRunOutcome], None] | None = None, + on_change: ChangeObserver | None = None, + ) -> None: + self._target = target + self._args = args + self._kwargs = dict(kwargs or {}) + self._executor: Executor = executor if executor is not None else ThreadExecutor() + self._retry = retry if retry is not None else RetryPolicy() + self._timeout = timeout if timeout is not None else Timeout() + self._clock: Clock = clock if clock is not None else utcnow + self._graceful_stop = graceful_stop + self._poll = poll + self._artifact_dir = artifact_dir + self._on_finished = on_finished + self._priority = priority + + resolved_id = task_id if task_id is not None else uuid.uuid4().hex[:12] + resolved_name = name if name is not None else getattr(target, "__name__", "task") + self._machine = StateMachine.create( + resolved_id, + resolved_name, + max_attempts=self._retry.max_attempts, + tags=frozenset(tags), + bucket=bucket, + bucket_key=bucket_key, + pipeline_run_id=pipeline_run_id, + clock=self._clock, + on_change=on_change, + ) + + self._lock = threading.RLock() + self._handle: ExecutionHandle | None = None + self._thread: threading.Thread | None = None + self._result: TaskRunOutcome | None = None + self._stop_requested = False + self._stop_reason: str | None = None + self._cancel_requested = False + self._cancel_reason: str | None = None + self._inputs: Mapping[str, Any] = {} + self._wake = threading.Event() + + # ------------------------------------------------------------------ + # 只读视图 + # ------------------------------------------------------------------ + + @property + def snapshot(self) -> TaskSnapshot: + """当前快照，可在任务运行期间安全读取。""" + with self._lock: + return self._machine.snapshot + + @property + def status(self) -> TaskStatus: + """当前状态。""" + with self._lock: + return self._machine.status + + @property + def transitions(self) -> tuple[Transition, ...]: + """完整的状态迁移历史，可审计。""" + with self._lock: + return self._machine.history + + @property + def started(self) -> bool: + """是否已经启动过。 + + 排队中（尚未获得执行槽）的任务为 ``False``；此时调用 :meth:`wait` 会报错， + 因为根本没有可等待的执行。 + """ + return self._thread is not None + + @property + def priority(self) -> int: + """排队时的优先级；数值越大越先被取走（同优先级按提交顺序）。 + + 只在被 :class:`~chongming_task.controller.Controller` 排队调度时起作用—— + 直接 :meth:`start` 一个任务没有队列，也就用不到它。 + """ + return self._priority + + def refresh_activity(self) -> TaskSnapshot: + """把执行载体的**真实心跳**同步进快照，并返回它。 + + 为什么必须同步：心跳由执行句柄的 :class:`~chongming_task.runtime.heartbeat.Heartbeat` + 持有，而快照是另一份数据。若不同步，下一次状态迁移的全量 ``save`` 会把 + ``touch`` 刚写进去的心跳**覆盖回 ``None``**——同一行有两条写入路径，这是最 + 典型的坑，而且是静默的（看板上心跳列永远是空的）。 + + 注意取的是**载体记录的心跳时刻**，不是"现在"：后者会让卡死的任务也显得心跳 + 新鲜，``idle`` 超时就永不触发。 + """ + with self._lock: + handle = self._handle + if handle is None: # pragma: no cover - 尚未启动 + return self.snapshot + with self._lock: + return self._machine.heartbeat(at=handle.heartbeat.last_beat) + + # ------------------------------------------------------------------ + # 生命周期 + # ------------------------------------------------------------------ + + def enqueue(self) -> None: + """标记为"已就绪、等待执行槽"（``queued``）。 + + 由调度方（如 :class:`~chongming_task.controller.Controller`）在**排队**时 + 调用，而不是在真正启动时。这样"提交了但排不上队"的任务状态是 ``queued`` + 而非 ``pending``——后者会让人以为它还没被受理。 + """ + with self._lock: + if self._machine.status is TaskStatus.PENDING: + self._machine.mark_queued() + + def block(self, *, reason: str | None = None) -> None: + """标记为"等待上游"（``blocked``）。 + + 与 ``queued`` 的区别是它**不会**被执行器取走：上游还没满足，现在跑它没有 + 意义。管道调度器负责在上游就绪后调用 :meth:`unblock`。 + """ + with self._lock: + if self._machine.status is TaskStatus.PENDING: + self._machine.mark_blocked(reason=reason) + + def unblock(self) -> None: + """上游已满足，回到"等待执行槽"（``blocked`` → ``queued``）。 + + 只对 ``blocked`` 生效：对已经在跑或已落终态的任务调用它是无操作，而不是报错 + ——调度器的重新评估可能重入，宽进严出比抛异常更合适。 + """ + with self._lock: + if self._machine.status is TaskStatus.BLOCKED: + self._machine.mark_queued() + + def start(self) -> None: + """启动受监督执行。立即返回，不等待任务结束。""" + with self._lock: + if self._thread is not None: + raise RuntimeError("该 TaskRun 已经启动过；请新建一个实例") + # 调度方可能已经用 enqueue() 标过 queued；只有仍是 pending 才需要迁移。 + if self._machine.status is TaskStatus.PENDING: + self._machine.mark_queued() + self._thread = threading.Thread( + target=self._run, + name=f"run-{self._machine.snapshot.task_id}", + daemon=True, + ) + self._thread.start() + + def wait(self, timeout: float | None = None) -> TaskRunOutcome: + """等待结束并返回结果。 + + ``timeout`` 到期只抛 :class:`TimeoutError`，**不改变任务状态**——把"我没 + 等到"当成"任务失败"是最常见的误用。 + """ + thread = self._thread + if thread is None: + raise RuntimeError("请先调用 start()") + thread.join(timeout) + if thread.is_alive(): + raise TimeoutError(f"等待超时（{timeout} 秒）；任务仍在运行，状态未被改变") + result = self._result + if result is None: # pragma: no cover - 监督循环必然留下结果 + raise RuntimeError("监督循环结束却没有留下结果") + return result + + def stop(self, *, reason: str | None = None) -> None: + """请求停止。不阻塞，也不直接改状态。 + + 这里只置位信号并让令牌抛错，**状态迁移仍由监督循环（唯一写者）完成**—— + 否则控制线程与监督线程会并发写状态机。 + """ + with self._lock: + if self._stop_requested: + return + self._stop_requested = True + self._stop_reason = reason + handle = self._handle + self._wake.set() + if handle is not None: + handle.token.request_stop(reason) + + def set_inputs(self, values: Mapping[str, Any]) -> None: + """注入上游节点的返回值（管道数据传递）。 + + 只能在任务**开始执行之前**调用：调度器在上游满足后才放行下游，因此这是自然 + 的时机。任务一旦跑起来，它已经拿到上下文里的那份快照，再改也影响不到它—— + 与其让调用方以为改了有用，不如明确这一点。 + + 需要显式加锁是因为状态机快照与这里的 ``_inputs`` 会被两个线程读：调度线程写、 + 监督线程读。 + """ + with self._lock: + if self._machine.status not in ( + TaskStatus.PENDING, + TaskStatus.BLOCKED, + TaskStatus.QUEUED, + ): + raise RuntimeError("任务已经开始执行，不能再注入上游数据") + self._inputs = dict(values) + + @property + def result(self) -> TaskRunOutcome | None: + """已完成时的结果；尚未结束或从未启动时为 ``None``。 + + 与 :meth:`wait` 的区别：``wait`` 会阻塞并要求任务已启动，而本属性只是读一眼 + 当前状态——管道调度器用它取上游的返回值。 + """ + with self._lock: + return self._result + + def _record_progress(self, progress: Progress) -> None: + """进度回写：只更新快照，不碰状态与版本号。 + + 存储落地由 :meth:`Controller._flush_activity` 经 ``touch`` 完成——进度是高 + 频写入，不能挤进状态迁移那条带乐观并发的路径。 + """ + with self._lock: + if self._machine.snapshot.is_terminal: + return + self._machine.update_progress(progress) + + def cancel(self, *, reason: str | None = None) -> None: + """**编排层级联取消**（上游失败、整条管道被取消）。 + + 与 :meth:`stop` 的区别**只在结局**，但那个区别很重要： + + * ``stop`` 是"用户要求的停止"，任务最终落 ``STOPPED``； + * ``cancel`` 是编排层的取消，任务最终落 ``CANCELLED``。 + + 责任方不同，告警与统计也必须能分开——"值班同学手动停的"和"上游挂了被连坐的" + 是两件完全不同的事，混成一个状态就再也分不出来了。 + + 因此这里**不置** ``_stop_requested``：监督循环据此区分"我们请求的"与"别人 + 取消的"（见 ``_settle``）。 + """ + with self._lock: + if self._machine.snapshot.is_terminal: + return + self._cancel_requested = True + self._cancel_reason = reason + status = self._machine.status + handle = self._handle + if status in (TaskStatus.PENDING, TaskStatus.BLOCKED): + # 从未交给执行器、也不会再交出去：直接落终态，不必走中间态缓冲。 + self._machine.mark_cancelled(reason=reason, failure=_cancelled_failure(reason)) + return + self._wake.set() + if handle is not None: + handle.token.request_stop(reason) + + def skip(self, *, reason: str | None = None) -> None: + """按策略跳过（``pending`` / ``blocked`` / ``queued`` → ``skipped``）。 + + 与 :meth:`cancel` 的区别在**意图**：``skip`` 是"它本来就不该跑"（上游没成功、 + 条件不成立），``cancel`` 是"本来要跑，但不跑了"。两者都不是失败，但把原因 + 混在一起，事后就分不清"被跳过"和"被连坐"。 + + 已经占用执行资源的任务无法跳过（状态机没有那条边）：到那一步只能 ``cancel``。 + """ + with self._lock: + if self._machine.status in ( + TaskStatus.PENDING, + TaskStatus.BLOCKED, + TaskStatus.QUEUED, + ): + self._machine.mark_skipped(reason=reason) + + def pause(self, *, timeout: float = _DEFAULT_GRACEFUL_STOP_SECONDS) -> bool: + """请求暂停并等待确认。 + + 返回 ``False`` 表示任务未到达检查点、暂停**没有生效**：此时状态回滚为 + ``RUNNING``，绝不假装已暂停。 + + 注意这是**就地暂停**——任务阻塞在 ``ctx.checkpoint()`` 里，并**继续占用 + 执行槽**。我们无法把任意的 Python 调用栈换出内存再换回来，所以暂停不释放 + 槽位；暂停大量任务可能耗尽并发额度。 + """ + with self._lock: + handle = self._handle + if handle is None or self._machine.status is not TaskStatus.RUNNING: + return False + self._machine.request_pause(actor="user") + + if not handle.pause(timeout=timeout): + with self._lock: + self._machine.abort_pause(reason="任务未在窗口内到达检查点") + return False + + with self._lock: + try: + self._machine.mark_paused() + except InvalidTransition: + # 任务恰好在"确认暂停"与"迁移状态"之间结束了，如实放弃这次暂停。 + return False + return True + + def resume(self) -> None: + """撤销暂停。 + + 就地暂停的任务回到 ``RUNNING``；从未开始执行就被暂停的任务（``QUEUED`` + 或 ``RETRY_WAIT`` 期间暂停）回到 ``QUEUED``。 + """ + with self._lock: + handle = self._handle + if handle is None or self._machine.status is not TaskStatus.PAUSED: + return + handle.resume() + self._machine.resume(actor="user") + self._wake.set() + + # ------------------------------------------------------------------ + # 监督循环：唯一写者 + # ------------------------------------------------------------------ + + def _run(self) -> None: + try: + self._supervise() + except BaseException as exc: + # 监督循环静默死掉是最糟的失败模式：wait() 会挂住并抛出一个与真实原因 + # 毫无关系的错误。这里把异常落成一次失败，并保留完整 traceback。 + self._result = self._record_supervisor_crash(exc) + finally: + self._notify_finished() + + def _notify_finished(self) -> None: + """通知外部（如 Controller 的调度器）本次运行已结束、可以让出并发额度。""" + callback = self._on_finished + result = self._result + if callback is None or result is None: + return + with contextlib.suppress(Exception): + # 回调是外部代码；它抛异常不该影响任务本身的结论。 + callback(result) + + def _record_supervisor_crash(self, exc: BaseException) -> TaskRunOutcome: + failure = Failure( + kind=FailureKind.EXCEPTION, + message=f"监督循环内部错误：{type(exc).__name__}: {exc}", + traceback="".join(traceback.format_exception(exc)), + retryable=False, + ) + with self._lock: + if not self._machine.snapshot.is_terminal: + with contextlib.suppress(InvalidTransition): + self._machine.mark_failed(failure) + return TaskRunOutcome(snapshot=self._machine.snapshot) + + def _supervise(self) -> None: + attempt = 0 + while True: + with self._lock: + if self._machine.snapshot.is_terminal: + # 在"已出队、尚未启动"的窗口里被取消或跳过：状态已经落定， + # 绝不能再走 mark_starting——那会撞上非法状态转移。 + self._result = TaskRunOutcome(snapshot=self._machine.snapshot) + return + not_started = self._machine.status in ( + TaskStatus.PENDING, + TaskStatus.QUEUED, + TaskStatus.BLOCKED, + TaskStatus.PAUSED, + ) + # 编排层级联取消优先于用户停止：取消是"这条管道不跑了"，更根本。 + if self._cancel_requested and not_started: + self._machine.mark_cancelled( + reason=self._cancel_reason, + failure=_cancelled_failure(self._cancel_reason), + ) + self._result = TaskRunOutcome(snapshot=self._machine.snapshot) + return + # 还没开始跑就已被要求停止：直接收场，不要白白消耗一次尝试。 + if self._stop_requested and not_started: + self._machine.request_stop(actor="user", reason=self._stop_reason) + self._result = TaskRunOutcome(snapshot=self._machine.snapshot) + return + + attempt += 1 + launch_failure = self._begin_attempt(attempt) + outcome = launch_failure if launch_failure is not None else self._await_attempt() + settled = self._settle(attempt, outcome) + if settled is not None: + self._result = settled + return + + def _begin_attempt(self, attempt: int) -> ExecutionOutcome | None: + """起一次尝试。返回 ``None`` 表示已拉起，否则返回起始就失败的结论。""" + with self._lock: + if self._machine.status is TaskStatus.PENDING: + self._machine.mark_queued() + self._machine.mark_starting(executor=self._executor.name) + snapshot = self._machine.snapshot + inputs = self._inputs + + request = ExecutionRequest( + task_id=snapshot.task_id, + name=snapshot.name, + target=self._target, + attempt=attempt, + args=self._args, + kwargs=self._kwargs, + artifact_dir=self._artifact_dir, + inputs=inputs, + on_progress=self._record_progress, + clock=self._clock, + ) + try: + handle = self._executor.launch(request) + except BaseException as exc: + # 执行器没能把任务拉起来：配置错（目标不可 pickle、任务没注册）、资源耗尽…… + # 必须在这里接住。异常若逃进监督线程，任务会以"监督循环内部错误"收场—— + # 那等于把一个**任务侧**的问题说成框架的 bug，指向完全错误的排查方向。 + return ExecutionOutcome(failure=classify_exception(exc)) + with self._lock: + self._handle = handle + # pid 来自句柄：同进程执行器为 None（任务就跑在本进程里），进程执行器给出 + # 子进程的进程号——事后 `ps` 对得上号的只有后者。 + self._machine.mark_running(pid=handle.pid) + return None + + def _mark_stopping_if_needed(self) -> None: + """把"已请求停止"落成 ``STOPPING``。 + + 必须在**交出结果之前**调用。停止请求发出后，任务可能在下一次 + ``checkpoint()`` 立刻抛错结束——那样监督循环会直接从 ``handle.wait`` + 拿到结果，从未走过 ``STOPPING``，于是结果会被错误地记成"级联取消"而不是 + "用户停止"。这个竞态在第一次运行时就真实发生了。 + """ + with self._lock: + if self._machine.status in ( + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.PAUSING, + ): + self._machine.request_stop(actor="user", reason=self._stop_reason) + + def _await_attempt(self) -> ExecutionOutcome: + """等待本次尝试结束，期间处理停止请求与超时。""" + handle = self._must_handle() + started_at = self._clock() + + while True: + # 先取到局部变量：否则 mypy 会把 self._stop_requested 收窄为 False， + # 并把循环末尾的同名检查判成不可达——而那个检查正是"任务抢在停止分支 + # 之前就结束"这个竞态的兜底。 + stop_requested = self._stop_requested + if stop_requested: + self._mark_stopping_if_needed() + return handle.stop(graceful_timeout=self._graceful_stop, reason=self._stop_reason) + + # 暂停（含暂停中）期间不判超时：被暂停的任务不是"卡住"的， + # 而它此时本来就不会推进心跳。 + with self._lock: + pause_in_flight = self._machine.status in ( + TaskStatus.PAUSING, + TaskStatus.PAUSED, + ) + if not pause_in_flight: + scope = self._expired_scope(handle, started_at) + if scope is not None: + limit = self._limit_for(scope) + stopped = handle.stop( + graceful_timeout=self._graceful_stop, + reason=f"{scope.value} 超时", + ) + return ExecutionOutcome( + failure=Failure( + kind=FailureKind.for_timeout(scope), + message=(f"任务超时（{scope.value}）：已超过 {limit} 秒，已请求停止"), + retryable=True, + ), + abandoned=stopped.abandoned, + ) + + outcome = handle.wait(timeout=self._poll) + if outcome is not None: + # 停止请求可能已生效但还没被上面的分支看到：任务抢先结束了。 + if self._stop_requested: + self._mark_stopping_if_needed() + return outcome + + def _expired_scope(self, handle: ExecutionHandle, started_at: datetime) -> TimeoutScope | None: + deadline = next_deadline( + self._timeout, + started_at=started_at, + last_beat_at=handle.heartbeat.last_beat, + ) + if deadline is None or not is_expired(deadline, now=self._clock()): + return None + return deadline.scope + + def _limit_for(self, scope: TimeoutScope) -> float | None: + if scope is TimeoutScope.ATTEMPT: + return self._timeout.per_attempt + if scope is TimeoutScope.IDLE: + return self._timeout.idle + return self._timeout.start + + def _settle(self, attempt: int, outcome: ExecutionOutcome) -> TaskRunOutcome | None: + """把执行结果落成状态。返回 ``None`` 表示"还要再试一次"。""" + if outcome.ok: + problem = describe_serialization_problem(outcome.value) + if problem is not None: + # 返回值会被持久化，因此必须可 JSON 序列化。**必须在写库之前拦下来**： + # 否则失败会发生在写穿路径上，而那条路径的失败只被记录、不会让任务失败 + # ——结果是进程内显示 succeeded、存储里停在 running，重启恢复再把它判成 + # LOST。把一个真正成功过的任务报成"结果未知"是最糟的一种错。 + # + # 不重试：同一个返回值重跑多少次都还是不可序列化。 + failure = Failure( + kind=FailureKind.RESULT_INVALID, + message=( + f"返回值不满足持久化契约：{problem}。" + "返回值会写入存储，请返回可 JSON 序列化的值；大数据请传指针" + "（对象存储 key、表名 + 主键、文件路径）。" + ), + retryable=False, + ) + with self._lock: + self._machine.mark_failed(failure) + return TaskRunOutcome(snapshot=self._machine.snapshot) + with self._lock: + self._machine.mark_succeeded(outcome={"value": outcome.value}) + return TaskRunOutcome(snapshot=self._machine.snapshot, value=outcome.value) + + failure = outcome.failure or Failure(kind=FailureKind.LOST, message="执行器未给出结果") + + if outcome.abandoned: + # 结果未知：任务可能仍在后台运行并产生副作用，绝不能记成"已停止"。 + # 失败原因必须一并记录，否则事后无从判断该重跑还是该人工核对。 + with self._lock: + self._machine.mark_lost(reason=failure.message, failure=failure) + return TaskRunOutcome(snapshot=self._machine.snapshot) + + if failure.kind is FailureKind.CANCELLED: + with self._lock: + if self._stop_requested: + # 我们自己请求的停止：必须走过 STOPPING，最终落 STOPPED。 + self._mark_stopping_if_needed() + self._machine.mark_stopped(reason=self._stop_reason, failure=failure) + else: + # 别人取消的（编排层级联），落 CANCELLED。 + self._machine.mark_cancelled(reason=failure.message, failure=failure) + return TaskRunOutcome(snapshot=self._machine.snapshot) + + if failure.retryable and self._retry.should_retry(attempt): + with self._lock: + self._machine.mark_retry_wait(failure) + if not self._backoff(attempt): + with self._lock: + self._mark_stopping_if_needed() + self._machine.mark_stopped(reason=self._stop_reason, failure=failure) + return TaskRunOutcome(snapshot=self._machine.snapshot) + return None + + with self._lock: + self._machine.mark_failed(failure) + return TaskRunOutcome(snapshot=self._machine.snapshot) + + def _backoff(self, attempt: int) -> bool: + """退避等待。返回 ``False`` 表示等待期间被要求停止。 + + 用可唤醒的事件等待，而不是 ``time.sleep``：否则一次 300 秒的退避会让 + ``stop()`` 也等上 300 秒才生效。 + """ + deadline = time.monotonic() + self._retry.delay_for(attempt) + while True: + if self._stop_requested: + return False + remaining = deadline - time.monotonic() + if remaining <= 0: + return True + self._wake.wait(min(remaining, self._poll)) + + def _must_handle(self) -> ExecutionHandle: + handle = self._handle + if handle is None: # pragma: no cover - 监督循环内必然已启动 + raise RuntimeError("执行尚未启动") + return handle + + +def run_task( + target: Callable[..., Any], + /, + *args: Any, + task_id: str | None = None, + name: str | None = None, + executor: Executor | None = None, + retry: RetryPolicy | None = None, + timeout: Timeout | None = None, + artifact_dir: Path | None = None, + clock: Clock | None = None, + graceful_stop: float = _DEFAULT_GRACEFUL_STOP_SECONDS, + **kwargs: Any, +) -> TaskRunOutcome: + """跑一个任务并等它结束，返回最终结果。 + + 这是 :class:`TaskRun` 的阻塞式便捷包装。需要运行期控制（边跑边暂停、停止） + 时请直接用 :class:`TaskRun`。 + + ``target`` 的第一个位置参数由执行器注入执行期上下文:: + + def extract(ctx, source: str) -> list[Row]: ... + """ + run = TaskRun( + target, + args, + kwargs=kwargs, + task_id=task_id, + name=name, + executor=executor, + retry=retry, + timeout=timeout, + artifact_dir=artifact_dir, + clock=clock, + graceful_stop=graceful_stop, + ) + run.start() + return run.wait() diff --git a/src/chongming_task/runtime/__init__.py b/src/chongming_task/runtime/__init__.py new file mode 100644 index 0000000..806aeb9 --- /dev/null +++ b/src/chongming_task/runtime/__init__.py @@ -0,0 +1,5 @@ +"""运行时：取消令牌、心跳、超时判定、执行器。 + +本层可以依赖 :mod:`chongming_task.core`，但**不反向被依赖**：领域内核不认识 +线程、超时与执行器，因此它才能被属性测试穷举验证。 +""" diff --git a/src/chongming_task/runtime/cancellation.py b/src/chongming_task/runtime/cancellation.py new file mode 100644 index 0000000..9d4779a --- /dev/null +++ b/src/chongming_task/runtime/cancellation.py @@ -0,0 +1,179 @@ +"""协作式取消与暂停。 + +Python 既不能安全强杀线程，也没有跨平台的抢占式暂停手段（见设计文档 §11）。 +因此停止与暂停都必须是**任务主动配合**的：任务在长耗时循环里调用 +:meth:`CancellationToken.checkpoint`，由它响应信号。 + +这个选择带来一条硬约束，必须让使用者知道： + + 不调用 ``checkpoint()`` 的任务，既无法优雅停止，也无法暂停。 + +本模块只管信号，不管状态迁移——``RUNNING -> STOPPING`` 之类的写入仍由 +:class:`~chongming_task.core.state_machine.StateMachine` 独占。 +""" + +from __future__ import annotations + +import asyncio +import threading +import time + +from ..errors import TaskCancelled + +__all__ = ["CancellationToken"] + + +class CancellationToken: + """可跨线程使用的取消/暂停信号。 + + 线程安全：同一个令牌会被任务线程（调用 :meth:`checkpoint`）与调度线程 + （调用 :meth:`request_stop` / :meth:`request_pause`）同时访问。 + """ + + __slots__ = ("_ack", "_cond", "_pause", "_stop", "_stop_reason") + + def __init__(self) -> None: + self._cond = threading.Condition(threading.Lock()) + self._stop = False + self._stop_reason: str | None = None + self._pause = False + self._ack = False + """是否已有 checkpoint 真正停在了暂停点上。""" + + # ------------------------------------------------------------------ + # 只读 + # ------------------------------------------------------------------ + + @property + def is_stop_requested(self) -> bool: + """是否已请求停止。""" + with self._cond: + return self._stop + + @property + def is_pause_requested(self) -> bool: + """是否已请求暂停。""" + with self._cond: + return self._pause + + @property + def is_pause_acknowledged(self) -> bool: + """任务是否已确认暂停（某个 checkpoint 正在暂停点上阻塞）。""" + with self._cond: + return self._ack + + # ------------------------------------------------------------------ + # 请求侧（调度线程） + # ------------------------------------------------------------------ + + def request_stop(self, reason: str | None = None) -> None: + """请求停止。 + + 会唤醒所有正阻塞在暂停点上的任务，使它们抛 :class:`TaskCancelled`—— + 否则"暂停中的任务停不掉"。重复调用是幂等的，且**保留首个原因**， + 因为最先给出的那个通常最接近真实起因。 + """ + with self._cond: + if self._stop: + return + self._stop = True + self._stop_reason = reason + self._cond.notify_all() + + def request_pause(self) -> None: + """请求暂停。任务将在下一个 :meth:`checkpoint` 挂起。""" + with self._cond: + if self._pause: + return + self._pause = True + self._ack = False + + def resume(self) -> None: + """撤销暂停，唤醒阻塞中的任务。""" + with self._cond: + if not self._pause: + return + self._pause = False + self._ack = False + self._cond.notify_all() + + def wait_paused(self, timeout: float | None = None) -> bool: + """等待任务确认已暂停，返回是否在超时前确认。 + + 调度器用它区分"已进入 ``PAUSED``"与"暂停没能生效"。后者必须回滚为 + ``RUNNING`` 并告警，而不是假装成功——假装会让用户以为资源已释放。 + """ + deadline = None if timeout is None else time.monotonic() + timeout + with self._cond: + while not self._ack: + if self._stop: + return False + if deadline is None: + self._cond.wait() + continue + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + self._cond.wait(remaining) + return True + + # ------------------------------------------------------------------ + # 任务侧 + # ------------------------------------------------------------------ + + def _raise_if_stopped_locked(self) -> None: + """须在持有 ``_cond`` 时调用。 + + 抽成独立方法不只是为了去重：若把它内联进 :meth:`checkpoint`，mypy 会把 + 第一次检查后的 ``self._stop`` 收窄为 ``Literal[False]``，进而把循环之后 + 的复查判为不可达语句——而那个复查恰恰是"暂停中被停止"能生效的关键。 + """ + if self._stop: + raise TaskCancelled(self._stop_reason) + + def checkpoint(self) -> None: + """协作点：响应停止与暂停。 + + 这是任务与运行时之间的**唯一约定**。长耗时循环必须周期性调用它。 + + 停止优先于暂停：即使任务正处于暂停挂起状态，停止请求也会立刻生效并 + 抛出 :class:`TaskCancelled`——否则"暂停中的任务停不掉"。 + """ + with self._cond: + self._raise_if_stopped_locked() + while self._pause and not self._stop: + # ``_ack`` 必须在**循环体内**设置，不能只在进入时设一次。 + # + # 否则会有一次丢唤醒：``resume()`` 把 ``_pause`` 置回 False 并通知后， + # 任务线程还没来得及重新取到锁，新的 ``request_pause()`` 就把 + # ``_pause`` 又置成了 True —— 线程重新检查循环条件时发现"还要暂停"， + # 于是继续等待，而它已经越过了设置 ``_ack`` 的那一行。结果 + # ``wait_paused()`` 永远等不到确认、暂停超时回滚。 + # + # 表现出来就是"暂停 → 恢复 → 再暂停"里第二次暂停永久失效。 + self._ack = True + self._cond.notify_all() + self._cond.wait() + self._raise_if_stopped_locked() + + async def acheckpoint(self) -> None: + """:meth:`checkpoint` 的异步版本。 + + 非阻塞快速路径（绝大多数调用）保持同步执行，避免在紧密循环里为每次 + 检查都付一次线程调度成本；只有真的需要挂起时才把阻塞等待交给线程。 + """ + with self._cond: + self._raise_if_stopped_locked() + if not self._pause: + return + await asyncio.to_thread(self.checkpoint) + + def raise_if_stopped(self) -> None: + """只检查停止信号，不阻塞、不响应暂停。 + + 用于无法安全挂起的位置（例如 ``except`` 块的清理逻辑中），此时阻塞会 + 导致清理永远无法完成。 + """ + with self._cond: + if self._stop: + raise TaskCancelled(self._stop_reason) diff --git a/src/chongming_task/runtime/context.py b/src/chongming_task/runtime/context.py new file mode 100644 index 0000000..7902420 --- /dev/null +++ b/src/chongming_task/runtime/context.py @@ -0,0 +1,219 @@ +"""执行期上下文：任务与运行时之间的**唯一协作入口**。 + +任务作者看到的就是这个对象（公开 API 里叫 ``TaskContext``，同一个类）： + +```python +def extract(ctx, source: str) -> list[Row]: + ctx.log(f"开始读取 {source}") + rows = fetch(source) + for index, row in enumerate(rows): + ctx.checkpoint() # 响应停止/暂停 + 刷新心跳 + ctx.progress(index + 1, total=len(rows)) + return rows +``` + +三件事收敛在这里，而且都必须在这里： + +* **停止 / 暂停 / 心跳**——与执行载体强相关，只有执行器知道载体是什么； +* **进度**——高频写入，走旁路而不是状态机（见 ``touch``）； +* **上游数据**——管道节点之间传值，只有调度器知道谁是谁的上游。 + +## 为什么上下文必须是注入的 + +见 :class:`~chongming_task.runtime.executors.base.ExecutionRequest` 的说明：上下文里 +有 :class:`~chongming_task.runtime.cancellation.CancellationToken`，它**每次尝试才 +诞生**（由执行器在启动那一刻创建），所以任务根本无从捕获——它只能从第一个位置参数 +拿到。这不是风格问题：换成闭包捕获就得让令牌活到任务定义之外，指针一旦指向"上一次 +尝试"的令牌，停止指令就会发给一个已经结束的执行。 +""" + +from __future__ import annotations + +import logging +from collections import deque +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from ..models import Progress +from .cancellation import CancellationToken +from .heartbeat import Heartbeat + +__all__ = ["RuntimeContext", "TaskContext"] + +#: 上下文在内存里保留的日志条数上限。 +_LOG_BUFFER = 200 + +#: 任务日志的 logger 名称前缀。 +LOGGER_PREFIX = "chongming_task.task" + + +@dataclass(slots=True) +class RuntimeContext: + """任务执行期可用的运行时能力。""" + + task_id: str + name: str + attempt: int + token: CancellationToken + heartbeat: Heartbeat + artifact_dir: Path | None = None + """本次尝试的产物目录。按 attempt 隔离，重试不会污染上一次的产物。""" + + inputs: Mapping[str, Any] = field(default_factory=dict) + """上游节点的返回值，按节点 id 索引。非管道任务为空。 + + 只有**成功**的上游才会出现在这里：失败或跳过的上游没有值可传，与其塞一个 + ``None`` 让人误以为"上游返回了 None"，不如干脆不出现。用 :meth:`input` 取则 + 可以给出默认值。 + """ + + on_progress: Callable[[Progress], None] | None = None + """进度回写通道，由执行器注入。为 ``None`` 时 :meth:`progress` 静默丢弃。""" + + clock: Callable[[], datetime] | None = None + """时钟。与任务本身共用同一个，避免时间戳来自两个不同的时间源。""" + + logs: deque[str] = field(default_factory=lambda: deque(maxlen=_LOG_BUFFER)) + """本任务最近的日志（内存中，最多 200 条）。 + + 它**不是**日志存储方案：进程退出即消失。它的用途是让"任务失败时最后几条日志是 + 什么"在事后可查——把完整日志落盘/入库是应用自己的事（挂个 ``logging`` handler + 即可）。 + """ + + # ------------------------------------------------------------------ + # 协作点 + # ------------------------------------------------------------------ + + def checkpoint(self) -> None: + """协作点：响应停止与暂停，并刷新心跳。 + + 把心跳绑在这里是有意的：``checkpoint()`` 就是"我还在推进"的信号。 + 不调用它的任务既不响应停止，也会被 ``idle`` 超时判定为疑似卡死。 + + 收到停止请求时抛 :class:`~chongming_task.errors.TaskCancelled`。 + """ + self.token.checkpoint() + self.heartbeat.beat() + + async def acheckpoint(self) -> None: + """:meth:`checkpoint` 的异步版本，供原生 ``async`` 任务使用。""" + await self.token.acheckpoint() + self.heartbeat.beat() + + @property + def is_stop_requested(self) -> bool: + """是否已请求停止。""" + return self.token.is_stop_requested + + @property + def is_pause_requested(self) -> bool: + """是否已请求暂停。""" + return self.token.is_pause_requested + + # ------------------------------------------------------------------ + # 进度 + # ------------------------------------------------------------------ + + def progress( + self, + done: int, + *, + total: int | None = None, + unit: str | None = None, + message: str | None = None, + ) -> None: + """上报进度。 + + **不参与状态机**，也不递增 ``status_version``：进度可能每秒几十次，若参与 + 乐观并发会与状态写入持续冲突，把版本号变成噪声。 + + 它同时也刷新心跳吗？**不。** 心跳是"我还在推进"的整体信号，进度是"推进到哪 + 了"。两者都调用一次是常见写法，但把心跳藏进进度里会让一个只报进度、不做实际 + 工作的任务看起来永远健康——那正是 ``idle`` 超时要抓的东西。 + """ + sink = self.on_progress + if sink is None: + return + # 这两个字段会被写进存储。类型错误必须在**调用点**炸掉：让它流到写穿路径上， + # 失败只会被记录下来、任务继续跑，表现是"看板的进度永远是旧的"——离原因很远。 + if unit is not None and not isinstance(unit, str): + raise TypeError(f"unit 必须是 str 或 None，收到 {type(unit).__name__}") + if message is not None and not isinstance(message, str): + raise TypeError(f"message 必须是 str 或 None，收到 {type(message).__name__}") + moment = self.clock() if self.clock is not None else datetime.now(UTC) + sink( + Progress( + done=done, + total=total, + unit=unit, + message=message, + updated_at=moment, + ) + ) + + # ------------------------------------------------------------------ + # 日志 + # ------------------------------------------------------------------ + + @property + def logger(self) -> logging.Logger: + """本任务的 logger。 + + 名字是 ``chongming_task.task.``，因此可以在应用侧按任务维度单独配 + handler / 级别，而不必给每个任务各写一套。 + """ + return logging.getLogger(f"{LOGGER_PREFIX}.{self.task_id}") + + def log( + self, + message: str, + *, + level: int = logging.INFO, + exc_info: bool = False, + ) -> None: + """记一条日志。 + + 走标准库 ``logging``，并把 ``task_id`` / ``task_name`` / ``attempt`` 作为 + ``extra`` 附上，因此 handler 可以据此过滤或加字段。同时保留最近 + :data:`_LOG_BUFFER` 条在 :attr:`logs` 里，供事后查看。 + """ + self.logs.append(message) + self.logger.log( + level, + message, + exc_info=exc_info, + extra={ + "task_id": self.task_id, + "task_name": self.name, + "attempt": self.attempt, + }, + ) + + def warning(self, message: str) -> None: + """:meth:`log` 的 ``WARNING`` 快捷方式。""" + self.log(message, level=logging.WARNING) + + def error(self, message: str) -> None: + """:meth:`log` 的 ``ERROR`` 快捷方式。""" + self.log(message, level=logging.ERROR) + + # ------------------------------------------------------------------ + # 上游数据 + # ------------------------------------------------------------------ + + @property + def upstream(self) -> Mapping[str, Any]: + """上游节点的返回值，按节点 id 索引。只读。""" + return self.inputs + + def input(self, node_id: str, default: Any = None) -> Any: + """取某个上游节点的返回值；没有就给 ``default``。""" + return self.inputs.get(node_id, default) + + +#: 公开 API 里的名字。**同一个类**，不是包装——任务作者只面对一个 ``ctx``。 +TaskContext = RuntimeContext diff --git a/src/chongming_task/runtime/executors/__init__.py b/src/chongming_task/runtime/executors/__init__.py new file mode 100644 index 0000000..145646e --- /dev/null +++ b/src/chongming_task/runtime/executors/__init__.py @@ -0,0 +1 @@ +"""执行器实现。""" diff --git a/src/chongming_task/runtime/executors/base.py b/src/chongming_task/runtime/executors/base.py new file mode 100644 index 0000000..595d40f --- /dev/null +++ b/src/chongming_task/runtime/executors/base.py @@ -0,0 +1,333 @@ +"""执行器抽象。 + +执行器只负责四件事：把一段可调用对象跑起来、报告它是否还在跑、能被要求停止、 +能被要求暂停。 + +它**不**做两件事，这是刻意划下的边界： + +* 不决定状态迁移——那是 :class:`~chongming_task.core.state_machine.StateMachine` + 的独占职责； +* 不决定并发上限——那是调度器的职责（``ConcurrencyPolicy``）。因此执行器内部 + 不再叠一层线程池，否则两层限制互相掩盖，很难推理。 +""" + +from __future__ import annotations + +import abc +import asyncio +import inspect +import traceback +from collections.abc import Awaitable, Callable, Mapping +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Protocol, runtime_checkable + +from ...clock import Clock +from ...errors import ConfigurationError, TaskCancelled, TaskTimeout +from ...models import Failure, FailureKind, Progress +from ..cancellation import CancellationToken +from ..context import RuntimeContext +from ..heartbeat import Heartbeat + +__all__ = [ + "ExecutionHandle", + "ExecutionOutcome", + "ExecutionRequest", + "Executor", + "LocalExecutionHandle", + "classify_exception", + "run_target", +] + + +@dataclass(frozen=True, slots=True) +class ExecutionRequest: + """一次尝试的执行请求。 + + ``target`` 的**第一个位置参数是执行期上下文**，由执行器注入:: + + def extract(ctx, source: str) -> list[Row]: ... + + 这个约定是刻意的：上下文（尤其是取消令牌）属于**某一次尝试**，在启动那一刻才 + 由执行器造出来，因此任务只能被喂给它，不可能自己攥着一个。若靠闭包捕获，任务 + 定义里就得提前握住某个令牌，而那多半是上一次尝试留下的——停止指令会发给一个 + 已经结束的执行，且完全静默。 + """ + + task_id: str + name: str + target: Callable[..., Any] + attempt: int = 1 + args: tuple[Any, ...] = () + kwargs: Mapping[str, Any] = field(default_factory=dict, hash=False) + artifact_dir: Path | None = None + inputs: Mapping[str, Any] = field(default_factory=dict, hash=False) + """上游节点的返回值，按节点 id 索引。管道数据传递靠它。""" + on_progress: Callable[[Progress], None] | None = None + """进度回写通道：进度不参与状态机，由调用方决定怎么落地。""" + clock: Clock | None = None + """时钟。与自己的任务共用同一个，避免时间戳来自两个不同的时间源。""" + + +@dataclass(frozen=True, slots=True) +class ExecutionOutcome: + """一次执行的结果。""" + + value: Any = None + failure: Failure | None = None + abandoned: bool = False + """执行载体未能停止、已被放弃。 + + 为 ``True`` 时结果**未知**：任务可能仍在后台运行并产生副作用。调用方应据此 + 落 ``LOST`` 而不是 ``STOPPED``——把"没停住"记成"已停止"不只是不准确， + 而是会让运维误判资源已释放。 + """ + + @property + def ok(self) -> bool: + """是否正常完成。""" + return self.failure is None and not self.abandoned + + +def run_target(request: ExecutionRequest, context: RuntimeContext) -> Any: + """调用任务体并取出返回值；目标是协程函数时，在本线程内把它跑完。 + + 线程执行器与内联执行器共用这一个入口，语义才不会分叉。**必须真的等到协程结束**： + 直接把协程对象当返回值的话，任务会"看起来成功"（结果是 ````）， + 然后在写存储时以"返回值不可序列化"失败，或者在没有存储时静默通过——两种都难查。 + + 协程在**本线程**新起一个事件循环来跑，不借用调用方已有的循环：因此不要求调用方 + 是异步的，也不会与调用方的循环互相干扰，``ctx.acheckpoint()`` 这类异步协作点 + 照常工作。调用方若本身已在事件循环里（只有内联执行器可能遇到），``asyncio.run`` + 会抛 ``RuntimeError``——如实报错，不猜。 + """ + value = request.target(context, *request.args, **request.kwargs) + if not inspect.isawaitable(value): + return value + return asyncio.run(_drive_awaitable(value)) + + +async def _drive_awaitable(value: Awaitable[Any]) -> Any: + """把任意可等待对象包成协程——``asyncio.run`` 只接受协程。""" + return await value + + +def classify_exception(exc: BaseException) -> Failure: + """把任务抛出的异常转成结构化失败信息。 + + ``TaskCancelled`` 继承自 ``BaseException``，必须先于普通异常分支处理。 + 它被标记为**不可重试**：取消不是失败，重试它等于无视停止指令。 + """ + detail = "".join(traceback.format_exception(type(exc), exc, exc.__traceback__)) + + if isinstance(exc, TaskCancelled): + return Failure( + kind=FailureKind.CANCELLED, + message=str(exc) or "任务已被取消", + traceback=detail, + retryable=False, + ) + if isinstance(exc, TaskTimeout): + return Failure( + kind=FailureKind.for_timeout(exc.scope), + message=str(exc), + traceback=detail, + retryable=True, + ) + if isinstance(exc, ConfigurationError): + # 配置类错误（目标不可 pickle、任务没注册……）重试多少次都是同一个结果， + # 再试只会白烧重试预算，还会把"改配置"这件事拖到重试耗尽之后才暴露。 + return Failure( + kind=FailureKind.EXCEPTION, + message=str(exc), + traceback=detail, + retryable=False, + ) + return Failure( + kind=FailureKind.EXCEPTION, + message=f"{type(exc).__name__}: {exc}", + traceback=detail, + retryable=True, + ) + + +@runtime_checkable +class ExecutionHandle(Protocol): + """一次执行的句柄。 + + 实时控制（停止、暂停、恢复）都长在句柄上：只有句柄同时持有令牌、心跳与 + 执行载体。这样控制面就不必自己拼装这三者，也不会出现"令牌与载体对不上号"。 + """ + + @property + def token(self) -> CancellationToken: + """本次执行的取消/暂停令牌。""" + ... + + @property + def heartbeat(self) -> Heartbeat: + """本次执行的心跳。""" + ... + + @property + def pid(self) -> int | None: + """执行载体的操作系统进程号；**同进程**执行器为 ``None``。 + + 进程执行器下它是子进程的 pid，会写进任务快照——事后排障时"这个任务当时在哪个 + 进程里"是第一个要问的问题（看板、`ps`、`top` 都靠它对上号）。同进程执行器里 + 任务就跑在监督进程自己身上，再存一个 pid 只是重复。 + """ + ... + + @property + def running(self) -> bool: + """是否仍在运行。 + + 被放弃之后依然返回 ``True``——它确实还在跑。 + """ + ... + + def wait(self, timeout: float | None = None) -> ExecutionOutcome | None: + """等待结束；``timeout`` 内未结束则返回 ``None``。""" + ... + + def stop(self, *, graceful_timeout: float, reason: str | None = None) -> ExecutionOutcome: + """请求停止，最多等待 ``graceful_timeout`` 秒给出结论。 + + Python 无法安全地终止线程，所以超时之后**没有兜底动作**：只能放弃等待，并 + 如实报告 :attr:`ExecutionOutcome.abandoned`（结果未知，由上层落 ``LOST``）。 + 谎报成"已停止"会让运维以为资源已经释放。 + """ + ... + + def pause(self, *, timeout: float) -> bool: + """请求暂停并等待任务确认；返回是否在 ``timeout`` 内确认。 + + 返回 ``False`` 表示暂停**没有生效**（任务未到达检查点）。调用方应据此 + 回滚为 ``RUNNING`` 并告警，而不是假装已暂停。 + """ + ... + + def resume(self) -> None: + """撤销暂停。""" + ... + + +@runtime_checkable +class Executor(Protocol): + """执行器。""" + + name: str + """标识，用于快照的 ``executor`` 字段与排障。""" + + def launch(self, request: ExecutionRequest) -> ExecutionHandle: + """启动一次执行并立即返回句柄（**不得**阻塞到任务结束）。""" + ... + + +class LocalExecutionHandle(abc.ABC): + """同进程执行器的句柄基类（线程与内联执行器实现它）。 + + 集中实现三件与具体执行器无关、但语义必须完全一致的事：令牌与心跳的持有、 + ``pause`` / ``resume`` 的握手、``stop`` 的"请求 → 优雅窗口 → 兜底"三段式。 + + 子类只需回答两个问题：还在跑吗（:attr:`running`），以及优雅窗口用尽之后 + 怎么办（:meth:`_on_stop_timeout`）。 + """ + + __slots__ = ("_forced_outcome", "_heartbeat", "_request", "_token") + + def __init__(self, request: ExecutionRequest, *, clock: Clock | None = None) -> None: + self._request = request + self._token = self._make_token() + self._heartbeat = Heartbeat(clock) + self._forced_outcome: ExecutionOutcome | None = None + + def _make_token(self) -> CancellationToken: + """造本次执行的取消令牌。 + + 子类可换掉实现——进程执行器用它把本地信号**转发**给子进程：控制面有几条路径 + 直接调 ``handle.token.request_stop()``（见 :meth:`~chongming_task.run.TaskRun.stop`）， + 绕过了句柄的 ``stop()``；令牌不转发的话，子进程永远收不到那次停止， + 表现出来就是"停止按钮没反应"。 + """ + return CancellationToken() + + @property + def request(self) -> ExecutionRequest: + """原始请求。""" + return self._request + + @property + def token(self) -> CancellationToken: + """本次执行的取消/暂停令牌。""" + return self._token + + @property + def pid(self) -> int | None: + """同进程执行器没有独立进程号——任务就跑在本进程里。""" + return None + + @property + def heartbeat(self) -> Heartbeat: + """本次执行的心跳。""" + return self._heartbeat + + def build_context(self) -> RuntimeContext: + """构造交给任务的执行期上下文。""" + return RuntimeContext( + task_id=self._request.task_id, + name=self._request.name, + attempt=self._request.attempt, + token=self._token, + heartbeat=self._heartbeat, + artifact_dir=self._request.artifact_dir, + inputs=self._request.inputs, + on_progress=self._request.on_progress, + clock=self._request.clock, + ) + + def pause(self, *, timeout: float) -> bool: + """请求暂停并等待任务确认。""" + self._token.request_pause() + return self._token.wait_paused(timeout) + + def resume(self) -> None: + """撤销暂停。""" + self._token.resume() + + def stop(self, *, graceful_timeout: float, reason: str | None = None) -> ExecutionOutcome: + """请求停止，三段式收场。""" + self._token.request_stop(reason) + outcome = self.wait(graceful_timeout) + if outcome is not None: + return outcome + self._forced_outcome = self._on_stop_timeout(graceful_timeout) + return self._forced_outcome + + def wait(self, timeout: float | None = None) -> ExecutionOutcome | None: + """等待结束。 + + 兜底结论一旦产生就是**粘性**的：后台载体日后返回的结果不会改写已记录的 + ``LOST``，否则审计记录会自相矛盾。 + """ + if self._forced_outcome is not None: + return self._forced_outcome + return self._wait_impl(timeout) + + @property + @abc.abstractmethod + def running(self) -> bool: + """是否仍在运行。被放弃的任务依然返回 ``True``。""" + + @abc.abstractmethod + def _wait_impl(self, timeout: float | None) -> ExecutionOutcome | None: + """实际的等待实现；超时返回 ``None``。""" + + @abc.abstractmethod + def _on_stop_timeout(self, graceful_timeout: float) -> ExecutionOutcome: + """优雅窗口用尽后的兜底。 + + 线程不可强杀，所以实现只能是"放弃等待"，并如实报告 + :attr:`ExecutionOutcome.abandoned`。 + """ diff --git a/src/chongming_task/runtime/executors/inline.py b/src/chongming_task/runtime/executors/inline.py new file mode 100644 index 0000000..52f1046 --- /dev/null +++ b/src/chongming_task/runtime/executors/inline.py @@ -0,0 +1,95 @@ +"""内联执行器：在调用线程内直接跑完。 + +零调度开销，适合微秒级的纯计算步骤，也便于确定性测试。 + +**它刻意违反 ``Executor`` 的"launch 不得阻塞"约定**：``launch`` 会一直执行到 +任务结束才返回。这就是它存在的意义，但也意味着**异步调度器不得使用它**—— +否则单个任务会卡住整个调度循环。它的用途是同步运行路径与测试。 + +停止与暂停对它意义有限：整个执行发生在调用线程内，没有第二个线程可以投递 +信号。任务仍可通过 ``ctx.is_stop_requested`` 自行检查，但外部无法在它运行期间 +介入——因为"运行期间"调用方根本没拿到控制权。 + +``async def`` 任务同样能跑（在本线程新起一个事件循环跑完），但**调用方若已经在事件 +循环里**就不能用它——``asyncio.run`` 不允许嵌套。异步应用请用线程执行器。 +""" + +from __future__ import annotations + +from typing import Any + +from ...clock import Clock +from .base import ( + ExecutionOutcome, + ExecutionRequest, + LocalExecutionHandle, + classify_exception, + run_target, +) + +__all__ = ["InlineExecutionHandle", "InlineExecutor"] + + +class InlineExecutor: + """在调用线程内同步执行任务。""" + + name = "inline" + + def __init__(self, *, clock: Clock | None = None) -> None: + self._clock = clock + + def launch(self, request: ExecutionRequest) -> InlineExecutionHandle: + """同步执行到结束，返回一个已完成的句柄。 + + 注意这与 ``Executor`` 协议的约定不同（见模块文档），是有意为之。 + """ + handle = InlineExecutionHandle(request, clock=self._clock) + handle.run() + return handle + + +class InlineExecutionHandle(LocalExecutionHandle): + """内联执行的句柄。它在 ``launch`` 返回时就已经结束。""" + + __slots__ = ("_error", "_value") + + def __init__(self, request: ExecutionRequest, *, clock: Clock | None = None) -> None: + super().__init__(request, clock=clock) + self._value: Any = None + self._error: BaseException | None = None + + def run(self) -> None: + """执行任务。由 :class:`InlineExecutor` 调用，不应直接使用。""" + context = self.build_context() + try: + self._value = run_target(self._request, context) + except BaseException as exc: + self._error = exc + + @property + def running(self) -> bool: + """内联执行在 ``launch`` 返回时必然已结束。""" + return False + + def pause(self, *, timeout: float) -> bool: + """内联执行无法被暂停：没有第二个线程能投递信号。 + + 返回 ``False`` 是**如实回答**——调用方据此回滚为 ``RUNNING``， + 而不是以为暂停成功了。 + """ + del timeout + return False + + def _wait_impl(self, timeout: float | None) -> ExecutionOutcome | None: + del timeout # 已完成，无需等待 + if self._error is None: + return ExecutionOutcome(value=self._value) + return ExecutionOutcome(failure=classify_exception(self._error)) + + def _on_stop_timeout(self, graceful_timeout: float) -> ExecutionOutcome: + """不会被调用：内联执行在 ``stop`` 之前就已结束。""" + return ExecutionOutcome( + failure=classify_exception( + RuntimeError(f"内联执行无法停止（已等待 {graceful_timeout} 秒）") + ) + ) diff --git a/src/chongming_task/runtime/executors/process.py b/src/chongming_task/runtime/executors/process.py new file mode 100644 index 0000000..96cee1e --- /dev/null +++ b/src/chongming_task/runtime/executors/process.py @@ -0,0 +1,668 @@ +"""进程执行器：每个任务一个守护**子进程**。 + +它换来三件线程给不了的事： + +* **硬性停止保证**：优雅窗口用尽后 ``terminate()`` → ``kill()``，任务是真的停了， + 不是"放弃等待"（线程只能放弃，并如实记 ``LOST``）； +* **绕开 GIL**：CPU 密集的任务能真正并行； +* **崩溃隔离**：任务里加载的原生扩展段错误只会带走子进程，监督进程活着， + 能如实记下一次失败并按策略重试。 + +代价也是三条，都是**结构性的**，不是实现偷懒： + +1. **target / 参数 / 上游数据必须可 pickle**。``lambda``、闭包、局部函数统统不行 + ——它们在子进程里找不到。违反时在 ``launch()`` 当场抛 + :class:`~chongming_task.errors.ConfigurationError`，并告诉调用方改用什么， + 而不是等子进程里神秘失败。 +2. **子进程是守护进程，不能再生子进程**（``multiprocessing`` 的硬规定，为的是让 + 解释器退出时不被卡住）。任务内部要并行，请自己开线程，或改用线程执行器。 +3. **停止与暂停有轮询延迟**（当前 20ms）：信号得靠 ``multiprocessing`` 的事件过界， + 子进程由一个轻量轮询线程把它映射到本地令牌上。线程执行器是即时的。 + +刻意选择 **spawn** 而不是 POSIX 上的 ``fork``：监督进程是**多线程**的，fork 出来的 +子进程会继承一堆状态不一致的锁，那是死锁的经典来源。spawn 起的是全新解释器，代价是 +启动慢一些（毫秒到百毫秒级，还要重新 import 目标模块）。 + +于是有两条使用要求，与 ``multiprocessing`` 本身一模一样：目标函数必须是**模块级** +的，且调用方的脚本要有 ``if __name__ == "__main__":`` 守卫——否则子进程会把主模块的 +顶层代码再跑一遍。 + +## 父进程怎么知道子进程在干什么 + +一条**单向管道**（子 → 父）+ 两个 ``Event``（父 → 子）： + +| 方向 | 载体 | 内容 | +|---|---|---| +| 子 → 父 | 管道 | ``started`` / ``beat`` / ``paused`` / ``progress`` / ``result`` / ``failure`` | +| 父 → 子 | ``Event`` | 停止、暂停（子进程的轮询线程映射成本地令牌信号） | + +**父进程必须持续排空管道**，而且与有没有人在 ``wait()`` 无关：心跳就在那条管道上， +只在 ``wait()`` 里读的话，``idle`` 超时会在任务健康时也触发。所以句柄起一个读取线程， +它同时负责收集进度与最终结论。 + +管道在子进程死亡时给出 ``EOFError``——父进程靠它把"子进程没了"和"子进程给了结论" +区分开，这也正是"崩溃隔离"能如实上报的原因。 +""" + +from __future__ import annotations + +import contextlib +import logging +import os +import pickle +import threading +import time +from dataclasses import dataclass, replace +from multiprocessing import get_context +from multiprocessing.connection import PipeConnection +from multiprocessing.process import BaseProcess +from typing import Any + +from ...clock import Clock, utcnow +from ...errors import ConfigurationError +from ...models import Failure, FailureKind, Progress +from ..cancellation import CancellationToken +from ..context import RuntimeContext +from ..heartbeat import Heartbeat +from .base import ( + ExecutionOutcome, + ExecutionRequest, + LocalExecutionHandle, + classify_exception, + run_target, +) + +__all__ = ["ProcessExecutionHandle", "ProcessExecutor"] + +#: spawn 上下文。用 spawn 而不是 fork 的理由见模块文档。 +_SPAWN = get_context("spawn") + +#: 读取线程等子进程真正退出的时间（避免留下僵尸进程）。 +_JOIN_TIMEOUT = 2.0 + +#: ``terminate()`` 之后留给子进程的时间；仍活着就升级为 ``kill()``。 +#: +#: POSIX 上 ``terminate()`` 发 ``SIGTERM``（本执行器不装信号处理器，因此几乎立即生效， +#: 这点时间只是收尸余量）；Windows 上它就是 ``TerminateProcess``，毫无缓冲余地。 +_TERMINATE_GRACE = 0.2 + +#: 子进程里镜像父进程信号的轮询周期。这就是"停止/暂停有 20ms 延迟"的来源： +#: 用一点延迟换掉"信号本身不可跨进程"这个硬约束。 +_SIGNAL_POLL = 0.02 + +#: 心跳节流间隔：紧凑循环里的 ``checkpoint()`` 不该每次都写一次管道。 +_BEAT_INTERVAL = 0.05 + +#: 读取线程的轮询周期。它同时决定"发现子进程没了"的延迟——管道不给 EOF 的那些情形 +#: （见 :meth:`ProcessExecutionHandle._read_loop` 的说明）全靠它兜住。 +_READ_POLL = 0.05 + +_log = logging.getLogger("chongming_task.executor.process") + + +class ProcessExecutor: + """用子进程执行任务。""" + + name = "process" + + def __init__(self, *, clock: Clock | None = None) -> None: + """``clock`` 只影响父进程这侧的心跳时间戳；子进程用系统时钟。""" + self._clock = clock + + def launch(self, request: ExecutionRequest) -> ProcessExecutionHandle: + """校验载荷可 pickle，然后启动一个守护子进程。""" + _ensure_picklable(request) + handle = ProcessExecutionHandle(request, clock=self._clock) + handle.start() + return handle + + +@dataclass(frozen=True, slots=True) +class _ChildJob: + """发到子进程的载荷：只包含可 pickle 的部分。 + + ``on_progress`` 与 ``clock`` 刻意置空——它们是**闭包**，跨不过进程边界。子进程 + 用自己的时钟，并把进度经管道发回来（父进程收到后再回调真正的 ``on_progress``）。 + """ + + request: ExecutionRequest + + +class _Channel: + """子进程这侧的管道：把消息发给父进程。 + + 管道不是线程安全的：任务的 ``ctx.checkpoint()``（心跳）与轮询线程（暂停期心跳） + 会并发写同一根管道，不加锁内容就会交错损坏。锁只保护 ``send``，不保护任何业务 + 逻辑；发不出去（父进程已经走了）**绝不让任务失败**——可观测性坏了不是任务的错。 + """ + + __slots__ = ("_conn", "_last_beat", "_lock") + + def __init__(self, conn: PipeConnection) -> None: + self._conn = conn + self._lock = threading.Lock() + self._last_beat = 0.0 + + def send(self, kind: str, payload: Any = None) -> None: + with self._lock: + self._conn.send((kind, payload)) + + def beat(self) -> None: + """回报一次心跳，按 :data:`_BEAT_INTERVAL` 节流。""" + now = time.monotonic() + if now - self._last_beat < _BEAT_INTERVAL: + return + self._last_beat = now + with contextlib.suppress(Exception): + # 父进程已经走了：心跳没人收，也绝不能让任务因此失败。 + self.send("beat") + + def progress(self, progress: Progress) -> None: + with contextlib.suppress(Exception): + # 同上：进度丢一条不是任务的问题。 + self.send("progress", progress) + + def close(self) -> None: + with contextlib.suppress(Exception): + self._conn.close() + + +class _MirrorToken(CancellationToken): + """子进程这侧的令牌：把父进程的跨进程信号映射成本地信号。 + + ``CancellationToken`` 内部是 ``threading.Condition``，**不可跨进程传递**，所以 + 子进程自建一个，由 :class:`_SignalMirror` 轮询父进程的 ``Event``，再调用 + ``request_stop`` / ``request_pause`` / ``resume`` 把状态映射进来。 + + 映射（而不是直接解析事件）的好处是：暂停、停止、以及"暂停中的任务也要停得掉" + 这些语义一条都不用重写——跨进程与同进程的行为由**同一个** ``checkpoint()`` 决定。 + """ + + __slots__ = ("_channel",) + + def __init__(self, channel: _Channel) -> None: + super().__init__() + self._channel = channel + + def checkpoint(self) -> None: + """本地协作点 + 一次心跳回传。 + + 心跳挂在这里是有意的：``checkpoint()`` 就是"我还在推进"的唯一信号，``idle`` + 超时靠它在跨进程时依然有效——而它是唯一能发现"卡住但没报错"的机制。 + """ + super().checkpoint() + self._channel.beat() + + +class _SignalMirror(threading.Thread): + """子进程里的轮询线程：把父进程的 ``Event`` 映射到本地令牌，并在暂停期代为心跳。 + + 两个职责都必须在**任务线程之外**： + + * 镜像信号：任务线程可能正阻塞在暂停点上，只有别的线程能叫醒它； + * 暂停期心跳：被暂停的任务不是卡死的，可它自己不会去调 ``checkpoint()``——心跳 + 得由这里代发，否则父进程的 ``idle`` 超时会把一个正常暂停的任务判成卡死。 + """ + + def __init__( + self, + token: _MirrorToken, + channel: _Channel, + stop_event: Any, + pause_event: Any, + ) -> None: + super().__init__(name="task-signal-mirror", daemon=True) + self._token = token + self._channel = channel + self._stop_event = stop_event + self._pause_event = pause_event + self._done = threading.Event() + self._paused_reported = False + + def shutdown(self) -> None: + self._done.set() + + def run(self) -> None: + while not self._done.wait(_SIGNAL_POLL): + try: + self._mirror_once() + except Exception: # pragma: no cover - 父进程没了不该影响任务 + return + + def _mirror_once(self) -> None: + if self._stop_event.is_set(): + self._token.request_stop() + # 停止之后不再需要心跳：任务要么在收尾，要么马上要被强杀。 + return + if self._pause_event.is_set(): + self._token.request_pause() + if self._token.is_pause_acknowledged: + if not self._paused_reported: + # 报一次确认——父进程等的就是这个：它据此把状态落成 PAUSED， + # 而不是以为暂停没生效（那会让两边对同一个任务的判断分叉）。 + self._paused_reported = True + with contextlib.suppress(Exception): + self._channel.send("paused") + # 已经停在暂停点上：代为心跳，别让父进程以为它卡死了。 + self._channel.beat() + return + self._paused_reported = False + self._token.resume() + + +class ProcessExecutionHandle(LocalExecutionHandle): + """子进程执行的句柄。 + + 它兑现了 :class:`~chongming_task.runtime.executors.base.ExecutionHandle` 里那句 + "超时之后如何收场由具体执行器决定"：线程只能放弃并记 ``abandoned``，而这里 + **真的把任务停掉**——所以结论是一个已知的 ``CANCELLED``（任务未完成）， + 而不是"结果未知"。 + """ + + def __init__(self, request: ExecutionRequest, *, clock: Clock | None = None) -> None: + # 这两个事件必须先于 super().__init__() 建好：基类在构造时就会通过 + # _make_token() 取令牌，而转发令牌正需要它们。 + self._stop_event = _SPAWN.Event() + self._pause_event = _SPAWN.Event() + self._pause_acked = threading.Event() + self._settled = threading.Event() + self._lock = threading.Lock() + self._verdict: ExecutionOutcome | None = None + self._child_started = False + self._terminating = threading.Event() + self._process: BaseProcess | None = None + self._reader: threading.Thread | None = None + self._conn: PipeConnection | None = None + super().__init__(request, clock=clock) + + # ------------------------------------------------------------------ + # 控制面 + # ------------------------------------------------------------------ + + def _make_token(self) -> CancellationToken: + """父进程这侧的令牌要**转发**信号：有几条控制路径直接调 ``handle.token``。""" + return _RelayToken(self._stop_event, self._pause_event) + + @property + def pid(self) -> int | None: + """子进程的进程号；尚未启动时为 ``None``。""" + process = self._process + return None if process is None else process.pid + + @property + def process(self) -> BaseProcess | None: + """底层 ``multiprocessing.Process``，供诊断与测试使用。""" + return self._process + + @property + def running(self) -> bool: + """是否仍在运行。 + + 判据是**载体**：子进程还活着，且我们还没拿到结论。被强杀之后立刻为 ``False`` + ——它确实已经死了；谎报成 ``True`` 会让上层以为资源还占着。 + """ + process = self._process + if process is None: + return False + return process.is_alive() and not self._settled.is_set() + + def pause(self, *, timeout: float) -> bool: + """请求暂停，等子进程在下一个 ``checkpoint()`` 确认。 + + 超时**必须回滚**：撤掉暂停请求，让子进程继续跑。不回滚的话，调用方已经按 + "暂停没生效"把状态回滚成 ``RUNNING``，子进程却会在稍后的某个检查点自己停住 + ——那就成了一个没人知道、也没人会恢复的挂起。 + """ + self._pause_acked.clear() + self._token.request_pause() + if self._pause_acked.wait(timeout): + return True + self._token.resume() + return False + + def resume(self) -> None: + """撤销暂停。幂等。""" + self._token.resume() + self._pause_acked.clear() + + def _on_stop_timeout(self, graceful_timeout: float) -> ExecutionOutcome: + """优雅窗口用尽：**真的把子进程杀掉**。 + + 结论是 ``CANCELLED`` 而不是 ``LOST``：任务确实没有完成，而且我们知道这一点 + （线程那边做不到，只能放弃并寄希望于它自己收工）。 + """ + self._terminating.set() + self._terminate_and_reap() + return self._terminated_outcome(graceful_timeout) + + def _terminated_outcome(self, graceful_timeout: float) -> ExecutionOutcome: + return ExecutionOutcome( + failure=Failure( + kind=FailureKind.CANCELLED, + message=( + f"任务在 {graceful_timeout} 秒内未响应停止请求，已强制终止子进程" + f"（pid={self.pid}）" + ), + retryable=False, + ) + ) + + # ------------------------------------------------------------------ + # 生命周期 + # ------------------------------------------------------------------ + + def start(self) -> None: + """起子进程与读取线程。""" + # 单向管道：返回 (recv, send)。子进程拿 send，父进程留 recv。 + self._conn, child_conn = _SPAWN.Pipe(duplex=False) + request = self._request + # 闭包过不去：进度回调与时钟都置空，子进程用自己的。 + job = _ChildJob(request=replace(request, on_progress=None, clock=None)) + self._process = _SPAWN.Process( + target=_child_entry, + args=(job, child_conn, self._stop_event, self._pause_event), + name=f"task-{request.task_id}", + daemon=True, + ) + self._process.start() + # 父进程这侧的写端必须关掉：否则子进程死后管道不会给出 EOF， + # "载体没了"这件事就永远发现不了。 + child_conn.close() + self._reader = threading.Thread( + target=self._read_loop, name=f"task-reader-{request.task_id}", daemon=True + ) + self._reader.start() + + def _wait_impl(self, timeout: float | None) -> ExecutionOutcome | None: + if not self._settled.wait(timeout): + return None + with self._lock: + return self._verdict + + def _read_loop(self) -> None: + """持续排空管道——**与有没有人在 wait 无关**。 + + 只在 ``wait()`` 里读的话，心跳就只在调用方主动轮询时推进，而 ``idle`` 超时靠 + 的正是心跳：一个健康的长期任务会被判成卡死。所以这里是一个独立的线程。 + + **判活以 ``Process.is_alive()`` 为准，不靠管道断开。** 原因实测过：子进程若在 + 反序列化参数时就死掉（目标模块 import 失败、参数过不来），父进程那侧的写端 + 句柄**从未交出去**，管道永远不给 EOF——只等 EOF 的话这里会一直阻塞，子进程 + 早已死了而父进程毫无察觉。管道只负责搬消息。 + """ + conn = self._conn + process = self._process + assert conn is not None and process is not None # start() 先建好它们 + try: + while True: + if conn.poll(_READ_POLL): + try: + kind, payload = conn.recv() + except (EOFError, OSError): + # 管道断了：子进程正常退出时会走到这里——那时结论已在手上。 + break + if not self._dispatch(kind, payload): + break + continue + if not process.is_alive(): + # 载体没了。先把管道里剩下的读完：它可能还来得及把结论发出来。 + self._drain(conn) + break + finally: + conn.close() + self._reap() + if not self._settled.is_set(): + # 没给出结论就没了：崩溃、被外部杀掉、或 os._exit。 + self._settle(self._on_child_vanished()) + + def _drain(self, conn: PipeConnection) -> None: + """子进程已经死了，把管道里残留的消息读完（没有残留就直接返回）。""" + while conn.poll(0): + try: + kind, payload = conn.recv() + except (EOFError, OSError): + return + if not self._dispatch(kind, payload): + return + + def _dispatch(self, kind: str, payload: Any) -> bool: + """处理一条消息；返回 ``False`` 表示"这是最后一条，读完收工"。""" + if kind == "beat": + # 记录的是**父进程收到**的时刻：子进程一旦停摆（哪怕它没崩），这里就不再 + # 更新，idle 超时照常触发。 + self._heartbeat.beat() + return True + if kind == "started": + self._child_started = True + return True + if kind == "paused": + self._pause_acked.set() + return True + if kind == "progress": + self._forward_progress(payload) + return True + if kind == "result": + self._settle(ExecutionOutcome(value=payload)) + return False + if kind == "failure": + self._settle(ExecutionOutcome(failure=payload)) + return False + _log.warning("进程执行器收到未知消息 %r，已忽略", kind) + return True + + def _forward_progress(self, progress: Progress) -> None: + """把子进程的进度交给父进程的 ``on_progress``。 + + 进度是**可观测性**，它的失败绝不能让任务失败——与"存储故障不让任务失败" + 是同一条原则。 + """ + callback = self._request.on_progress + if callback is None: + return + try: + callback(progress) + except Exception as exc: # pragma: no cover - 调用方自己的回调 + _log.warning("进度回调失败：%s: %s", type(exc).__name__, exc) + + def _settle(self, outcome: ExecutionOutcome) -> None: + with self._lock: + if self._verdict is None: + self._verdict = outcome + self._settled.set() + + def _reap(self) -> None: + """等子进程真正退出，避免留下僵尸进程。""" + process = self._process + if process is not None: + process.join(_JOIN_TIMEOUT) + + def _on_child_vanished(self) -> ExecutionOutcome: + """子进程没给结论就没了。 + + "有没有报告过 started"决定了失败的性质，也决定了运维下一步该做什么： + + * **没报告过** —— 子进程根本没跑起来（多半是载荷在子进程里无法反序列化， + 或目标模块 import 失败）。任务没有执行过，重试是干净的。 + * **报告过** —— 任务跑着跑着载体没了（段错误、``os._exit``、被外部杀掉）。 + **结果未知**：副作用可能已经发生了一部分。 + """ + if self._terminating.is_set(): + # 是我们自己按停止请求杀的，结论以那次停止为准。 + return self._terminated_outcome(_TERMINATE_GRACE) + code = self._process.exitcode if self._process is not None else None + if code is None or code == 0: + return ExecutionOutcome( + failure=Failure( + kind=FailureKind.LOST, + message=( + f"子进程（pid={self.pid}）在给出结论前退出（退出码 {code}）；" + "任务可能已在别处产生副作用，结果未知" + ), + retryable=True, + ) + ) + if self._child_started: + message = ( + f"子进程（pid={self.pid}）以退出码 {code} 结束且未给出结论：" + "任务可能在崩溃前已产生部分副作用，结果未知" + ) + else: + message = ( + f"子进程（pid={self.pid}）以退出码 {code} 结束，任务从未开始：" + "通常是 target 或参数在子进程里无法导入/反序列化" + "（子进程的报错已打印到本进程的 stderr）" + ) + return ExecutionOutcome( + failure=Failure(kind=FailureKind.NONZERO_EXIT, message=message, exit_code=code) + ) + + def _terminate_and_reap(self) -> None: + """``terminate()`` → 宽限 → ``kill()``。""" + process = self._process + if process is None: # pragma: no cover - 只会在未启动时发生 + return + process.terminate() + process.join(_TERMINATE_GRACE) + if process.is_alive(): + process.kill() + process.join(_TERMINATE_GRACE) + if self._reader is not None: + # 读取线程会因 EOF 自行退出；这里只是等它收工，好让结论落定。 + self._reader.join(_JOIN_TIMEOUT) + + +class _RelayToken(CancellationToken): + """父进程这侧的令牌：本地置位的同时把信号转发给子进程。 + + 两条控制路径会**直接**调 ``handle.token.request_stop()``（见 + :meth:`~chongming_task.run.TaskRun.stop`），绕过了句柄的 ``stop()``。令牌不转发的话 + 子进程永远收不到那次停止——表现出来就是"停止按钮没反应"。 + """ + + __slots__ = ("_pause_event", "_stop_event") + + def __init__(self, stop_event: Any, pause_event: Any) -> None: + super().__init__() + self._stop_event = stop_event + self._pause_event = pause_event + + def request_stop(self, reason: str | None = None) -> None: + super().request_stop(reason) + # 暂停中的任务也必须停得掉：先解除暂停，否则子进程会一直挂在暂停点上。 + self._pause_event.clear() + self._stop_event.set() + + def request_pause(self) -> None: + super().request_pause() + self._pause_event.set() + + def resume(self) -> None: + super().resume() + self._pause_event.clear() + + +# ---------------------------------------------------------------------- +# 子进程侧 +# ---------------------------------------------------------------------- + + +def _child_entry( + job: _ChildJob, + conn: PipeConnection, + stop_event: Any, + pause_event: Any, +) -> None: + """子进程入口。**必须是模块级函数**：spawn 靠"模块 + 限定名"定位它。 + + 它自己拼装运行期上下文（父进程的令牌、心跳、闭包都过不来），跑完把结果经管道 + 发回去，然后正常退出——父进程那侧读到的 EOF 就是"载体结束了"。 + """ + _ensure_child_logging() + request = job.request + channel = _Channel(conn) + token = _MirrorToken(channel) + mirror = _SignalMirror(token, channel, stop_event, pause_event) + mirror.start() + try: + context = RuntimeContext( + task_id=request.task_id, + name=request.name, + attempt=request.attempt, + token=token, + heartbeat=Heartbeat(), + artifact_dir=request.artifact_dir, + inputs=request.inputs, + # 进度经管道回传，由父进程回调真正的 on_progress。 + on_progress=channel.progress, + clock=utcnow, + ) + channel.send("started", os.getpid()) + try: + value = run_target(request, context) + except BaseException as exc: + # TaskCancelled 是控制流信号、不继承 Exception，必须一起接住：让它逃出 + # 子进程的话，父进程只会看到"子进程消失了"，结论就变成未知。 + channel.send("failure", classify_exception(exc)) + return + try: + channel.send("result", value) + except Exception as exc: + channel.send( + "failure", + Failure( + kind=FailureKind.RESULT_INVALID, + message=( + f"返回值无法跨进程传回（{type(exc).__name__}: {exc}）。" + "进程执行器要求返回值可 pickle，比线程执行器的" + "'可 JSON 序列化'更严——请返回可 pickle 的值，" + "或改用线程执行器。" + ), + retryable=False, + ), + ) + finally: + mirror.shutdown() + channel.close() + + +def _ensure_child_logging() -> None: + """子进程不继承父进程的 logging 配置（spawn 起的是全新解释器）。 + + 不补一个 handler 的话，``ctx.log`` 这类 INFO 级日志会被 ``lastResort`` 直接丢掉 + ——任务在子进程里"什么日志都没有"，是最难查的一种现场。父进程的 stderr 由子进程 + 继承，所以挂到 stderr 上就是"日志还看得见"。已经有配置就不动它。 + """ + root = logging.getLogger() + if root.handlers: + return + handler = logging.StreamHandler() + handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s")) + root.addHandler(handler) + root.setLevel(logging.INFO) + + +def _ensure_picklable(request: ExecutionRequest) -> None: + """在起子进程**之前**验证载荷可 pickle。 + + 为什么要提前查：真让它失败在 ``Process.start()`` 的 spawn 机制深处，报出来的是一句 + 不知从何下手的 ``PicklingError``（甚至只是子进程里的一行 traceback）。在这里拦下 + 就能直接告诉调用方"改成什么"。 + """ + try: + pickle.dumps( + ( + request.target, + request.args, + dict(request.kwargs), + request.artifact_dir, + dict(request.inputs), + ) + ) + except Exception as exc: + raise ConfigurationError( + f"进程执行器的 target / 参数 / 上游数据必须可 pickle：" + f"{type(exc).__name__}: {exc}\n" + "lambda、闭包、局部函数都不可 pickle——这类任务请改用线程执行器" + "（ThreadExecutor）。进程执行器只在需要**强制停止、绕开 GIL、" + "崩溃隔离**时才值得付出这个代价。" + ) from exc diff --git a/src/chongming_task/runtime/executors/thread.py b/src/chongming_task/runtime/executors/thread.py new file mode 100644 index 0000000..b62ac89 --- /dev/null +++ b/src/chongming_task/runtime/executors/thread.py @@ -0,0 +1,139 @@ +"""线程执行器：每个任务一个守护线程。 + +**这是本项目唯一的执行载体**（另有 :class:`~chongming_task.runtime.executors.inline.InlineExecutor` +用于测试同步化）。选它的理由： + +* 线程创建开销在几十微秒量级，进程是毫秒到几十毫秒； +* 线程共享内存，可以直接复用连接池、缓存、已加载的模型； +* 跨进程的方案需要 pickle 参数与返回值，闭包、``lambda``、局部函数、装饰器产物 + 统统用不了——而 ``@task`` 装饰过的函数恰恰都是这类；参数还必须跨进程可传输。 + +代价只有一条：**无法强制停止**。Python 不能安全地终止线程，所以优雅窗口超时后 +只能放弃等待，并如实报告 ``abandoned=True``（上层落 ``LOST``，不谎报 ``STOPPED``）。 + +于是约束落回任务本身，两条都要照做： + +* 长耗时循环里**必须**调 ``ctx.checkpoint()``——否则停不了，也就会被判成卡死； +* 任务要**幂等或可断点续传**：超时与放弃之后它可能仍在后台跑，重跑是调用方的选择， + 后果要能承受。 + +CPU 密集的活请在三处之一处理：把并行放进任务内部（例如任务里自建进程池），把它 +拆成多个任务，或者干脆单独部署到一个节点上——**不要**指望线程能绕开 GIL。这里 +不再提供进程执行器：一个"能强杀但要 pickle、还不能用闭包"的执行器，与这个项目的 +任务形态（装饰器 + 闭包 + 共享连接池）长期相冲，维护两套语义的代价高于收益。 +""" + +from __future__ import annotations + +import threading +from typing import Any + +from ...clock import Clock +from ...models import Failure, FailureKind +from .base import ( + ExecutionOutcome, + ExecutionRequest, + LocalExecutionHandle, + classify_exception, + run_target, +) + +__all__ = ["ThreadExecutionHandle", "ThreadExecutor"] + + +class ThreadExecutor: + """用线程执行任务。""" + + name = "thread" + + def __init__(self, *, clock: Clock | None = None) -> None: + self._clock = clock + + def launch(self, request: ExecutionRequest) -> ThreadExecutionHandle: + """启动守护线程并立即返回。 + + 刻意**不用** ``concurrent.futures.ThreadPoolExecutor``，两个理由： + + 1. 它的工作线程会在解释器退出时被 ``join``。一个不响应取消的任务会让 + 整个进程**无法退出**——对一个专门用来监督"可能不配合"的任务的工具 + 而言，这是致命缺陷。 + 2. 它的线程无法放弃：池子会一直持有那个卡住的线程，无法回收。 + + 并发上限由调度器统一控制（``ConcurrencyPolicy.max_parallel``），执行器 + 不再叠一层池。 + """ + handle = ThreadExecutionHandle(request, clock=self._clock) + handle.start() + return handle + + +class ThreadExecutionHandle(LocalExecutionHandle): + """线程执行的句柄。 + + 线程安全：工作线程只写 ``_value`` / ``_error`` 并在最后 ``set`` 事件； + 其余状态由调度线程读写，二者通过 ``_done`` 建立 happens-before 关系。 + """ + + __slots__ = ("_done", "_error", "_thread", "_value") + + def __init__(self, request: ExecutionRequest, *, clock: Clock | None = None) -> None: + super().__init__(request, clock=clock) + self._done = threading.Event() + self._value: Any = None + self._error: BaseException | None = None + self._thread = threading.Thread( + target=self._body, + name=f"task-{request.task_id}", + daemon=True, + ) + + @property + def thread(self) -> threading.Thread: + """底层线程，供诊断与测试使用。""" + return self._thread + + def start(self) -> None: + """启动线程。""" + self._thread.start() + + def _body(self) -> None: + context = self.build_context() + try: + self._value = run_target(self._request, context) + except BaseException as exc: + # 必须捕获 BaseException：TaskCancelled 是控制流信号、不继承 Exception。 + # 若任由它逃出工作线程，解释器只会打印一行日志然后丢弃它，调度器 + # 将永远等不到结论。 + self._error = exc + finally: + self._done.set() + + @property + def running(self) -> bool: + """是否仍在运行。被放弃的任务依然返回 ``True``。""" + return not self._done.is_set() + + def _wait_impl(self, timeout: float | None) -> ExecutionOutcome | None: + if not self._done.wait(timeout): + return None + return self._collect() + + def _on_stop_timeout(self, graceful_timeout: float) -> ExecutionOutcome: + """线程无法强杀，因此这里只能放弃——架构级限制，不是实现偷懒。""" + return ExecutionOutcome( + abandoned=True, + failure=Failure( + kind=FailureKind.LOST, + message=( + f"任务在 {graceful_timeout} 秒内未响应停止请求；" + "线程无法强制终止，已放弃等待" + "（任务可能仍在后台运行并产生副作用）" + ), + retryable=True, + ), + ) + + def _collect(self) -> ExecutionOutcome: + if self._error is None: + return ExecutionOutcome(value=self._value) + return ExecutionOutcome(failure=classify_exception(self._error)) diff --git a/src/chongming_task/runtime/heartbeat.py b/src/chongming_task/runtime/heartbeat.py new file mode 100644 index 0000000..76e1959 --- /dev/null +++ b/src/chongming_task/runtime/heartbeat.py @@ -0,0 +1,49 @@ +"""心跳。 + +心跳由 ``checkpoint()`` 驱动，**不额外起线程**。理由：一个不调用 checkpoint 的 +任务本来就无法被暂停或优雅停止，再给它伪造心跳只会掩盖问题。 + +代价必须明确：``idle`` 超时衡量的是 **checkpoint 间隔**，而不是"CPU 是否在干活"。 +一段长时间不 checkpoint 的重计算会被判定为疑似卡死。这是我们希望暴露的信号， +但也要求任务作者让 checkpoint 频率与 ``idle`` 超时相匹配。 +""" + +from __future__ import annotations + +from datetime import datetime + +from ..clock import Clock, utcnow + +__all__ = ["Heartbeat"] + + +class Heartbeat: + """最近一次活动时间。 + + 线程安全性与令牌一致：由持有该任务执行权的单个线程写入。 + """ + + __slots__ = ("_clock", "_last") + + def __init__(self, clock: Clock | None = None) -> None: + self._clock: Clock = clock if clock is not None else utcnow + self._last = self._clock() + + def beat(self) -> None: + """记录一次活动。""" + self._last = self._clock() + + @property + def last_beat(self) -> datetime: + """最近一次活动的时刻。""" + return self._last + + def idle_seconds(self) -> float: + """距最近一次活动已过去的秒数。""" + return (self._clock() - self._last).total_seconds() + + def is_stale(self, idle_timeout: float) -> bool: + """静默时长是否已达到 ``idle_timeout``。""" + if idle_timeout <= 0: + raise ValueError(f"idle_timeout 必须为正数，收到 {idle_timeout}") + return self.idle_seconds() >= idle_timeout diff --git a/src/chongming_task/runtime/timeouts.py b/src/chongming_task/runtime/timeouts.py new file mode 100644 index 0000000..2d98329 --- /dev/null +++ b/src/chongming_task/runtime/timeouts.py @@ -0,0 +1,86 @@ +"""超时判定。 + +三个作用域相互独立，取**最早**到期者： + +* ``start`` —— 从提交到真正开始执行，一旦开始执行即失效； +* ``per_attempt`` —— 单次尝试的总时长； +* ``idle`` —— 距最近一次心跳的静默时长（语义见 :mod:`~chongming_task.runtime.heartbeat`）。 + +本模块只做**纯计算**，不做等待、不设计时循环——那属于调度器。这样判定逻辑可以 +用 ``FakeClock`` 精确测试，而调度器只负责在到期时唤醒。 +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timedelta + +from ..models import FailureKind, TimeoutScope +from ..policies import Timeout + +__all__ = ["Deadline", "is_expired", "next_deadline", "remaining_seconds"] + + +@dataclass(frozen=True, slots=True) +class Deadline: + """一个已确定的到期点。""" + + scope: TimeoutScope + at: datetime + """到期时刻。""" + + @property + def kind(self) -> FailureKind: + """由作用域得到对应的失败归类，避免调用处手写映射而拼错。""" + return FailureKind.for_timeout(self.scope) + + +def next_deadline( + timeout: Timeout, + *, + submitted_at: datetime | None = None, + started_at: datetime | None = None, + last_beat_at: datetime | None = None, +) -> Deadline | None: + """返回下一个到期点；没有任何生效的超时则返回 ``None``。 + + 传 ``None`` 表示对应阶段尚未开始，该作用域自动失效——例如 ``started_at`` + 一旦给出，``start`` 超时就不再参与竞争。 + """ + if not timeout.is_enabled: + # 一档超时都没配：这条路径在监督循环里每轮都会走，早退掉省下建表与排序。 + return None + + candidates: list[Deadline] = [] + + if timeout.start is not None and submitted_at is not None and started_at is None: + candidates.append( + Deadline(TimeoutScope.START, submitted_at + timedelta(seconds=timeout.start)) + ) + if timeout.per_attempt is not None and started_at is not None: + candidates.append( + Deadline(TimeoutScope.ATTEMPT, started_at + timedelta(seconds=timeout.per_attempt)) + ) + # idle 必须要求已开始执行：任务还没跑起来就谈不上"卡住"， + # 否则排队等待的时长会被误判成静默超时。 + if timeout.idle is not None and last_beat_at is not None and started_at is not None: + candidates.append( + Deadline(TimeoutScope.IDLE, last_beat_at + timedelta(seconds=timeout.idle)) + ) + + if not candidates: + return None + return min(candidates, key=lambda deadline: deadline.at) + + +def is_expired(deadline: Deadline, *, now: datetime) -> bool: + """``now`` 是否已到达或越过 ``deadline``。""" + return now >= deadline.at + + +def remaining_seconds(deadline: Deadline, *, now: datetime) -> float: + """距到期还剩多少秒；已过期则为负值。 + + 调度器用它决定下一次唤醒的等待时长——**不要把负数直接交给 sleep**。 + """ + return (deadline.at - now).total_seconds() diff --git a/src/chongming_task/states.py b/src/chongming_task/states.py new file mode 100644 index 0000000..06dc950 --- /dev/null +++ b/src/chongming_task/states.py @@ -0,0 +1,239 @@ +"""任务生命周期状态与合法转移表。 + +设计要点 +-------- +1. ``TaskStatus`` 只表达**生命周期阶段**。重试次数、进度、心跳健康、失败原因属于 + 正交维度，分别由 :mod:`chongming_task.models` 中的 ``attempt`` / ``progress`` / + ``last_heartbeat_at`` / ``failure`` 承载。把它们的组合塞进同一个枚举会迅速导致 + 状态爆炸，并且无法表达"已超时但正在重试"这类情形。 +2. 转移表以**数据**表达而非 ``if/else``，因此可被属性测试穷举验证、可内省、 + 可由工具生成文档。 +3. 终态在表中显式声明为空集，使"终态不可离开"成为可断言的事实，而不是散落在 + 各处的隐式约定。复活只能经由显式的 ``retry()`` / ``reset()``。 +""" + +from __future__ import annotations + +from collections.abc import Mapping +from enum import StrEnum +from types import MappingProxyType +from typing import Final + +__all__ = [ + "NON_TERMINAL", + "RESOURCE_HOLDING", + "REVIVABLE", + "TERMINAL", + "TRANSITIONS", + "TaskStatus", + "allowed_from", + "can_transition", + "is_terminal", +] + + +class TaskStatus(StrEnum): + """任务生命周期状态。终态集合见 :data:`TERMINAL`。""" + + # ---- 非终态 ---- + PENDING = "pending" + """已创建，依赖尚未解析。""" + BLOCKED = "blocked" + """依赖未满足，等待上游。""" + QUEUED = "queued" + """就绪，等待执行槽；暂停后恢复也回到此状态。""" + STARTING = "starting" + """执行器启动中（spawn 进程 / 建立连接 / 加载资源）。""" + RUNNING = "running" + PAUSING = "pausing" + """已请求暂停，等待任务到达静默点。""" + PAUSED = "paused" + """已静止，不再消耗执行资源。""" + RETRY_WAIT = "retry_wait" + """本次尝试失败，正在退避等待。""" + STOPPING = "stopping" + """已请求停止，处于优雅停止窗口内。""" + # ---- 终态 ---- + SUCCEEDED = "succeeded" + """正常完成。""" + FAILED = "failed" + """出错且重试耗尽，或遇到不可重试的错误。""" + STOPPED = "stopped" + """人工主动停止。语义上不是错误，与 CANCELLED 区分责任方。""" + CANCELLED = "cancelled" + """编排层级联取消（上游失败 / 整条管道被取消）。""" + SKIPPED = "skipped" + """条件为假，或依策略跳过。""" + LOST = "lost" + """执行载体失联（进程消失、心跳超时），结果未知。""" + + # 兼容早期版本中语义模糊的 FINISHED：它无法区分"成功结束"与"任意终止"。 + FINISHED = "succeeded" + + +#: 合法状态转移表：``当前状态 -> 允许的后续状态``。 +#: +#: 终态显式声明为空 ``frozenset``，因此 :data:`TERMINAL` 可以由本表推导， +#: 无需两处维护（避免表与集合不一致这一最常见缺陷）。 +TRANSITIONS: Final[Mapping[TaskStatus, frozenset[TaskStatus]]] = MappingProxyType( + { + # **统一规则**：控制动作是否需要中间态，取决于任务是否**已占用执行资源**。 + # 未占用（PENDING/BLOCKED/QUEUED/PAUSED/RETRY_WAIT）→ 立即生效； + # 已占用（STARTING/RUNNING/PAUSING）→ 经中间态缓冲。 + # 让未启动的任务显示"停止中/暂停中"是虚假信息，会让运维误以为需要等待。 + TaskStatus.PENDING: frozenset( + { + TaskStatus.BLOCKED, + TaskStatus.QUEUED, + TaskStatus.SKIPPED, + TaskStatus.CANCELLED, + TaskStatus.STOPPED, + } + ), + TaskStatus.BLOCKED: frozenset( + { + TaskStatus.QUEUED, + TaskStatus.SKIPPED, + TaskStatus.CANCELLED, + TaskStatus.STOPPED, + } + ), + TaskStatus.QUEUED: frozenset( + { + TaskStatus.STARTING, + TaskStatus.PAUSED, + TaskStatus.SKIPPED, + TaskStatus.STOPPED, + TaskStatus.CANCELLED, + } + ), + TaskStatus.STARTING: frozenset( + { + TaskStatus.RUNNING, + TaskStatus.RETRY_WAIT, + TaskStatus.PAUSING, + TaskStatus.STOPPING, + TaskStatus.FAILED, + TaskStatus.CANCELLED, + TaskStatus.LOST, + } + ), + # CANCELLED 是编排层的**级联**取消，运行中的任务同样可能被级联取消 + # （例如同批次的兄弟任务失败）。这与"用户主动 stop"是不同的路径： + # 后者必须经过 STOPPING，最终落 STOPPED。 + TaskStatus.RUNNING: frozenset( + { + TaskStatus.SUCCEEDED, + TaskStatus.FAILED, + TaskStatus.RETRY_WAIT, + TaskStatus.PAUSING, + TaskStatus.STOPPING, + TaskStatus.CANCELLED, + TaskStatus.LOST, + } + ), + # RUNNING 表示暂停请求被撤销/回滚；STOPPING 表示暂停期间收到停止请求。 + TaskStatus.PAUSING: frozenset( + { + TaskStatus.PAUSED, + TaskStatus.RUNNING, + TaskStatus.STOPPING, + TaskStatus.FAILED, + TaskStatus.CANCELLED, + TaskStatus.LOST, + } + ), + # 恢复的目标取决于**暂停发生在哪一层**： + # 就地暂停（STARTING/RUNNING -> PAUSING -> PAUSED）→ RUNNING， + # 任务始终占着执行槽，调用栈还在内存里； + # 未启动即暂停（QUEUED/RETRY_WAIT -> PAUSED）→ QUEUED，重新排队。 + # 无 STOPPING 出边：暂停中的任务没有占用新的执行资源，stop() 直接落 STOPPED。 + TaskStatus.PAUSED: frozenset( + { + TaskStatus.QUEUED, + TaskStatus.RUNNING, + TaskStatus.STOPPED, + TaskStatus.CANCELLED, + } + ), + # 无 PAUSING/STOPPING 出边：退避等待中的任务没有占用执行资源， + # 暂停与停止都立即生效。 + TaskStatus.RETRY_WAIT: frozenset( + { + TaskStatus.STARTING, + TaskStatus.QUEUED, + TaskStatus.PAUSED, + TaskStatus.STOPPED, + TaskStatus.FAILED, + TaskStatus.CANCELLED, + } + ), + # SUCCEEDED 是"恰好跑完":停止请求发出后任务自然结束，属于合法结果。 + TaskStatus.STOPPING: frozenset( + { + TaskStatus.STOPPED, + TaskStatus.SUCCEEDED, + TaskStatus.FAILED, + TaskStatus.LOST, + } + ), + # ---- 终态：无出边 ---- + TaskStatus.SUCCEEDED: frozenset(), + TaskStatus.FAILED: frozenset(), + TaskStatus.STOPPED: frozenset(), + TaskStatus.CANCELLED: frozenset(), + TaskStatus.SKIPPED: frozenset(), + TaskStatus.LOST: frozenset(), + } +) + +#: 终态，由转移表推导（出边为空者）。 +TERMINAL: Final[frozenset[TaskStatus]] = frozenset( + status for status, successors in TRANSITIONS.items() if not successors +) + +#: 非终态。注意 ``frozenset(TaskStatus)`` 只包含规范成员，不含 ``FINISHED`` 之类的别名。 +NON_TERMINAL: Final[frozenset[TaskStatus]] = frozenset(TaskStatus) - TERMINAL + +#: 允许经 ``retry()`` 重新进入执行流程的终态。 +#: +#: ``SUCCEEDED`` 与 ``SKIPPED`` 不在其中：前者重跑没有意义，后者是策略决定的结果。 +REVIVABLE: Final[frozenset[TaskStatus]] = frozenset( + {TaskStatus.FAILED, TaskStatus.STOPPED, TaskStatus.CANCELLED, TaskStatus.LOST} +) + +#: **占用执行资源**的非终态：这些状态下存在一个活的执行载体（线程/进程/远端作业）。 +#: +#: 它是重启恢复的判据之一：这类任务若失去监督者，不是"等待中"而是**已经丢了**—— +#: 载体随进程一起消失，谁也不会再推进它，只能落 ``LOST``。相对的，``queued`` / +#: ``paused`` / ``retry_wait`` 不占资源，失去监督者只是"没人接手"，不是"丢了"。 +#: +#: 注意与 ``state_machine._PAUSE_IS_DEFERRED`` 的区别：那个集合更小（只有 +#: ``starting`` / ``running``），语义是"暂停要等到检查点"，而不是"占着资源"—— +#: 已经 ``pausing`` 的任务不必再等一次检查点。 +RESOURCE_HOLDING: Final[frozenset[TaskStatus]] = frozenset( + { + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.PAUSING, + TaskStatus.STOPPING, + } +) + + +def is_terminal(status: TaskStatus) -> bool: + """``status`` 是否为终态（不可能再自行变化）。""" + return status in TERMINAL + + +def allowed_from(status: TaskStatus) -> frozenset[TaskStatus]: + """从 ``status`` 出发允许到达的状态集合；终态返回空集合。""" + return TRANSITIONS[status] + + +def can_transition(source: TaskStatus, target: TaskStatus) -> bool: + """``source`` 是否可以合法转移到 ``target``。 + + 仅依据转移表判断，不涉及守卫条件（如并发槽、依赖是否满足）。 + """ + return target in TRANSITIONS[source] diff --git a/src/chongming_task/stores/__init__.py b/src/chongming_task/stores/__init__.py new file mode 100644 index 0000000..176b208 --- /dev/null +++ b/src/chongming_task/stores/__init__.py @@ -0,0 +1,69 @@ +"""持久层实现。 + +:mod:`~chongming_task.stores.protocol` 定义契约，本包提供： + +* :class:`~chongming_task.stores.memory.InMemoryTaskStore` —— 单进程 / 测试 / 参照物； +* :class:`~chongming_task.stores.sqlite.SqliteTaskStore` —— 跨进程与持久化（标准库）； +* :class:`~chongming_task.stores.postgres.PostgresTaskStore` —— 跨机部署（可选依赖）。 + +**换持久层只需要做两件事**： + +1. 实现 :class:`TaskStore` 契约里的方法（语义要求见 ``docs/store-guide.md``）； +2. 用 ``chongming_task.testing.store_contract.StoreContract`` 自证。 + +代码里用不到继承或注册——``Controller(store=...)`` 只做结构化检查，任何满足契约的 +对象都能直接塞进去。序列化细节可以复用 :mod:`~chongming_task.stores.codec`，不必 +自己重写一遍 ``datetime`` / ``frozenset`` / 枚举 / 嵌套结构的往返。 + +``PostgresTaskStore`` 需要可选的 psycopg，因此它**惰性导入**（既不在 ``__all__`` 里， +也不在包导入时求值）——装没装驱动都不影响 ``import chongming_task``。 +""" + +from __future__ import annotations + +from typing import Any + +from .codec import ( + command_to_row, + decode_dt, + encode_dt, + row_to_command, + row_to_snapshot, + row_to_transition, + snapshot_to_row, + transition_to_row, +) +from .memory import InMemoryTaskStore +from .protocol import Command, CommandKind, TaskQuery, TaskStore +from .sqlite import SqliteTaskStore + +__all__ = [ + "Command", + "CommandKind", + "InMemoryTaskStore", + "SqliteTaskStore", + "TaskQuery", + "TaskStore", + "command_to_row", + "decode_dt", + "encode_dt", + "row_to_command", + "row_to_snapshot", + "row_to_transition", + "snapshot_to_row", + "transition_to_row", +] + + +def __getattr__(name: str) -> Any: + """惰性导出可选依赖的实现。 + + 见模块文档：``PostgresTaskStore`` 需要 psycopg，把它放进 ``__all__`` 会让 + ``from chongming_task.stores import *`` 在没装驱动时报错——那是核心包不该有的 + 依赖。懒加载保证"用到了才需要装"。 + """ + if name == "PostgresTaskStore": + from .postgres import PostgresTaskStore + + return PostgresTaskStore + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/src/chongming_task/stores/codec.py b/src/chongming_task/stores/codec.py new file mode 100644 index 0000000..af55f7c --- /dev/null +++ b/src/chongming_task/stores/codec.py @@ -0,0 +1,300 @@ +"""把领域对象与"扁平的原语映射"互相转换。**公开契约的一部分。** + +任何要落地持久化的后端都躲不开这件事：:class:`~chongming_task.models.TaskSnapshot` +里有 ``datetime``、``frozenset``、枚举和嵌套的 ``Progress`` / ``Failure``，而数据库行、 +JSON 文档、Redis Hash 只认原语。把这些细节收敛在这里，后端作者就不必各自重写一遍， +也不会各自重写错一遍。 + +映射的形状是**扁平的一层字典**（列名 → 原语），因为这是最容易落到任何后端的形式： + +* 关系库 → 直接就是一行； +* 文档库 / JSON 文件 → 存成一个子对象，或按需展平； +* KV（Redis Hash）→ 键值对直接可写。 + +往返保真是**契约要求**，不是"尽量"：``row_to_snapshot(snapshot_to_row(s)) == s`` +必须成立。``StoreContract`` 里有一条用例专门用"每个字段都填满"的快照验证它—— +只测几个字段会漏掉时区、``frozenset``、嵌套结构这类静默丢失。 +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from datetime import UTC, datetime +from typing import Any + +from ..core.state_machine import Transition +from ..models import Failure, FailureKind, Progress, RequestedAction, TaskSnapshot +from ..states import TaskStatus +from .protocol import Command, CommandKind + +__all__ = [ + "command_to_row", + "decode_dt", + "decode_failure", + "decode_progress", + "decode_tags", + "encode_dt", + "encode_failure", + "encode_progress", + "encode_tags", + "row_to_command", + "row_to_snapshot", + "row_to_transition", + "snapshot_to_row", + "transition_to_row", +] + + +# ---------------------------------------------------------------------- +# 原语 +# ---------------------------------------------------------------------- + + +def encode_dt(value: datetime | None) -> str | None: + """存成 UTC 的 ISO 字符串。 + + 一律先转 UTC：跨时区部署时把本地时间原样写进去，排序与比较都会错，而且不报错。 + """ + return value.astimezone(UTC).isoformat() if value is not None else None + + +def decode_dt(raw: str | None) -> datetime | None: + """读回时间。``None`` 原样返回。""" + return datetime.fromisoformat(raw) if raw else None + + +def encode_tags(tags: frozenset[str]) -> str: + """标签存成 JSON 数组。 + + 排序后再写：不排序的话，内容相同的两个集合会产生不同的字节串，让"判断是否变化" + 这类比较失效。 + """ + return json.dumps(sorted(tags), ensure_ascii=False) + + +def decode_tags(raw: str) -> frozenset[str]: + """读回标签集合。""" + return frozenset(json.loads(raw)) + + +def _dump(value: Any) -> str | None: + return json.dumps(value, ensure_ascii=False, sort_keys=True) if value is not None else None + + +def _load(raw: str | None) -> Any: + return json.loads(raw) if raw else None + + +# ---------------------------------------------------------------------- +# 嵌套结构 +# ---------------------------------------------------------------------- + + +def encode_progress(progress: Progress | None) -> str | None: + """进度 → JSON 文本。""" + if progress is None: + return None + return _dump( + { + "done": progress.done, + "total": progress.total, + "unit": progress.unit, + "message": progress.message, + "updated_at": encode_dt(progress.updated_at), + } + ) + + +def decode_progress(raw: str | None) -> Progress | None: + """JSON 文本 → 进度。""" + data = _load(raw) + if data is None: + return None + return Progress( + done=data["done"], + total=data["total"], + unit=data["unit"], + message=data["message"], + updated_at=decode_dt(data["updated_at"]), + ) + + +def encode_failure(failure: Failure | None) -> str | None: + """失败信息 → JSON 文本。 + + ``FailureKind`` 是 ``StrEnum``，必须显式取 ``.value``：直接把枚举丢给 + ``json.dumps`` 在部分版本上会写成一个不可读的形式。 + """ + if failure is None: + return None + return _dump( + { + "kind": failure.kind.value, + "message": failure.message, + "traceback": failure.traceback, + "exit_code": failure.exit_code, + "retryable": failure.retryable, + } + ) + + +def decode_failure(raw: str | None) -> Failure | None: + """JSON 文本 → 失败信息。""" + data = _load(raw) + if data is None: + return None + return Failure( + kind=FailureKind(data["kind"]), + message=data["message"], + traceback=data["traceback"], + exit_code=data["exit_code"], + retryable=data["retryable"], + ) + + +# ---------------------------------------------------------------------- +# 快照 +# ---------------------------------------------------------------------- + + +def snapshot_to_row(snapshot: TaskSnapshot) -> dict[str, Any]: + """快照 → 扁平原语映射。 + + 键的顺序与 :data:`chongming_task.stores.sqlite._TASK_COLUMNS` 一致，关系库可以 + 直接拿它拼 SQL。 + """ + return { + "task_id": snapshot.task_id, + "name": snapshot.name, + "pipeline_run_id": snapshot.pipeline_run_id, + "tags": encode_tags(snapshot.tags), + "bucket": snapshot.bucket, + "bucket_key": snapshot.bucket_key, + "status": snapshot.status.value, + "status_version": snapshot.status_version, + "requested_action": ( + snapshot.requested_action.value if snapshot.requested_action else None + ), + "attempt": snapshot.attempt, + "max_attempts": snapshot.max_attempts, + "created_at": encode_dt(snapshot.created_at), + "queued_at": encode_dt(snapshot.queued_at), + "started_at": encode_dt(snapshot.started_at), + "ended_at": encode_dt(snapshot.ended_at), + "last_heartbeat_at": encode_dt(snapshot.last_heartbeat_at), + "progress": encode_progress(snapshot.progress), + "failure": encode_failure(snapshot.failure), + "outcome": _dump(dict(snapshot.outcome)) if snapshot.outcome is not None else None, + "executor": snapshot.executor, + "pid": snapshot.pid, + "overridden": int(snapshot.overridden), + "lease_owner": snapshot.lease_owner, + "lease_until": encode_dt(snapshot.lease_until), + } + + +def row_to_snapshot(row: Mapping[str, Any]) -> TaskSnapshot: + """扁平原语映射 → 快照。 + + ``overridden`` 从整数读回布尔：SQLite 没有布尔类型，写进去的是 0/1。 + """ + requested = row["requested_action"] + return TaskSnapshot( + task_id=row["task_id"], + name=row["name"], + pipeline_run_id=row["pipeline_run_id"], + tags=decode_tags(row["tags"]), + bucket=row["bucket"], + bucket_key=row["bucket_key"], + status=TaskStatus(row["status"]), + status_version=row["status_version"], + requested_action=RequestedAction(requested) if requested else None, + attempt=row["attempt"], + max_attempts=row["max_attempts"], + created_at=decode_dt(row["created_at"]), + queued_at=decode_dt(row["queued_at"]), + started_at=decode_dt(row["started_at"]), + ended_at=decode_dt(row["ended_at"]), + last_heartbeat_at=decode_dt(row["last_heartbeat_at"]), + progress=decode_progress(row["progress"]), + failure=decode_failure(row["failure"]), + outcome=_load(row["outcome"]), + executor=row["executor"], + pid=row["pid"], + overridden=bool(row["overridden"]), + lease_owner=row["lease_owner"], + lease_until=decode_dt(row["lease_until"]), + ) + + +# ---------------------------------------------------------------------- +# 迁移记录（审计轨迹，只追加） +# ---------------------------------------------------------------------- + + +def transition_to_row(transition: Transition) -> dict[str, Any]: + """迁移记录 → 扁平原语映射。""" + return { + "task_id": transition.task_id, + "version": transition.version, + "from_status": transition.from_status.value, + "to_status": transition.to_status.value, + "at": encode_dt(transition.at), + "trigger": transition.trigger, + "actor": transition.actor, + "reason": transition.reason, + } + + +def row_to_transition(row: Mapping[str, Any]) -> Transition: + """扁平原语映射 → 迁移记录。 + + ``at`` 缺失时回退到"现在"：宁可时间戳不准，也不要因为一条历史记录读不出来而让 + 整个任务查询失败。 + """ + return Transition( + task_id=row["task_id"], + from_status=TaskStatus(row["from_status"]), + to_status=TaskStatus(row["to_status"]), + at=decode_dt(row["at"]) or datetime.now(UTC), + trigger=row["trigger"], + actor=row["actor"], + reason=row["reason"], + version=row["version"], + ) + + +# ---------------------------------------------------------------------- +# 控制指令 +# ---------------------------------------------------------------------- + + +def command_to_row(item: Command) -> dict[str, Any]: + """控制指令 → 扁平原语映射。""" + return { + "command_id": item.command_id, + "task_id": item.task_id, + "kind": item.kind.value, + "created_at": encode_dt(item.created_at), + "requested_by": item.requested_by, + "reason": item.reason, + "lease_until": encode_dt(item.lease_until), + "finished_at": encode_dt(item.finished_at), + "outcome": item.outcome, + } + + +def row_to_command(row: Mapping[str, Any]) -> Command: + """扁平原语映射 → 控制指令。""" + return Command( + command_id=row["command_id"], + task_id=row["task_id"], + kind=CommandKind(row["kind"]), + created_at=decode_dt(row["created_at"]) or datetime.min.replace(tzinfo=UTC), + requested_by=row["requested_by"], + reason=row["reason"], + lease_until=decode_dt(row["lease_until"]), + finished_at=decode_dt(row["finished_at"]), + outcome=row["outcome"], + ) diff --git a/src/chongming_task/stores/memory.py b/src/chongming_task/stores/memory.py new file mode 100644 index 0000000..1a4bfd0 --- /dev/null +++ b/src/chongming_task/stores/memory.py @@ -0,0 +1,224 @@ +"""内存存储实现。 + +适合：单进程嵌入（不关心重启）、单元测试、以及作为其他实现的**参照物**—— +新写一个后端时，先用它对照 :class:`~chongming_task.stores.protocol.TaskStore` 的 +语义，比读文档可靠。 + +不适合：跨进程查询（内存不共享）。那正是数据库实现存在的理由。 +""" + +from __future__ import annotations + +import threading +from dataclasses import replace +from datetime import datetime, timedelta + +from ..core.state_machine import Transition +from ..errors import ConcurrencyConflict, StoreError +from ..models import Progress, TaskSnapshot +from ..states import TERMINAL, TaskStatus +from .protocol import Command, TaskQuery + +__all__ = ["InMemoryTaskStore"] + + +class InMemoryTaskStore: + """把任务、迁移历史与控制指令放在进程内存里。""" + + def __init__(self) -> None: + self._lock = threading.RLock() + self._tasks: dict[str, TaskSnapshot] = {} + self._history: dict[str, dict[int, Transition]] = {} + self._commands: dict[str, Command] = {} + + # ------------------------------------------------------------------ + # 1. 任务快照 + # ------------------------------------------------------------------ + + def create(self, snapshot: TaskSnapshot) -> None: + """写入新任务；``task_id`` 已存在则抛错。""" + with self._lock: + if snapshot.task_id in self._tasks: + raise StoreError(f"任务已存在：{snapshot.task_id}") + self._tasks[snapshot.task_id] = snapshot + self._history.setdefault(snapshot.task_id, {}) + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + """按乐观并发更新快照。 + + **保留租约列**：它们由 :meth:`touch` 独占。直接整体替换会让一次状态迁移把租约 + 清空，于是重启恢复把一个活得好好的任务误判为失去监督者——契约测试里有一条 + 用例专门盯这件事。 + """ + with self._lock: + current = self._tasks.get(snapshot.task_id) + if current is None: + raise StoreError(f"任务不存在：{snapshot.task_id}") + if current.status_version != expect_version: + raise ConcurrencyConflict(snapshot.task_id, expect_version, current.status_version) + self._tasks[snapshot.task_id] = replace( + snapshot, + lease_owner=current.lease_owner, + lease_until=current.lease_until, + ) + + def get(self, task_id: str) -> TaskSnapshot | None: + """读取单个任务。""" + with self._lock: + return self._tasks.get(task_id) + + def query(self, query: TaskQuery | None = None) -> list[TaskSnapshot]: + """按条件查询。""" + condition = query if query is not None else TaskQuery() + with self._lock: + found = list(self._tasks.values()) + + if condition.statuses is not None: + found = [item for item in found if item.status in condition.statuses] + if condition.name is not None: + found = [item for item in found if item.name == condition.name] + if condition.name_prefix is not None: + found = [item for item in found if item.name.startswith(condition.name_prefix)] + if condition.tag is not None: + found = [item for item in found if condition.tag in item.tags] + if condition.buckets is not None: + found = [item for item in found if item.bucket in condition.buckets] + if condition.pipeline_run_id is not None: + found = [item for item in found if item.pipeline_run_id == condition.pipeline_run_id] + + found.sort( + key=lambda item: (item.created_at is None, item.created_at or datetime.min), + reverse=condition.newest_first, + ) + return found[: condition.limit] if condition.limit is not None else found + + def counts(self) -> dict[TaskStatus, int]: + """各状态计数。""" + with self._lock: + result: dict[TaskStatus, int] = {} + for snapshot in self._tasks.values(): + result[snapshot.status] = result.get(snapshot.status, 0) + 1 + return result + + # ------------------------------------------------------------------ + # 2. 审计轨迹 + # ------------------------------------------------------------------ + + def append(self, transition: Transition) -> None: + """追加一条迁移记录；同一版本重复写入是幂等的。""" + with self._lock: + entries = self._history.setdefault(transition.task_id, {}) + entries.setdefault(transition.version, transition) + + def history(self, task_id: str, *, since_version: int = 0) -> list[Transition]: + """读取迁移历史。""" + with self._lock: + entries = self._history.get(task_id, {}) + return [entry for version, entry in sorted(entries.items()) if version > since_version] + + # ------------------------------------------------------------------ + # 3. 高频旁路 + # ------------------------------------------------------------------ + + def touch( + self, + task_id: str, + *, + progress: Progress | None = None, + heartbeat_at: datetime | None = None, + lease_owner: str | None = None, + lease_until: datetime | None = None, + ) -> None: + """只更新进度、心跳与监督者租约，不校验也不递增版本号。""" + with self._lock: + current = self._tasks.get(task_id) + if current is None: + return + changes: dict[str, object] = {} + if progress is not None: + changes["progress"] = progress + if heartbeat_at is not None: + changes["last_heartbeat_at"] = heartbeat_at + if lease_owner is not None: + changes["lease_owner"] = lease_owner + if lease_until is not None: + changes["lease_until"] = lease_until + if changes: + self._tasks[task_id] = replace(current, **changes) # type: ignore[arg-type] + + # ------------------------------------------------------------------ + # 4. 控制指令 + # ------------------------------------------------------------------ + + def submit_command(self, command: Command) -> None: + """写入一条控制指令。""" + with self._lock: + if command.command_id in self._commands: + raise StoreError(f"指令已存在：{command.command_id}") + self._commands[command.command_id] = command + + def claim_commands( + self, *, now: datetime, lease_seconds: float, limit: int = 50 + ) -> list[Command]: + """取走可执行指令并设置租约。""" + with self._lock: + claimable = [ + command + for command in self._commands.values() + if command.is_open and (command.lease_until is None or command.lease_until <= now) + ] + claimable.sort(key=lambda command: command.created_at) + claimed: list[Command] = [] + for command in claimable[:limit]: + leased = replace(command, lease_until=now + timedelta(seconds=lease_seconds)) + self._commands[command.command_id] = leased + claimed.append(leased) + return claimed + + def complete_command( + self, command_id: str, *, now: datetime, outcome: str | None = None + ) -> None: + """标记指令已完成。""" + with self._lock: + command = self._commands.get(command_id) + if command is None: + return + self._commands[command_id] = replace( + command, finished_at=now, outcome=outcome, lease_until=None + ) + + def release_command(self, command_id: str, *, now: datetime) -> None: + """交还一条已取走但无人认领的指令。""" + del now + with self._lock: + command = self._commands.get(command_id) + if command is None or not command.is_open: + return + self._commands[command_id] = replace(command, lease_until=None) + + def commands(self) -> list[Command]: + """全部指令，供诊断与测试使用（不属于 :class:`TaskStore` 契约）。""" + with self._lock: + return sorted(self._commands.values(), key=lambda item: item.created_at) + + # ------------------------------------------------------------------ + # 运维 + # ------------------------------------------------------------------ + + def prune(self, *, keep_terminal: int) -> int: + """丢弃最旧的终态任务及其历史。""" + with self._lock: + terminal = [ + snapshot for snapshot in self._tasks.values() if snapshot.status in TERMINAL + ] + terminal.sort(key=lambda item: item.created_at or datetime.min) + excess = len(terminal) - max(0, keep_terminal) + if excess <= 0: + return 0 + for snapshot in terminal[:excess]: + del self._tasks[snapshot.task_id] + self._history.pop(snapshot.task_id, None) + return excess + + def close(self) -> None: + """空操作：内存实现没有需要释放的资源。""" diff --git a/src/chongming_task/stores/postgres.py b/src/chongming_task/stores/postgres.py new file mode 100644 index 0000000..aaff843 --- /dev/null +++ b/src/chongming_task/stores/postgres.py @@ -0,0 +1,688 @@ +"""Postgres 存储实现（可选依赖：``psycopg`` 3 + ``psycopg-pool``）。 + +为什么是可选依赖 +---------------- +核心包保持**零第三方运行时依赖**：``import chongming_task`` 与内存 / SQLite 两个 +内置实现都不需要驱动。本模块只在显式导入时才要求 psycopg:: + + pip install "chongming-task[postgres]" + + from chongming_task import PostgresTaskStore + + store = PostgresTaskStore("postgresql://user@host:5432/tasks") + controller = Controller(store=store, owner="worker-1") + +它是**跨机部署的第 0 步**（SQLite 文件不能跨机共享，见分布式设计说明 §15）： +把持久层换成网络后端，上层（Controller、CLI、看板）一行都不用改。 + +与 SQLite 实现的三个刻意差异 +---------------------------- +1. **连接池**。SQLite 用"一把锁 + 单连接"串行化；Postgres 用 + ``psycopg_pool.ConnectionPool``，多个线程真的并行。``close()`` 的语义见方法文档。 +2. **认领指令用 ``FOR UPDATE SKIP LOCKED``**：一条语句完成"挑选 + 打租约"， + 并发认领互不阻塞、也不会重复取到同一条。SQLite 用 ``BEGIN IMMEDIATE`` 是同一 + 保证的不同实现。 +3. **原生类型**：时间列 ``TIMESTAMPTZ``（无时区的 ``timestamp`` 跨机必然出错）， + ``tags`` / ``progress`` / ``failure`` / ``outcome`` 用 ``JSONB``，标签过滤下推为 + ``tags @> '["io"]'::jsonb`` 并吃到 GIN 索引。 + +三条硬约束（与 SQLite 实现相同，任何后端都不能破） +-------------------------------------------------- +* ``save`` 的"校验 + 写入"是一条 ``UPDATE ... WHERE status_version = ?``， + "校验 + 写入"不是两步； +* ``save`` **不写租约列**（``lease_owner`` / ``lease_until`` 由 :meth:`touch` + 独占）。否则一次状态迁移就会把租约覆盖回旧值，重启恢复会把活着的任务误判失联； +* ``append`` 幂等（``ON CONFLICT DO NOTHING``）：崩溃恢复会重放。 + +时钟：租约判定用**调用方传入的 now** +------------------------------------ +契约把 ``now`` 作为参数传入（``claim_commands(*, now, ...)``），因此"租约过期了吗" +以调用方的时钟为准。分布式设计说明 §15.1 曾建议"租约判定一律用存储侧 ``now()``"—— +那与契约冲突：可注入的 ``Clock`` 是这个项目可测试性的地基，服务端时钟会让 +``StoreContract`` 里"过期 / 未过期"的用例无法构造。契约优先，此处以此为准。 + +表结构：建在 ``search_path`` 的第一个 schema +-------------------------------------------- +DDL 使用不带 schema 限定的表名，因此建表位置由连接的 ``search_path`` 决定。要隔离 +到独立 schema，把 ``options=-csearch_path=xxx`` 放进 DSN（见 +:func:`chongming_task.stores.codec` 的用法示例），或用独立数据库。 + +**建表是条件性的**：三张表都已存在时不执行任何 DDL——只读账号（无 CREATE 权限） +因此可以正常使用本存储（查询、下发指令）。表缺失时才建，多节点同时首次启动由一把 +咨询锁串行化。注意这**不是迁移**：表结构变更需要手工迁移，``IF NOT EXISTS`` 只负责 +首次建表。 + +:: note + 本实现跑通随包发布的 ``StoreContract``（``tests/integration/test_postgres_store.py``）， + 并在其上自建并发用例（N 连接并发 save / 并发认领 / 杀掉持有者后端）——契约测试 + 只覆盖单连接语义，跨进程原子性必须单独验证（见设计说明 §15.4）。 +""" + +from __future__ import annotations + +import json +from collections.abc import Iterator, Mapping +from contextlib import contextmanager +from datetime import datetime, timedelta +from typing import Any + +try: + import psycopg + from psycopg.conninfo import conninfo_to_dict + from psycopg.rows import dict_row + from psycopg_pool import ConnectionPool, PoolClosed, PoolTimeout +except ImportError as exc: # pragma: no cover - 依赖环境，无法在装好驱动的测试里走到 + raise ImportError( + "PostgresTaskStore 需要可选依赖 psycopg 与 psycopg-pool。安装：\n" + ' pip install "chongming-task[postgres]"\n' + '或：pip install "psycopg[binary]" psycopg-pool' + ) from exc + +from ..core.state_machine import Transition +from ..errors import ConcurrencyConflict, ConfigurationError, StoreError +from ..models import Progress, TaskSnapshot +from ..states import TERMINAL, TaskStatus +from .codec import ( + command_to_row, + decode_dt, + encode_dt, + encode_progress, + row_to_command, + row_to_snapshot, + row_to_transition, + snapshot_to_row, + transition_to_row, +) +from .protocol import Command, TaskQuery + +__all__ = ["PostgresTaskStore"] + +#: 与 ``codec.snapshot_to_row`` 的键**逐字一致**（顺序也一致）。 +#: 单元测试会把它与 codec 的输出比对，防止两边各自漂移。 +_TASK_COLUMNS = ( + "task_id", + "name", + "pipeline_run_id", + "tags", + "bucket", + "bucket_key", + "status", + "status_version", + "requested_action", + "attempt", + "max_attempts", + "created_at", + "queued_at", + "started_at", + "ended_at", + "last_heartbeat_at", + "progress", + "failure", + "outcome", + "executor", + "pid", + "overridden", + "lease_owner", + "lease_until", +) + +#: ``save`` 可以写的列 = 全部列 **减去租约列**。理由见模块文档。 +_SAVE_COLUMNS = tuple(column for column in _TASK_COLUMNS if not column.startswith("lease_")) + +#: 存成 JSONB 的列。绑定参数时用 ``::jsonb`` 显式转型；读回来时包回 JSON 文本。 +_JSON_COLUMNS = frozenset({"tags", "progress", "failure", "outcome"}) + +#: 存成 TIMESTAMPTZ 的列。写：ISO 文本 → datetime；读：datetime → ISO 文本。 +#: 三张表的集合取并集即可：列名在各表间同义，没有歧义。 +_DATETIME_COLUMNS = frozenset( + { + "created_at", + "queued_at", + "started_at", + "ended_at", + "last_heartbeat_at", + "lease_until", + "at", + "finished_at", + } +) + +#: 建表前的建议锁 key。多个节点同时首次启动时，``CREATE TABLE IF NOT EXISTS`` +#: 会竞争系统目录（``pg_type`` 的唯一键），拿它把 DDL 串行化。 +_SCHEMA_LOCK_KEY = "chongming_task.schema" + +#: 表是否齐全。齐全就完全跳过 DDL——只读账号因此不需要 CREATE 权限。 +_SCHEMA_READY_SQL = """ + SELECT to_regclass('tasks') IS NOT NULL + AND to_regclass('transitions') IS NOT NULL + AND to_regclass('commands') IS NOT NULL AS ready +""" + +_DDL_STATEMENTS: tuple[str, ...] = ( + """ + CREATE TABLE IF NOT EXISTS tasks ( + task_id TEXT PRIMARY KEY, + name TEXT NOT NULL, + pipeline_run_id TEXT, + tags JSONB NOT NULL, + bucket INTEGER, + bucket_key TEXT, + status TEXT NOT NULL, + status_version INTEGER NOT NULL, + requested_action TEXT, + attempt INTEGER NOT NULL, + max_attempts INTEGER NOT NULL, + created_at TIMESTAMPTZ, + queued_at TIMESTAMPTZ, + started_at TIMESTAMPTZ, + ended_at TIMESTAMPTZ, + last_heartbeat_at TIMESTAMPTZ, + progress JSONB, + failure JSONB, + outcome JSONB, + executor TEXT, + pid INTEGER, + overridden BOOLEAN NOT NULL DEFAULT FALSE, + -- 监督者租约：与 last_heartbeat_at 是两件事，见 TaskSnapshot 的说明。 + lease_owner TEXT, + lease_until TIMESTAMPTZ + ) + """, + # 索引只建现有查询形状用得到的那些。给 tasks 建 `(lease_until) WHERE status IN (…)` + # 之类的部分索引是**没用的**：租约判定发生在 Python 里（reconcile 取回非终态任务后 + # 逐条比较），而通用 query() 拼的是 `status = ANY($1)`，规划器证明不了它等价于单个 + # 状态值——索引永远不会被选中，白付写入开销。理由与"什么时候该加"记在 + # docs/distributed-design.md §15.3。 + "CREATE INDEX IF NOT EXISTS idx_tasks_status ON tasks (status)", + "CREATE INDEX IF NOT EXISTS idx_tasks_created_at ON tasks (created_at)", + "CREATE INDEX IF NOT EXISTS idx_tasks_bucket ON tasks (bucket)", + "CREATE INDEX IF NOT EXISTS idx_tasks_pipeline_run ON tasks (pipeline_run_id)", + # 标签精确匹配（`tags @> '["io"]'`）走 GIN；这是 JSONB 相对 TEXT + LIKE 多出来的东西。 + "CREATE INDEX IF NOT EXISTS idx_tasks_tags ON tasks USING GIN (tags)", + """ + CREATE TABLE IF NOT EXISTS transitions ( + task_id TEXT NOT NULL, + version INTEGER NOT NULL, + from_status TEXT NOT NULL, + to_status TEXT NOT NULL, + at TIMESTAMPTZ NOT NULL, + trigger TEXT NOT NULL, + actor TEXT NOT NULL, + reason TEXT, + PRIMARY KEY (task_id, version) + ) + """, + """ + CREATE TABLE IF NOT EXISTS commands ( + command_id TEXT PRIMARY KEY, + task_id TEXT NOT NULL, + kind TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL, + requested_by TEXT NOT NULL, + reason TEXT, + lease_until TIMESTAMPTZ, + finished_at TIMESTAMPTZ, + outcome TEXT + ) + """, + "CREATE INDEX IF NOT EXISTS idx_commands_task ON commands (task_id)", + # 认领查询只关心未完成的指令，而表在持续变大：部分索引更小也更快。 + "CREATE INDEX IF NOT EXISTS idx_commands_pending ON commands (created_at) " + "WHERE finished_at IS NULL", +) + +#: "挑选 + 打租约"必须是原子的。``FOR UPDATE SKIP LOCKED`` 让并发认领互不阻塞， +#: 也不会重复取到同一条指令——SQLite 版本用 ``BEGIN IMMEDIATE`` 换来的正是这个保证。 +_CLAIM_SQL = """ +WITH claimable AS ( + SELECT command_id + FROM commands + WHERE finished_at IS NULL + AND (lease_until IS NULL OR lease_until <= %(now)s) + ORDER BY created_at ASC, command_id + LIMIT %(limit)s + FOR UPDATE SKIP LOCKED +) +UPDATE commands SET lease_until = %(until)s +FROM claimable +WHERE commands.command_id = claimable.command_id +RETURNING commands.* +""" + + +def _describe_dsn(dsn: str) -> str: + """给报错用的 DSN 摘要：**不含密码**（连接串经常被整体写进日志）。""" + try: + info = conninfo_to_dict(dsn) + except psycopg.Error: + return "<无法解析的 DSN>" + user = info.get("user", "") + host = info.get("host", "localhost") + port = info.get("port", "5432") + dbname = info.get("dbname", user) + return f"{user}@{host}:{port}/{dbname}" if user else f"{host}:{port}/{dbname}" + + +def _to_db_params(row: Mapping[str, Any]) -> dict[str, Any]: + """codec 的"扁平原语映射" → 可直接绑定的参数。 + + 两处类型差异在这里收口，而不是散落到每条 SQL： + + * 时间列：ISO 文本 → ``datetime``（让驱动按 TIMESTAMPTZ 绑定，而不是依赖 + PG 对文本参数的隐式解析）； + * ``overridden``：codec 写的是 0/1（SQLite 没有布尔类型），PG 的 BOOLEAN + 列**不接受** smallint（``DatatypeMismatch``），显式转 bool。 + """ + params = dict(row) + for column in _DATETIME_COLUMNS: + if column in params: + params[column] = decode_dt(params[column]) + if "overridden" in params: + params["overridden"] = bool(params["overridden"]) + return params + + +def _from_db_row(row: Mapping[str, Any]) -> dict[str, Any]: + """数据库行 → codec 认识的"扁平原语映射"。 + + ``datetime`` 转回 UTC ISO 文本、JSONB 的 Python 对象转回 JSON 文本——codec 的 + ``decode_*`` 系列吃的是文本。JSONB 的规范化（键序、空白）不影响语义：解码后 + 比较的是对象而不是字节。 + """ + data = dict(row) + for column in _DATETIME_COLUMNS: + value = data.get(column) + if isinstance(value, datetime): + data[column] = encode_dt(value) + for column in _JSON_COLUMNS: + value = data.get(column) + if value is not None: + data[column] = json.dumps(value, ensure_ascii=False) + return data + + +def _placeholder(column: str) -> str: + """一个列的绑定占位符；JSON 列显式 ``::jsonb`` 转型（文本参数不会自动转）。""" + return f"%({column})s::jsonb" if column in _JSON_COLUMNS else f"%({column})s" + + +class PostgresTaskStore: + """把任务、迁移历史与控制指令存进 PostgreSQL。 + + ``dsn`` 是 libpq 连接串（URL 或 key=value 两种写法都行）。连接池默认 + ``min_size=1, max_size=8``：按需增长，空闲时只留一条。 + """ + + def __init__( + self, + dsn: str, + *, + min_size: int = 1, + max_size: int = 8, + timeout: float = 5.0, + ) -> None: + if min_size < 1: + raise ConfigurationError(f"min_size 至少为 1，收到 {min_size}") + if max_size < min_size: + raise ConfigurationError(f"max_size({max_size}) 不能小于 min_size({min_size})") + if timeout <= 0: + raise ConfigurationError(f"timeout 必须为正数，收到 {timeout}") + + self._dsn = dsn + self._closed = False + connect_timeout = max(1, int(timeout)) + + # 先直连一次：连接池的失败信息藏在日志里，直连能把真实原因（认证失败、 + # 库不存在、网络不通）带进异常。 + try: + with psycopg.connect(dsn, connect_timeout=connect_timeout): + pass + except psycopg.Error as exc: + raise StoreError(f"无法连接 Postgres（{_describe_dsn(dsn)}）：{exc}") from exc + + self._pool: ConnectionPool[psycopg.Connection[dict[str, Any]]] = ConnectionPool( + dsn, + kwargs={ + "row_factory": dict_row, + "application_name": "chongming-task", + "connect_timeout": connect_timeout, + }, + min_size=min_size, + max_size=max_size, + timeout=timeout, + open=False, + # 空闲连接可能被对端悄悄杀掉（重启、超时踢线、网络抖动）。取出时做一次 + # 轻量探活，坏连接会被丢弃并补新的——代价是每次借出多一次空查询。 + check=ConnectionPool.check_connection, + ) + try: + self._pool.open(wait=True, timeout=timeout) + except PoolTimeout as exc: # pragma: no cover - 直连已通过，这里基本只剩竞态 + self._pool.close() + raise StoreError(f"连接池初始化超时（{_describe_dsn(dsn)}）") from exc + + self._ensure_schema() + + # ------------------------------------------------------------------ + # 内部 + # ------------------------------------------------------------------ + + @contextmanager + def _connection(self) -> Iterator[psycopg.Connection[dict[str, Any]]]: + """借一条池连接；正常退出即提交。``close()`` 之后再用抛 :class:`StoreError`。""" + if self._closed: + raise StoreError("PostgresTaskStore 已关闭：close() 之后不能再使用。") + try: + with self._pool.connection() as conn: + yield conn + except PoolClosed as exc: + raise StoreError("PostgresTaskStore 已关闭：close() 之后不能再使用。") from exc + + def _ensure_schema(self) -> None: + """表缺失时才建；已齐全则一个 DDL 都不执行（只读账号也能用）。""" + try: + with self._connection() as conn: + if self._tables_ready(conn): + return + # 多节点同时首次启动会竞争系统目录。拿一把事务级建议锁串行化， + # 拿到锁后**再查一次**——别人可能已经建好了。 + conn.execute( + "SELECT pg_advisory_xact_lock(hashtext(%(key)s))", {"key": _SCHEMA_LOCK_KEY} + ) + if self._tables_ready(conn): + return + for statement in _DDL_STATEMENTS: + conn.execute(statement) + except psycopg.Error as exc: + raise StoreError( + f"初始化 Postgres 表结构失败（{_describe_dsn(self._dsn)}）：{exc}\n" + "需要建表权限；若用只读账号，请先由管理员在目标 schema 里建好三张表。" + ) from exc + + @staticmethod + def _tables_ready(conn: psycopg.Connection[dict[str, Any]]) -> bool: + row = conn.execute(_SCHEMA_READY_SQL).fetchone() + return bool(row is not None and row["ready"]) + + # ------------------------------------------------------------------ + # 1. 任务快照 + # ------------------------------------------------------------------ + + def create(self, snapshot: TaskSnapshot) -> None: + """写入新任务；``task_id`` 已存在则抛错。""" + params = _to_db_params(snapshot_to_row(snapshot)) + columns = ", ".join(_TASK_COLUMNS) + values = ", ".join(_placeholder(column) for column in _TASK_COLUMNS) + try: + with self._connection() as conn: + conn.execute(f"INSERT INTO tasks ({columns}) VALUES ({values})", params) + except psycopg.errors.UniqueViolation as exc: + raise StoreError(f"任务已存在：{snapshot.task_id}") from exc + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + """按乐观并发更新快照。 + + ``UPDATE ... WHERE task_id = ? AND status_version = ?`` 让"版本校验 + 写入" + 成为**同一条语句**——拆成先查后写，跨进程就必然丢更新。 + + **不写租约列**（见 ``_SAVE_COLUMNS``）。 + """ + params = _to_db_params(snapshot_to_row(snapshot)) + params["expect_version"] = expect_version + assignments = ", ".join( + f"{column} = {_placeholder(column)}" for column in _SAVE_COLUMNS if column != "task_id" + ) + with self._connection() as conn: + cursor = conn.execute( + f"UPDATE tasks SET {assignments} " + "WHERE task_id = %(task_id)s AND status_version = %(expect_version)s", + params, + ) + if cursor.rowcount == 1: + return + existing = conn.execute( + "SELECT status_version FROM tasks WHERE task_id = %(task_id)s", + {"task_id": snapshot.task_id}, + ).fetchone() + if existing is None: + raise StoreError(f"任务不存在：{snapshot.task_id}") + raise ConcurrencyConflict(snapshot.task_id, expect_version, existing["status_version"]) + + def get(self, task_id: str) -> TaskSnapshot | None: + """读取单个任务。""" + with self._connection() as conn: + row = conn.execute( + "SELECT * FROM tasks WHERE task_id = %(task_id)s", {"task_id": task_id} + ).fetchone() + return row_to_snapshot(_from_db_row(row)) if row is not None else None + + def query(self, query: TaskQuery | None = None) -> list[TaskSnapshot]: + """按条件查询。过滤**下推到 SQL**，不是在 Python 里筛。 + + 排序与 SQLite 实现对齐：``created_at`` 升序时 NULL 在前、降序时在后 + （SQLite 把 NULL 当作最小值，PG 默认相反，所以显式写 ``NULLS FIRST/LAST``）。 + ``created_at`` 相同时按 ``task_id`` 决出先后，保证结果稳定可复现。 + """ + condition = query if query is not None else TaskQuery() + clauses: list[str] = [] + params: dict[str, Any] = {} + + if condition.statuses is not None: + clauses.append("status = ANY(%(statuses)s)") + params["statuses"] = [status.value for status in condition.statuses] + if condition.name is not None: + clauses.append("name = %(name)s") + params["name"] = condition.name + if condition.name_prefix is not None: + # 与 SQLite 实现同样不转义 LIKE 通配符：两边的语义保持一致。 + clauses.append("name LIKE %(name_prefix)s") + params["name_prefix"] = f"{condition.name_prefix}%" + if condition.tag is not None: + # JSONB 包含：`["cpu", "io"] @> ["io"]` 为真，`["ios"] @> ["io"]` 为假。 + # 这就是"标签精确匹配"的正确表达（不要退回 LIKE 子串匹配）。 + clauses.append("tags @> %(tag)s::jsonb") + params["tag"] = json.dumps([condition.tag], ensure_ascii=False) + if condition.buckets is not None: + clauses.append("bucket = ANY(%(buckets)s)") + params["buckets"] = list(condition.buckets) + if condition.pipeline_run_id is not None: + clauses.append("pipeline_run_id = %(pipeline_run_id)s") + params["pipeline_run_id"] = condition.pipeline_run_id + + where = f" WHERE {' AND '.join(clauses)}" if clauses else "" + direction = "DESC NULLS LAST" if condition.newest_first else "ASC NULLS FIRST" + statement = f"SELECT * FROM tasks{where} ORDER BY created_at {direction}, task_id" + if condition.limit is not None: + statement += " LIMIT %(limit)s" + params["limit"] = condition.limit + + with self._connection() as conn: + rows = conn.execute(statement, params).fetchall() + return [row_to_snapshot(_from_db_row(row)) for row in rows] + + def counts(self) -> dict[TaskStatus, int]: + """各状态计数（``GROUP BY``，不把全部记录读回来数）。""" + with self._connection() as conn: + rows = conn.execute( + "SELECT status, COUNT(*) AS total FROM tasks GROUP BY status" + ).fetchall() + return {TaskStatus(row["status"]): int(row["total"]) for row in rows} + + # ------------------------------------------------------------------ + # 2. 审计轨迹 + # ------------------------------------------------------------------ + + def append(self, transition: Transition) -> None: + """追加迁移记录；同一 ``(task_id, version)`` 重复写入被忽略（幂等）。""" + params = _to_db_params(transition_to_row(transition)) + with self._connection() as conn: + conn.execute( + "INSERT INTO transitions " + "(task_id, version, from_status, to_status, at, trigger, actor, reason) " + "VALUES (%(task_id)s, %(version)s, %(from_status)s, %(to_status)s, %(at)s, " + "%(trigger)s, %(actor)s, %(reason)s) " + "ON CONFLICT (task_id, version) DO NOTHING", + params, + ) + + def history(self, task_id: str, *, since_version: int = 0) -> list[Transition]: + """读取迁移历史，``version > since_version``，按版本升序。""" + with self._connection() as conn: + rows = conn.execute( + "SELECT * FROM transitions " + "WHERE task_id = %(task_id)s AND version > %(since_version)s " + "ORDER BY version", + {"task_id": task_id, "since_version": since_version}, + ).fetchall() + return [row_to_transition(_from_db_row(row)) for row in rows] + + # ------------------------------------------------------------------ + # 3. 高频旁路 + # ------------------------------------------------------------------ + + def touch( + self, + task_id: str, + *, + progress: Progress | None = None, + heartbeat_at: datetime | None = None, + lease_owner: str | None = None, + lease_until: datetime | None = None, + ) -> None: + """只更新进度、心跳与监督者租约，**不碰** ``status_version``。 + + 任务不存在时静默忽略（心跳晚到一步不该报错）。四个参数各自独立可选。 + """ + assignments: list[str] = [] + params: dict[str, Any] = {"task_id": task_id} + if progress is not None: + assignments.append("progress = %(progress)s::jsonb") + params["progress"] = encode_progress(progress) + if heartbeat_at is not None: + assignments.append("last_heartbeat_at = %(heartbeat_at)s") + params["heartbeat_at"] = heartbeat_at + if lease_owner is not None: + assignments.append("lease_owner = %(lease_owner)s") + params["lease_owner"] = lease_owner + if lease_until is not None: + assignments.append("lease_until = %(lease_until)s") + params["lease_until"] = lease_until + if not assignments: + return + with self._connection() as conn: + conn.execute( + f"UPDATE tasks SET {', '.join(assignments)} WHERE task_id = %(task_id)s", + params, + ) + + # ------------------------------------------------------------------ + # 4. 控制指令 + # ------------------------------------------------------------------ + + def submit_command(self, command: Command) -> None: + """写入一条控制指令。存储层只传递，不解释、不执行。""" + params = _to_db_params(command_to_row(command)) + try: + with self._connection() as conn: + conn.execute( + "INSERT INTO commands " + "(command_id, task_id, kind, created_at, requested_by, reason, " + " lease_until, finished_at, outcome) " + "VALUES (%(command_id)s, %(task_id)s, %(kind)s, %(created_at)s, " + "%(requested_by)s, %(reason)s, %(lease_until)s, %(finished_at)s, " + "%(outcome)s)", + params, + ) + except psycopg.errors.UniqueViolation as exc: + raise StoreError(f"指令已存在：{command.command_id}") from exc + + def claim_commands( + self, *, now: datetime, lease_seconds: float, limit: int = 50 + ) -> list[Command]: + """取走一批可执行指令并设置租约；单条 SQL，天然原子。 + + 结果按 ``(created_at, command_id)`` 排序后再返回：``RETURNING`` 的行序没有 + 保证，而契约的用词是"按 created_at 升序取"，排序把它变成确定的。 + """ + params = { + "now": now, + "until": now + timedelta(seconds=lease_seconds), + "limit": limit, + } + with self._connection() as conn: + rows = conn.execute(_CLAIM_SQL, params).fetchall() + rows.sort(key=lambda row: (row["created_at"], row["command_id"])) + return [row_to_command(_from_db_row(row)) for row in rows] + + def complete_command( + self, command_id: str, *, now: datetime, outcome: str | None = None + ) -> None: + """标记指令已完成，使其不再被取走。""" + with self._connection() as conn: + conn.execute( + "UPDATE commands SET finished_at = %(now)s, outcome = %(outcome)s, " + "lease_until = NULL WHERE command_id = %(command_id)s", + {"command_id": command_id, "now": now, "outcome": outcome}, + ) + + def release_command(self, command_id: str, *, now: datetime) -> None: + """交还一条已取走但无人认领的指令：只清租约，**不**标记完成。 + + 标记完成会让真正的持有者永远收不到这次请求；等租约过期又会被持续续租饿死。 + 所以需要第三种结局，见 :meth:`TaskStore.release_command`。 + """ + del now # 交还与时间无关；接口保留参数以维持契约形状 + with self._connection() as conn: + conn.execute( + "UPDATE commands SET lease_until = NULL " + "WHERE command_id = %(command_id)s AND finished_at IS NULL", + {"command_id": command_id}, + ) + + # ------------------------------------------------------------------ + # 运维 + # ------------------------------------------------------------------ + + def prune(self, *, keep_terminal: int) -> int: + """丢弃最旧的终态任务及其历史，返回丢弃数量。非终态任务绝不触碰。 + + 一条 CTE 语句完成"选受害者 → 删历史 → 删任务"：两个 DELETE 与选择看到的是 + 同一份物化结果，不会出现"历史删了任务没删"的中间态（引用两次的 CTE 会被 + Postgres 物化一次；数据修改型 CTE 即使不被引用也保证执行）。 + + 顺序：按 ``created_at`` **降序**保留最新的 ``keep_terminal`` 条，``OFFSET`` + 跳过它们，剩下的就是"最旧的"要删的。升序 + ``OFFSET`` 会删反——那是把最新 + 的丢掉。NULL 在 SQLite 里是最小值（视为最旧），所以这里用 ``NULLS LAST`` + 对齐同一语义。 + """ + params = { + "statuses": [status.value for status in TERMINAL], + "keep": max(0, keep_terminal), + } + statement = """ + WITH doomed AS ( + SELECT task_id FROM tasks + WHERE status = ANY(%(statuses)s) + ORDER BY created_at DESC NULLS LAST, task_id DESC + OFFSET %(keep)s + ), purged AS ( + DELETE FROM transitions WHERE task_id IN (SELECT task_id FROM doomed) + ) + DELETE FROM tasks WHERE task_id IN (SELECT task_id FROM doomed) + RETURNING task_id + """ + with self._connection() as conn: + rows = conn.execute(statement, params).fetchall() + return len(rows) + + def close(self) -> None: + """关闭连接池。**幂等**；关闭后任何操作都会抛 :class:`StoreError`。 + + 正在被借出的连接会在归还时关闭（``psycopg_pool`` 的语义），所以调用方应确保 + 没有线程还在使用——``Controller.close(close_store=True)`` 已经等任务收尾， + 直接用它即可。 + """ + if self._closed: + return + self._closed = True + self._pool.close() diff --git a/src/chongming_task/stores/protocol.py b/src/chongming_task/stores/protocol.py new file mode 100644 index 0000000..053d5a2 --- /dev/null +++ b/src/chongming_task/stores/protocol.py @@ -0,0 +1,231 @@ +"""存储抽象。 + +:class:`TaskStore` 是整个持久层的**唯一契约**。内存实现、SQLite 实现、以及将来任何 +第三方实现（Postgres、Redis、对象存储……）都只需满足它，上层（:class:`Controller`、 +CLI、看板）不必知道数据从哪来。 + +本模块只依赖标准库，**不依赖 pytest**：契约测试套件在 +``chongming_task.testing.store_contract``，第三方实现应当拿它验证自己。 + +四类数据，写入频次与一致性要求完全不同，因此刻意分开 +---------------------------------------------------- + +1. **任务快照**（``create`` / ``save`` / ``get`` / ``query``）——查询面，参与**乐观 + 并发**（``status_version``）。写入频率低（每次状态迁移一次）。 +2. **审计轨迹**（``append`` / ``history``）——**只追加，永不修改**。这是状态机的 + 外部证据，删改它就等于篡改审计。 +3. **高频旁路**（``touch``）——进度与心跳，可能每秒几十次。它们**不参与乐观并发**： + 否则会与状态写入持续冲突，把版本号变成噪声。 +4. **控制指令**（``submit_command`` / ``claim_commands`` / ``complete_command``）—— + 跨进程控制的媒介。CLI 无法给别的进程发信号，只能写一条指令让持有者取走。 + +关于"谁来执行指令" +------------------ +存储层**只负责传递**，不解释、不执行。执行者是持有该任务的进程（见 +:class:`~chongming_task.controller.Controller`）。这样存储实现可以保持极简， +也可以换成任何能 CAS 的后端。 +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from enum import StrEnum +from typing import Protocol, runtime_checkable + +from ..core.state_machine import Transition +from ..models import Progress, TaskSnapshot +from ..states import TaskStatus + +__all__ = [ + "Command", + "CommandKind", + "TaskQuery", + "TaskStore", +] + + +class CommandKind(StrEnum): + """跨进程控制动作。""" + + STOP = "stop" + PAUSE = "pause" + RESUME = "resume" + + +@dataclass(frozen=True, slots=True) +class Command: + """一条待执行的控制指令。 + + ``lease_until`` 是**租约**：取走指令的一方获得一段独占时间，期间别的进程不会 + 重复取到它。若持有者在租约到期前没有 ``complete_command``（例如进程崩了）， + 指令会被重新放出——这样既不会重复执行，也不会因为一次崩溃就永久卡住。 + """ + + command_id: str + task_id: str + kind: CommandKind + created_at: datetime + requested_by: str = "unknown" + reason: str | None = None + lease_until: datetime | None = None + finished_at: datetime | None = None + outcome: str | None = None + + @property + def is_open(self) -> bool: + """是否尚未完成。""" + return self.finished_at is None + + +@dataclass(frozen=True, slots=True) +class TaskQuery: + """查询条件。所有字段都是可选的，``None`` 表示不限制。""" + + statuses: frozenset[TaskStatus] | None = None + name_prefix: str | None = None + tag: str | None = None + name: str | None = None + buckets: frozenset[int] | None = None + """只取这些桶里的任务。 + + 为将来的分布式预留：一台机器只关心自己拥有的那组桶，应当能在存储层直接过滤， + 而不是取回全部再筛。单桶查询传单元素集合即可。 + """ + limit: int | None = None + newest_first: bool = False + pipeline_run_id: str | None = None + """只取属于某次管道运行的任务。 + + 没有它，"把这条管道的所有节点列出来"就只能全表拉回来自己筛——而管道恰恰是最需要 + 按整体查看的东西。 + """ + + +@runtime_checkable +class TaskStore(Protocol): + """持久层契约。 + + 实现者请用 ``chongming_task.testing.store_contract.StoreContract`` 验证自己： + 同一套用例必须对每个实现都通过，否则"换后端"就是空话。 + """ + + # ------------------------------------------------------------------ + # 1. 任务快照（参与乐观并发） + # ------------------------------------------------------------------ + + def create(self, snapshot: TaskSnapshot) -> None: + """写入一个新任务。``task_id`` 已存在则抛 :class:`~chongming_task.errors.StoreError`。""" + ... + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + """更新任务快照。 + + ``expect_version`` 是写入前读到的 ``status_version``；不一致则抛 + :class:`~chongming_task.errors.ConcurrencyConflict`，**不做自动重试**—— + 每个任务的写入者应当唯一，冲突意味着存在第二个写入者，那是设计缺陷。 + """ + ... + + def get(self, task_id: str) -> TaskSnapshot | None: + """读取单个任务；不存在返回 ``None``。""" + ... + + def query(self, query: TaskQuery | None = None) -> list[TaskSnapshot]: + """按条件查询，按 ``created_at`` 升序（``newest_first=True`` 则降序）。""" + ... + + def counts(self) -> dict[TaskStatus, int]: + """各状态的任务数。""" + ... + + # ------------------------------------------------------------------ + # 2. 审计轨迹（只追加） + # ------------------------------------------------------------------ + + def append(self, transition: Transition) -> None: + """追加一条状态迁移记录。 + + 应当按 ``task_id`` + ``version`` **幂等**写入：重放同一批记录（例如崩溃后 + 恢复）不应产生重复行。 + """ + ... + + def history(self, task_id: str, *, since_version: int = 0) -> list[Transition]: + """读取迁移历史，``version > since_version``，按版本升序。""" + ... + + # ------------------------------------------------------------------ + # 3. 高频旁路（不参与乐观并发） + # ------------------------------------------------------------------ + + def touch( + self, + task_id: str, + *, + progress: Progress | None = None, + heartbeat_at: datetime | None = None, + lease_owner: str | None = None, + lease_until: datetime | None = None, + ) -> None: + """仅更新进度、心跳与监督者租约，**不校验也不递增** ``status_version``。 + + 两个可选信号是**独立的**，实现不得用一个去推导另一个： + + * ``heartbeat_at`` 由任务自己的 ``checkpoint()`` 驱动，回答"任务在推进吗"。 + * ``lease_owner`` / ``lease_until`` 由监督进程定期续租，回答"还有人在管它吗"。 + + 任务不存在时静默忽略：任务可能刚被清理，而心跳晚到一步不该报错。 + + 实现注意：``save`` **不得**写租约列（由本方法独占），否则一次状态迁移就会把 + 租约覆盖回旧值，让重启恢复把活着的任务误判为失联。 + """ + ... + + # ------------------------------------------------------------------ + # 4. 控制指令（跨进程） + # ------------------------------------------------------------------ + + def submit_command(self, command: Command) -> None: + """写入一条控制指令。存储层不解释它。""" + ... + + def claim_commands( + self, *, now: datetime, lease_seconds: float, limit: int = 50 + ) -> list[Command]: + """取走一批可执行指令，并为其设置租约。 + + 应当**同时**满足两个条件才可被取走：未被完成，且（无租约或租约已过期）。 + 实现应保证并发调用下同一条指令不会被两个进程同时取到。 + """ + ... + + def complete_command( + self, command_id: str, *, now: datetime, outcome: str | None = None + ) -> None: + """标记指令已完成，使其不再被取走。""" + ... + + def release_command(self, command_id: str, *, now: datetime) -> None: + """放弃一条已取走的指令，交还给别人。 + + 没有它，多进程下会出问题：任何**能取指令**的进程都可能取到不属于自己的 + 任务（它手上没有那个任务）。若它把指令标记完成，真正的持有者就永远收不到 + 这次停止请求——一个静默的丢指令。若它什么都不做、只等租约过期，那么只要 + 它持续轮询就会不断续租，真正的持有者会被无限期饿死。 + + 因此需要第三种结局：``claimed → release``。取到不属于自己的指令时立即交还。 + """ + ... + + # ------------------------------------------------------------------ + # 运维 + # ------------------------------------------------------------------ + + def prune(self, *, keep_terminal: int) -> int: + """丢弃最旧的终态任务（含其历史），返回丢弃数量。""" + ... + + def close(self) -> None: + """释放资源。对内存实现是空操作。""" + ... diff --git a/src/chongming_task/stores/sqlite.py b/src/chongming_task/stores/sqlite.py new file mode 100644 index 0000000..03c83a9 --- /dev/null +++ b/src/chongming_task/stores/sqlite.py @@ -0,0 +1,490 @@ +"""SQLite 存储实现（标准库 ``sqlite3``，不引入任何第三方依赖）。 + +为什么默认选它 +-------------- +* **跨进程可见**——这正是它存在的首要理由：另一个进程里的 CLI 要能查询到任务。 +* **无 IPC、无网络层**，天然跨平台，一个文件就是全部状态。 +* **顺带就是持久化**，重启后可查。 + +并发模型 +-------- +* 进程内：所有操作由一把可重入锁串行化，使"读—判断—写"在进程内是原子的。 +* 跨进程：可能被多个进程同时写（例如任务进程写状态、CLI 写指令），因此 + 取指令等复合操作用 ``BEGIN IMMEDIATE`` + ``COMMIT``，并依赖 + ``status_version`` 做乐观并发。``busy_timeout`` 让短暂争抢自动重试。 + +三个刻意的取舍 +-------------- +1. **不用 ORM。** 表结构与 :class:`~chongming_task.models.TaskSnapshot` 一一对应， + 读起来就是 SQL，排障时不用先翻译一层。 +2. **时间一律以 UTC ISO 字符串存储。** 本地时间原样写入会在跨时区排序时出错。 +3. **指令只有租约，没有优先级。** 需要优先级时应当在上层排队，而不是让存储层 + 长出调度语义。 +""" + +from __future__ import annotations + +import json +import sqlite3 +import threading +from dataclasses import replace +from datetime import datetime, timedelta +from pathlib import Path +from typing import Any + +from ..core.state_machine import Transition +from ..errors import ConcurrencyConflict, StoreError +from ..models import Progress, TaskSnapshot +from ..states import TERMINAL, TaskStatus +from .codec import ( + command_to_row, + encode_dt, + row_to_command, + row_to_snapshot, + row_to_transition, + snapshot_to_row, + transition_to_row, +) +from .protocol import Command, TaskQuery + +__all__ = ["SqliteTaskStore"] + +_TASK_COLUMNS = ( + "task_id", + "name", + "pipeline_run_id", + "tags", + "bucket", + "bucket_key", + "status", + "status_version", + "requested_action", + "attempt", + "max_attempts", + "created_at", + "queued_at", + "started_at", + "ended_at", + "last_heartbeat_at", + "progress", + "failure", + "outcome", + "executor", + "pid", + "overridden", + "lease_owner", + "lease_until", +) + +#: ``save`` 可以写的列 = 全部列 **减去租约列**。 +#: +#: 租约由 :meth:`SqliteTaskStore.touch` 独占写入。若 ``save`` 也写它，一次状态迁移 +#: 就会把租约覆盖成内存快照里的旧值（甚至 ``None``）——同一行两条写入路径互相打架， +#: 和心跳曾经踩过的是同一个坑：静默、且只有重启恢复时才会以"任务被误判失联"的形式 +#: 暴露出来。 +_SAVE_COLUMNS = tuple(column for column in _TASK_COLUMNS if not column.startswith("lease_")) + +_SCHEMA = """ +CREATE TABLE IF NOT EXISTS tasks ( + task_id TEXT PRIMARY KEY, + name TEXT NOT NULL, + pipeline_run_id TEXT, + tags TEXT NOT NULL, + bucket INTEGER, + bucket_key TEXT, + status TEXT NOT NULL, + status_version INTEGER NOT NULL, + requested_action TEXT, + attempt INTEGER NOT NULL, + max_attempts INTEGER NOT NULL, + created_at TEXT, + queued_at TEXT, + started_at TEXT, + ended_at TEXT, + last_heartbeat_at TEXT, + progress TEXT, + failure TEXT, + outcome TEXT, + executor TEXT, + pid INTEGER, + overridden INTEGER NOT NULL DEFAULT 0, + -- 监督者租约：与 last_heartbeat_at 是两件事，见 TaskSnapshot 的说明。 + lease_owner TEXT, + lease_until TEXT +); +CREATE INDEX IF NOT EXISTS idx_tasks_status ON tasks(status); +CREATE INDEX IF NOT EXISTS idx_tasks_created_at ON tasks(created_at); +-- 分桶索引是为将来的分布式预留的：一台机器"取我拥有的那些桶"会是一条高频查询， +-- 没有索引就只能全表扫。 +CREATE INDEX IF NOT EXISTS idx_tasks_bucket ON tasks(bucket); +-- 管道索引同理：看板/CLI 最常问的就是"这条管道的节点现在怎样了"。 +CREATE INDEX IF NOT EXISTS idx_tasks_pipeline_run ON tasks(pipeline_run_id); + +CREATE TABLE IF NOT EXISTS transitions ( + task_id TEXT NOT NULL, + version INTEGER NOT NULL, + from_status TEXT NOT NULL, + to_status TEXT NOT NULL, + at TEXT NOT NULL, + trigger TEXT NOT NULL, + actor TEXT NOT NULL, + reason TEXT, + PRIMARY KEY (task_id, version) +); + +CREATE TABLE IF NOT EXISTS commands ( + command_id TEXT PRIMARY KEY, + task_id TEXT NOT NULL, + kind TEXT NOT NULL, + created_at TEXT NOT NULL, + requested_by TEXT NOT NULL, + reason TEXT, + lease_until TEXT, + finished_at TEXT, + outcome TEXT +); +CREATE INDEX IF NOT EXISTS idx_commands_pending ON commands(finished_at, lease_until); +CREATE INDEX IF NOT EXISTS idx_commands_task ON commands(task_id); +""" + + +class SqliteTaskStore: + """把任务、迁移历史与控制指令存进一个 SQLite 文件。 + + 传 ``":memory:"`` 可得到一个进程内容量库，语义与文件库完全一致——因此同一套 + 契约测试可以同时跑在两者上。 + """ + + def __init__(self, path: str | Path = "chongming-tasks.db", *, timeout: float = 5.0) -> None: + self._path = str(path) + if self._path != ":memory:": + parent = Path(self._path).expanduser().resolve().parent + if not parent.is_dir(): + # SQLite 只会报 "unable to open database file"，完全看不出是目录不存在。 + # 这是最容易浪费时间的错误之一，所以在这里就说清楚。 + raise StoreError(f"数据库所在目录不存在：{parent}。请先创建它，或改用别的路径。") + self._lock = threading.RLock() + self._conn = sqlite3.connect( + self._path, + timeout=timeout, + check_same_thread=False, # 由自己的锁串行化，见模块文档 + isolation_level=None, # 显式管理事务 + ) + self._conn.row_factory = sqlite3.Row + # WAL 让读不阻塞写；busy_timeout 让跨进程争抢自动重试而不是立刻报错。 + self._conn.execute("PRAGMA journal_mode=WAL") + self._conn.execute("PRAGMA synchronous=NORMAL") + self._conn.execute(f"PRAGMA busy_timeout={int(timeout * 1000)}") + self._conn.executescript(_SCHEMA) + + @property + def path(self) -> str: + """数据库文件路径，供诊断使用。""" + return self._path + + # ------------------------------------------------------------------ + # 1. 任务快照 + # ------------------------------------------------------------------ + + def create(self, snapshot: TaskSnapshot) -> None: + """写入新任务；``task_id`` 已存在则抛错。""" + row = snapshot_to_row(snapshot) + placeholders = ", ".join("?" for _ in _TASK_COLUMNS) + with self._lock: + try: + self._conn.execute( + f"INSERT INTO tasks ({', '.join(_TASK_COLUMNS)}) VALUES ({placeholders})", + tuple(row[column] for column in _TASK_COLUMNS), + ) + except sqlite3.IntegrityError as exc: + raise StoreError(f"任务已存在：{snapshot.task_id}") from exc + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + """按乐观并发更新快照。 + + ``UPDATE ... WHERE status_version = ?`` 让"版本校验"和"写入"成为同一条 + SQL——拆成先查后写就会在跨进程场景下产生竞态。 + + **不写租约列**（见 ``_SAVE_COLUMNS``）：那些列由 :meth:`touch` 独占，否则一次 + 状态迁移就会把租约覆盖回旧值，导致重启恢复误判任务失联。 + """ + row = snapshot_to_row(snapshot) + assignments = ", ".join(f"{column} = ?" for column in _SAVE_COLUMNS if column != "task_id") + values = [row[column] for column in _SAVE_COLUMNS if column != "task_id"] + with self._lock: + cursor = self._conn.execute( + f"UPDATE tasks SET {assignments} WHERE task_id = ? AND status_version = ?", + (*values, snapshot.task_id, expect_version), + ) + if cursor.rowcount == 1: + return + existing = self._conn.execute( + "SELECT status_version FROM tasks WHERE task_id = ?", (snapshot.task_id,) + ).fetchone() + if existing is None: + raise StoreError(f"任务不存在：{snapshot.task_id}") + raise ConcurrencyConflict(snapshot.task_id, expect_version, existing["status_version"]) + + def get(self, task_id: str) -> TaskSnapshot | None: + """读取单个任务。""" + with self._lock: + row = self._conn.execute("SELECT * FROM tasks WHERE task_id = ?", (task_id,)).fetchone() + return row_to_snapshot(dict(row)) if row is not None else None + + def query(self, query: TaskQuery | None = None) -> list[TaskSnapshot]: + """按条件查询。""" + condition = query if query is not None else TaskQuery() + clauses: list[str] = [] + params: list[Any] = [] + + if condition.statuses is not None: + placeholders = ", ".join("?" for _ in condition.statuses) + clauses.append(f"status IN ({placeholders})") + params.extend(status.value for status in condition.statuses) + if condition.name is not None: + clauses.append("name = ?") + params.append(condition.name) + if condition.name_prefix is not None: + clauses.append("name LIKE ?") + params.append(f"{condition.name_prefix}%") + if condition.tag is not None: + # 标签存成 JSON 数组；用 LIKE 匹配 "tag" 带引号的形式，避免 "io" 命中 "ios"。 + clauses.append("tags LIKE ?") + params.append(f'%"{condition.tag}"%') + if condition.buckets is not None: + placeholders = ", ".join("?" for _ in condition.buckets) + clauses.append(f"bucket IN ({placeholders})") + params.extend(condition.buckets) + if condition.pipeline_run_id is not None: + clauses.append("pipeline_run_id = ?") + params.append(condition.pipeline_run_id) + + where = f" WHERE {' AND '.join(clauses)}" if clauses else "" + direction = "DESC" if condition.newest_first else "ASC" + statement = f"SELECT * FROM tasks{where} ORDER BY created_at {direction}" + if condition.limit is not None: + statement += " LIMIT ?" + params.append(condition.limit) + with self._lock: + rows = self._conn.execute(statement, params).fetchall() + return [row_to_snapshot(dict(row)) for row in rows] + + def counts(self) -> dict[TaskStatus, int]: + """各状态计数。""" + with self._lock: + rows = self._conn.execute( + "SELECT status, COUNT(*) AS total FROM tasks GROUP BY status" + ).fetchall() + return {TaskStatus(row["status"]): row["total"] for row in rows} + + # ------------------------------------------------------------------ + # 2. 审计轨迹 + # ------------------------------------------------------------------ + + def append(self, transition: Transition) -> None: + """追加迁移记录；同一版本重复写入被忽略（幂等）。""" + row = transition_to_row(transition) + with self._lock: + self._conn.execute( + "INSERT OR IGNORE INTO transitions " + "(task_id, version, from_status, to_status, at, trigger, actor, reason) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + ( + row["task_id"], + row["version"], + row["from_status"], + row["to_status"], + row["at"], + row["trigger"], + row["actor"], + row["reason"], + ), + ) + + def history(self, task_id: str, *, since_version: int = 0) -> list[Transition]: + """读取迁移历史。""" + with self._lock: + rows = self._conn.execute( + "SELECT * FROM transitions WHERE task_id = ? AND version > ? ORDER BY version", + (task_id, since_version), + ).fetchall() + return [row_to_transition(dict(row)) for row in rows] + + # ------------------------------------------------------------------ + # 3. 高频旁路 + # ------------------------------------------------------------------ + + def touch( + self, + task_id: str, + *, + progress: Progress | None = None, + heartbeat_at: datetime | None = None, + lease_owner: str | None = None, + lease_until: datetime | None = None, + ) -> None: + """只更新进度、心跳与监督者租约。 + + 刻意不碰 ``status_version``：它每次跳动都会让状态写入的乐观并发失败， + 把版本号变成噪声。任务不存在时静默忽略——心跳晚到一步不该报错。 + + 租约与心跳是**两个独立的信号**，见 :class:`~chongming_task.models.TaskSnapshot` + 的说明：心跳回答"任务在推进吗"，租约回答"还有人在管它吗"。用一个去推另一个 + 都会出错。 + """ + assignments: list[str] = [] + params: list[Any] = [] + if progress is not None: + assignments.append("progress = ?") + params.append( + json.dumps( + { + "done": progress.done, + "total": progress.total, + "unit": progress.unit, + "message": progress.message, + "updated_at": encode_dt(progress.updated_at), + }, + ensure_ascii=False, + ) + ) + if heartbeat_at is not None: + assignments.append("last_heartbeat_at = ?") + params.append(encode_dt(heartbeat_at)) + if lease_owner is not None: + assignments.append("lease_owner = ?") + params.append(lease_owner) + if lease_until is not None: + assignments.append("lease_until = ?") + params.append(encode_dt(lease_until)) + if not assignments: + return + with self._lock: + self._conn.execute( + f"UPDATE tasks SET {', '.join(assignments)} WHERE task_id = ?", + (*params, task_id), + ) + + # ------------------------------------------------------------------ + # 4. 控制指令 + # ------------------------------------------------------------------ + + def submit_command(self, command: Command) -> None: + """写入一条控制指令。""" + row = command_to_row(command) + with self._lock: + try: + self._conn.execute( + "INSERT INTO commands " + "(command_id, task_id, kind, created_at, requested_by, reason, " + " lease_until, finished_at, outcome) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + row["command_id"], + row["task_id"], + row["kind"], + row["created_at"], + row["requested_by"], + row["reason"], + row["lease_until"], + row["finished_at"], + row["outcome"], + ), + ) + except sqlite3.IntegrityError as exc: + raise StoreError(f"指令已存在：{command.command_id}") from exc + + def claim_commands( + self, *, now: datetime, lease_seconds: float, limit: int = 50 + ) -> list[Command]: + """取走可执行指令并设置租约。 + + 整个"挑选 + 打租约"必须在一个 ``BEGIN IMMEDIATE`` 事务里完成，否则两个 + 进程可能同时取到同一条指令，于是同一次停止被下发两次。 + """ + with self._lock: + self._conn.execute("BEGIN IMMEDIATE") + try: + rows = self._conn.execute( + "SELECT * FROM commands " + "WHERE finished_at IS NULL AND (lease_until IS NULL OR lease_until <= ?) " + "ORDER BY created_at LIMIT ?", + (encode_dt(now), limit), + ).fetchall() + + claimed: list[Command] = [] + for row in rows: + command = row_to_command(dict(row)) + leased = replace(command, lease_until=now + timedelta(seconds=lease_seconds)) + self._conn.execute( + "UPDATE commands SET lease_until = ? WHERE command_id = ?", + (encode_dt(leased.lease_until), command.command_id), + ) + claimed.append(leased) + self._conn.execute("COMMIT") + except BaseException: + self._conn.execute("ROLLBACK") + raise + return claimed + + def complete_command( + self, command_id: str, *, now: datetime, outcome: str | None = None + ) -> None: + """标记指令已完成。""" + with self._lock: + self._conn.execute( + "UPDATE commands SET finished_at = ?, outcome = ?, lease_until = NULL " + "WHERE command_id = ?", + (encode_dt(now), outcome, command_id), + ) + + def release_command(self, command_id: str, *, now: datetime) -> None: + """交还一条已取走但无人认领的指令。""" + del now + with self._lock: + self._conn.execute( + "UPDATE commands SET lease_until = NULL " + "WHERE command_id = ? AND finished_at IS NULL", + (command_id,), + ) + + def commands(self) -> list[Command]: + """全部指令，供诊断与测试使用（不属于 :class:`TaskStore` 契约）。""" + with self._lock: + rows = self._conn.execute("SELECT * FROM commands ORDER BY created_at").fetchall() + return [row_to_command(dict(row)) for row in rows] + + # ------------------------------------------------------------------ + # 运维 + # ------------------------------------------------------------------ + + def prune(self, *, keep_terminal: int) -> int: + """丢弃最旧的终态任务及其历史。""" + terminal_values = tuple(status.value for status in TERMINAL) + placeholders = ", ".join("?" for _ in terminal_values) + with self._lock: + self._conn.execute("BEGIN IMMEDIATE") + try: + rows = self._conn.execute( + f"SELECT task_id FROM tasks WHERE status IN ({placeholders}) " + "ORDER BY created_at ASC", + terminal_values, + ).fetchall() + excess = len(rows) - max(0, keep_terminal) + if excess <= 0: + self._conn.execute("COMMIT") + return 0 + victims = [row["task_id"] for row in rows[:excess]] + marks = ", ".join("?" for _ in victims) + self._conn.execute(f"DELETE FROM transitions WHERE task_id IN ({marks})", victims) + self._conn.execute(f"DELETE FROM tasks WHERE task_id IN ({marks})", victims) + self._conn.execute("COMMIT") + except BaseException: + self._conn.execute("ROLLBACK") + raise + return excess + + def close(self) -> None: + """关闭连接。""" + with self._lock: + self._conn.close() diff --git a/src/chongming_task/testing/__init__.py b/src/chongming_task/testing/__init__.py new file mode 100644 index 0000000..5d56011 --- /dev/null +++ b/src/chongming_task/testing/__init__.py @@ -0,0 +1,12 @@ +"""随包发布的测试工具。 + +放在包内而非 ``tests/`` 之下，是为了让**使用者的测试**也能用上确定性时钟与 +断言助手——否则每个使用方都要自己重写一遍 :class:`FakeClock`，而重写出来的 +版本往往无法与运行时的取时点完全对齐。 +""" + +from __future__ import annotations + +from .clock import FakeClock + +__all__ = ["FakeClock"] diff --git a/src/chongming_task/testing/clock.py b/src/chongming_task/testing/clock.py new file mode 100644 index 0000000..6c8dfd1 --- /dev/null +++ b/src/chongming_task/testing/clock.py @@ -0,0 +1,41 @@ +"""确定性时钟。""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +__all__ = ["FakeClock"] + + +class FakeClock: + """可显式推进的时钟。 + + 刻意**不**自动前进：测试必须显式调用 :meth:`advance`。这样"时间没有流逝" + 与"忘记推进时间"不会混为一谈——后者会让超时测试静默地永远不触发。 + """ + + __slots__ = ("_now",) + + def __init__(self, start: datetime | None = None) -> None: + self._now = start if start is not None else datetime(2026, 1, 1, tzinfo=UTC) + + def __call__(self) -> datetime: + return self._now + + @property + def now(self) -> datetime: + """当前时刻。""" + return self._now + + def advance(self, seconds: float) -> datetime: + """向前推进 ``seconds`` 秒并返回新时刻。""" + if seconds < 0: + raise ValueError(f"时间不能倒流：{seconds}") + self._now += timedelta(seconds=seconds) + return self._now + + def set(self, moment: datetime) -> None: + """直接设为 ``moment``（只能向前）。""" + if moment < self._now: + raise ValueError(f"时间不能倒流：{moment} < {self._now}") + self._now = moment diff --git a/src/chongming_task/testing/store_contract.py b/src/chongming_task/testing/store_contract.py new file mode 100644 index 0000000..c05b041 --- /dev/null +++ b/src/chongming_task/testing/store_contract.py @@ -0,0 +1,528 @@ +"""``TaskStore`` 契约测试套件。 + +**随包发布**，因为"可扩展的持久层"只有配上可执行的契约才有意义：第三方实现 +（Postgres、Redis、对象存储……）只要能通过这一套用例，就可以放心替换掉内置实现。 + +刻意**不依赖 pytest**：只用普通断言与一个 :func:`expect` 助手，因此这个模块不会 +给运行期引入任何第三方依赖。pytest 仍会自动收集子类里的 ``test_*`` 方法。 + +用法:: + + from chongming_task.testing.store_contract import StoreContract + + class TestMyStore(StoreContract): + def make_store(self): + return MyStore(...) +""" + +from __future__ import annotations + +from dataclasses import replace +from datetime import UTC, datetime, timedelta +from typing import Any + +from ..core.state_machine import Transition +from ..errors import ConcurrencyConflict, StoreError +from ..models import Failure, FailureKind, Progress, TaskSnapshot +from ..states import TaskStatus +from ..stores.protocol import Command, CommandKind, TaskQuery, TaskStore + +__all__ = ["StoreContract"] + +_T0 = datetime(2026, 1, 1, tzinfo=UTC) + + +def expect(exception: type[BaseException], action: Any, *, because: str = "") -> BaseException: + """断言 ``action`` 抛出 ``exception``，并把它返回以便进一步检查。 + + 不用 ``pytest.raises``，是为了让本模块保持零依赖。 + """ + try: + action() + except exception as caught: + return caught + except BaseException as other: + raise AssertionError( + f"{because}：期望 {exception.__name__}，实际 {type(other).__name__}: {other}" + ) from other + raise AssertionError(f"{because}：期望抛出 {exception.__name__}，但没有") + + +def full_snapshot(task_id: str = "t-full") -> TaskSnapshot: + """一个**每个字段都填满**的快照。 + + 契约测试用它做往返比对：只测几个字段会漏掉序列化里的静默丢失（例如时区、 + frozenset、嵌套的 progress / failure）。 + """ + return TaskSnapshot( + task_id=task_id, + name="抽取", + pipeline_run_id="run-1", + tags=frozenset({"io", "nightly"}), + bucket=7, + bucket_key="tenant-42", + status=TaskStatus.FAILED, + status_version=7, + requested_action=None, + attempt=2, + max_attempts=3, + created_at=_T0, + queued_at=_T0 + timedelta(seconds=1), + started_at=_T0 + timedelta(seconds=1), + ended_at=_T0 + timedelta(seconds=5), + last_heartbeat_at=_T0 + timedelta(seconds=4), + progress=Progress( + done=3, + total=10, + unit="rows", + message="半途", + updated_at=_T0 + timedelta(seconds=3), + ), + failure=Failure( + kind=FailureKind.TIMEOUT_ATTEMPT, + message="超时了", + traceback="Traceback (most recent call last): ...", + exit_code=3, + retryable=True, + ), + outcome={"rows": 3}, + executor="process", + pid=4321, + overridden=False, + lease_owner="worker-1", + lease_until=_T0 + timedelta(seconds=30), + ) + + +def simple_snapshot( + task_id: str, + *, + status: TaskStatus = TaskStatus.RUNNING, + version: int = 1, + created_offset: float = 0.0, + name: str = "任务", + tags: frozenset[str] = frozenset(), + bucket: int | None = None, +) -> TaskSnapshot: + """一个便于构造查询场景的最小快照。""" + return TaskSnapshot( + task_id=task_id, + name=name, + tags=tags, + bucket=bucket, + bucket_key=None if bucket is None else f"key-{task_id}", + status=status, + status_version=version, + attempt=1, + max_attempts=1, + created_at=_T0 + timedelta(seconds=created_offset), + started_at=_T0 + timedelta(seconds=created_offset), + ) + + +def transition(task_id: str, version: int, *, reason: str | None = None) -> Transition: + """构造一条迁移记录。""" + return Transition( + task_id=task_id, + from_status=TaskStatus.QUEUED, + to_status=TaskStatus.RUNNING, + at=_T0 + timedelta(seconds=version), + trigger="executor:launched", + actor="system", + reason=reason, + version=version, + ) + + +def command( + command_id: str, + task_id: str = "t1", + *, + kind: CommandKind = CommandKind.STOP, + created_offset: float = 0.0, +) -> Command: + """构造一条控制指令。""" + return Command( + command_id=command_id, + task_id=task_id, + kind=kind, + created_at=_T0 + timedelta(seconds=created_offset), + requested_by="ops", + reason="测试", + ) + + +class StoreContract: + """每个 :class:`TaskStore` 实现都必须通过的行为契约。 + + 子类只需实现 :meth:`make_store`，并保证每次调用返回**互相隔离**的干净存储。 + """ + + store: TaskStore + + def make_store(self) -> TaskStore: + """返回一个全新的、隔离的存储实例。""" + raise NotImplementedError + + def setup_method(self) -> None: + self.store = self.make_store() + + def teardown_method(self) -> None: + self.store.close() + + # ------------------------------------------------------------------ + # 契约的形式要件 + # ------------------------------------------------------------------ + + def test_implements_the_protocol(self) -> None: + assert isinstance(self.store, TaskStore) + + # ------------------------------------------------------------------ + # 1. 任务快照 + # ------------------------------------------------------------------ + + def test_full_snapshot_round_trips_without_loss(self) -> None: + """每个字段都要原样回来：时区、frozenset、嵌套的 progress / failure。""" + original = full_snapshot() + self.store.create(original) + restored = self.store.get(original.task_id) + assert restored is not None + assert restored == original + assert restored.tags == frozenset({"io", "nightly"}) + assert restored.progress is not None + assert restored.progress.message == "半途" + assert restored.failure is not None + assert restored.failure.kind is FailureKind.TIMEOUT_ATTEMPT + assert restored.failure.retryable is True + assert restored.outcome == {"rows": 3} + + def test_get_returns_none_for_unknown_task(self) -> None: + assert self.store.get("nope") is None + + def test_create_rejects_duplicate_task_id(self) -> None: + self.store.create(simple_snapshot("dup")) + expect(StoreError, lambda: self.store.create(simple_snapshot("dup")), because="重复创建") + + def test_save_updates_when_version_matches(self) -> None: + self.store.create(simple_snapshot("t1", version=1)) + updated = simple_snapshot("t1", status=TaskStatus.SUCCEEDED, version=2) + self.store.save(updated, expect_version=1) + restored = self.store.get("t1") + assert restored is not None + assert restored.status is TaskStatus.SUCCEEDED + assert restored.status_version == 2 + + def test_save_rejects_stale_version(self) -> None: + """乐观并发是"换后端"能成立的前提：冲突必须被拒绝，而不是静默覆盖。""" + self.store.create(simple_snapshot("t1", version=1)) + self.store.save(simple_snapshot("t1", version=2), expect_version=1) + conflict = expect( + ConcurrencyConflict, + lambda: self.store.save(simple_snapshot("t1", version=3), expect_version=1), + because="用过期的版本号写入", + ) + assert isinstance(conflict, ConcurrencyConflict) + assert self.store.get("t1").status_version == 2 # type: ignore[union-attr] + + def test_save_on_missing_task_raises(self) -> None: + expect( + StoreError, + lambda: self.store.save(simple_snapshot("ghost"), expect_version=1), + because="更新不存在的任务", + ) + + # ------------------------------------------------------------------ + # 2. 查询 + # ------------------------------------------------------------------ + + def test_query_filters_by_status(self) -> None: + self.store.create(simple_snapshot("a", status=TaskStatus.RUNNING)) + self.store.create(simple_snapshot("b", status=TaskStatus.SUCCEEDED)) + found = self.store.query(TaskQuery(statuses=frozenset({TaskStatus.RUNNING}))) + assert [item.task_id for item in found] == ["a"] + + def test_query_filters_by_tag_exactly(self) -> None: + """标签必须精确匹配：``io`` 不能命中 ``ios``。 + + 存储层常见的退化实现是子串匹配，它在标签里会悄悄错配。 + """ + self.store.create(simple_snapshot("a", tags=frozenset({"io"}))) + self.store.create(simple_snapshot("b", tags=frozenset({"ios"}))) + self.store.create(simple_snapshot("c", tags=frozenset({"cpu", "io"}))) + found = self.store.query(TaskQuery(tag="io")) + assert sorted(item.task_id for item in found) == ["a", "c"] + + def test_query_filters_by_name_and_prefix(self) -> None: + self.store.create(simple_snapshot("a", name="抽取-甲")) + self.store.create(simple_snapshot("b", name="抽取-乙")) + self.store.create(simple_snapshot("c", name="加载")) + assert [i.task_id for i in self.store.query(TaskQuery(name="加载"))] == ["c"] + assert sorted(i.task_id for i in self.store.query(TaskQuery(name_prefix="抽取"))) == [ + "a", + "b", + ] + + def test_query_orders_and_limits(self) -> None: + for index, task_id in enumerate(("a", "b", "c")): + self.store.create(simple_snapshot(task_id, created_offset=float(index))) + assert [i.task_id for i in self.store.query()] == ["a", "b", "c"] + assert [i.task_id for i in self.store.query(TaskQuery(newest_first=True))] == [ + "c", + "b", + "a", + ] + assert [i.task_id for i in self.store.query(TaskQuery(limit=2))] == ["a", "b"] + + def test_query_filters_by_bucket_set(self) -> None: + """分布式下"取我拥有的那组桶"会是一条高频查询，因此它必须能下推到存储层。""" + for index, (task_id, bucket) in enumerate((("a", 1), ("b", 2), ("c", 1))): + self.store.create(simple_snapshot(task_id, created_offset=float(index), bucket=bucket)) + assert [i.task_id for i in self.store.query(TaskQuery(buckets=frozenset({1})))] == [ + "a", + "c", + ] + assert [i.task_id for i in self.store.query(TaskQuery(buckets=frozenset({1, 2})))] == [ + "a", + "b", + "c", + ] + assert self.store.query(TaskQuery(buckets=frozenset({9}))) == [] + + def test_task_without_bucket_key_has_no_bucket(self) -> None: + self.store.create(simple_snapshot("a")) + found = self.store.query()[0] + assert found.bucket is None + assert found.bucket_key is None + + def test_query_on_empty_store(self) -> None: + assert self.store.query() == [] + assert self.store.counts() == {} + + def test_counts_by_status(self) -> None: + self.store.create(simple_snapshot("a", status=TaskStatus.RUNNING)) + self.store.create(simple_snapshot("b", status=TaskStatus.RUNNING)) + self.store.create(simple_snapshot("c", status=TaskStatus.FAILED)) + counts = self.store.counts() + assert counts[TaskStatus.RUNNING] == 2 + assert counts[TaskStatus.FAILED] == 1 + + # ------------------------------------------------------------------ + # 3. 审计轨迹 + # ------------------------------------------------------------------ + + def test_history_is_ordered_by_version(self) -> None: + self.store.create(simple_snapshot("t1")) + for version in (3, 1, 2): + self.store.append(transition("t1", version)) + assert [item.version for item in self.store.history("t1")] == [1, 2, 3] + + def test_history_filters_by_since_version(self) -> None: + self.store.create(simple_snapshot("t1")) + for version in (1, 2, 3): + self.store.append(transition("t1", version)) + assert [i.version for i in self.store.history("t1", since_version=1)] == [2, 3] + + def test_append_is_idempotent(self) -> None: + """重放同一批记录（崩溃恢复）不应产生重复行。""" + self.store.create(simple_snapshot("t1")) + self.store.append(transition("t1", 1)) + self.store.append(transition("t1", 1)) + assert len(self.store.history("t1")) == 1 + + def test_history_round_trips_all_fields(self) -> None: + self.store.create(simple_snapshot("t1")) + original = transition("t1", 5, reason="发布窗口") + self.store.append(original) + restored = self.store.history("t1")[0] + assert restored == original + assert restored.reason == "发布窗口" + + def test_history_of_unknown_task_is_empty(self) -> None: + assert self.store.history("nope") == [] + + # ------------------------------------------------------------------ + # 4. 高频旁路 + # ------------------------------------------------------------------ + + def test_touch_updates_progress_and_heartbeat_without_bumping_version(self) -> None: + """进度与心跳若参与乐观并发，会与状态写入持续冲突。""" + self.store.create(simple_snapshot("t1", version=4)) + moment = _T0 + timedelta(seconds=30) + self.store.touch( + "t1", + progress=Progress(done=5, total=10, updated_at=moment), + heartbeat_at=moment, + ) + restored = self.store.get("t1") + assert restored is not None + assert restored.status_version == 4, "touch 不应改动版本号" + assert restored.last_heartbeat_at == moment + assert restored.progress is not None + assert restored.progress.done == 5 + + def test_touch_on_missing_task_is_ignored(self) -> None: + """心跳晚到一步（任务已被清理）不该报错。""" + self.store.touch("ghost", heartbeat_at=_T0) + + def test_touch_writes_the_supervisor_lease(self) -> None: + """租约与心跳是两个独立信号，都能经 touch 落库。""" + self.store.create(simple_snapshot("t1")) + until = _T0 + timedelta(seconds=30) + self.store.touch("t1", lease_owner="worker-1", lease_until=until) + + restored = self.store.get("t1") + assert restored is not None + assert restored.lease_owner == "worker-1" + assert restored.lease_until == until + + def test_touch_can_set_owner_and_lease_independently(self) -> None: + """只给 owner 不给 until（或反之）也应当被接受，不能要求调用方凑齐两个。""" + self.store.create(simple_snapshot("t1")) + self.store.touch("t1", lease_owner="worker-1") + restored = self.store.get("t1") + assert restored is not None + assert restored.lease_owner == "worker-1" + assert restored.lease_until is None + + def test_save_does_not_clobber_the_lease(self) -> None: + """**这条是重启恢复的关键。** ``save`` 不写租约列。 + + 否则一次状态迁移就会把租约覆盖回内存快照里的旧值（甚至 ``None``），于是 + 重启恢复会把一个活得好好的任务误判为"失去监督者"并判 ``LOST``——把一个正常 + 任务杀掉，而且只在重启时才暴露。 + """ + self.store.create(simple_snapshot("t1", version=1)) + until = _T0 + timedelta(seconds=30) + self.store.touch("t1", lease_owner="worker-1", lease_until=until) + + # 内存里的快照并不知道租约（它来自 touch），于是保存时会带一个空租约。 + plain = simple_snapshot("t1", status=TaskStatus.SUCCEEDED, version=2) + assert plain.lease_owner is None + self.store.save(plain, expect_version=1) + + restored = self.store.get("t1") + assert restored is not None + assert restored.status is TaskStatus.SUCCEEDED + assert restored.lease_owner == "worker-1", "save 覆盖了租约，重启恢复会误判失联" + assert restored.lease_until == until + + # ------------------------------------------------------------------ + # 5. 控制指令 + # ------------------------------------------------------------------ + + def test_claim_then_complete(self) -> None: + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + claimed = self.store.claim_commands(now=_T0, lease_seconds=30) + assert [item.command_id for item in claimed] == ["c1"] + assert claimed[0].lease_until is not None + + self.store.complete_command("c1", now=_T0, outcome="ok") + assert self.store.claim_commands(now=_T0 + timedelta(seconds=60), lease_seconds=30) == [] + + def test_leased_command_is_not_handed_out_twice(self) -> None: + """同一次停止被下发两次会让任务收到重复指令。""" + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + assert len(self.store.claim_commands(now=_T0, lease_seconds=30)) == 1 + assert self.store.claim_commands(now=_T0 + timedelta(seconds=1), lease_seconds=30) == [] + + def test_expired_lease_can_be_reclaimed(self) -> None: + """持有者崩溃后，指令必须能被重新取走，否则会永久卡住。""" + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + assert len(self.store.claim_commands(now=_T0, lease_seconds=10)) == 1 + reclaimed = self.store.claim_commands(now=_T0 + timedelta(seconds=11), lease_seconds=10) + assert [item.command_id for item in reclaimed] == ["c1"] + + def test_claims_respect_limit_and_creation_order(self) -> None: + self.store.create(simple_snapshot("t1")) + for index in range(3): + self.store.submit_command(command(f"c{index}", created_offset=float(index))) + claimed = self.store.claim_commands(now=_T0, lease_seconds=30, limit=2) + assert [item.command_id for item in claimed] == ["c0", "c1"] + + def test_command_kind_and_metadata_survive(self) -> None: + self.store.create(simple_snapshot("t1")) + original = command("c1", kind=CommandKind.PAUSE) + self.store.submit_command(original) + claimed = self.store.claim_commands(now=_T0, lease_seconds=30)[0] + assert claimed.kind is CommandKind.PAUSE + assert claimed.task_id == "t1" + assert claimed.requested_by == "ops" + assert claimed.reason == "测试" + + def test_released_command_becomes_claimable_immediately(self) -> None: + """取到不属于自己的指令时应当立即交还。 + + 否则要么被错误地标记完成（真正的持有者永远收不到这次停止请求），要么被 + 持续续租（真正的持有者被无限期饿死）。 + """ + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + assert len(self.store.claim_commands(now=_T0, lease_seconds=300)) == 1 + + self.store.release_command("c1", now=_T0) + reclaimed = self.store.claim_commands(now=_T0, lease_seconds=30) + assert [item.command_id for item in reclaimed] == ["c1"] + + def test_released_command_is_still_open(self) -> None: + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + self.store.claim_commands(now=_T0, lease_seconds=300) + self.store.release_command("c1", now=_T0) + + claimed = self.store.claim_commands(now=_T0, lease_seconds=300) + assert claimed[0].is_open + + def test_release_does_not_revive_a_finished_command(self) -> None: + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + self.store.claim_commands(now=_T0, lease_seconds=30) + self.store.complete_command("c1", now=_T0, outcome="ok") + + self.store.release_command("c1", now=_T0) + assert self.store.claim_commands(now=_T0, lease_seconds=30) == [] + + def test_query_filters_by_pipeline_run(self) -> None: + self.store.create(replace(simple_snapshot("t1"), pipeline_run_id="run-a")) + self.store.create(replace(simple_snapshot("t2"), pipeline_run_id="run-a")) + self.store.create(replace(simple_snapshot("t3"), pipeline_run_id="run-b")) + self.store.create(simple_snapshot("t4")) + + found = self.store.query(TaskQuery(pipeline_run_id="run-a")) + assert [item.task_id for item in found] == ["t1", "t2"] + assert self.store.query(TaskQuery(pipeline_run_id="nope")) == [] + + def test_duplicate_command_id_is_rejected(self) -> None: + self.store.create(simple_snapshot("t1")) + self.store.submit_command(command("c1")) + expect(StoreError, lambda: self.store.submit_command(command("c1")), because="重复指令") + + # ------------------------------------------------------------------ + # 6. 运维 + # ------------------------------------------------------------------ + + def test_prune_keeps_only_the_newest_terminal_tasks(self) -> None: + for index, task_id in enumerate(("a", "b", "c")): + self.store.create( + simple_snapshot( + task_id, + status=TaskStatus.SUCCEEDED, + created_offset=float(index), + ) + ) + self.store.append(transition(task_id, 1)) + + assert self.store.prune(keep_terminal=2) == 1 + assert [item.task_id for item in self.store.query()] == ["b", "c"] + assert self.store.history("a") == [], "被清理任务的历史也应一并删除" + assert self.store.prune(keep_terminal=2) == 0 + + def test_prune_never_touches_unfinished_tasks(self) -> None: + """运行中的任务被清掉就等于把状态机扔了。""" + self.store.create(simple_snapshot("running", status=TaskStatus.RUNNING)) + self.store.create(simple_snapshot("done", status=TaskStatus.SUCCEEDED)) + assert self.store.prune(keep_terminal=0) == 1 + assert self.store.get("running") is not None + assert self.store.get("done") is None diff --git a/tests/contract/test_store_contract.py b/tests/contract/test_store_contract.py new file mode 100644 index 0000000..bb6bda1 --- /dev/null +++ b/tests/contract/test_store_contract.py @@ -0,0 +1,39 @@ +"""内置存储实现的契约一致性测试。 + +两个实现（内存 / SQLite）跑**完全相同**的用例。这是"持久层可替换"这句话能成立的 +唯一证据——否则换后端就是在赌。 +""" + +from __future__ import annotations + +import tempfile +from pathlib import Path + +from chongming_task.stores.memory import InMemoryTaskStore +from chongming_task.stores.protocol import TaskStore +from chongming_task.stores.sqlite import SqliteTaskStore +from chongming_task.testing.store_contract import StoreContract + + +class TestInMemoryStore(StoreContract): + """内存实现。""" + + def make_store(self) -> TaskStore: + return InMemoryTaskStore() + + +class TestSqliteFileStore(StoreContract): + """SQLite 文件库——真实使用场景（跨进程、可重启）。""" + + _tempdir: tempfile.TemporaryDirectory[str] + + def make_store(self) -> TaskStore: + self._tempdir = tempfile.TemporaryDirectory(prefix="chongming-store-") + return SqliteTaskStore(Path(self._tempdir.name) / "tasks.db") + + +class TestSqliteMemoryStore(StoreContract): + """SQLite 内存库——语义必须与文件库一致。""" + + def make_store(self) -> TaskStore: + return SqliteTaskStore(":memory:") diff --git a/tests/integration/test_cli.py b/tests/integration/test_cli.py new file mode 100644 index 0000000..c9b97d2 --- /dev/null +++ b/tests/integration/test_cli.py @@ -0,0 +1,766 @@ +"""命令行测试。 + +CLI 是**另一个进程**，所以这里全部通过 ``--db`` 指向真实的 SQLite 文件，而不是把 +``Controller`` 塞进去——那样测的就不是 CLI 了。 + +重点覆盖两件容易静默出错的事： + +* **输出契约**（``--json`` 的形状、退出码），因为脚本要依赖它； +* **错误路径**，尤其是"给一个不存在的任务下指令"——它曾经会成功并留下一条永远不会 + 被执行的指令。 +""" + +from __future__ import annotations + +import json +import os +from datetime import UTC, datetime +from pathlib import Path + +import pytest + +from chongming_task.cli.main import ( + EXIT_ERROR, + EXIT_OK, + EXIT_TASK_NOT_SUCCESSFUL, + build_parser, + main, +) +from chongming_task.errors import ConfigurationError +from chongming_task.models import TaskSnapshot +from chongming_task.states import TaskStatus +from chongming_task.stores.sqlite import SqliteTaskStore + +_TASKS_MODULE = "tasks_demo" + + +@pytest.fixture +def db(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> str: + """把示例任务模块加进导入路径，并返回一个全新的库路径。""" + import sys + + examples = Path(__file__).resolve().parents[2] / "examples" + monkeypatch.syspath_prepend(str(examples)) + sys.modules.pop(_TASKS_MODULE, None) + return str(tmp_path / "tasks.db") + + +def _run_cli(db: str, *argv: str) -> int: + return main(["--db", db, *argv]) + + +# --------------------------------------------------------------------------- +# 参数解析 +# --------------------------------------------------------------------------- + + +def test_parser_requires_a_subcommand() -> None: + with pytest.raises(SystemExit): + build_parser().parse_args([]) + + +def test_parser_help_mentions_every_command() -> None: + text = build_parser().format_help() + for command in ( + "tasks", + "run", + "ps", + "status", + "history", + "stats", + "buckets", + "stop", + "pause", + "resume", + ): + assert command in text + + +# --------------------------------------------------------------------------- +# 定义 +# --------------------------------------------------------------------------- + + +def test_tasks_lists_the_registered_definitions( + db: str, capsys: pytest.CaptureFixture[str] +) -> None: + assert main(["tasks", "--tasks", _TASKS_MODULE]) == EXIT_OK + out = capsys.readouterr().out + assert "sync" in out + assert "add" in out + assert "两数相加。" in out, "注册项的 docstring 应当作为描述显示" + + +def test_tasks_on_an_unimportable_module_fails_clearly( + capsys: pytest.CaptureFixture[str], +) -> None: + assert main(["tasks", "--tasks", "no.such.module"]) == EXIT_ERROR + assert "无法导入任务模块" in capsys.readouterr().err + + +def test_a_module_not_found_error_explains_sys_path( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """模块自身没找到时，必须说清"为什么找不到"。 + + 最常踩的坑：``chongming-task`` 是**入口脚本**，不会把当前目录放进 ``sys.path``—— + 把模块放在当前目录、然后直接 ``--tasks mymodule`` 会失败，而 "No module named" + 本身完全不解释这件事。 + """ + monkeypatch.delenv("PYTHONPATH", raising=False) + assert main(["tasks", "--tasks", "definitely_not_installed"]) == EXIT_ERROR + err = capsys.readouterr().err + assert "PYTHONPATH" in err + assert "不会**把当前目录放进 sys.path" in err + assert "python -m chongming_task.cli.main" in err + + +def test_a_module_found_but_failing_to_import_says_so( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """模块存在、但内部导入失败：不能把矛头指向 sys.path，那是它自己的依赖问题。""" + module = tmp_path / "broken_tasks.py" + module.write_text("import no_such_dependency_here\n", encoding="utf-8") + monkeypatch.syspath_prepend(str(tmp_path)) + + assert main(["tasks", "--tasks", "broken_tasks"]) == EXIT_ERROR + err = capsys.readouterr().err + assert "找到了，但它导入失败" in err + assert "PYTHONPATH" not in err + + +def test_the_cli_module_can_be_run_with_python_m() -> None: + """备选入口：``python -m`` 会把当前目录放进 ``sys.path``，因此根目录的模块也能用。 + + 这条用例钉住"备选入口存在"这个事实——它已经写进了错误提示里，不能悄悄消失。 + """ + source = Path(__file__).resolve().parents[2] / "src" / "chongming_task" / "cli" / "main.py" + assert source.is_file() + assert 'if __name__ == "__main__"' in source.read_text(encoding="utf-8") + + +def test_tasks_on_a_module_without_register_fails_clearly( + capsys: pytest.CaptureFixture[str], +) -> None: + """约定必须说清楚——否则用户只会看到一句 AttributeError。""" + assert main(["tasks", "--tasks", "json"]) == EXIT_ERROR + assert "register(controller)" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# 提交与执行 +# --------------------------------------------------------------------------- + + +def test_run_can_execute_in_a_child_process(db: str, capsys: pytest.CaptureFixture[str]) -> None: + """``--executor process``：任务在子进程里跑，快照因此带上 pid。 + + 这是"能被真的杀掉"那条能力在命令行上的入口——前台 ``run`` 遇到不配合的任务时， + 另一个终端发 ``stop`` 就能把它连根拔掉（线程只能放弃等待）。 + """ + code = _run_cli( + db, + "run", + "--tasks", + _TASKS_MODULE, + "add", + "--arg", + "left=2", + "--arg", + "right=3", + "--executor", + "process", + ) + assert code == EXIT_OK + assert "succeeded" in capsys.readouterr().out + + store = SqliteTaskStore(db) + try: + snapshot = store.query()[0] + assert snapshot.executor == "process" + assert snapshot.pid is not None + assert snapshot.pid != os.getpid(), "任务确实跑在另一个进程里" + finally: + store.close() + + +def test_run_executes_and_exits_zero(db: str, capsys: pytest.CaptureFixture[str]) -> None: + code = _run_cli( + db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=2", "--arg", "right=3" + ) + assert code == EXIT_OK + out = capsys.readouterr().out + assert "succeeded" in out + + store = SqliteTaskStore(db) + try: + snapshots = store.query() + assert len(snapshots) == 1 + assert snapshots[0].outcome == {"value": 5} + finally: + store.close() + + +def test_run_parses_json_typed_arguments(db: str) -> None: + """``--arg n=3`` 应当得到整数 3，而不是字符串 "3"。""" + assert ( + _run_cli( + db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=20", "--arg", "right=22" + ) + == EXIT_OK + ) + + store = SqliteTaskStore(db) + try: + assert store.query()[0].outcome == {"value": 42} + finally: + store.close() + + +def test_run_on_a_failing_task_exits_three(db: str) -> None: + """命令成功、任务失败——两者用不同退出码区分，脚本才好判分支。""" + code = _run_cli(db, "run", "--tasks", _TASKS_MODULE, "flaky", "--arg", "fail_times=9") + assert code == EXIT_TASK_NOT_SUCCESSFUL + + +def test_run_rejects_a_malformed_arg(db: str, capsys: pytest.CaptureFixture[str]) -> None: + code = _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "noequals") + assert code == EXIT_ERROR + assert "K=V" in capsys.readouterr().err + + +def test_run_records_the_bucket(db: str) -> None: + code = _run_cli( + db, + "run", + "--tasks", + _TASKS_MODULE, + "add", + "--arg", + "left=1", + "--arg", + "right=1", + "--bucket-key", + "tenant-9", + ) + assert code == EXIT_OK + + store = SqliteTaskStore(db) + try: + snapshot = store.query()[0] + assert snapshot.bucket_key == "tenant-9" + assert snapshot.bucket is not None + finally: + store.close() + + +# --------------------------------------------------------------------------- +# 查询 +# --------------------------------------------------------------------------- + + +def test_ps_defaults_to_unfinished_and_all_shows_everything( + db: str, capsys: pytest.CaptureFixture[str] +) -> None: + assert ( + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=1", "--arg", "right=1") + == EXIT_OK + ) + capsys.readouterr() + + assert _run_cli(db, "ps") == EXIT_OK + assert "没有匹配的任务" in capsys.readouterr().out, "默认只显示未落终态的" + + assert _run_cli(db, "ps", "--all") == EXIT_OK + assert "succeeded" in capsys.readouterr().out + + +def test_status_shows_detail_and_history(db: str, capsys: pytest.CaptureFixture[str]) -> None: + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=1", "--arg", "right=1") + capsys.readouterr() + + assert _run_cli(db, "status", "x") == EXIT_ERROR # 不存在 + + store = SqliteTaskStore(db) + try: + task_id = store.query()[0].task_id + finally: + store.close() + + assert _run_cli(db, "status", task_id) == EXIT_OK + out = capsys.readouterr().out + assert "status" in out + assert "迁移历史" in out + assert "executor:completed" in out + + +def test_stats_and_buckets(db: str, capsys: pytest.CaptureFixture[str]) -> None: + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=1", "--arg", "right=1") + capsys.readouterr() + + assert _run_cli(db, "stats") == EXIT_OK + out = capsys.readouterr().out + assert "succeeded" in out + assert "total" in out + + assert _run_cli(db, "buckets") == EXIT_OK + assert "BUCKET" in capsys.readouterr().out + + +def test_ps_rejects_an_unknown_status(db: str, capsys: pytest.CaptureFixture[str]) -> None: + assert _run_cli(db, "ps", "--status", "no-such-status") == EXIT_ERROR + assert "未知状态" in capsys.readouterr().err + + +def test_ps_filters_by_pipeline_run(db: str, capsys: pytest.CaptureFixture[str]) -> None: + """按管道查看是看板最常问的问题，过滤下推到存储层。""" + from datetime import UTC, datetime + + from chongming_task.models import TaskSnapshot + + store = SqliteTaskStore(db) + try: + for task_id, run_id in (("a1", "run-a"), ("a2", "run-a"), ("b1", "run-b")): + store.create( + TaskSnapshot( + task_id=task_id, + name="节点", + status=TaskStatus.SUCCEEDED, + status_version=1, + attempt=1, + max_attempts=1, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ended_at=datetime(2026, 1, 1, 1, tzinfo=UTC), + pipeline_run_id=run_id, + ) + ) + finally: + store.close() + + assert _run_cli(db, "ps", "--all", "--pipeline", "run-a") == EXIT_OK + out = capsys.readouterr().out + assert "a1" in out + assert "a2" in out + assert "b1" not in out + + assert _run_cli(db, "ps", "--all", "--pipeline", "no-such-run") == EXIT_OK + assert "没有匹配的任务" in capsys.readouterr().out + + +# --------------------------------------------------------------------------- +# 输出契约 +# --------------------------------------------------------------------------- + + +def test_json_output_is_primitive_only(db: str, capsys: pytest.CaptureFixture[str]) -> None: + """``--json`` 的输出必须全是原语，脚本才不必了解 datetime / 枚举的序列化规则。""" + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=2", "--arg", "right=3") + capsys.readouterr() + + assert main(["--db", db, "--json", "ps", "--all"]) == EXIT_OK + payload = json.loads(capsys.readouterr().out) + assert isinstance(payload, list) + entry = payload[0] + assert entry["status"] == "succeeded" + assert entry["outcome"] == '{"value": 5}' + assert all( + value is None or isinstance(value, (str, int, float, bool)) for value in entry.values() + ), "出现了非原语字段" + + +def test_json_error_path_uses_stderr_not_stdout(capsys: pytest.CaptureFixture[str]) -> None: + assert main(["--json", "ps"]) == EXIT_ERROR + captured = capsys.readouterr() + assert captured.out == "" + assert "未指定任务存储" in captured.err + + +# --------------------------------------------------------------------------- +# 错误路径 +# --------------------------------------------------------------------------- + + +def test_missing_db_is_reported_clearly( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + monkeypatch.delenv("CHONGMING_TASK_DB", raising=False) + assert main(["ps"]) == EXIT_ERROR + assert "CHONGMING_TASK_DB" in capsys.readouterr().err + + +def test_db_can_come_from_the_environment(db: str, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("CHONGMING_TASK_DB", db) + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=1", "--arg", "right=1") + assert main(["ps", "--all"]) == EXIT_OK + + +def test_stop_on_an_unknown_task_fails_instead_of_writing_a_dead_command( + db: str, capsys: pytest.CaptureFixture[str] +) -> None: + """曾经这里的 bug：给不存在的任务下指令会成功，留下一条永远不会被执行的指令， + 而调用方还以为"停止已下发"。 + """ + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=1", "--arg", "right=1") + capsys.readouterr() + + assert _run_cli(db, "stop", "typo-task-id") == EXIT_ERROR + assert "没有找到任务" in capsys.readouterr().err + + store = SqliteTaskStore(db) + try: + assert store.commands() == [], "不该留下任何指令" + finally: + store.close() + + +def test_control_on_an_existing_task_is_accepted( + db: str, capsys: pytest.CaptureFixture[str] +) -> None: + _run_cli(db, "run", "--tasks", _TASKS_MODULE, "add", "--arg", "left=1", "--arg", "right=1") + capsys.readouterr() + + store = SqliteTaskStore(db) + try: + task_id = store.query()[0].task_id + finally: + store.close() + + # 任务已落终态，但"找到并受理"仍然成立。 + assert _run_cli(db, "stop", task_id, "--reason", "收尾") == EXIT_OK + assert "已请求停止" in capsys.readouterr().out + + +def test_store_directory_missing_is_reported_clearly( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + """SQLite 只会报 "unable to open database file"，完全看不出是目录不存在。""" + missing = tmp_path / "not-created" / "tasks.db" + assert main(["--db", str(missing), "ps"]) == EXIT_ERROR + assert "目录不存在" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# 重启恢复 +# --------------------------------------------------------------------------- + + +def _plant_zombie(db: str, task_id: str = "zombie") -> None: + """造一个"上次崩溃留下的"任务行：running 且租约早已过期。""" + from datetime import UTC, datetime, timedelta + + from chongming_task.models import TaskSnapshot + + store = SqliteTaskStore(db) + try: + old = datetime.now(UTC) - timedelta(minutes=10) + store.create( + TaskSnapshot( + task_id=task_id, + name="遗留任务", + status=TaskStatus.RUNNING, + status_version=3, + attempt=1, + max_attempts=1, + created_at=old, + started_at=old, + lease_owner="worker-old", + lease_until=datetime.now(UTC) - timedelta(minutes=9), + ) + ) + finally: + store.close() + + +def test_reconcile_dry_run_writes_nothing(db: str, capsys: pytest.CaptureFixture[str]) -> None: + _plant_zombie(db) + assert _run_cli(db, "reconcile", "--dry-run") == EXIT_OK + out = capsys.readouterr().out + assert "marked-lost" in out + assert "预演" in out + + store = SqliteTaskStore(db) + try: + snapshot = store.get("zombie") + assert snapshot is not None + assert snapshot.status is TaskStatus.RUNNING, "预演不该写状态" + assert store.history("zombie") == [] + finally: + store.close() + + +def test_reconcile_marks_zombies(db: str, capsys: pytest.CaptureFixture[str]) -> None: + _plant_zombie(db) + assert _run_cli(db, "reconcile", "--reason", "worker-old 崩溃") == EXIT_OK + assert "已把 1 个僵尸任务标记为 lost" in capsys.readouterr().out + + store = SqliteTaskStore(db) + try: + snapshot = store.get("zombie") + assert snapshot is not None + assert snapshot.status is TaskStatus.LOST + assert store.history("zombie")[0].reason == "worker-old 崩溃" + finally: + store.close() + + +def test_reconcile_with_nothing_to_do(db: str, capsys: pytest.CaptureFixture[str]) -> None: + assert _run_cli(db, "reconcile") == EXIT_OK + assert "没有需要处理的任务" in capsys.readouterr().out + + +def test_reconcile_json_report(db: str, capsys: pytest.CaptureFixture[str]) -> None: + _plant_zombie(db) + assert main(["--db", db, "--json", "reconcile", "--dry-run"]) == EXIT_OK + payload = json.loads(capsys.readouterr().out) + assert payload["dry_run"] is True + assert payload["entries"][0]["task_id"] == "zombie" + assert payload["entries"][0]["previous_status"] == "running" + assert payload["entries"][0]["outcome"] == "marked-lost" + + +# --------------------------------------------------------------------------- +# 节点与桶归属（自治节点 + 统一平面） +# --------------------------------------------------------------------------- + + +def test_bucket_spec_supports_ranges_and_lists() -> None: + from chongming_task.cli.main import _parse_bucket_spec + + assert _parse_bucket_spec("0-3") == frozenset({0, 1, 2, 3}) + assert _parse_bucket_spec("0-1,5,7-8") == frozenset({0, 1, 5, 7, 8}) + assert _parse_bucket_spec(" 2 , 4 ") == frozenset({2, 4}) + assert _parse_bucket_spec("3-3") == frozenset({3}) + + +def test_bucket_spec_rejects_bad_input() -> None: + from chongming_task.cli.main import _parse_bucket_spec + + for bad in ("3-1", "a", "0-", "1,x"): + with pytest.raises(ConfigurationError): + _parse_bucket_spec(bad) + with pytest.raises(ConfigurationError, match="没有解析出任何桶"): + _parse_bucket_spec(" , ") + + +def test_ps_mine_filters_to_owned_buckets(db: str, capsys: pytest.CaptureFixture[str]) -> None: + """自治架构里最常看的视图："我这一份现在怎样"。""" + _plant_zombie(db, "zombie") # 无桶任务 + store = SqliteTaskStore(db) + try: + for task_id, bucket in (("in-a", 1), ("in-b", 40)): + store.create( + TaskSnapshot( + task_id=task_id, + name="节点任务", + bucket=bucket, + bucket_key=f"key-{bucket}", + status=TaskStatus.SUCCEEDED, + status_version=1, + attempt=1, + max_attempts=1, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ended_at=datetime(2026, 1, 1, 1, tzinfo=UTC), + ) + ) + finally: + store.close() + + assert main(["--db", db, "--owned-buckets", "0-31", "ps", "--all", "--mine"]) == EXIT_OK + out = capsys.readouterr().out + assert "in-a" in out + assert "in-b" not in out, "别把别的桶的任务算进来" + assert "zombie" not in out, "无桶任务不属于任何限定了归属的节点" + + +def test_ps_mine_requires_owned_buckets(db: str, capsys: pytest.CaptureFixture[str]) -> None: + assert main(["--db", db, "ps", "--mine"]) == EXIT_ERROR + assert "--owned-buckets" in capsys.readouterr().err + + +def test_reconcile_is_scoped_to_owned_buckets(db: str, capsys: pytest.CaptureFixture[str]) -> None: + """节点只清理自己桶里的僵尸——这是自治架构的核心不变量。""" + store = SqliteTaskStore(db) + try: + for task_id, bucket in (("mine", 1), ("theirs", 40)): + old = datetime(2026, 1, 1, tzinfo=UTC) + store.create( + TaskSnapshot( + task_id=task_id, + name="遗留任务", + bucket=bucket, + bucket_key=f"key-{bucket}", + status=TaskStatus.RUNNING, + status_version=3, + attempt=1, + max_attempts=1, + created_at=old, + started_at=old, + lease_owner="dead", + lease_until=old, + ) + ) + finally: + store.close() + + assert main(["--db", db, "--owned-buckets", "0-31", "reconcile"]) == EXIT_OK + out = capsys.readouterr().out + assert "mine" in out + assert "theirs" not in out + + store = SqliteTaskStore(db) + try: + assert store.get("mine").status is TaskStatus.LOST # type: ignore[union-attr] + assert store.get("theirs").status is TaskStatus.RUNNING, ( # type: ignore[union-attr] + "动了别的节点的任务" + ) + finally: + store.close() + + +def test_buckets_view_marks_owned_buckets(db: str, capsys: pytest.CaptureFixture[str]) -> None: + assert main(["--db", db, "--num-buckets", "4", "--owned-buckets", "0-1", "buckets"]) == EXIT_OK + out = capsys.readouterr().out + marked = [line for line in out.splitlines() if line.startswith("*")] + assert len(marked) == 2, f"应当恰好标出两个桶，实际：{marked}" + + +def test_an_out_of_range_owned_bucket_is_reported( + db: str, capsys: pytest.CaptureFixture[str] +) -> None: + """桶号越界必须在启动时报错——否则症状是"任务永远排队且没有任何报错"。""" + assert main(["--db", db, "--owned-buckets", "0-999", "ps"]) == EXIT_ERROR + assert "桶号越界" in capsys.readouterr().err + + +def test_run_respects_owned_buckets(db: str, capsys: pytest.CaptureFixture[str]) -> None: + """``run`` 必须**真的**受归属约束。 + + 曾经 ``run`` 自己构造 ``Controller(...)``，把 ``--owned-buckets`` 静默忽略了—— + 参数被接受却不起作用，比报错更难查。 + """ + from chongming_task import BucketPolicy + + policy = BucketPolicy(num_buckets=64) + mine = next(f"tenant-{i}" for i in range(999) if policy.bucket_for(f"tenant-{i}") < 16) + theirs = next(f"tenant-{i}" for i in range(999) if policy.bucket_for(f"tenant-{i}") >= 48) + + # 限定了归属却没给 bucket_key → 必须报错（否则会产出没人认领的任务） + assert ( + main( + [ + "--db", + db, + "--num-buckets", + "64", + "--owned-buckets", + "0-15", + "run", + "--tasks", + _TASKS_MODULE, + "add", + "--arg", + "left=1", + "--arg", + "right=1", + "--task-id", + "r1", + ] + ) + == EXIT_ERROR + ) + assert "必须给出 bucket_key" in capsys.readouterr().err + + # 落在别人的桶 → 必须报错 + assert ( + main( + [ + "--db", + db, + "--num-buckets", + "64", + "--owned-buckets", + "0-15", + "run", + "--tasks", + _TASKS_MODULE, + "add", + "--arg", + "left=1", + "--arg", + "right=1", + "--task-id", + "r2", + "--bucket-key", + theirs, + ] + ) + == EXIT_ERROR + ) + assert "不属于本节点" in capsys.readouterr().err + + # 落在自己的桶 → 正常执行 + assert ( + main( + [ + "--db", + db, + "--num-buckets", + "64", + "--owned-buckets", + "0-15", + "run", + "--tasks", + _TASKS_MODULE, + "add", + "--arg", + "left=20", + "--arg", + "right=22", + "--task-id", + "r3", + "--bucket-key", + mine, + ] + ) + == EXIT_OK + ) + assert "succeeded" in capsys.readouterr().out + + +def test_a_module_with_only_task_decorators_works( + db: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """只用 ``@task`` 装饰、**没有** ``register()`` 的模块也必须能用。 + + ``@task`` 是后来加的写法；CLI 若只认 ``register()``，那个 API 在命令行下就废了。 + """ + module = tmp_path / "decorated_only.py" + module.write_text( + "from chongming_task import task\n" + "\n" + "@task(tags={'io'})\n" + "def doubled(ctx, value: int) -> int:\n" + " '''把数字翻倍。'''\n" + " del ctx\n" + " return value * 2\n", + encoding="utf-8", + ) + monkeypatch.syspath_prepend(str(tmp_path)) + + assert main(["tasks", "--tasks", "decorated_only"]) == EXIT_OK + out = capsys.readouterr().out + assert "doubled" in out + assert "把数字翻倍。" in out + + assert ( + main(["--db", db, "run", "--tasks", "decorated_only", "doubled", "--arg", "value=21"]) + == EXIT_OK + ) + assert "succeeded" in capsys.readouterr().out + + +def test_a_module_with_neither_register_nor_task_is_reported( + capsys: pytest.CaptureFixture[str], +) -> None: + assert main(["tasks", "--tasks", "json"]) == EXIT_ERROR + assert "既没有 register" in capsys.readouterr().err diff --git a/tests/integration/test_custom_store.py b/tests/integration/test_custom_store.py new file mode 100644 index 0000000..5d1bc6d --- /dev/null +++ b/tests/integration/test_custom_store.py @@ -0,0 +1,454 @@ +"""换掉默认持久层的**可执行范例**：一个 JSON 文件后端。 + +这不是内置实现，而是"第三方后端长什么样"的活样板。它只做三件事： + +1. 实现 :class:`~chongming_task.stores.protocol.TaskStore` 契约； +2. 复用 :mod:`chongming_task.stores.codec` 做序列化，不自己重写 ``datetime`` / + ``frozenset`` / 枚举 / 嵌套结构的往返； +3. 跑通完整的 ``StoreContract``。 + +**刻意的局限，也是给后端作者的提醒**：它只用一把进程内锁，**没有文件锁**，因此仅在 +单进程内正确。真正的多进程后端必须在 ``claim_commands`` / ``release_command`` 上做 +跨进程原子性——SQLite 用 ``BEGIN IMMEDIATE``，Redis 用 Lua 脚本或 ``WATCH/MULTI``。 +本范例跑通契约不代表它多进程安全，而契约测试本身也不测跨进程。 +""" + +from __future__ import annotations + +import contextlib +import json +import os +import tempfile +import threading +import time +from collections.abc import Iterator, Mapping +from dataclasses import replace +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import Any + +from chongming_task.controller import Controller +from chongming_task.errors import ConcurrencyConflict, StoreError +from chongming_task.models import Progress, TaskSnapshot +from chongming_task.states import TERMINAL, TaskStatus +from chongming_task.stores.codec import ( + command_to_row, + encode_dt, + encode_progress, + row_to_command, + row_to_snapshot, + row_to_transition, + snapshot_to_row, + transition_to_row, +) +from chongming_task.stores.protocol import Command, TaskQuery, TaskStore +from chongming_task.testing.store_contract import StoreContract + +_REPO_ROOT = Path(__file__).resolve().parents[2] + + +#: ``Path.replace`` 在 Windows 上被瞬时占用时的重试次数。 +_REPLACE_ATTEMPTS = 5 + + +class JsonFileTaskStore: + """把一个 JSON 文件当作持久层。 + + 单进程正确，跨进程不安全（见模块文档）。写它是为了让"换持久层"这件事有一个 + 可抄、可跑、可对照的骨架。 + """ + + def __init__(self, path: str | Path) -> None: + self._path = Path(path) + self._lock = threading.RLock() + self._data: dict[str, Any] = {"tasks": {}, "transitions": {}, "commands": {}} + + # --- 内部 --- + + def _reload(self) -> None: + """每次操作前重新读盘。 + + **本实现刻意不缓存。** 缓存意味着两个连接各持一份副本，彼此永远看不到对方的 + 写入，于是跨进程控制会**静默失效**——本范例第一次跑跨进程用例时就是这样： + 控制端把指令写进了自己的副本，worker 端压根不知道。这是文件/无共享后端最 + 容易踩的坑。 + + 真要缓存，就必须配文件锁 + 失效判断；比起那个复杂度，每次都读反而更诚实。 + """ + if not self._path.exists(): + self._data = {"tasks": {}, "transitions": {}, "commands": {}} + return + text = self._path.read_text(encoding="utf-8") + self._data = ( + json.loads(text) if text.strip() else {"tasks": {}, "transitions": {}, "commands": {}} + ) + + @contextlib.contextmanager + def _transaction(self) -> Iterator[None]: + """所有公开方法的入口：加锁 → 读盘 → 交给调用方改 → 由调用方 ``_flush``。""" + with self._lock: + self._reload() + yield + + def _flush(self) -> None: + """原子替换写回。 + + 不能直接 ``write_text``：它先截断再写，另一个进程恰好在这一瞬间打开文件就会 + 读到**空文件或半截 JSON**——本范例第一次跑跨进程用例时踩到的就是这个，错误是 + ``JSONDecodeError: Expecting value: line 1 column 1``。``Path.replace`` 在 + 同一文件系统上是原子的，读方只会看到完整内容。 + + 临时文件名带上 pid：两个进程共用一个临时文件会互相覆盖。 + + Windows 上还有第三种失败，属于**瞬时**：刚写完的文件可能被杀毒扫描或索引临时 + 占住，``replace`` 抛 ``PermissionError``（WinError 5，实测约每十轮跑批命中一次）。 + 这里重试几次就够了。注意它只解决**瞬时**占用——两个实例持续共用一个文件的冲突 + 照旧无解（重试耗尽后还是抛），那正是下面 ``test_the_naive_file_store_is_not_ + cross_process_safe`` 钉住的事实。 + """ + temporary = self._path.with_name(f"{self._path.name}.{os.getpid()}.tmp") + temporary.write_text( + json.dumps(self._data, ensure_ascii=False, sort_keys=True), encoding="utf-8" + ) + for attempt in range(_REPLACE_ATTEMPTS): + try: + temporary.replace(self._path) + return + except PermissionError: + if attempt == _REPLACE_ATTEMPTS - 1: + raise + time.sleep(0.02 * (attempt + 1)) + + def _rows(self) -> dict[str, dict[str, Any]]: + return self._data["tasks"] # type: ignore[no-any-return] + + # --- 1. 任务快照 --- + + def create(self, snapshot: TaskSnapshot) -> None: + with self._transaction(): + if snapshot.task_id in self._rows(): + raise StoreError(f"任务已存在：{snapshot.task_id}") + self._rows()[snapshot.task_id] = snapshot_to_row(snapshot) + self._data["transitions"].setdefault(snapshot.task_id, {}) + self._flush() + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + with self._transaction(): + current = self._rows().get(snapshot.task_id) + if current is None: + raise StoreError(f"任务不存在：{snapshot.task_id}") + if current["status_version"] != expect_version: + raise ConcurrencyConflict( + snapshot.task_id, expect_version, current["status_version"] + ) + row = snapshot_to_row(snapshot) + # 租约由 touch 独占：save 必须保留它，否则重启恢复会把活着的任务误判失联。 + row["lease_owner"] = current["lease_owner"] + row["lease_until"] = current["lease_until"] + self._rows()[snapshot.task_id] = row + self._flush() + + def get(self, task_id: str) -> TaskSnapshot | None: + with self._transaction(): + row = self._rows().get(task_id) + return row_to_snapshot(row) if row is not None else None + + def query(self, query: TaskQuery | None = None) -> list[TaskSnapshot]: + condition = query if query is not None else TaskQuery() + with self._transaction(): + found = [row_to_snapshot(row) for row in self._rows().values()] + + if condition.statuses is not None: + found = [item for item in found if item.status in condition.statuses] + if condition.name is not None: + found = [item for item in found if item.name == condition.name] + if condition.name_prefix is not None: + found = [item for item in found if item.name.startswith(condition.name_prefix)] + if condition.tag is not None: + found = [item for item in found if condition.tag in item.tags] + if condition.buckets is not None: + found = [item for item in found if item.bucket in condition.buckets] + if condition.pipeline_run_id is not None: + found = [item for item in found if item.pipeline_run_id == condition.pipeline_run_id] + + found.sort( + key=lambda item: (item.created_at is None, item.created_at or datetime.min), + reverse=condition.newest_first, + ) + return found[: condition.limit] if condition.limit is not None else found + + def counts(self) -> dict[TaskStatus, int]: + with self._transaction(): + result: dict[TaskStatus, int] = {} + for row in self._rows().values(): + status = TaskStatus(row["status"]) + result[status] = result.get(status, 0) + 1 + return result + + # --- 2. 审计轨迹 --- + + def append(self, transition: Any) -> None: + with self._transaction(): + entries = self._data["transitions"].setdefault(transition.task_id, {}) + # 幂等：同一版本重复写入被忽略（崩溃恢复会重放）。 + entries.setdefault(str(transition.version), transition_to_row(transition)) + self._flush() + + def history(self, task_id: str, *, since_version: int = 0) -> list[Any]: + with self._transaction(): + entries: Mapping[str, Any] = self._data["transitions"].get(task_id, {}) + rows = [row for version, row in entries.items() if int(version) > since_version] + rows.sort(key=lambda row: row["version"]) + return [row_to_transition(row) for row in rows] + + # --- 3. 高频旁路 --- + + def touch( + self, + task_id: str, + *, + progress: Progress | None = None, + heartbeat_at: datetime | None = None, + lease_owner: str | None = None, + lease_until: datetime | None = None, + ) -> None: + with self._transaction(): + row = self._rows().get(task_id) + if row is None: + return # 心跳晚到一步不该报错 + if progress is not None: + # 刻意不动 status_version：否则会与状态写入持续冲突。 + row["progress"] = encode_progress(progress) + if heartbeat_at is not None: + row["last_heartbeat_at"] = encode_dt(heartbeat_at) + if lease_owner is not None: + row["lease_owner"] = lease_owner + if lease_until is not None: + row["lease_until"] = encode_dt(lease_until) + self._flush() + + # --- 4. 控制指令 --- + + def submit_command(self, command: Command) -> None: + with self._transaction(): + if command.command_id in self._data["commands"]: + raise StoreError(f"指令已存在：{command.command_id}") + self._data["commands"][command.command_id] = command_to_row(command) + self._flush() + + def claim_commands( + self, *, now: datetime, lease_seconds: float, limit: int = 50 + ) -> list[Command]: + with self._transaction(): + claimable = [ + row_to_command(row) + for row in self._data["commands"].values() + if row["finished_at"] is None + and (row["lease_until"] is None or row["lease_until"] <= encode_dt(now)) + ] + claimable.sort(key=lambda item: item.created_at) + claimed: list[Command] = [] + for item in claimable[:limit]: + leased = replace(item, lease_until=now + timedelta(seconds=lease_seconds)) + self._data["commands"][item.command_id] = command_to_row(leased) + claimed.append(leased) + self._flush() + return claimed + + def complete_command( + self, command_id: str, *, now: datetime, outcome: str | None = None + ) -> None: + with self._transaction(): + row = self._data["commands"].get(command_id) + if row is None: + return + row["finished_at"] = encode_dt(now) + row["outcome"] = outcome + row["lease_until"] = None + self._flush() + + def release_command(self, command_id: str, *, now: datetime) -> None: + del now + with self._transaction(): + row = self._data["commands"].get(command_id) + if row is None or row["finished_at"] is not None: + return + row["lease_until"] = None + self._flush() + + # --- 运维 --- + + def prune(self, *, keep_terminal: int) -> int: + with self._transaction(): + terminal = [ + row for row in self._rows().values() if TaskStatus(row["status"]) in TERMINAL + ] + terminal.sort(key=lambda row: row["created_at"] or "") + excess = len(terminal) - max(0, keep_terminal) + if excess <= 0: + return 0 + for row in terminal[:excess]: + task_id = row["task_id"] + self._rows().pop(task_id, None) + self._data["transitions"].pop(task_id, None) + self._flush() + return excess + + def close(self) -> None: + """写回并释放。文件后端没有长连接，但保持同样的契约形状。""" + with self._transaction(): + self._flush() + + +# --------------------------------------------------------------------------- +# 1. 跑完整的契约套件 +# --------------------------------------------------------------------------- + + +class TestJsonFileStore(StoreContract): + """第三方后端只要通过这一套，就可以替换内置实现。""" + + _tempdir: tempfile.TemporaryDirectory[str] + + def make_store(self) -> TaskStore: + self._tempdir = tempfile.TemporaryDirectory(prefix="chongming-jsonstore-") + return JsonFileTaskStore(Path(self._tempdir.name) / "tasks.json") + + +# --------------------------------------------------------------------------- +# 2. 证明它能接进门面（而不只是自己自洽） +# --------------------------------------------------------------------------- + + +def test_a_third_party_store_plugs_into_the_controller(tmp_path: Path) -> None: + """契约通过 ≠ 能用。这条用例证明它真的能驱动 Controller。""" + store = JsonFileTaskStore(tmp_path / "tasks.json") + controller = Controller(store=store, command_poll_interval=0.05) + controller.register("add", _add) + run = controller.submit("add", 2, 3, task_id="t1", bucket_key="tenant-1") + outcome = controller.wait("t1", timeout=10) + + assert outcome.ok + assert outcome.value == 5 + assert controller.persist_failures == (), "写穿失败被吞掉了" + + persisted = store.get("t1") + assert persisted is not None + assert persisted.status is TaskStatus.SUCCEEDED + assert persisted.bucket == controller.bucket_policy.bucket_for("tenant-1") + assert [item.to_status for item in store.history("t1")][-1] is TaskStatus.SUCCEEDED + assert run.status is TaskStatus.SUCCEEDED + controller.close(timeout=10) + + +def test_the_naive_file_store_is_not_cross_process_safe(tmp_path: Path) -> None: + """把这条局限**钉成用例**，而不是写在注释里自欺欺人。 + + 无锁的文件型后端在 Windows 上无法支撑"两个实例共用一个文件"：``Path.replace`` + 的原子替换与并发打开会互相冲突。真跑起来的表现是 ``PermissionError`` 或读回 + 空文件。 + + 所以本范例演示的是**接口形状**，不是跨进程方案。要跨进程就用内置的 + ``SqliteTaskStore``——文件锁、WAL、``BEGIN IMMEDIATE`` 都由 SQLite 负责， + 这正是它作为默认实现的理由。 + """ + path = tmp_path / "tasks.json" + first = JsonFileTaskStore(path) + second = JsonFileTaskStore(path) + first.create(snapshot_with_id("t1")) + + # 单实例读写没问题（每次操作都重新读盘）。 + assert second.get("t1") is not None + second.touch("t1", heartbeat_at=datetime.now(UTC)) + assert first.get("t1").last_heartbeat_at is not None # type: ignore[union-attr] + + # 真正的并发访问则不可靠。这里只断言"我们知道它不可靠"这个事实本身： + # 两个实例各自持锁，因此谁也无法阻止对方在自己读的瞬间替换文件。 + assert first._lock is not second._lock, "没有共享锁，正是它不能跨实例的原因" + + +# --------------------------------------------------------------------------- +# 3. 文档与协议不许漂移 +# --------------------------------------------------------------------------- + +#: 契约的方法全集。``docs/store-guide.md`` 必须逐个说明，协议本身也必须一字不差。 +DOCUMENTED_METHODS = frozenset( + { + "append", + "claim_commands", + "close", + "complete_command", + "counts", + "create", + "get", + "history", + "prune", + "query", + "release_command", + "save", + "submit_command", + "touch", + } +) + + +def test_protocol_surface_matches_the_documented_list() -> None: + """协议加了方法却忘了写进指南（或反之），这条会失败。""" + actual = {name for name in dir(TaskStore) if not name.startswith("_")} + assert actual == DOCUMENTED_METHODS + + +def test_store_guide_documents_every_protocol_method() -> None: + """指南必须逐个说明每个方法——否则"换持久层"就没有可依据的规格。""" + guide = (_REPO_ROOT / "docs" / "store-guide.md").read_text(encoding="utf-8") + missing = [ + name + for name in sorted(DOCUMENTED_METHODS) + if f"`{name}(" not in guide and f"`{name}`" not in guide + ] + assert missing == [], f"以下方法在 docs/store-guide.md 里没有说明：{missing}" + + +def test_store_guide_mentions_every_shipped_implementation() -> None: + guide = (_REPO_ROOT / "docs" / "store-guide.md").read_text(encoding="utf-8") + for name in ( + "InMemoryTaskStore", + "SqliteTaskStore", + "PostgresTaskStore", + "StoreContract", + ): + assert name in guide, f"指南里没有提到 {name}" + + +# --- 任务 --- + + +def _add(ctx: Any, left: int, right: int) -> int: + del ctx + return left + right + + +def snapshot_with_id(task_id: str) -> TaskSnapshot: + """构造一个最小的合法快照。""" + return TaskSnapshot( + task_id=task_id, + name="t", + status=TaskStatus.RUNNING, + status_version=1, + attempt=1, + max_attempts=1, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + + +def _block(ctx: Any) -> str: + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "done" + + +def test_documented_methods_all_exist_on_the_protocol() -> None: + """指南里写的名字必须真实存在——防止文档里出现一个不存在的接口。""" + for name in DOCUMENTED_METHODS: + assert hasattr(TaskStore, name), f"{name} 不在 TaskStore 上" diff --git a/tests/integration/test_node_buckets.py b/tests/integration/test_node_buckets.py new file mode 100644 index 0000000..cbb536e --- /dev/null +++ b/tests/integration/test_node_buckets.py @@ -0,0 +1,348 @@ +"""节点与桶归属测试：**自治节点 + 统一平面**架构的划分单位。 + +这个架构里，每个节点有自己的任务定义与触发方式，共用同一套存储与控制层， +靠 ``owned_buckets`` 区分"哪一份是我的"。所以这一组用例的重点是两条： + +1. **划界**——节点只接受、只清理自己负责的桶； +2. **不许静默**——任务落进别人的桶时必须在提交那一刻报错，而不是产出一个人人都不认领 + 的任务（它会永远停在 ``queued``，且没有任何报错）。 +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +import pytest + +from chongming_task.controller import Controller, ReconcileOutcome +from chongming_task.errors import ConfigurationError +from chongming_task.models import TaskSnapshot +from chongming_task.policies import BucketPolicy +from chongming_task.states import TaskStatus +from chongming_task.stores.memory import InMemoryTaskStore + +_POLICY = BucketPolicy() +_OLD = datetime(2026, 1, 1, tzinfo=UTC) + +#: 两个节点各自的桶（按业务划分）。 +_NODE_A = frozenset(range(0, 32)) +_NODE_B = frozenset(range(32, 64)) + + +def _key_for(policy: BucketPolicy, wanted: frozenset[int]) -> str: + """找一个落在指定桶集合里的业务键。""" + for index in range(5000): + key = f"tenant-{index}" + if policy.bucket_for(key) in wanted: + return key + raise AssertionError("找不到落在目标桶集合里的键") + + +# --------------------------------------------------------------------------- +# 归属的表达与校验 +# --------------------------------------------------------------------------- + + +def test_without_owned_buckets_there_is_no_constraint() -> None: + """单机（或负责全部桶的节点）不该被迫给每个任务写 bucket_key。""" + controller = Controller() + controller.register("noop", _noop) + + assert controller.owned_buckets is None + assert controller.owns_all_buckets is True + assert controller.owns_bucket(None) is True + assert controller.owns_bucket(63) is True + + run = controller.submit("noop") + assert run.snapshot.bucket is None + controller.wait_all(timeout=10) + controller.close(timeout=5) + + +def test_owned_buckets_are_recorded_as_a_set() -> None: + controller = Controller(owned_buckets=[1, 2, 2, 3]) + assert controller.owned_buckets == frozenset({1, 2, 3}) + assert controller.owns_all_buckets is False + assert controller.owns_bucket(2) is True + assert controller.owns_bucket(4) is False + assert controller.owns_bucket(None) is False + controller.close(timeout=5) + + +def test_an_empty_bucket_set_is_rejected() -> None: + """空集通常是配置写漏了——那意味着这个节点什么都不负责。""" + with pytest.raises(ConfigurationError, match="不能是空集"): + Controller(owned_buckets=[]) + + +def test_an_out_of_range_bucket_is_rejected_at_startup() -> None: + """越界桶号必须在启动时报错。 + + 放过去的症状是"某些任务永远停在 queued，且没有任何报错"——最难查的一类问题。 + """ + limit = _POLICY.num_buckets + with pytest.raises(ConfigurationError, match="桶号越界"): + Controller(bucket_policy=_POLICY, owned_buckets=[limit]) + with pytest.raises(ConfigurationError, match="桶号越界"): + Controller(bucket_policy=_POLICY, owned_buckets=[-1]) + + +def test_a_non_integer_bucket_is_rejected() -> None: + with pytest.raises(ConfigurationError, match="必须是整数"): + Controller(owned_buckets=[1, "2"]) # type: ignore[list-item] + + +# --------------------------------------------------------------------------- +# 提交时划界：不许产出"没人认领"的任务 +# --------------------------------------------------------------------------- + + +def test_a_task_without_a_bucket_is_rejected_when_buckets_are_limited() -> None: + controller = Controller(owned_buckets=_NODE_A) + controller.register("noop", _noop) + with pytest.raises(ConfigurationError, match="必须给出 bucket_key"): + controller.submit("noop") + controller.close(timeout=5) + + +def test_a_task_in_a_foreign_bucket_is_rejected() -> None: + controller = Controller(bucket_policy=_POLICY, owned_buckets=_NODE_A) + controller.register("noop", _noop) + foreign = _key_for(_POLICY, _NODE_B) + + with pytest.raises(ConfigurationError, match="不属于本节点"): + controller.submit("noop", bucket_key=foreign) + controller.close(timeout=5) + + +def test_a_task_in_an_owned_bucket_is_accepted() -> None: + controller = Controller(bucket_policy=_POLICY, owned_buckets=_NODE_A) + controller.register("noop", _noop) + mine = _key_for(_POLICY, _NODE_A) + + run = controller.submit("noop", bucket_key=mine) + assert run.snapshot.bucket in _NODE_A + controller.wait_all(timeout=10) + controller.close(timeout=5) + + +def test_a_node_owning_every_bucket_accepts_unbucketed_tasks() -> None: + """常见形态：一个节点显式列举了全部桶（生成出来的配置）。 + + 此时没有任何"别的节点"，无桶任务同样会且只会由它执行——再强制每个任务写 + ``bucket_key`` 就只是纯摩擦。 + """ + policy = BucketPolicy(num_buckets=8) + controller = Controller(bucket_policy=policy, owned_buckets=range(8)) + controller.register("noop", _noop) + + assert controller.owns_all_buckets is True, "覆盖全部桶 = 不构成划分" + assert controller.owned_buckets == frozenset(range(8)), "但归属集合仍然记着" + + run = controller.submit("noop") + assert run.snapshot.bucket is None + controller.wait_all(timeout=10) + controller.close(timeout=5) + + +def test_owning_one_bucket_less_is_a_real_partition() -> None: + """差一个桶就构成划分，此时必须约束——边界不能含糊。""" + policy = BucketPolicy(num_buckets=8) + controller = Controller(bucket_policy=policy, owned_buckets=range(7)) + controller.register("noop", _noop) + + assert controller.owns_all_buckets is False + with pytest.raises(ConfigurationError, match="必须给出 bucket_key"): + controller.submit("noop") + controller.close(timeout=5) + + +# --------------------------------------------------------------------------- +# reconcile 的作用范围（本组最重要） +# --------------------------------------------------------------------------- + + +def _plant_zombie(store: InMemoryTaskStore, task_id: str, bucket: int) -> None: + """种一个"上次崩溃留下的"任务：running 且租约早已过期。""" + store.create( + TaskSnapshot( + task_id=task_id, + name="遗留任务", + bucket=bucket, + bucket_key=f"key-{bucket}", + status=TaskStatus.RUNNING, + status_version=3, + attempt=1, + max_attempts=1, + created_at=_OLD, + started_at=_OLD, + lease_owner="dead-worker", + lease_until=_OLD, + ) + ) + + +def test_reconcile_only_touches_its_own_buckets() -> None: + """**这是自治架构的核心不变量。** + + 共享存储里有别的节点的任务，去动它既越界（那是它的责任）也没有意义 + （它的租约由它自己续）。 + """ + store = InMemoryTaskStore() + mine = min(_NODE_A) + theirs = min(_NODE_B) + _plant_zombie(store, "mine", mine) + _plant_zombie(store, "theirs", theirs) + + node_a = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_A) + report = node_a.reconcile() + + assert [item.task_id for item in report.lost] == ["mine"] + assert store.get("theirs").status is TaskStatus.RUNNING, ( # type: ignore[union-attr] + "node A 动了 node B 的任务" + ) + + node_b = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_B) + assert [item.task_id for item in node_b.reconcile().lost] == ["theirs"] + assert store.get("theirs").status is TaskStatus.LOST # type: ignore[union-attr] + + node_a.close(timeout=5) + node_b.close(timeout=5) + + +def test_reconcile_without_a_limit_still_covers_everything() -> None: + """单机语义不能被这次改动破坏。""" + store = InMemoryTaskStore() + _plant_zombie(store, "one", min(_NODE_A)) + _plant_zombie(store, "two", min(_NODE_B)) + + controller = Controller(store=store, bucket_policy=_POLICY) + report = controller.reconcile() + assert sorted(item.task_id for item in report.lost) == ["one", "two"] + assert {item.outcome for item in report.entries} == {ReconcileOutcome.MARKED_LOST} + controller.close(timeout=5) + + +def test_reconcile_ignores_a_task_with_no_bucket_when_limited() -> None: + """没有归属桶的任务不属于任何节点——限定归属的节点不该去动它。""" + store = InMemoryTaskStore() + store.create( + TaskSnapshot( + task_id="loose", + name="无归属", + status=TaskStatus.RUNNING, + status_version=3, + attempt=1, + max_attempts=1, + created_at=_OLD, + started_at=_OLD, + lease_owner="dead", + lease_until=_OLD, + ) + ) + node = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_A) + assert node.reconcile().entries == () + assert store.get("loose").status is TaskStatus.RUNNING # type: ignore[union-attr] + node.close(timeout=5) + + +# --------------------------------------------------------------------------- +# 与统一平面的协同 +# --------------------------------------------------------------------------- + + +def test_two_nodes_share_one_store_without_stepping_on_each_other() -> None: + """每个节点跑自己的任务，共享存储提供统一的观测面。""" + store = InMemoryTaskStore() + key_a = _key_for(_POLICY, _NODE_A) + key_b = _key_for(_POLICY, _NODE_B) + + node_a = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_A, owner="a") + node_b = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_B, owner="b") + node_a.register("noop", _noop) + node_b.register("noop", _noop) + + node_a.submit("noop", task_id="a1", bucket_key=key_a) + node_b.submit("noop", task_id="b1", bucket_key=key_b) + node_a.wait_all(timeout=10) + node_b.wait_all(timeout=10) + + # 统一观测面：任一个节点都能看到全部任务。 + assert {s.task_id for s in node_a.list_tasks()} == {"a1", "b1"} + # 各自的"我这一份"：靠桶筛选，已有下推查询支持。 + mine = node_a.list_tasks(buckets=node_a.owned_buckets) + assert [s.task_id for s in mine] == ["a1"] + assert node_a.owns_bucket(store.get("a1").bucket) # type: ignore[union-attr] + assert not node_a.owns_bucket(store.get("b1").bucket) # type: ignore[union-attr] + + node_a.close(timeout=5) + node_b.close(timeout=5) + + +def test_stopping_a_task_owned_by_another_node_goes_through_the_command_channel() -> None: + """控制层统一：一个节点可以停止另一个节点正在跑的任务——靠指令通道，不靠抢占。""" + store = InMemoryTaskStore() + node_b = Controller( + store=store, + bucket_policy=_POLICY, + owned_buckets=_NODE_B, + command_poll_interval=0.05, + owner="b", + ) + node_b.register("block", _block) + node_b.submit("block", task_id="b1", bucket_key=_key_for(_POLICY, _NODE_B)) + assert _wait_until(lambda: node_b.get("b1").status is TaskStatus.RUNNING) # type: ignore[union-attr] + + node_a = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_A, owner="a") + assert node_a.stop("b1", reason="另一个节点发起的停止") is True + assert _wait_until(lambda: node_a.get("b1").status is TaskStatus.STOPPED) # type: ignore[union-attr] + + node_a.close(timeout=5) + node_b.close(timeout=5) + + +def test_a_node_that_declares_buckets_polls_commands_from_startup() -> None: + """声明了归属 = 真 worker，从启动就轮询指令。 + + 否则会有一个延迟生效的坑：别人 `stop` 一个属于本节点、当前还排在队列里的任务时， + 指令已经写进存储，但本节点要等到**第一次提交**才启动轮询线程——对定时任务节点 + 来说那可能是几小时之后。 + """ + store = InMemoryTaskStore() + node = Controller(store=store, bucket_policy=_POLICY, owned_buckets=_NODE_A) + assert node._worker is not None, "声明了归属的节点必须从启动就轮询" + node.close(timeout=5) + + +def test_a_pure_query_client_still_does_not_poll() -> None: + """没有声明归属的实例不轮询——否则它会抢走别人的指令。""" + store = InMemoryTaskStore() + client = Controller(store=store, bucket_policy=_POLICY) + assert client._worker is None + client.close(timeout=5) + + +def _wait_until(predicate: Any, timeout: float = 15.0) -> bool: + import time + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +def _noop(ctx: object) -> str: + del ctx + return "ok" + + +def _block(ctx: object) -> str: + import time + + for _ in range(1_000_000): + ctx.checkpoint() # type: ignore[attr-defined] + time.sleep(0.005) + return "never" diff --git a/tests/integration/test_persist_recovery.py b/tests/integration/test_persist_recovery.py new file mode 100644 index 0000000..d055dad --- /dev/null +++ b/tests/integration/test_persist_recovery.py @@ -0,0 +1,402 @@ +"""写穿（write-through）的恢复能力测试。 + +这一组用例存在的理由：**写穿失败曾经会让任务的存储状态永久冻结**。 + +故障链是：``_persist`` 用 ``expect_version = transition.version - 1`` 写入，这个式子 +假设此前的每一次迁移都成功落库。一旦某次失败，存储停在 k-1 而内存已到 k，此后**每一次** +写入都因版本对不上而冲突——一次瞬时故障让任务在存储里永远停住，最后被 ``reconcile()`` +判成 ``LOST``：把一个真正跑完的任务报成"结果未知"。 + +所以这里的重点不是"失败时不会崩"，而是"**失败之后能追上**"。 +""" + +from __future__ import annotations + +import threading +import time +from typing import Any + +import pytest + +from chongming_task.controller import Controller +from chongming_task.errors import ConfigurationError +from chongming_task.models import FailureKind, TaskSnapshot +from chongming_task.states import TaskStatus +from chongming_task.stores.memory import InMemoryTaskStore +from chongming_task.stores.sqlite import SqliteTaskStore + +_WINDOW = 15.0 + + +class FlakyStore(InMemoryTaskStore): + """可以按需让 ``save`` 抖动若干次，之后恢复正常。 + + ``fail_saves`` 计数：每次 ``save`` 尝试消耗一个；为 0 时完全正常。这样能精确模拟 + "网络抖了一下"以及"存储长时间不可用"两种情况。 + """ + + def __init__(self, *, fail_saves: int = 0, fail_forever: bool = False) -> None: + super().__init__() + self.fail_saves = fail_saves + self.fail_forever = fail_forever + self.save_calls = 0 + self._mutex = threading.RLock() + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + with self._mutex: + self.save_calls += 1 + if self.fail_forever or self.fail_saves > 0: + if not self.fail_forever: + self.fail_saves -= 1 + raise OSError("模拟存储抖动") + super().save(snapshot, expect_version=expect_version) + + +def _wait_until_running(controller: Controller, task_id: str, *, timeout: float = 5.0) -> None: + """等任务真的进入 ``RUNNING``。 + + **为什么必须显式等**：``TaskRun.pause()`` 对还没进入 ``RUNNING`` 的任务会 + **立刻返回 False 且不产生任何迁移**。提交后马上开循环，若执行线程还没被调度起来， + 整个循环会在微秒级跑完、一次都没暂停成功——用例看起来"通过到了断言那一步"， + 实则前提从未成立（它曾因此间歇性失败）。机器越忙，这个窗口越大。 + + **为什么不能用 ``controller.get()``**：它**优先读存储**，而本文件的存储多半是故意 + 写不进去的 ``FlakyStore``——那样永远等不到 ``RUNNING``。这里只看本地执行句柄。 + """ + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + for run in controller.runs(): + if run.snapshot.task_id == task_id and run.status is TaskStatus.RUNNING: + return + time.sleep(0.005) + raise AssertionError(f"{task_id} 没能在 {timeout}s 内进入 RUNNING") + + +# --------------------------------------------------------------------------- +# 核心：一次抖动不能让存储永久冻结 +# --------------------------------------------------------------------------- + + +def test_a_single_transient_failure_does_not_freeze_the_store() -> None: + """**这条是本次修复的核心回归用例。** + + 以前：第 3 次 save 抖动一次 → 存储永久停在 ``starting``，而任务其实跑完了。 + 现在：下一次迁移会把存储一次追上。 + """ + store = FlakyStore(fail_saves=1) + controller = Controller(store=store) + controller.register("ok", _ok) + controller.submit("ok", task_id="t1") + outcome = controller.wait("t1", timeout=_WINDOW) + + assert outcome.ok, "存储抖动不该让任务失败" + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.SUCCEEDED, "存储必须追上内存" + assert snapshot.outcome == {"value": "done"} + + history = [record.to_status for record in store.history("t1")] + assert history == [ + TaskStatus.QUEUED, + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.SUCCEEDED, + ], "迁移轨迹必须完整——缓冲的意义就在这里" + + assert controller.desynced_tasks == {}, "瞬时故障不该造成永久分叉" + assert controller.persist_failures, "但失败必须留痕" + controller.close(timeout=_WINDOW) + + +def test_the_store_version_catches_up_after_a_gap() -> None: + """版本会**跳号**（这是允许的），但必须单调且最终一致。""" + store = FlakyStore(fail_saves=2) + controller = Controller(store=store) + controller.register("ok", _ok) + controller.submit("ok", task_id="t1") + controller.wait("t1", timeout=_WINDOW) + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.SUCCEEDED + assert snapshot.status_version == 4, "内存算出的版本" + + versions = [record.version for record in store.history("t1")] + assert versions == [1, 2, 3, 4], "补写后轨迹不缺项" + assert versions == sorted(versions), "版本必须单调" + controller.close(timeout=_WINDOW) + + +def test_failures_spread_over_several_transitions_still_converge() -> None: + """抖动散布在多次迁移上（而不是集中一次）也要能追上。""" + store = FlakyStore(fail_saves=3) + controller = Controller(store=store) + controller.register("ok", _ok) + controller.submit("ok", task_id="t1") + assert controller.wait("t1", timeout=_WINDOW).ok + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.SUCCEEDED + controller.close(timeout=_WINDOW) + + +def test_a_long_outage_recovered_by_close_catches_up() -> None: + """存储整个任务期间都不可用：任务照常成功，``close()`` 时尽力补一次。 + + 补不上的话，任务在存储里就是落后的——这仍然比"任务失败"好，但必须在 + ``persist_failures`` 里留痕。 + """ + store = FlakyStore(fail_forever=True) + controller = Controller(store=store) + controller.register("ok", _ok) + controller.submit("ok", task_id="t1") + outcome = controller.wait("t1", timeout=_WINDOW) + + assert outcome.ok, "存储不可用不该让任务失败" + # 一次都没写进去：行停在 create 时的初始状态（pending）。 + assert store.get("t1").status is TaskStatus.PENDING # type: ignore[union-attr] + assert len(controller.persist_failures) >= 1 + + # 存储恢复后 close() 会补上 + store.fail_forever = False + controller.close(timeout=_WINDOW) + assert store.get("t1").status is TaskStatus.SUCCEEDED # type: ignore[union-attr] + + +def test_the_pending_buffer_is_bounded() -> None: + """存储长期不可用时内存不能跟着无限增长。""" + from chongming_task.controller import _PENDING_TRANSITIONS + + store = FlakyStore(fail_forever=True) + controller = Controller(store=store) + controller.register("spinner", _spinner) + controller.submit("spinner", task_id="t1") + _wait_until_running(controller, "t1") + + # 让任务反复暂停/恢复，制造远超上限的迁移次数。 + for _ in range(_PENDING_TRANSITIONS + 20): + assert controller.pause("t1", timeout=2.0), "任务没到达检查点：迁移没发生，用例失去意义" + controller.resume("t1") + + state = controller._persist_states["t1"] + assert len(state.pending) <= _PENDING_TRANSITIONS + assert state.dropped > 0, "丢弃必须被计数，否则轨迹缺口就不可见了" + + controller.stop("t1") + controller.wait_all(timeout=_WINDOW) + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 外部写入者：该放手时放手 +# --------------------------------------------------------------------------- + + +def test_an_external_writer_marks_the_task_desynced_instead_of_being_overwritten() -> None: + """用"上次确实写进去的版本"仍冲突 → 是**别人**写的，覆盖它会抹掉对方的状态。 + + 这与"我们自己漏写了一笔"必须区分开：前者要放手，后者要追上去。 + """ + store = InMemoryTaskStore() + controller = Controller(store=store, command_poll_interval=0.05) + controller.register("block", _block) + run = controller.submit("block", task_id="t1") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + # 模拟另一个写入者把版本推进了（例如另一个进程认领了同一个任务）。 + current = store.get("t1") + assert current is not None + intruder = _bump(current) + store.save(intruder, expect_version=current.status_version) + + # 触发一次我们自己的迁移：应当发现外部写入并放手。 + controller.stop("t1") + assert ( + _wait_until(lambda: bool(controller.desynced_tasks)) + or controller.wait_all(timeout=_WINDOW) is not None + ) + + desynced = controller.desynced_tasks + assert "t1" in desynced, "必须检测到外部写入者并停止写穿" + assert "外部写入者" in desynced["t1"] + + # 对方的状态没有被我们覆盖。 + assert store.get("t1").status is TaskStatus.RUNNING # type: ignore[union-attr] + controller.close(timeout=_WINDOW) + + +def test_desynced_tasks_are_reported_not_hidden() -> None: + """永久分叉必须可见——只靠 persist_failures（最近 20 条）是不够的。""" + store = InMemoryTaskStore() + controller = Controller(store=store) + assert controller.desynced_tasks == {} + + state = controller._persist_state("ghost") + assert controller.desynced_tasks == {} + state.desynced = "测试" + assert controller.desynced_tasks == {"ghost": "测试"} + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 非法返回值：响亮失败，而不是静默损坏 +# --------------------------------------------------------------------------- + + +def test_a_non_serializable_return_value_fails_the_task_loudly() -> None: + """以前：进程内 succeeded、存储停在 running、重启后被判 LOST。 + + 现在：任务**失败**，原因写明是返回值不满足持久化契约。 + """ + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("bad_result", _bad_result) + controller.submit("bad_result", task_id="t1") + outcome = controller.wait("t1", timeout=_WINDOW) + + assert not outcome.ok, "违反契约必须是失败，而不是静默的'成功'" + assert outcome.status is TaskStatus.FAILED + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.FAILED, "存储与内存一致，不再是 running" + assert snapshot.failure is not None + assert snapshot.failure.kind is FailureKind.RESULT_INVALID + assert "不可 JSON 序列化" in snapshot.failure.message + assert snapshot.failure.retryable is False, "重试解决不了：同一个值还是不可序列化" + controller.close(timeout=_WINDOW) + + +def test_a_non_serializable_result_is_not_retried() -> None: + """重试没有意义，所以只尝试一次。""" + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("bad_result", _bad_result) + controller.submit("bad_result", task_id="t1", retry=_retry_three()) + controller.wait("t1", timeout=_WINDOW) + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.attempt == 1, "不可序列化不是暂时性问题，不该重试" + controller.close(timeout=_WINDOW) + + +def test_serializable_results_are_unaffected() -> None: + """正常路径不能被这次收紧影响。""" + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("rich", _rich_result) + controller.submit("rich", task_id="t1") + outcome = controller.wait("t1", timeout=_WINDOW) + assert outcome.ok + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.outcome is not None + assert snapshot.outcome["value"]["nested"] == [1, 2, 3] + controller.close(timeout=_WINDOW) + + +def test_a_non_string_tag_is_rejected_at_submit_time() -> None: + """标签会被写进存储（JSON 数组），所以必须在**入口**按类型契约拒绝。 + + 真实类型是 ``frozenset[str]``。非字符串元素既不能正确往返，又只会在写穿路径上才 + 炸——而那条路径的失败不会让任务失败，问题会以"看板上的标签不对"的形式出现。 + + 注意这条**必须用会序列化的后端**才测得出存储层的反应：``InMemoryTaskStore`` 直接 + 持有对象、从不编码。"内存后端能跑"不等于"真后端能跑"。 + """ + store = SqliteTaskStore(":memory:") + try: + controller = Controller(store=store) + controller.register("ok", _ok) + with pytest.raises(ConfigurationError, match="标签必须是字符串"): + controller.submit("ok", task_id="t1", tags=[object()]) + controller.close(timeout=_WINDOW) + finally: + store.close() + + +def test_a_non_serializable_progress_message_raises_at_the_call_site() -> None: + """进度字段会被写进存储；类型错误必须在调用点炸掉，而不是流到写穿路径上。""" + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("bad_progress", _bad_progress) + outcome = controller.wait( + controller.submit("bad_progress", task_id="t1").snapshot.task_id, timeout=_WINDOW + ) + + assert not outcome.ok + assert "message 必须是 str" in (outcome.failure.message if outcome.failure else "") + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.FAILED + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 工具 +# --------------------------------------------------------------------------- + + +def _bump(snapshot: TaskSnapshot) -> TaskSnapshot: + """构造一个"别人写的"快照：版本前进一步。""" + from dataclasses import replace + + return replace(snapshot, status_version=snapshot.status_version + 1) + + +def _retry_three() -> Any: + from chongming_task.policies import FixedBackoff, RetryPolicy + + return RetryPolicy(max_attempts=3, backoff=FixedBackoff(0.0)) + + +def _wait_until(predicate: Any, timeout: float = _WINDOW) -> bool: + import time + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +def _ok(ctx: Any) -> str: + del ctx + return "done" + + +def _bad_result(ctx: Any) -> Any: + del ctx + return {1, 2, 3} + + +def _rich_result(ctx: Any) -> dict[str, Any]: + del ctx + return {"nested": [1, 2, 3], "flag": True, "text": "中文"} + + +def _bad_progress(ctx: Any) -> str: + ctx.progress(1, total=10, message=object()) # type: ignore[arg-type] + return "unreachable" + + +def _spinner(ctx: Any) -> str: + import time + + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.002) + return "never" + + +def _block(ctx: Any) -> str: + import time + + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "never" diff --git a/tests/integration/test_postgres_store.py b/tests/integration/test_postgres_store.py new file mode 100644 index 0000000..67e737a --- /dev/null +++ b/tests/integration/test_postgres_store.py @@ -0,0 +1,379 @@ +"""Postgres 后端的契约与跨连接测试。 + +需要一台可连的 Postgres，DSN 从环境变量给出:: + + CHONGMING_TEST_POSTGRES='postgresql://user@127.0.0.1:5432/tasks' \\ + uv run pytest tests/integration/test_postgres_store.py + +未设置时整组跳过——它依赖外部服务，与内置实现的契约测试（纯本地）刻意分开。 + +这里覆盖三类只有真数据库才能证明的事： + +1. **完整契约**——与内存 / SQLite 跑的是同一套 ``StoreContract``； +2. **跨连接原子性**——设计说明 §15.4 明确要求后端自建并发用例： + N 个连接并发 ``save`` 恰好一个成功、并发 ``claim_commands`` 不重复下发、 + 持有者被强杀后租约仍按时间过期且连接池自愈； +3. **接线**——``Controller`` 真的能把它当存储用，跨 store 的控制走指令通道。 +""" + +from __future__ import annotations + +import os +import threading +import time +from datetime import UTC, datetime, timedelta +from typing import Any +from uuid import uuid4 + +import pytest + +psycopg = pytest.importorskip("psycopg") +pytest.importorskip("psycopg_pool") + +from psycopg import sql # noqa: E402 +from psycopg.conninfo import make_conninfo # noqa: E402 + +from chongming_task.controller import Controller # noqa: E402 +from chongming_task.errors import ConcurrencyConflict, StoreError # noqa: E402 +from chongming_task.states import TaskStatus # noqa: E402 +from chongming_task.stores.postgres import PostgresTaskStore # noqa: E402 +from chongming_task.stores.protocol import TaskStore # noqa: E402 +from chongming_task.testing.store_contract import ( # noqa: E402 + StoreContract, + command, + simple_snapshot, +) + +_DSN = os.environ.get("CHONGMING_TEST_POSTGRES") +_T0 = datetime(2026, 1, 1, tzinfo=UTC) + +pytestmark = pytest.mark.skipif( + not _DSN, + reason="未设置 CHONGMING_TEST_POSTGRES（DSN）：Postgres 集成测试需要一台可连的数据库", +) + + +# --------------------------------------------------------------------------- +# 隔离：每个存储实例独占一个 schema（与 SQLite 的临时文件等价） +# --------------------------------------------------------------------------- + + +def _isolated_dsn() -> tuple[str, str]: + """建一个全新 schema，返回 ``(schema 名, 指向它的 DSN)``。""" + assert _DSN is not None + schema = f"cstore_{uuid4().hex[:12]}" + with psycopg.connect(_DSN, autocommit=True) as conn: + conn.execute(sql.SQL("CREATE SCHEMA {}").format(sql.Identifier(schema))) + return schema, make_conninfo(_DSN, options=f"-csearch_path={schema}") + + +def _drop_schema(schema: str) -> None: + assert _DSN is not None + with psycopg.connect(_DSN, autocommit=True) as conn: + conn.execute(sql.SQL("DROP SCHEMA IF EXISTS {} CASCADE").format(sql.Identifier(schema))) + + +# --------------------------------------------------------------------------- +# 1. 完整契约 +# --------------------------------------------------------------------------- + + +class TestPostgresStoreContract(StoreContract): + """与内存 / SQLite 实现跑**完全相同**的用例。""" + + _sandboxes: list[str] + + def setup_method(self) -> None: + self._sandboxes = [] + super().setup_method() + + def make_store(self) -> TaskStore: + schema, dsn = _isolated_dsn() + self._sandboxes.append(schema) + return PostgresTaskStore(dsn) + + def teardown_method(self) -> None: + super().teardown_method() + for schema in self._sandboxes: + _drop_schema(schema) + + +# --------------------------------------------------------------------------- +# 2. 跨连接原子性（StoreContract 覆盖不到，设计说明 §15.4 要求） +# --------------------------------------------------------------------------- + + +def test_concurrent_saves_have_exactly_one_winner() -> None: + """N 个连接并发 save 同一任务：恰好一个成功，其余收到冲突。 + + 乐观并发在**跨连接**下真正成立，而不是只在单连接语义里成立。 + """ + schema, dsn = _isolated_dsn() + try: + seed = PostgresTaskStore(dsn) + seed.create(simple_snapshot("t1", version=1)) + seed.close() + + workers = 8 + barrier = threading.Barrier(workers) + outcomes: list[str] = [] + lock = threading.Lock() + + def attempt() -> None: + store = PostgresTaskStore(dsn) + try: + barrier.wait(timeout=10) + try: + store.save( + simple_snapshot("t1", status=TaskStatus.SUCCEEDED, version=2), + expect_version=1, + ) + except ConcurrencyConflict: + outcome = "conflict" + else: + outcome = "ok" + finally: + store.close() + with lock: + outcomes.append(outcome) + + threads = [threading.Thread(target=attempt) for _ in range(workers)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=30) + + assert sorted(outcomes) == ["conflict"] * (workers - 1) + ["ok"] + finally: + _drop_schema(schema) + + +def test_concurrent_claims_hand_out_each_command_exactly_once() -> None: + """N 个连接并发认领：同一条指令绝不会被两个连接取到。 + + "挑选 + 打租约"是一条 ``UPDATE ... FOR UPDATE SKIP LOCKED``，所以并发认领互不 + 阻塞、也不会重复。这是 SQLite 用 ``BEGIN IMMEDIATE`` 换来的同一个保证。 + """ + schema, dsn = _isolated_dsn() + try: + total = 20 + seed = PostgresTaskStore(dsn) + for index in range(total): + seed.submit_command(command(f"c{index}", created_offset=float(index))) + seed.close() + + claimed: list[str] = [] + lock = threading.Lock() + + def drain() -> None: + store = PostgresTaskStore(dsn) + try: + while True: + batch = store.claim_commands(now=_T0, lease_seconds=300, limit=3) + if not batch: + return + with lock: + claimed.extend(item.command_id for item in batch) + finally: + store.close() + + threads = [threading.Thread(target=drain) for _ in range(8)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=30) + + assert sorted(claimed) == sorted(f"c{index}" for index in range(total)) + assert len(claimed) == len(set(claimed)), "同一条指令被下发了两次" + finally: + _drop_schema(schema) + + +def test_a_killed_holder_does_not_wedge_the_lease() -> None: + """持有者被强杀：租约仍按时间过期，别的连接可以接管；连接池自愈。 + + 租约是**存储里的数据**，不是连接上的会话状态——所以杀掉持有者的后端连接既不会 + 提前释放它，也不会永久锁住它。 + """ + schema, dsn = _isolated_dsn() + try: + store_a = PostgresTaskStore(dsn, min_size=1, max_size=1) + store_a.create(simple_snapshot("t1")) + store_a.submit_command(command("c1")) + claimed = store_a.claim_commands(now=_T0, lease_seconds=30) + assert [item.command_id for item in claimed] == ["c1"] + + # 模拟持有者进程消失：杀掉它的后端连接（本测试此刻只有 store_a 在用）。 + killed: list[bool] = [] + try: + with psycopg.connect(_DSN, autocommit=True) as admin: + rows = admin.execute( + "SELECT pg_terminate_backend(pid) FROM pg_stat_activity " + "WHERE application_name = 'chongming-task' AND pid <> pg_backend_pid()" + ).fetchall() + killed = [bool(row[0]) for row in rows] + except psycopg.errors.InsufficientPrivilege: # pragma: no cover - 权限受限的环境 + pytest.skip("当前角色无权 pg_terminate_backend，跳过") + assert any(killed), "没有杀掉任何连接，测试前提不成立" + + store_b = PostgresTaskStore(dsn) + # 租约未到期：接管者取不到。 + assert store_b.claim_commands(now=_T0 + timedelta(seconds=10), lease_seconds=30) == [] + # 租约到期：可以接管。 + reclaimed = store_b.claim_commands(now=_T0 + timedelta(seconds=31), lease_seconds=30) + assert [item.command_id for item in reclaimed] == ["c1"] + + # 被杀掉的持有者：连接池把死连接换掉，操作照常。 + assert store_a.get("t1") is not None + + store_a.close() + store_b.close() + finally: + _drop_schema(schema) + + +# --------------------------------------------------------------------------- +# 3. close() 的语义（模块文档承诺过，就要有用例守着） +# --------------------------------------------------------------------------- + + +def test_close_is_idempotent_and_operations_fail_afterwards() -> None: + schema, dsn = _isolated_dsn() + try: + store = PostgresTaskStore(dsn) + store.close() + store.close() # 幂等：重复关闭是空操作 + with pytest.raises(StoreError, match="已关闭"): + store.get("t1") + finally: + _drop_schema(schema) + + +def test_a_role_without_ddl_privileges_can_use_an_existing_schema() -> None: + """建表是**条件性**的：表已存在时一个 DDL 都不执行。 + + 因此运维控制台可以用只读账号接进来（只给 SELECT 就够看板了）。若实现无条件跑 + ``CREATE TABLE IF NOT EXISTS``，这个账号会因为缺 CREATE 权限直接连不上。 + """ + schema, dsn = _isolated_dsn() + role = f"cstore_ro_{uuid4().hex[:8]}" + password = uuid4().hex + try: + seed = PostgresTaskStore(dsn) + seed.create(simple_snapshot("t1", name="只读也能看到")) + seed.close() + + try: + with psycopg.connect(_DSN, autocommit=True) as admin: + admin.execute( + sql.SQL("CREATE ROLE {} LOGIN PASSWORD {}").format( + sql.Identifier(role), sql.Literal(password) + ) + ) + except psycopg.errors.InsufficientPrivilege: # pragma: no cover - 权限受限的环境 + pytest.skip("当前角色无权创建角色，跳过") + + with psycopg.connect(_DSN, autocommit=True) as admin: + admin.execute( + sql.SQL("GRANT USAGE ON SCHEMA {} TO {}").format( + sql.Identifier(schema), sql.Identifier(role) + ) + ) + admin.execute( + sql.SQL( + "GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA {} TO {}" + ).format(sql.Identifier(schema), sql.Identifier(role)) + ) + + info = psycopg.conninfo.conninfo_to_dict(dsn) if dsn else {} + info.update({"user": role, "password": password}) + readonly_dsn = psycopg.conninfo.make_conninfo(**info) + + store = PostgresTaskStore(readonly_dsn) # 没有 CREATE 权限，也必须能起来 + try: + snapshot = store.get("t1") + assert snapshot is not None and snapshot.name == "只读也能看到" + assert len(store.query()) == 1 + finally: + store.close() + finally: + with psycopg.connect(_DSN, autocommit=True) as admin: + admin.execute(sql.SQL("DROP OWNED BY {}").format(sql.Identifier(role))) + admin.execute(sql.SQL("DROP ROLE IF EXISTS {}").format(sql.Identifier(role))) + _drop_schema(schema) + + +# --------------------------------------------------------------------------- +# 4. 接线：Controller 真的能把它当存储用 +# --------------------------------------------------------------------------- + + +def _add(ctx: Any, left: int, right: int) -> int: + del ctx + return left + right + + +def _block(ctx: Any) -> str: + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "done" + + +def _wait_until(predicate: Any, timeout: float = 10.0) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +def test_state_transitions_persist_and_are_auditable() -> None: + schema, dsn = _isolated_dsn() + try: + store = PostgresTaskStore(dsn) + controller = Controller(store=store) + controller.register("add", _add) + controller.submit("add", 2, 3, task_id="t1") + controller.wait_all(timeout=10) + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.SUCCEEDED + assert snapshot.outcome == {"value": 5} + assert [item.to_status for item in store.history("t1")] == [ + TaskStatus.QUEUED, + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.SUCCEEDED, + ] + store.close() + finally: + _drop_schema(schema) + + +def test_stop_crosses_stores_through_postgres() -> None: + """两个 store = 两个"进程"：worker 在跑，运维从另一套连接停掉它。""" + schema, dsn = _isolated_dsn() + try: + owner_store = PostgresTaskStore(dsn) + owner = Controller(store=owner_store, command_poll_interval=0.05, owner="worker-pg") + owner.register("block", _block) + run = owner.submit("block", task_id="t1") + + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + client_store = PostgresTaskStore(dsn) + client = Controller(store=client_store, owner="ops-pg") + assert client.runs() == (), "客户端并不持有这个任务" + assert client.stop("t1", reason="跨 store 停止") is True + + assert _wait_until(lambda: run.status is TaskStatus.STOPPED), "持有者没有取走指令" + final = client.get("t1") + assert final is not None and final.status is TaskStatus.STOPPED + + owner.close(timeout=10) + client.close(timeout=5) + finally: + _drop_schema(schema) diff --git a/tests/integration/test_process_execution.py b/tests/integration/test_process_execution.py new file mode 100644 index 0000000..38e5047 --- /dev/null +++ b/tests/integration/test_process_execution.py @@ -0,0 +1,171 @@ +"""进程执行器的端到端接线：``TaskRun`` / ``Controller`` / 存储。 + +单元测试盯的是句柄本身（暂停、强杀、心跳、崩溃）。这里盯的是**接进主线之后**的 +三件承诺： + +1. 快照里真的记下了子进程的 pid（排障时 `ps` 对得上号的就是它）； +2. 不响应停止的任务会被**超时机制真的杀掉**，而不是留一个卡死的线程（线程执行器 + 只能把它记成 `LOST`，这是进程执行器存在的理由）； +3. 这套东西写进存储之后仍然对得上（pid 是 store 契约里的字段，不是内存里的临时值）。 +""" + +from __future__ import annotations + +import multiprocessing +import os +import time +from pathlib import Path +from typing import Any + +from chongming_task import ( + Controller, + FailureKind, + FixedBackoff, + ProcessExecutor, + RetryPolicy, + TaskStatus, + Timeout, + run_task, +) +from chongming_task.stores.sqlite import SqliteTaskStore + +_WINDOW = 20.0 + + +def _add(ctx: Any, left: int, right: int) -> int: + """两数相加。""" + del ctx + return left + right + + +def _boom(ctx: Any) -> None: + """总是失败：用来让任务反复重试。""" + del ctx + raise RuntimeError("总是失败") + + +def _deaf_loop(ctx: Any) -> str: + """**不**调用 checkpoint：停不掉，只能强杀。""" + del ctx + end = time.monotonic() + 120 + while time.monotonic() < end: + pass + return "never" + + +def test_a_task_run_records_the_child_pid_in_its_snapshot() -> None: + outcome = run_task( + _add, + 2, + 40, + task_id="proc-1", + executor=ProcessExecutor(), + timeout=Timeout(per_attempt=_WINDOW), + ) + + assert outcome.ok, outcome.failure + assert outcome.value == 42 + snapshot = outcome.snapshot + assert snapshot.executor == "process" + assert snapshot.pid is not None + assert snapshot.pid != os.getpid(), "任务跑在另一个进程里" + + +def test_an_idle_task_is_actually_killed_by_the_timeout() -> None: + """线程执行器下这条只能"放弃等待"并落 `LOST`（线程杀不掉）；进程执行器**真的**把它停掉。 + + 超时仍然如实记成超时（不是"已停止"）——但载体确实没了，资源也确实释放了。 + """ + started = time.monotonic() + outcome = run_task( + _deaf_loop, + task_id="proc-deaf", + executor=ProcessExecutor(), + timeout=Timeout(idle=1), + retry=RetryPolicy(max_attempts=1), + graceful_stop=0.5, + ) + elapsed = time.monotonic() - started + + assert outcome.status is TaskStatus.FAILED + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.TIMEOUT_IDLE + assert outcome.snapshot.pid is not None + + # 它自己那个循环要跑两分钟，这里几秒就回来了——只有"真的被停掉"解释得通。 + assert elapsed < _WINDOW + # 载体不在了：那个 pid 已经不再是本进程活着的子进程。 + alive = {child.pid for child in multiprocessing.active_children()} + assert outcome.snapshot.pid not in alive, "子进程必须真的被终止并回收" + + +def test_a_controller_can_run_tasks_in_child_processes(tmp_path: Path) -> None: + store = SqliteTaskStore(tmp_path / "proc.db") + controller = Controller(store=store, executor=ProcessExecutor(), owner="node-a") + try: + controller.register("add", _add) + controller.submit("add", 20, 22, task_id="t1") + outcome = controller.wait("t1", timeout=_WINDOW) + + assert outcome.ok and outcome.value == 42 + assert outcome.snapshot.executor == "process" + + # 落库之后仍然对得上：pid 是 store 契约里的字段，不是内存里的临时值。 + stored = store.get("t1") + assert stored is not None + assert stored.pid == outcome.snapshot.pid + assert stored.executor == "process" + finally: + controller.close(timeout=_WINDOW, close_store=True) + + +def test_a_target_that_cannot_cross_the_boundary_fails_fast() -> None: + """不可 pickle 的目标：任务落 ``failed`` 并带上诊断，而不是让调用方干等。 + + （直接调 ``ProcessExecutor.launch()`` 会当场抛 ``ConfigurationError``；这条测的是 + 走 ``run_task`` 那条路——异常在那里必须变成任务的结论。） + """ + + def local_closure(ctx: Any) -> str: # 局部函数：子进程里找不到它 + del ctx + return "nope" + + started = time.monotonic() + outcome = run_task(local_closure, task_id="bad-1", executor=ProcessExecutor()) + elapsed = time.monotonic() - started + + assert outcome.status is TaskStatus.FAILED + assert outcome.failure is not None + assert "可 pickle" in outcome.failure.message + assert outcome.failure.retryable is False + assert elapsed < _WINDOW, "它必须立刻有结论，而不是等到天荒地老" + + +def test_a_retry_does_not_carry_the_previous_attempts_pid() -> None: + """重试之间采一次快照：上一次的进程号必须已经清掉。 + + 留着它的危害很具体：运维看到一个 pid 就去 `ps`，而对上的可能是一个早在别处跑着的 + 无关进程——比"没有 pid"更容易误导。 + """ + controller = Controller(executor=ProcessExecutor()) + controller.register("boom", _boom) + try: + controller.submit( + "boom", + task_id="t-retry", + retry=RetryPolicy(max_attempts=3, backoff=FixedBackoff(2.0)), + ) + deadline = time.monotonic() + _WINDOW + while time.monotonic() < deadline: + snapshot = controller.get("t-retry") + if snapshot is not None and snapshot.status is TaskStatus.RETRY_WAIT: + break + time.sleep(0.02) + else: # pragma: no cover - 没等到就说明退避策略没生效 + raise AssertionError("任务没有进入 retry_wait") + + assert snapshot is not None + assert snapshot.pid is None, "上一次尝试的进程号必须清掉" + assert snapshot.executor == "process", "执行器是这一轮的配置，不随尝试变化" + finally: + controller.close(timeout=_WINDOW) diff --git a/tests/integration/test_reconcile.py b/tests/integration/test_reconcile.py new file mode 100644 index 0000000..5859d11 --- /dev/null +++ b/tests/integration/test_reconcile.py @@ -0,0 +1,404 @@ +"""重启恢复（``reconcile``）测试。 + +这一块最容易出错的地方是**误判**：把一个活得好好的任务判成丢失。所以用例的重点不是 +"能标记僵尸"，而是"**不**标记活着的任务"： + +* 有监督者（租约新鲜）→ 一动不动； +* 长时间不调 ``checkpoint()`` 的健康任务 → 同样一动不动（心跳停滞 ≠ 进程死了）； +* 还在排队的任务 → 保住状态，只报告； +* 原来的监督者又活过来并抢先写入 → 放手，绝不覆盖。 +""" + +from __future__ import annotations + +import tempfile +import time +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import Any + +from chongming_task.controller import ( + Controller, + ReconcileOutcome, + ReconcileReport, +) +from chongming_task.errors import ConfigurationError +from chongming_task.models import FailureKind, TaskSnapshot +from chongming_task.states import TaskStatus +from chongming_task.stores.memory import InMemoryTaskStore +from chongming_task.stores.sqlite import SqliteTaskStore + +_WINDOW = 10.0 +_OLD = datetime(2026, 1, 1, tzinfo=UTC) + + +def _wait_until(predicate: Any, timeout: float = _WINDOW) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +def _orphan( + task_id: str, + status: TaskStatus, + *, + lease_until: datetime | None = None, + lease_owner: str | None = None, + started_at: datetime | None = _OLD, +) -> TaskSnapshot: + """构造一个"上次进程留下的"任务行。""" + return TaskSnapshot( + task_id=task_id, + name="遗留任务", + status=status, + status_version=3, + attempt=1, + max_attempts=1, + created_at=_OLD, + started_at=started_at, + last_heartbeat_at=_OLD, + lease_owner=lease_owner, + lease_until=lease_until, + ) + + +# --------------------------------------------------------------------------- +# 基本语义 +# --------------------------------------------------------------------------- + + +def test_reconcile_requires_a_store() -> None: + """没有共享存储就无从判断谁还在管——如实报错，不要猜。""" + controller = Controller() + try: + controller.reconcile() + except ConfigurationError as exc: + assert "共享存储" in str(exc) + return + raise AssertionError("没有存储时 reconcile 应当报错") + + +def test_stale_running_task_is_marked_lost() -> None: + """占着执行资源却没有监督者：载体已随进程消失。""" + store = InMemoryTaskStore() + store.create(_orphan("zombie", TaskStatus.RUNNING)) + controller = Controller(store=store) + + report = controller.reconcile() + assert [item.task_id for item in report.lost] == ["zombie"] + + restored = store.get("zombie") + assert restored is not None + assert restored.status is TaskStatus.LOST + assert restored.failure is not None + assert restored.failure.kind is FailureKind.LOST + assert "失联" in restored.failure.message + assert restored.ended_at is not None, "终态必须带 ended_at" + assert restored.validate() == () + + +def test_reconcile_records_an_audit_trail() -> None: + """判丢失必须留下可追溯的迁移记录，而不是悄悄改一个字段。""" + store = InMemoryTaskStore() + store.create(_orphan("zombie", TaskStatus.RUNNING)) + controller = Controller(store=store) + controller.reconcile(reason="机房断电") + + history = store.history("zombie") + assert [item.to_status for item in history] == [TaskStatus.LOST] + assert history[0].from_status is TaskStatus.RUNNING + assert history[0].reason == "机房断电" + + +def test_all_resource_holding_states_are_reconciled() -> None: + """``starting`` / ``running`` / ``pausing`` / ``stopping`` 都占着载体。""" + store = InMemoryTaskStore() + statuses = ( + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.PAUSING, + TaskStatus.STOPPING, + ) + for index, status in enumerate(statuses): + store.create(_orphan(f"z{index}", status)) + controller = Controller(store=store) + + report = controller.reconcile() + assert len(report.lost) == len(statuses) + for index in range(len(statuses)): + restored = store.get(f"z{index}") + assert restored is not None + assert restored.status is TaskStatus.LOST + + +def test_queued_task_is_reported_but_keeps_its_status() -> None: + """不占资源的任务只是"没人接手"，不是"丢了"——改状态会把它的排队事实抹掉。""" + store = InMemoryTaskStore() + store.create(_orphan("waiting", TaskStatus.QUEUED)) + controller = Controller(store=store) + + report = controller.reconcile() + assert report.lost == () + assert [item.task_id for item in report.adoptable] == ["waiting"] + assert report.adoptable[0].previous_status is TaskStatus.QUEUED + + restored = store.get("waiting") + assert restored is not None + assert restored.status is TaskStatus.QUEUED, "不该被改动" + + +def test_paused_task_is_adoptable_not_lost() -> None: + store = InMemoryTaskStore() + store.create(_orphan("paused-task", TaskStatus.PAUSED)) + controller = Controller(store=store) + report = controller.reconcile() + assert report.lost == () + assert len(report.adoptable) == 1 + + +# --------------------------------------------------------------------------- +# 不该动的东西（这部分的用例更重要） +# --------------------------------------------------------------------------- + + +def test_a_task_with_a_fresh_lease_is_never_touched() -> None: + """有活着的监督者就必须放手。""" + store = InMemoryTaskStore() + store.create( + _orphan( + "alive", + TaskStatus.RUNNING, + lease_owner="worker-1", + lease_until=datetime.now(UTC) + timedelta(seconds=60), + ) + ) + controller = Controller(store=store) + + report = controller.reconcile() + assert report.entries == () + restored = store.get("alive") + assert restored is not None + assert restored.status is TaskStatus.RUNNING + + +def test_a_healthy_task_that_never_checkpoints_is_not_killed() -> None: + """**这是本模块最重要的一条。** + + 心跳由任务自己的 ``checkpoint()`` 驱动。一个长时间不调 checkpoint 的健康任务 + （纯 CPU 计算）心跳同样会停滞；若拿心跳判断"进程是否还活着"，它会被误判失联并 + 判 ``LOST``——把一个正常任务杀掉。 + + 判据必须是租约。这里心跳很旧、但租约新鲜，所以一动不动。 + """ + store = InMemoryTaskStore() + snapshot = _orphan( + "cpu-bound", + TaskStatus.RUNNING, + lease_owner="worker-1", + lease_until=datetime.now(UTC) + timedelta(seconds=60), + ) + assert snapshot.last_heartbeat_at == _OLD, "前提：心跳确实很旧" + store.create(snapshot) + controller = Controller(store=store) + + assert controller.reconcile().entries == () + restored = store.get("cpu-bound") + assert restored is not None + assert restored.status is TaskStatus.RUNNING + + +def test_a_just_started_task_gets_a_grace_period() -> None: + """刚提交、工作线程还没轮到第一次续租：不能立刻判它失联。""" + store = InMemoryTaskStore() + store.create(_orphan("fresh", TaskStatus.RUNNING, started_at=datetime.now(UTC))) + controller = Controller(store=store) + + report = controller.reconcile() + assert report.entries == (), "宽限期内不该动它" + + +def test_a_just_started_task_becomes_reconcilable_after_the_grace_period() -> None: + store = InMemoryTaskStore() + store.create(_orphan("fresh", TaskStatus.RUNNING, started_at=datetime.now(UTC))) + controller = Controller(store=store) + + report = controller.reconcile(stale_after=0.0) + assert [item.task_id for item in report.lost] == ["fresh"] + + +def test_tasks_owned_by_this_controller_are_skipped() -> None: + """自己正在跑的任务当然活着——不靠租约也该放过它。""" + store = InMemoryTaskStore() + controller = Controller(store=store, command_poll_interval=0.05) + controller.register("block", _block) + run = controller.submit("block", task_id="mine") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + report = controller.reconcile(stale_after=0.0) + assert report.lost == () + assert run.status is TaskStatus.RUNNING + + controller.stop("mine") + controller.wait_all(timeout=_WINDOW) + controller.close(timeout=_WINDOW) + + +def test_a_terminal_task_is_not_examined() -> None: + store = InMemoryTaskStore() + store.create(_orphan("done", TaskStatus.SUCCEEDED)) + controller = Controller(store=store) + assert controller.reconcile().entries == () + + +def test_reconcile_lets_go_when_someone_writes_first() -> None: + """原本的监督者又活过来并抢先写入 → 必须放手，强行覆盖会抹掉对方的状态。""" + store = InMemoryTaskStore() + store.create(_orphan("racy", TaskStatus.RUNNING, lease_owner="worker-1")) + + # 模拟"对方在我们读取之后、写入之前提交了一次迁移"。 + original_query = store.query + + def query_with_interference(query: Any = None) -> list[TaskSnapshot]: + found = original_query(query) + current = store.get("racy") + if current is not None and current.status is TaskStatus.RUNNING: + store.save( + TaskSnapshot( + task_id="racy", + name="遗留任务", + status=TaskStatus.SUCCEEDED, + status_version=current.status_version + 1, + attempt=1, + max_attempts=1, + created_at=_OLD, + started_at=_OLD, + ended_at=datetime.now(UTC), + ), + expect_version=current.status_version, + ) + return found + + store.query = query_with_interference # type: ignore[method-assign] + controller = Controller(store=store) + report = controller.reconcile() + + assert [item.outcome for item in report.skipped] == [ReconcileOutcome.SKIPPED] + restored = store.get("racy") + assert restored is not None + assert restored.status is TaskStatus.SUCCEEDED, "对方的状态被覆盖了" + + +# --------------------------------------------------------------------------- +# 预演与报告 +# --------------------------------------------------------------------------- + + +def test_dry_run_reports_without_writing() -> None: + store = InMemoryTaskStore() + store.create(_orphan("zombie", TaskStatus.RUNNING)) + controller = Controller(store=store) + + report = controller.reconcile(dry_run=True) + assert report.dry_run is True + assert [item.task_id for item in report.lost] == ["zombie"] + + restored = store.get("zombie") + assert restored is not None + assert restored.status is TaskStatus.RUNNING, "预演不该写任何状态" + assert store.history("zombie") == [] + + +def test_report_properties_partition_the_entries() -> None: + store = InMemoryTaskStore() + store.create(_orphan("zombie", TaskStatus.RUNNING)) + store.create(_orphan("waiting", TaskStatus.QUEUED)) + controller = Controller(store=store) + + report = controller.reconcile() + assert len(report.lost) == 1 + assert len(report.adoptable) == 1 + assert report.skipped == () + assert bool(report) is True + assert ReconcileReport().lost == () + assert bool(ReconcileReport()) is False + + +def test_reconcile_is_safe_to_run_twice() -> None: + """第二次已经没有僵尸了（第一个已落终态）。""" + store = InMemoryTaskStore() + store.create(_orphan("zombie", TaskStatus.RUNNING)) + controller = Controller(store=store) + + assert len(controller.reconcile().lost) == 1 + assert controller.reconcile().entries == () + + +# --------------------------------------------------------------------------- +# 与真实进程配合 +# --------------------------------------------------------------------------- + + +def test_leases_are_renewed_while_a_task_runs() -> None: + """监督者在跑就必须持续续租，否则它自己会被别人判成僵尸。""" + with tempfile.TemporaryDirectory(prefix="chongming-reconcile-") as directory: + store = SqliteTaskStore(Path(directory) / "tasks.db") + controller = Controller( + store=store, + command_poll_interval=0.05, + lease_ttl=30.0, + owner="worker-1", + ) + controller.register("block", _block) + run = controller.submit("block", task_id="t1") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + assert _wait_until( + lambda: (store.get("t1") or TaskSnapshot("x", "x")).lease_owner == "worker-1" + ), "监督者没有续租" + leased = store.get("t1") + assert leased is not None + assert leased.lease_until is not None + assert leased.lease_until > datetime.now(UTC) + + controller.stop("t1") + controller.wait_all(timeout=_WINDOW) + controller.close(timeout=_WINDOW, close_store=True) + + +def test_a_reconcile_run_after_a_crash_cleans_up(tmp_path: Path) -> None: + """端到端：worker 崩了（连接关闭、租约过期），新进程启动时判它丢失。""" + path = tmp_path / "tasks.db" + store = SqliteTaskStore(path) + store.create( + _orphan( + "crashed", + TaskStatus.RUNNING, + lease_owner="worker-1", + lease_until=datetime.now(UTC) - timedelta(seconds=5), + ) + ) + store.close() + + # 新进程：另一个连接，同一个文件。 + reopened = SqliteTaskStore(path) + try: + controller = Controller(store=reopened, owner="worker-2") + report = controller.reconcile() + assert [item.task_id for item in report.lost] == ["crashed"] + + restored = reopened.get("crashed") + assert restored is not None + assert restored.status is TaskStatus.LOST + assert controller.persist_failures == () + finally: + reopened.close() + + +def _block(ctx: Any) -> str: + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "done" diff --git a/tests/integration/test_store_wiring.py b/tests/integration/test_store_wiring.py new file mode 100644 index 0000000..adff99e --- /dev/null +++ b/tests/integration/test_store_wiring.py @@ -0,0 +1,455 @@ +"""存储接线测试：写穿、读穿、跨进程指令。 + +这里用**两个 ``SqliteTaskStore`` 连接同一个数据库文件**来模拟两个进程。这不是 +过度讲究：跨进程真正难的部分就是"两个连接看不到彼此的内存"，用同一个 store 对象 +测是测不出来的。 + +用例的重点是那些**会静默出错**的地方：写穿漏一次、指令被错误的进程吃掉、指令被 +错误地标记完成——它们都不会抛异常，只会让看板上的状态悄悄变成错的。 +""" + +from __future__ import annotations + +import tempfile +import time +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from chongming_task.controller import Controller +from chongming_task.errors import StoreError +from chongming_task.models import TaskSnapshot +from chongming_task.states import TaskStatus +from chongming_task.stores.memory import InMemoryTaskStore +from chongming_task.stores.protocol import Command, CommandKind +from chongming_task.stores.sqlite import SqliteTaskStore + +_WINDOW = 10.0 + + +def _wait_until(predicate: Any, timeout: float = _WINDOW) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +# --- 任务（模块级，便于复用） --- + + +def slow_loop(ctx: Any) -> str: + """持续小步推进直到被停止。""" + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "done" + + +def quick_add(ctx: Any, left: int, right: int) -> int: + del ctx + return left + right + + +def _shared_store(tmp: Path) -> tuple[SqliteTaskStore, SqliteTaskStore]: + """同一个数据库文件上的两个独立连接 = 两个"进程"。""" + return ( + SqliteTaskStore(tmp / "tasks.db"), + SqliteTaskStore(tmp / "tasks.db"), + ) + + +# --------------------------------------------------------------------------- +# 写穿 +# --------------------------------------------------------------------------- + + +def test_initial_snapshot_is_persisted_before_any_transition() -> None: + """必须先建行：之后每次迁移都是按版本更新的 UPDATE。 + + 顺序如果反了，第一次 ``save`` 会因"任务不存在"而失败。但注意：写穿失败是被 + 捕获并记进 ``persist_failures`` 的（存储故障不该让任务失败），所以**光断言行 + 存在是不够的**——必须同时断言没有任何写穿失败，否则这条用例会在实现有 bug 时 + 依然通过。 + """ + store = InMemoryTaskStore() + controller = Controller(store=store, max_parallel=1) + controller.register("block", slow_loop) + + controller.submit("block", task_id="t1", label="占住额度") + assert _wait_until( + lambda: (store.get("t1") or TaskSnapshot("x", "x")).status is TaskStatus.RUNNING + ) + + # 额度已被占满，这个任务停在 queued。 + controller.submit("block", task_id="t2", label="排队中") + persisted = store.get("t2") + assert persisted is not None + assert persisted.status is TaskStatus.QUEUED + assert persisted.name == "排队中" + assert controller.persist_failures == (), "建行与更新的顺序不对，写穿失败了" + + controller.stop_all() + controller.wait_all(timeout=_WINDOW) + assert store.get("t2").status is TaskStatus.STOPPED # type: ignore[union-attr] + + +def test_every_transition_is_written_through_and_is_auditable() -> None: + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("add", quick_add) + controller.submit("add", 2, 3, task_id="t1") + controller.wait_all(timeout=_WINDOW) + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.status is TaskStatus.SUCCEEDED + assert snapshot.outcome == {"value": 5} + assert snapshot.validate() == () + + history = store.history("t1") + assert [item.to_status for item in history] == [ + TaskStatus.QUEUED, + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.SUCCEEDED, + ] + assert [item.version for item in history] == [1, 2, 3, 4] + + +def test_bucket_and_key_survive_the_write_through() -> None: + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("add", quick_add) + controller.submit("add", 1, 1, task_id="t1", bucket_key="tenant-42") + controller.wait_all(timeout=_WINDOW) + + snapshot = store.get("t1") + assert snapshot is not None + assert snapshot.bucket == controller.bucket_policy.bucket_for("tenant-42") + assert snapshot.bucket_key == "tenant-42" + + +def test_progress_and_heartbeat_reach_the_store_without_bumping_the_version() -> None: + """进度走高频旁路：它若参与乐观并发，会与状态写入持续冲突。""" + store = InMemoryTaskStore() + controller = Controller(store=store, command_poll_interval=0.05) + controller.register("block", slow_loop) + controller.submit("block", task_id="t1") + + assert _wait_until( + lambda: (store.get("t1") or TaskSnapshot("x", "x")).status is TaskStatus.RUNNING + ) + before = store.get("t1") + assert before is not None + version = before.status_version + assert _wait_until( + lambda: (store.get("t1").last_heartbeat_at if store.get("t1") else None) is not None, + timeout=3.0, + ) + + after = store.get("t1") + assert after is not None + assert after.status_version == version, "心跳回写不应改动版本号" + + controller.stop("t1") + controller.wait_all(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 读穿 +# --------------------------------------------------------------------------- + + +def test_a_second_controller_sees_tasks_it_does_not_own() -> None: + """这就是存储接线存在的全部理由：跨进程可见。""" + with tempfile.TemporaryDirectory(prefix="chongming-wire-") as directory: + writer_store, reader_store = _shared_store(Path(directory)) + try: + writer = Controller(store=writer_store) + writer.register("add", quick_add) + writer.submit("add", 1, 1, task_id="t1", label="跨进程可见") + writer.wait_all(timeout=_WINDOW) + + reader = Controller(store=reader_store) + assert reader.runs() == (), "读方没有本地句柄" + + snapshot = reader.get("t1") + assert snapshot is not None + assert snapshot.name == "跨进程可见" + assert snapshot.status is TaskStatus.SUCCEEDED + + assert [s.task_id for s in reader.list_tasks()] == ["t1"] + assert reader.stats().of(TaskStatus.SUCCEEDED) == 1 + writer.close(timeout=_WINDOW) + finally: + writer_store.close() + reader_store.close() + + +def test_reader_filters_are_pushed_down_to_the_store() -> None: + with tempfile.TemporaryDirectory(prefix="chongming-wire-") as directory: + writer_store, reader_store = _shared_store(Path(directory)) + try: + writer = Controller(store=writer_store) + writer.register("add", quick_add) + writer.submit("add", 1, 1, task_id="a1", label="抽取-甲", bucket_key="tenant-A") + writer.submit("add", 1, 1, task_id="a2", label="加载", bucket_key="tenant-B") + writer.wait_all(timeout=_WINDOW) + + reader = Controller(store=reader_store) + assert [s.task_id for s in reader.list_tasks(name_prefix="抽取")] == ["a1"] + assert [ + s.task_id + for s in reader.list_tasks(bucket=writer.bucket_policy.bucket_for("tenant-B")) + ] == ["a2"] + assert reader.list_tasks(status=TaskStatus.FAILED) == [] + writer.close(timeout=_WINDOW) + finally: + writer_store.close() + reader_store.close() + + +# --------------------------------------------------------------------------- +# 跨进程控制 +# --------------------------------------------------------------------------- + + +def test_stop_crosses_the_process_boundary_via_a_command() -> None: + """一个进程无法给另一个进程的执行载体发信号，只能写指令让持有者取走。""" + with tempfile.TemporaryDirectory(prefix="chongming-wire-") as directory: + owner_store, client_store = _shared_store(Path(directory)) + try: + owner = Controller(store=owner_store, command_poll_interval=0.05, owner="owner-process") + owner.register("block", slow_loop) + owner.submit("block", task_id="t1", label="长任务") + + assert _wait_until( + lambda: ( + (owner_store.get("t1") or TaskSnapshot("x", "x")).status is TaskStatus.RUNNING + ) + ) + + client = Controller(store=client_store, owner="ops-client") + assert client.stop("t1", reason="跨进程停止") is True + assert client.runs() == (), "客户端并不持有这个任务" + + assert _wait_until( + lambda: ( + (owner_store.get("t1") or TaskSnapshot("x", "x")).status is TaskStatus.STOPPED + ) + ), "持有者没有取走指令" + + final = client.get("t1") + assert final is not None + assert final.status is TaskStatus.STOPPED + assert final.failure is not None + + owner.close(timeout=_WINDOW) + finally: + owner_store.close() + client_store.close() + + +def test_command_records_who_requested_it() -> None: + store = InMemoryTaskStore() + owner = Controller(store=store, command_poll_interval=0.05, owner="owner-1") + owner.register("block", slow_loop) + run = owner.submit("block", task_id="t1") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + client = Controller(store=store, owner="ops@laptop") + client.stop("t1", reason="发布窗口") + + assert _wait_until(lambda: run.status is TaskStatus.STOPPED) + owner.close(timeout=_WINDOW) + + +def test_pause_of_a_remote_task_reports_unconfirmed() -> None: + """跨进程没有回执通道，因此 `pause` 只能如实返回 False（未确认）。""" + store = InMemoryTaskStore() + owner = Controller(store=store, command_poll_interval=0.05) + owner.register("block", slow_loop) + run = owner.submit("block", task_id="t1") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + client = Controller(store=store) + assert client.pause("t1") is False + assert _wait_until(lambda: run.status is TaskStatus.PAUSED), "指令没有被执行" + + owner.close(timeout=_WINDOW) + + +def test_a_client_that_never_submitted_does_not_steal_commands() -> None: + """纯查询客户端若也去取指令，会把别人的停止请求抢走。 + + 即便它把指令交还，持续轮询也会让真正的持有者反复错过——所以轮询线程只在 + "提交过任务"之后才启动。 + """ + store = InMemoryTaskStore() + owner = Controller(store=store, command_poll_interval=0.05) + owner.register("block", slow_loop) + run = owner.submit("block", task_id="t1") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + # 一个只读客户端：只查询，不提交。 + client = Controller(store=store) + client.list_tasks() + client.get("t1") + client.stats() + assert client._worker is None, "没有提交过任务的实例不该启动指令轮询" + + owner.stop("t1") + assert _wait_until(lambda: run.status is TaskStatus.STOPPED) + owner.close(timeout=_WINDOW) + + +def test_a_command_for_a_task_we_do_not_own_is_released_not_completed() -> None: + """取到不属于自己的指令必须**交还**，否则真正的持有者永远收不到它。 + + 真实场景：另一个进程提交的任务，它的停止指令被**本**进程的轮询线程取到了—— + 本进程没有那个任务的本地句柄。所以这里直接往存储里写"别人的任务"和它的指令， + 而不是走 ``Controller.request``（后者会先校验任务存在，见下一条用例）。 + """ + store = InMemoryTaskStore() + controller = Controller(store=store, command_poll_interval=0.05) + controller.register("add", quick_add) + # 先提交一个自己的任务，让轮询线程启动起来。 + controller.submit("add", 1, 1, task_id="mine") + controller.wait_all(timeout=_WINDOW) + + store.create( + TaskSnapshot( + task_id="someone-elses", + name="别人的任务", + status=TaskStatus.RUNNING, + status_version=1, + attempt=1, + max_attempts=1, + ) + ) + store.submit_command( + Command( + command_id="c-foreign", + task_id="someone-elses", + kind=CommandKind.STOP, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + requested_by="other-process", + ) + ) + + def released() -> bool: + found = [item for item in store.commands() if item.command_id == "c-foreign"] + return bool(found) and found[0].is_open and found[0].lease_until is None + + assert _wait_until(released), "指令没有被交还，真正的持有者将永远收不到它" + controller.close(timeout=_WINDOW) + + +def test_request_refuses_a_task_that_does_not_exist() -> None: + """拼错的 task_id 不该写出一条**永远不会有人执行**的指令。 + + 没有这道校验时，``stop typo`` 会"成功"，于是存储里留下一条孤儿指令，而调用方 + 还以为停止已经下发。 + """ + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("add", quick_add) + controller.submit("add", 1, 1, task_id="mine") + controller.wait_all(timeout=_WINDOW) + + assert controller.request("typo-task-id", CommandKind.STOP) is False + assert store.commands() == [] + controller.close(timeout=_WINDOW) + + +def test_request_without_a_store_is_rejected() -> None: + """没有共享媒介就不可能跨进程控制，如实返回 False 而不是假装成功。""" + controller = Controller() + assert controller.request("t1", CommandKind.STOP) is False + + +# --------------------------------------------------------------------------- +# 存储故障不该让任务失败 +# --------------------------------------------------------------------------- + + +class _FlakyStore(InMemoryTaskStore): + """``save`` 总是失败，用来验证存储故障不会打断任务。""" + + def save(self, snapshot: TaskSnapshot, *, expect_version: int) -> None: + raise StoreError("模拟磁盘故障") + + +def test_store_failure_does_not_fail_the_task_but_is_recorded() -> None: + """任务本身没问题，坏的只是可观测性。 + + 但也绝不能静默——否则"看板上的状态是旧的"这件事毫无线索。 + """ + controller = Controller(store=_FlakyStore()) + controller.register("add", quick_add) + run = controller.submit("add", 2, 3, task_id="t1") + outcome = controller.wait("t1", timeout=_WINDOW) + + assert outcome.ok, "存储故障不该让任务失败" + assert outcome.value == 5 + assert run.status is TaskStatus.SUCCEEDED + + failures = controller.persist_failures + assert failures, "写穿失败必须留痕" + assert "StoreError" in failures[0] + assert "t1" in failures[0] + + +def test_persist_failures_are_bounded() -> None: + controller = Controller(store=_FlakyStore()) + controller.register("add", quick_add) + for index in range(30): + controller.submit("add", index, 1, task_id=f"t{index}") + controller.wait_all(timeout=_WINDOW) + + assert len(controller.persist_failures) <= 20 + + +# --------------------------------------------------------------------------- +# 与分桶、并发额度的协同 +# --------------------------------------------------------------------------- + + +def test_bucket_counts_read_through_the_store() -> None: + with tempfile.TemporaryDirectory(prefix="chongming-wire-") as directory: + writer_store, reader_store = _shared_store(Path(directory)) + try: + writer = Controller(store=writer_store) + writer.register("add", quick_add) + writer.submit("add", 1, 1, task_id="a1", bucket_key="tenant-A") + writer.submit("add", 1, 1, task_id="a2", bucket_key="tenant-A") + writer.submit("add", 1, 1, task_id="b1") + writer.wait_all(timeout=_WINDOW) + + reader = Controller(store=reader_store) + counts = reader.bucket_counts() + assert counts[writer.bucket_policy.bucket_for("tenant-A")] == 2 + assert counts[None] == 1 + writer.close(timeout=_WINDOW) + finally: + writer_store.close() + reader_store.close() + + +def test_local_concurrency_limit_still_applies_with_a_store() -> None: + """接了存储也不能绕过本地的额度：排队中的任务在存储里应当是 queued。""" + store = InMemoryTaskStore() + controller = Controller(store=store, max_parallel=1) + controller.register("block", slow_loop) + + first = controller.submit("block", task_id="t1") + assert _wait_until(lambda: first.status is TaskStatus.RUNNING) + second = controller.submit("block", task_id="t2") + assert second.status is TaskStatus.QUEUED + assert store.get("t2").status is TaskStatus.QUEUED # type: ignore[union-attr] + + controller.stop_all() + controller.wait_all(timeout=_WINDOW) + assert store.get("t2").status is TaskStatus.STOPPED # type: ignore[union-attr] diff --git a/tests/property/test_state_machine_properties.py b/tests/property/test_state_machine_properties.py new file mode 100644 index 0000000..a47a8b2 --- /dev/null +++ b/tests/property/test_state_machine_properties.py @@ -0,0 +1,118 @@ +"""用 hypothesis 随机生成事件序列，验证状态机的核心不变量。 + +比手写用例更强的地方：它能生成人手想不到的组合，例如 +"暂停中收到停止"、"重试等待中被重试"、"重试预算耗尽后又被派发"这类长尾序列。 + +被验证的不变量 +-------------- +1. 被拒绝的动作**完全不产生副作用**（快照对象不变、历史不增长）。 +2. 每次被接受的动作使 ``status_version`` 恰好 +1，并使历史恰好 +1 条。 +3. 除显式审计的复活路径外，实际发生的转移必在转移表内。 +4. 终态不可逃逸。 +5. 每次被接受的动作之后，快照都通过 :meth:`TaskSnapshot.validate` 的跨字段一致性检查。 +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st + +from chongming_task.core.state_machine import StateMachine, Trigger +from chongming_task.errors import AttemptsExhausted, InvalidTransition +from chongming_task.models import Failure, FailureKind +from chongming_task.states import TERMINAL, TRANSITIONS +from chongming_task.testing import FakeClock + +if TYPE_CHECKING: + from collections.abc import Callable + +_FAILURE = Failure(kind=FailureKind.EXCEPTION, message="boom") + +#: 刻意绕过转移表的触发源，只有这三条审计路径合法。 +_NON_TABLE_TRIGGERS = frozenset( + { + Trigger.RETRY_REQUESTED, + Trigger.RESET_REQUESTED, + Trigger.ADMIN_OVERRIDE, + } +) + +#: 随机动作集合。 +#: +#: 刻意不含 ``force_transition``：它按定义可跳到任意状态，与"随机事件序列"这一 +#: 模型正交，其行为由 tests/unit/test_state_machine.py 的定向用例覆盖。 +ACTIONS: tuple[tuple[str, Callable[[StateMachine], object]], ...] = ( + ("mark_blocked", lambda m: m.mark_blocked()), + ("mark_queued", lambda m: m.mark_queued()), + ("mark_starting", lambda m: m.mark_starting(executor="inline")), + ("mark_running", lambda m: m.mark_running()), + ("mark_succeeded", lambda m: m.mark_succeeded()), + ("mark_failed", lambda m: m.mark_failed(_FAILURE)), + ("mark_retry_wait", lambda m: m.mark_retry_wait(_FAILURE)), + ("mark_retry_exhausted", lambda m: m.mark_retry_exhausted(_FAILURE)), + ("mark_skipped", lambda m: m.mark_skipped()), + ("mark_cancelled", lambda m: m.mark_cancelled()), + ("mark_lost", lambda m: m.mark_lost()), + ("request_stop", lambda m: m.request_stop()), + ("mark_stopped", lambda m: m.mark_stopped()), + ("request_pause", lambda m: m.request_pause()), + ("mark_paused", lambda m: m.mark_paused()), + ("abort_pause", lambda m: m.abort_pause()), + ("resume", lambda m: m.resume()), + ("retry", lambda m: m.retry()), + ("force_retry", lambda m: m.retry(force=True)), + ("reset", lambda m: m.reset()), +) + + +@settings( + max_examples=500, + deadline=None, + suppress_health_check=[HealthCheck.too_slow], +) +@given( + max_attempts=st.integers(min_value=1, max_value=4), + picks=st.lists(st.integers(min_value=0, max_value=len(ACTIONS) - 1), max_size=30), +) +def test_invariants_hold_for_any_action_sequence(max_attempts: int, picks: list[int]) -> None: + clock = FakeClock() + machine = StateMachine.create("t", "x", max_attempts=max_attempts, clock=clock) + + for index in picks: + name, action = ACTIONS[index] + clock.advance(1) + before = machine.snapshot + history_len = len(machine.history) + + try: + action(machine) + except (InvalidTransition, AttemptsExhausted): + # 不变量 1：被拒绝的动作必须完全无副作用。 + assert machine.snapshot is before, f"{name} 被拒绝却改动了快照" + assert len(machine.history) == history_len, f"{name} 被拒绝却写了历史" + continue + + after = machine.snapshot + record = machine.history[-1] + + # 不变量 2：版本号与历史同步严格 +1。 + assert len(machine.history) == history_len + 1, name + assert after.status_version == before.status_version + 1, name + + # 不变量 3：只有审计路径可以绕过转移表。 + if record.trigger not in _NON_TABLE_TRIGGERS: + assert after.status in TRANSITIONS[before.status], ( + f"{name} 绕过了转移表：{before.status} -> {after.status}" + ) + + # 不变量 4：终态不可逃逸——**复活除外**，后者必须走审计路径。 + # 终态出边为空集，因此这一条其实是上一条的推论；显式写出是为了失败时 + # 能一眼分清"逃逸"与"绕表"。 + if record.trigger not in _NON_TABLE_TRIGGERS: + assert before.status not in TERMINAL, f"{name} 从终态 {before.status} 逃逸" + + # 不变量 5：快照跨字段一致。 + problems = after.validate() + assert problems == (), f"{name} 产生不一致快照：{problems}" diff --git a/tests/unit/test_buckets.py b/tests/unit/test_buckets.py new file mode 100644 index 0000000..18b51df --- /dev/null +++ b/tests/unit/test_buckets.py @@ -0,0 +1,139 @@ +"""分桶策略测试。 + +最重要的一条是**跨进程稳定性**：内置 ``hash()`` 带进程级随机盐，若图省事用它， +同一个键在不同进程里会落到不同的桶，分布式下"同桶任务在同一台机器"这条约定会 +**静默**失效。这里用真实子进程与不同的 ``PYTHONHASHSEED`` 把它钉死。 +""" + +from __future__ import annotations + +import os +import subprocess +import sys + +import pytest + +from chongming_task.policies import BucketPolicy + + +def _run_with_hash_seed(script: str, seed: str) -> str: + """在指定 ``PYTHONHASHSEED`` 的子进程里跑一段脚本，返回其标准输出。""" + environment = {**os.environ, "PYTHONHASHSEED": seed} + completed = subprocess.run( + [sys.executable, "-c", script], + capture_output=True, + text=True, + env=environment, + check=True, + timeout=60, + ) + return completed.stdout.strip() + + +# --------------------------------------------------------------------------- +# 稳定性（分桶能用于分布式的全部前提） +# --------------------------------------------------------------------------- + + +def test_same_key_always_maps_to_the_same_bucket() -> None: + policy = BucketPolicy(64) + for _ in range(50): + assert policy.bucket_for("tenant-42") == policy.bucket_for("tenant-42") + + +def test_assignment_is_stable_across_processes_with_different_hash_seeds() -> None: + """内置 ``hash()`` 带随机盐，本实现必须不受它影响。""" + script = ( + "from chongming_task.policies import BucketPolicy;" + "print(' '.join(str(BucketPolicy(64).bucket_for(f'tenant-{i}')) for i in range(20)))" + ) + first = _run_with_hash_seed(script, "0") + second = _run_with_hash_seed(script, "1") + assert first == second, "同一个键在不同进程里落到了不同的桶" + + local = " ".join(str(BucketPolicy(64).bucket_for(f"tenant-{i}")) for i in range(20)) + assert first == local, "子进程与本进程的结果不一致" + + +def test_builtin_hash_would_have_been_unusable() -> None: + """留一条反例，说明为什么不能图省事用内置 ``hash()``。 + + 不是"可能不一样"，而是**确实不一样**——这就是那个坑本身。 + """ + script = "print(' '.join(str(hash(f'k{i}') % 64) for i in range(20)))" + assert _run_with_hash_seed(script, "0") != _run_with_hash_seed(script, "1") + + +# --------------------------------------------------------------------------- +# 取值范围与分布 +# --------------------------------------------------------------------------- + + +def test_bucket_is_always_within_range() -> None: + for num_buckets in (1, 2, 7, 64, 1000): + policy = BucketPolicy(num_buckets) + for index in range(200): + bucket = policy.bucket_for(f"key-{index}") + assert 0 <= bucket < num_buckets + + +def test_single_bucket_policy_puts_everything_in_bucket_zero() -> None: + policy = BucketPolicy(1) + assert all(policy.bucket_for(key) == 0 for key in ("a", "b", "c")) + + +def test_distribution_is_reasonably_uniform() -> None: + """分桶明显倾斜会让某些机器空转、另一些排队。 + + 这里只做一个粗检：64 个桶、4096 个键，每个桶的落点不应当偏离均值太多。 + """ + policy = BucketPolicy(64) + counts = [0] * 64 + for index in range(4096): + counts[policy.bucket_for(f"tenant-{index}")] += 1 + + expected = 4096 / 64 + assert all(count > 0 for count in counts), "有桶完全没分到键，分布明显异常" + assert max(counts) < expected * 3, f"分布过于集中：max={max(counts)}" + + +def test_similar_keys_do_not_cluster() -> None: + """``tenant-1`` 与 ``tenant-2`` 这类只差一个字符的键必须被打散。""" + policy = BucketPolicy(64) + buckets = {policy.bucket_for(f"tenant-{index}") for index in range(50)} + assert len(buckets) > 20, "相邻键挤在少数几个桶里，哈希的雪崩性不足" + + +def test_policy_validates_bucket_range() -> None: + policy = BucketPolicy(8) + assert policy.is_valid(0) + assert policy.is_valid(7) + assert not policy.is_valid(8) + assert not policy.is_valid(-1) + + +def test_policy_exposes_all_buckets() -> None: + assert list(BucketPolicy(4).buckets) == [0, 1, 2, 3] + + +# --------------------------------------------------------------------------- +# 参数校验 +# --------------------------------------------------------------------------- + + +def test_num_buckets_must_be_positive() -> None: + with pytest.raises(ValueError, match="num_buckets"): + BucketPolicy(0) + + +def test_empty_key_is_rejected() -> None: + """空键会让所有"没填租户"的任务挤进同一个桶。""" + with pytest.raises(ValueError, match="不能为空"): + BucketPolicy().bucket_for("") + + +def test_policy_is_hashable_and_comparable() -> None: + """策略会作为配置传递，值语义让它能被比较与当作字典键。""" + assert BucketPolicy(64) == BucketPolicy(64) + assert BucketPolicy(64) != BucketPolicy(128) + assert len({BucketPolicy(64), BucketPolicy(64)}) == 1 diff --git a/tests/unit/test_cancellation.py b/tests/unit/test_cancellation.py new file mode 100644 index 0000000..237eaac --- /dev/null +++ b/tests/unit/test_cancellation.py @@ -0,0 +1,281 @@ +"""协作式取消令牌的行为测试。 + +这里必须用**真实线程**：令牌的价值恰恰在于跨线程正确握手（任务线程阻塞在 +暂停点、调度线程请求停止并唤醒它）。用假的线程模型测不出竞态。 + +因此这些用例带小超时并断言线程确实退出——**同时也在验证没有死锁**。 +""" + +from __future__ import annotations + +import asyncio +import contextlib +import threading + +import pytest + +from chongming_task.errors import TaskCancelled +from chongming_task.runtime.cancellation import CancellationToken + +_JOIN_TIMEOUT = 5.0 + + +def _run_in_thread(target: object) -> threading.Thread: + thread = threading.Thread(target=target) # type: ignore[arg-type] + thread.daemon = True + thread.start() + return thread + + +# --------------------------------------------------------------------------- +# 停止 +# --------------------------------------------------------------------------- + + +def test_checkpoint_is_noop_without_any_request() -> None: + """没有请求时 checkpoint 必须立即返回，否则每轮循环都在付代价。""" + CancellationToken().checkpoint() + + +def test_checkpoint_raises_when_stop_requested() -> None: + token = CancellationToken() + token.request_stop("发布窗口") + with pytest.raises(TaskCancelled) as info: + token.checkpoint() + assert info.value.reason == "发布窗口" + + +def test_stop_is_idempotent_and_keeps_the_first_reason() -> None: + """首个原因通常最接近真实起因，后续请求不应覆盖它。""" + token = CancellationToken() + token.request_stop("first") + token.request_stop("second") + with pytest.raises(TaskCancelled) as info: + token.checkpoint() + assert info.value.reason == "first" + + +def test_raise_if_stopped_ignores_pause() -> None: + """用于清理路径：只响应停止，绝不阻塞。""" + token = CancellationToken() + token.request_pause() + token.raise_if_stopped() # 不应阻塞 + token.request_stop() + with pytest.raises(TaskCancelled): + token.raise_if_stopped() + + +def test_is_stop_requested_reflects_state() -> None: + token = CancellationToken() + assert not token.is_stop_requested + token.request_stop() + assert token.is_stop_requested + + +# --------------------------------------------------------------------------- +# 暂停 +# --------------------------------------------------------------------------- + + +def test_pause_blocks_until_resume() -> None: + token = CancellationToken() + token.request_pause() # 先请求，消除"任务先跑到检查点"的竞态 + observed: list[str] = [] + + def worker() -> None: + token.checkpoint() + observed.append("resumed") + + thread = _run_in_thread(worker) + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is True + assert observed == [], "暂停请求生效前任务不应继续执行" + + token.resume() + thread.join(_JOIN_TIMEOUT) + assert not thread.is_alive(), "resume() 必须唤醒阻塞中的任务" + assert observed == ["resumed"] + + +def test_stop_wakes_a_paused_task() -> None: + """暂停中的任务必须能被停止，否则"暂停"就成了不可撤销的挂起。""" + token = CancellationToken() + token.request_pause() + reasons: list[str | None] = [] + + def worker() -> None: + try: + token.checkpoint() + except TaskCancelled as exc: + reasons.append(exc.reason) + + thread = _run_in_thread(worker) + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is True + + token.request_stop("运维介入") + thread.join(_JOIN_TIMEOUT) + assert not thread.is_alive(), "停止请求必须能唤醒暂停点" + assert reasons == ["运维介入"] + + +def test_repeated_pause_does_not_clear_acknowledgement() -> None: + """重复请求暂停不应把"已确认"重置掉，否则 wait_paused 会误判为未生效。""" + token = CancellationToken() + token.request_pause() + + def worker() -> None: + token.checkpoint() + + thread = _run_in_thread(worker) + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is True + token.request_pause() + assert token.is_pause_acknowledged is True + + token.resume() + thread.join(_JOIN_TIMEOUT) + + +def test_wait_paused_times_out_when_task_never_checkpoints() -> None: + """调度器必须能区分"已暂停"与"暂停没生效"——后者要回滚为 RUNNING 并告警。""" + token = CancellationToken() + token.request_pause() + assert token.wait_paused(timeout=0.05) is False + assert token.is_pause_acknowledged is False + + +def test_wait_paused_reports_false_when_stopped_instead() -> None: + """请求暂停后又被停止：这不是"暂停成功"，必须返回 False。""" + token = CancellationToken() + token.request_pause() + token.request_stop() + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is False + + +def test_resume_without_pause_is_noop() -> None: + token = CancellationToken() + token.resume() + assert not token.is_pause_requested + token.checkpoint() # 仍应立即返回 + + +def _start_pausing_worker(token: CancellationToken) -> threading.Event: + """起一个持续调用 ``checkpoint()`` 的工作线程，返回"完成过一轮"的事件。 + + 线程必须**持续**调用 checkpoint——只调一次就退出的话，之后的暂停请求自然没人 + 确认，那是测试写错而不是缺陷。 + """ + + def worker() -> None: + # 收尾时的 stop 会让 checkpoint 抛 TaskCancelled：那正是预期行为， + # 但线程里未捕获的异常会被 pytest 转成错误。 + with contextlib.suppress(TaskCancelled): + while True: + token.checkpoint() + ran.set() + + ran = threading.Event() + threading.Thread(target=worker, daemon=True).start() + return ran + + +def test_pause_is_acknowledged_again_after_a_resume() -> None: + """**回归用例**：暂停 → 恢复 → 再暂停里，第二次暂停曾经永久失效。 + + ``checkpoint()`` 原先只在**进入**暂停分支时设置 ``_ack``。``resume()`` 把 + ``_pause`` 置回 False 并通知后，任务线程还没来得及重新取到锁，新的 + ``request_pause()`` 就把 ``_pause`` 又置成了 True——线程重新检查循环条件时发现 + "还要暂停"，于是继续等待，而它已经越过了设置 ``_ack`` 的那一行。 + + 结果 ``wait_paused()`` 永远等不到确认，暂停超时回滚。表现出来就是"暂停一次成功， + 之后再也暂停不了"。 + """ + token = CancellationToken() + ran = _start_pausing_worker(token) + assert ran.wait(_JOIN_TIMEOUT), "工作线程没有跑起来" + + for round_index in range(5): + token.request_pause() + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is True, ( + f"第 {round_index + 1} 轮暂停没有得到确认" + ) + + ran.clear() + token.resume() + assert ran.wait(_JOIN_TIMEOUT), f"第 {round_index + 1} 轮恢复后任务没有继续推进" + + token.request_stop() + + +def test_a_pause_right_after_a_resume_is_still_acknowledged() -> None: + """恢复与再次暂停之间不留任何间隔——最容易踩中的时序。""" + token = CancellationToken() + ran = _start_pausing_worker(token) + assert ran.wait(_JOIN_TIMEOUT) + + token.request_pause() + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is True + + # 立刻恢复再立刻暂停：不给任务线程任何重新取锁的机会。 + token.resume() + token.request_pause() + assert token.wait_paused(timeout=_JOIN_TIMEOUT) is True, "紧接着的第二次暂停丢了" + + token.resume() + token.request_stop() + + +# --------------------------------------------------------------------------- +# 异步路径 +# --------------------------------------------------------------------------- + + +def test_acheckpoint_fast_path_does_not_offload_to_thread() -> None: + """紧密循环里的每次检查都丢线程会拖垮事件循环，快速路径必须留在本线程。""" + + async def main() -> str: + token = CancellationToken() + for _ in range(2000): + await token.acheckpoint() + return "done" + + assert asyncio.run(main()) == "done" + + +def test_acheckpoint_raises_when_stopped() -> None: + token = CancellationToken() + token.request_stop("已取消") + + async def main() -> None: + await token.acheckpoint() + + with pytest.raises(TaskCancelled) as info: + asyncio.run(main()) + assert info.value.reason == "已取消" + + +def test_acheckpoint_suspends_and_resumes_without_blocking_loop() -> None: + """暂停期间事件循环必须仍能推进——这正是异步版本存在的理由。""" + token = CancellationToken() + token.request_pause() + + async def main() -> list[str]: + events: list[str] = [] + + async def heartbeat() -> None: + while True: + events.append("tick") + await asyncio.sleep(0.01) + + ticker = asyncio.create_task(heartbeat()) + waiter = asyncio.create_task(token.acheckpoint()) + + while not token.is_pause_acknowledged: + await asyncio.sleep(0.01) + ticks_while_paused = len(events) + assert ticks_while_paused > 0, "暂停期间事件循环不应被阻塞" + + token.resume() + await asyncio.wait_for(waiter, timeout=_JOIN_TIMEOUT) + ticker.cancel() + return events + + assert asyncio.run(main()) diff --git a/tests/unit/test_controller.py b/tests/unit/test_controller.py new file mode 100644 index 0000000..fe828d8 --- /dev/null +++ b/tests/unit/test_controller.py @@ -0,0 +1,678 @@ +"""任务管理门面（Controller / Registry）的测试。 + +重点覆盖三件事：**白名单**（只能启动已注册的任务）、**并发额度**（超限排队而不是 +无限起线程）、以及**收尾的诚实性**（排队中的任务被取消时不应真的执行任务体）。 +""" + +from __future__ import annotations + +import time +from typing import Any + +import pytest + +from chongming_task.controller import Controller, StatusCounts, TaskRegistry, render_table +from chongming_task.errors import ConfigurationError +from chongming_task.policies import BucketPolicy, ConcurrencyPolicy +from chongming_task.run import TaskRun +from chongming_task.states import TaskStatus + +_WINDOW = 10.0 + + +def _wait_until(predicate: Any, timeout: float = _WINDOW) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +def _wait_for_status(run: TaskRun, status: TaskStatus, timeout: float = _WINDOW) -> bool: + return _wait_until(lambda: run.status is status, timeout) + + +# --- 注册用任务（模块级，便于复用） --- + + +def add(ctx: Any, left: int, right: int) -> int: + """两数相加。""" + del ctx + return left + right + + +def slow_loop(ctx: Any) -> str: + """持续小步推进直到被停止。""" + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "done" + + +def never_runs(ctx: Any) -> str: + """排队期间被取消时，这个函数体不应该被执行。""" + marker.append("ran") + return "ran" + + +marker: list[str] = [] + + +# --------------------------------------------------------------------------- +# 注册表：白名单 +# --------------------------------------------------------------------------- + + +def test_registry_rejects_duplicate_names() -> None: + """静默覆盖会让"我改了实现，跑的还是旧的"这类问题极难发现。""" + registry = TaskRegistry() + registry.register("add", add) + with pytest.raises(ConfigurationError, match="已注册"): + registry.register("add", add) + + +def test_registry_requires_a_callable() -> None: + with pytest.raises(ConfigurationError, match="可调用"): + TaskRegistry().register("bad", "not-callable") # type: ignore[arg-type] + + +def test_registry_rejects_blank_name() -> None: + with pytest.raises(ConfigurationError, match="不能为空"): + TaskRegistry().register(" ", add) + + +def test_registry_picks_up_the_docstring_as_description() -> None: + registration = TaskRegistry().register("add", add) + assert registration.description == "两数相加。" + + +def test_submitting_an_unregistered_task_is_rejected() -> None: + """安全底线：入口只接受**名字**，绝不接受代码。 + + 一个能启动任意可调用对象的入口等于任意代码执行——这是将来接 CLI 还是 + HTTP 都不能破的规则。 + """ + controller = Controller() + with pytest.raises(ConfigurationError, match="未注册"): + controller.submit("nope") + assert controller.registry.names() == () + + +def test_registry_lists_registered_names() -> None: + controller = Controller() + controller.register("add", add) + controller.register("slow", slow_loop) + assert controller.registry.names() == ("add", "slow") + assert len(controller.registry) == 2 + assert "add" in controller.registry + + +# --------------------------------------------------------------------------- +# 提交、查询、统计 +# --------------------------------------------------------------------------- + + +def test_submit_runs_and_reports_snapshot() -> None: + controller = Controller() + controller.register("add", add) + run = controller.submit("add", 2, 3, task_id="t1", label="加法") + assert isinstance(run, TaskRun) + + outcome = controller.wait("t1", timeout=_WINDOW) + assert outcome.ok + assert outcome.value == 5 + + snapshot = controller.get("t1") + assert snapshot is not None + assert snapshot.task_id == "t1" + assert snapshot.name == "加法" + assert snapshot.status is TaskStatus.SUCCEEDED + + +def test_get_returns_none_for_unknown_task() -> None: + assert Controller().get("nope") is None + + +def test_stats_counts_by_status() -> None: + controller = Controller() + controller.register("add", add) + for index in range(3): + controller.submit("add", index, 1, task_id=f"t{index}") + controller.wait_all(timeout=_WINDOW) + + stats = controller.stats() + assert isinstance(stats, StatusCounts) + assert stats.total == 3 + assert stats.of(TaskStatus.SUCCEEDED) == 3 + assert stats.finished == 3 + assert stats.unfinished == 0 + + +def test_list_filters_by_status_and_name_prefix() -> None: + controller = Controller() + controller.register("add", add) + controller.submit("add", 1, 1, task_id="a1", label="抽取-甲") + controller.submit("add", 1, 1, task_id="a2", label="抽取-乙") + controller.wait_all(timeout=_WINDOW) + + assert len(controller.list_tasks()) == 2 + assert len(controller.list_tasks(name_prefix="抽取")) == 2 + assert controller.list_tasks(name_prefix="加载") == [] + assert len(controller.list_tasks(status=TaskStatus.SUCCEEDED)) == 2 + assert controller.list_tasks(status=TaskStatus.FAILED) == [] + assert len(controller.list_tasks(limit=1)) == 1 + + +def test_list_filters_by_tag() -> None: + controller = Controller() + controller.register("add", add, tags={"io"}) + controller.submit("add", 1, 1, task_id="a1") + controller.submit("add", 1, 1, task_id="a2", tags={"cpu"}) + controller.wait_all(timeout=_WINDOW) + + assert len(controller.list_tasks(tag="io")) == 2, "注册时的标签应当被继承" + assert [s.task_id for s in controller.list_tasks(tag="cpu")] == ["a2"] + + +def test_duplicate_task_id_is_rejected() -> None: + controller = Controller() + controller.register("add", add) + controller.submit("add", 1, 1, task_id="dup") + with pytest.raises(ConfigurationError, match="已存在"): + controller.submit("add", 1, 1, task_id="dup") + + +# --------------------------------------------------------------------------- +# 并发额度 +# --------------------------------------------------------------------------- + + +def test_excess_submissions_queue_instead_of_starting() -> None: + """超限的任务状态是 ``queued``（已被受理、等待执行槽），不是 ``pending``。""" + controller = Controller(max_parallel=1) + controller.register("slow", slow_loop) + + first = controller.submit("slow", task_id="t1") + assert _wait_for_status(first, TaskStatus.RUNNING) + + second = controller.submit("slow", task_id="t2") + third = controller.submit("slow", task_id="t3") + + assert second.status is TaskStatus.QUEUED + assert third.status is TaskStatus.QUEUED + assert not second.started, "排队中的任务不应被启动" + assert controller.active_count == 1 + assert controller.queued_count == 2 + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + assert controller.queued_count == 0 + + +def test_queued_tasks_start_as_slots_free_up() -> None: + controller = Controller(max_parallel=2) + controller.register("add", add) + runs = [controller.submit("add", index, 1, task_id=f"t{index}") for index in range(4)] + + outcomes = controller.wait_all(timeout=_WINDOW) + assert len(outcomes) == 4 + assert all(outcome.ok for outcome in outcomes) + assert sorted(outcome.value for outcome in outcomes) == [1, 2, 3, 4] + assert controller.active_count == 0 + assert all(run.status is TaskStatus.SUCCEEDED for run in runs) + + +def test_max_parallel_must_be_positive() -> None: + with pytest.raises(ConfigurationError, match="max_parallel"): + Controller(max_parallel=0) + + +# --------------------------------------------------------------------------- +# 具名资源信号量与优先级（ConcurrencyPolicy） +# +# 这三个字段曾经只是"声明了、校验了、文档里写着支持"——调度器从不读它，于是 +# ``limits`` 与 ``priority`` 设了等于没设，而且**不报任何错**。下面几条把它们钉住。 +# --------------------------------------------------------------------------- + + +def test_named_resource_limit_caps_only_the_tagged_tasks() -> None: + """``limits={"gpu": 1}`` 只约束带 ``gpu`` 标签的任务，不是给全局加一道闸。""" + controller = Controller(concurrency=ConcurrencyPolicy(limits={"gpu": 1})) + controller.register("slow", slow_loop) + + first = controller.submit("slow", task_id="gpu-1", tags=["gpu"]) + assert _wait_for_status(first, TaskStatus.RUNNING) + second = controller.submit("slow", task_id="gpu-2", tags=["gpu"]) + assert second.status is TaskStatus.QUEUED, "gpu 名额已满，第二个必须排队" + + # 不带该标签的任务不受它约束——具名信号量不是全局额度。 + plain = controller.submit("slow", task_id="plain") + assert _wait_for_status(plain, TaskStatus.RUNNING) + assert controller.active_count == 2 + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_a_task_needs_room_in_every_limited_tag_it_carries() -> None: + """多标签任务要**每个**受限期资源都还有余量才放行。""" + controller = Controller(concurrency=ConcurrencyPolicy(limits={"gpu": 1, "db": 1})) + controller.register("slow", slow_loop) + + gpu = controller.submit("slow", task_id="gpu-holder", tags=["gpu"]) + assert _wait_for_status(gpu, TaskStatus.RUNNING) + db = controller.submit("slow", task_id="db-holder", tags=["db"]) + assert _wait_for_status(db, TaskStatus.RUNNING) + + both = controller.submit("slow", task_id="needs-both", tags=["gpu", "db"]) + assert both.status is TaskStatus.QUEUED, "两个资源都满了，带两个标签的任务更不能放行" + + # 只空出一个：db 还占着，所以它依旧不能启动。等额度记账真的落下来再断言， + # 否则这句话可能在调度器还没醒的时候"通过"，什么都没证明。 + assert controller.stop("gpu-holder") is True + controller.wait("gpu-holder", timeout=_WINDOW) + assert _wait_until(lambda: controller.active_count == 1), "额度应当已归还" + assert both.status is TaskStatus.QUEUED, "gpu 空了但 db 还占着，不能放行" + + # 两个都空出来才轮到它。 + assert controller.stop("db-holder") is True + controller.wait("db-holder", timeout=_WINDOW) + assert _wait_for_status(both, TaskStatus.RUNNING) + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_tags_without_a_limit_do_not_throttle_anything() -> None: + """没写进 ``limits`` 的标签不受约束（常见误解：以为打个标签就会被限流）。""" + controller = Controller(concurrency=ConcurrencyPolicy(limits={"gpu": 1})) + controller.register("slow", slow_loop) + + first = controller.submit("slow", task_id="io-1", tags=["io"]) + second = controller.submit("slow", task_id="io-2", tags=["io"]) + assert _wait_for_status(first, TaskStatus.RUNNING) + assert _wait_for_status(second, TaskStatus.RUNNING) + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_priority_picks_the_highest_first() -> None: + """都能启动时先取优先级最高的；数值越大越优先。""" + controller = Controller(max_parallel=1) + controller.register("slow", slow_loop) + + blocker = controller.submit("slow", task_id="blocker") + assert _wait_for_status(blocker, TaskStatus.RUNNING) + low = controller.submit("slow", task_id="low", priority=1) + high = controller.submit("slow", task_id="high", priority=5) + + assert controller.stop("blocker") is True + controller.wait("blocker", timeout=_WINDOW) + + assert _wait_for_status(high, TaskStatus.RUNNING), "高优先级的应当先被取走" + assert low.status is TaskStatus.QUEUED, "低优先级的不该插到前面" + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_equal_priority_keeps_submission_order() -> None: + """同优先级仍然先进先出——优先级机制不该打乱时间序。""" + controller = Controller(max_parallel=1) + controller.register("slow", slow_loop) + + blocker = controller.submit("slow", task_id="blocker") + assert _wait_for_status(blocker, TaskStatus.RUNNING) + first_queued = controller.submit("slow", task_id="first") + second_queued = controller.submit("slow", task_id="second") + + assert controller.stop("blocker") is True + controller.wait("blocker", timeout=_WINDOW) + + assert _wait_for_status(first_queued, TaskStatus.RUNNING), "先提交的应当先跑" + assert second_queued.status is TaskStatus.QUEUED + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_policy_default_priority_applies_to_submissions() -> None: + """``ConcurrencyPolicy.priority`` 是 ``submit`` 不给 ``priority`` 时的默认值。""" + controller = Controller(concurrency=ConcurrencyPolicy(max_parallel=1, priority=7)) + controller.register("slow", slow_loop) + + blocker = controller.submit("slow", task_id="blocker") + assert _wait_for_status(blocker, TaskStatus.RUNNING) + assert blocker.priority == 7, "没给 priority 时应当取策略里的默认值" + + explicit_low = controller.submit("slow", task_id="low", priority=1) + assert explicit_low.priority == 1, "显式给的优先级必须能覆盖默认值" + default_high = controller.submit("slow", task_id="high") + assert default_high.priority == 7 + + assert controller.stop("blocker") is True + controller.wait("blocker", timeout=_WINDOW) + assert _wait_for_status(default_high, TaskStatus.RUNNING) + assert explicit_low.status is TaskStatus.QUEUED + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_concurrency_policy_max_parallel_is_honoured() -> None: + """策略里的 ``max_parallel`` 与 ``Controller(max_parallel=...)`` 等价。""" + controller = Controller(concurrency=ConcurrencyPolicy(max_parallel=1)) + controller.register("slow", slow_loop) + + first = controller.submit("slow", task_id="t1") + assert _wait_for_status(first, TaskStatus.RUNNING) + second = controller.submit("slow", task_id="t2") + assert second.status is TaskStatus.QUEUED + assert controller.active_count == 1 + + controller.stop_all(reason="测试结束") + controller.wait_all(timeout=_WINDOW) + + +def test_policy_and_max_parallel_are_mutually_exclusive() -> None: + """两份配置只能给一份：谁说了算必须明确，不能靠"后者覆盖前者"这种猜法。""" + with pytest.raises(ConfigurationError, match="只能给一个"): + Controller(concurrency=ConcurrencyPolicy(max_parallel=2), max_parallel=1) + + +def test_policy_rejects_a_non_positive_resource_limit() -> None: + with pytest.raises(ValueError, match="并发上限"): + ConcurrencyPolicy(limits={"gpu": 0}) + + +def test_a_non_integer_priority_is_rejected_at_submit_time() -> None: + """非整数优先级必须在提交时拦下。 + + 放过去的话，它会在**派发线程**里抛 ``TypeError``——线程直接死掉，此后所有排队 + 任务都不再启动，而没有任何报错指向真正的原因。 + """ + controller = Controller() + controller.register("add", add) + with pytest.raises(ConfigurationError, match="priority"): + controller.submit("add", 1, 2, task_id="t1", priority="high") # type: ignore[arg-type] + + +# --------------------------------------------------------------------------- +# 控制 +# --------------------------------------------------------------------------- + + +def test_stop_routes_to_the_right_task() -> None: + controller = Controller() + controller.register("slow", slow_loop) + victim = controller.submit("slow", task_id="victim") + bystander = controller.submit("slow", task_id="bystander") + assert _wait_for_status(victim, TaskStatus.RUNNING) + assert _wait_for_status(bystander, TaskStatus.RUNNING) + + assert controller.stop("victim", reason="发布窗口") is True + outcome = controller.wait("victim", timeout=_WINDOW) + assert outcome.status is TaskStatus.STOPPED + assert bystander.status is TaskStatus.RUNNING, "不该误伤别的任务" + + controller.stop_all() + controller.wait_all(timeout=_WINDOW) + + +def test_stop_returns_false_for_unknown_task() -> None: + assert Controller().stop("nope") is False + + +def test_pause_and_resume_route_through_the_controller() -> None: + controller = Controller() + controller.register("slow", slow_loop) + run = controller.submit("slow", task_id="p1") + assert _wait_for_status(run, TaskStatus.RUNNING) + + assert controller.pause("p1", timeout=_WINDOW) is True + assert run.status is TaskStatus.PAUSED + + assert controller.resume("p1") is True + assert run.status is TaskStatus.RUNNING + + controller.stop_all() + controller.wait_all(timeout=_WINDOW) + + +def test_stop_all_reports_how_many_were_affected() -> None: + controller = Controller() + controller.register("slow", slow_loop) + for index in range(3): + controller.submit("slow", task_id=f"t{index}") + assert _wait_until(lambda: controller.active_count == 3) + + assert controller.stop_all(reason="批量收工") == 3 + controller.wait_all(timeout=_WINDOW) + assert controller.stop_all() == 0, "已无未落终态的任务" + + +def test_submit_after_close_is_rejected() -> None: + controller = Controller() + controller.register("add", add) + controller.close(timeout=_WINDOW) + with pytest.raises(ConfigurationError, match="已关闭"): + controller.submit("add", 1, 1) + + +# --------------------------------------------------------------------------- +# 收尾 +# --------------------------------------------------------------------------- + + +def test_close_does_not_execute_queued_task_bodies() -> None: + """排队中的任务被取消时不该真的跑起来——否则"停止"只是把副作用推迟了。""" + marker.clear() + controller = Controller(max_parallel=1) + controller.register("slow", slow_loop) + controller.register("never", never_runs) + + controller.submit("slow", task_id="t-block") + assert _wait_until(lambda: controller.active_count == 1) + controller.submit("never", task_id="t-queued") + assert controller.queued_count == 1 + + outcomes = controller.close(reason="收工", timeout=_WINDOW) + + assert marker == [], "排队中的任务被启动了，任务体被执行" + statuses = {outcome.snapshot.task_id: outcome.status for outcome in outcomes} + assert statuses["t-queued"] is TaskStatus.STOPPED + assert statuses["t-block"] is TaskStatus.STOPPED + + +def test_prune_drops_oldest_terminal_records() -> None: + controller = Controller() + controller.register("add", add) + for index in range(5): + controller.submit("add", index, 1, task_id=f"t{index}") + controller.wait_all(timeout=_WINDOW) + assert len(controller.list_tasks()) == 5 + + assert controller.prune(keep_terminal=2) == 3 + assert len(controller.list_tasks()) == 2 + assert controller.prune(keep_terminal=2) == 0 + + +# --------------------------------------------------------------------------- +# 渲染 +# --------------------------------------------------------------------------- + + +def test_render_table_aligns_chinese_names_by_display_width() -> None: + """中日韩全角字符占两列；不按显示宽度对齐，中文任务名会把整张表挤歪。""" + controller = Controller() + controller.register("add", add) + controller.submit("add", 1, 1, task_id="a1", label="抽取") + controller.submit("add", 1, 1, task_id="a2", label="a-much-longer-name") + controller.wait_all(timeout=_WINDOW) + + table = render_table(controller.list_tasks()) + lines = table.splitlines() + + assert lines[0].startswith("TASK_ID") + assert set(lines[1]) <= {"-", " "}, "表头下方应当是分隔线" + widths = {len(line.rstrip()) for line in lines} + assert len(widths) > 1, "各行列宽应当随内容变化，而不是靠不齐" + + +def test_render_table_reports_failure_kind() -> None: + def boom(ctx: Any) -> None: + raise RuntimeError("炸了") + + controller = Controller() + controller.register("boom", boom) + controller.submit("boom", task_id="b1") + controller.wait_all(timeout=_WINDOW) + + table = render_table(controller.list_tasks()) + assert "exception" in table + assert "failed" in table + + +# --------------------------------------------------------------------------- +# 分桶 +# --------------------------------------------------------------------------- + + +def _two_keys_in_different_buckets(policy: BucketPolicy) -> tuple[str, str]: + """找两个确实落在不同桶里的键。 + + 硬编码两个键是不行的：它们有约 1/64 的概率撞进同一个桶，那会让"分桶隔离" + 的用例变成一句空话，而且只在特定版本上偶发失败。 + """ + first = "tenant-0" + first_bucket = policy.bucket_for(first) + for index in range(1, 500): + candidate = f"tenant-{index}" + if policy.bucket_for(candidate) != first_bucket: + return first, candidate + raise AssertionError("在 500 个键里找不到第二个桶，分桶策略有问题") + + +def test_bucket_is_derived_from_the_key_and_recorded() -> None: + controller = Controller() + controller.register("add", add) + run = controller.submit("add", 1, 1, task_id="a1", bucket_key="tenant-42") + controller.wait_all(timeout=_WINDOW) + + snapshot = run.snapshot + assert snapshot.bucket == controller.bucket_policy.bucket_for("tenant-42") + assert snapshot.bucket_key == "tenant-42", "原始键必须留存，否则改桶数时无法迁移" + + +def test_same_key_lands_in_the_same_bucket() -> None: + controller = Controller() + controller.register("add", add) + for index in range(3): + controller.submit("add", index, 1, task_id=f"t{index}", bucket_key="tenant-9") + controller.wait_all(timeout=_WINDOW) + + buckets = {snapshot.bucket for snapshot in controller.list_tasks()} + assert len(buckets) == 1 + + +def test_task_without_bucket_key_is_unbucketed() -> None: + controller = Controller() + controller.register("add", add) + controller.submit("add", 1, 1, task_id="a1") + controller.wait_all(timeout=_WINDOW) + + assert controller.get("a1").bucket is None # type: ignore[union-attr] + assert controller.bucket_counts() == {None: 1} + + +def test_per_bucket_limit_isolates_buckets_from_each_other() -> None: + """一个桶把额度占满不应饿死其他桶——这是分桶在**本地**就有的价值。 + + 若调度器直接取队首而不是扫描可运行者，一个积压的桶会阻塞所有桶，把分桶要解决 + 的问题又原样引入一遍。 + """ + controller = Controller(max_parallel=10, max_parallel_per_bucket=1) + controller.register("slow", slow_loop) + hot, cold = _two_keys_in_different_buckets(controller.bucket_policy) + + first = controller.submit("slow", task_id="a1", bucket_key=hot) + second = controller.submit("slow", task_id="a2", bucket_key=hot) + other = controller.submit("slow", task_id="b1", bucket_key=cold) + + assert _wait_for_status(first, TaskStatus.RUNNING) + assert _wait_for_status(other, TaskStatus.RUNNING), "别的桶被积压的桶挡住了" + assert second.status is TaskStatus.QUEUED, "同桶超出额度应当排队" + assert controller.active_in_bucket(controller.bucket_policy.bucket_for(hot)) == 1 + + controller.stop_all() + controller.wait_all(timeout=_WINDOW) + + +def test_per_bucket_limit_still_respects_the_global_limit() -> None: + controller = Controller(max_parallel=1, max_parallel_per_bucket=5) + controller.register("slow", slow_loop) + hot, cold = _two_keys_in_different_buckets(controller.bucket_policy) + + first = controller.submit("slow", task_id="a1", bucket_key=hot) + second = controller.submit("slow", task_id="b1", bucket_key=cold) + assert _wait_for_status(first, TaskStatus.RUNNING) + assert second.status is TaskStatus.QUEUED, "总额度才是硬上限" + + controller.stop_all() + controller.wait_all(timeout=_WINDOW) + + +def test_per_bucket_limit_must_be_positive() -> None: + with pytest.raises(ConfigurationError, match="max_parallel_per_bucket"): + Controller(max_parallel_per_bucket=0) + + +def test_list_tasks_filters_by_bucket_and_bucket_set() -> None: + controller = Controller() + controller.register("add", add) + hot, cold = _two_keys_in_different_buckets(controller.bucket_policy) + controller.submit("add", 1, 1, task_id="a1", bucket_key=hot) + controller.submit("add", 1, 1, task_id="b1", bucket_key=cold) + controller.submit("add", 1, 1, task_id="c1") + controller.wait_all(timeout=_WINDOW) + + hot_bucket = controller.bucket_policy.bucket_for(hot) + cold_bucket = controller.bucket_policy.bucket_for(cold) + + assert [s.task_id for s in controller.list_tasks(bucket=hot_bucket)] == ["a1"] + assert [s.task_id for s in controller.list_tasks(buckets=[hot_bucket, cold_bucket])] == [ + "a1", + "b1", + ] + assert controller.list_tasks(bucket=9999) == [] + + +def test_bucket_counts_summarises_each_bucket() -> None: + controller = Controller() + controller.register("add", add) + hot, cold = _two_keys_in_different_buckets(controller.bucket_policy) + controller.submit("add", 1, 1, task_id="a1", bucket_key=hot) + controller.submit("add", 1, 1, task_id="a2", bucket_key=hot) + controller.submit("add", 1, 1, task_id="b1", bucket_key=cold) + controller.wait_all(timeout=_WINDOW) + + counts = controller.bucket_counts() + assert counts[controller.bucket_policy.bucket_for(hot)] == 2 + assert counts[controller.bucket_policy.bucket_for(cold)] == 1 + + +def test_render_table_shows_the_bucket_column() -> None: + controller = Controller() + controller.register("add", add) + controller.submit("add", 1, 1, task_id="a1", bucket_key="tenant-42") + controller.submit("add", 1, 1, task_id="a2") + controller.wait_all(timeout=_WINDOW) + + table = render_table(controller.list_tasks()) + lines = table.splitlines() + assert "BUCKET" in lines[0] + expected = str(controller.bucket_policy.bucket_for("tenant-42")) + assert expected in lines[2] + assert lines[3].split()[2] == "-", "未分桶的任务应当显示 -" diff --git a/tests/unit/test_inline_executor.py b/tests/unit/test_inline_executor.py new file mode 100644 index 0000000..7beb8ab --- /dev/null +++ b/tests/unit/test_inline_executor.py @@ -0,0 +1,119 @@ +"""内联执行器测试。 + +重点是两件事：它**确实**同步跑完（违反 launch 不阻塞的约定），以及它对无法 +支持的暂停能力**如实返回 False** 而不是假装成功。 +""" + +from __future__ import annotations + +import asyncio +from typing import Any + +from chongming_task.models import FailureKind +from chongming_task.runtime.executors.base import ExecutionRequest +from chongming_task.runtime.executors.inline import InlineExecutor + + +def _request(target: Any, *args: Any) -> ExecutionRequest: + return ExecutionRequest(task_id="t-inline", name="t-inline", target=target, args=args) + + +def test_launch_runs_to_completion_before_returning() -> None: + """这是它与 ``Executor`` 协议唯一的差异，也是它存在的意义。""" + order: list[str] = [] + + def work(ctx: Any) -> str: + del ctx + order.append("task") + return "done" + + handle = InlineExecutor().launch(_request(work)) + order.append("after-launch") + + assert order == ["task", "after-launch"], "内联执行必须在 launch 返回前跑完" + assert handle.running is False + outcome = handle.wait() + assert outcome is not None + assert outcome.value == "done" + + +def test_arguments_are_forwarded_after_the_context() -> None: + def work(ctx: Any, left: int, right: int, *, scale: int = 1) -> int: + del ctx + return (left + right) * scale + + handle = InlineExecutor().launch( + ExecutionRequest( + task_id="t-inline-args", + name="t-inline-args", + target=work, + args=(2, 3), + kwargs={"scale": 10}, + ) + ) + outcome = handle.wait() + assert outcome is not None + assert outcome.value == 50 + + +def test_failure_is_reported_like_any_other_executor() -> None: + def boom(ctx: Any) -> None: + del ctx + raise KeyError("missing") + + outcome = InlineExecutor().launch(_request(boom)).wait() + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.EXCEPTION + assert "KeyError" in outcome.failure.message + assert outcome.failure.traceback is not None + + +def test_pause_reports_false_because_it_cannot_be_honoured() -> None: + """如实返回 False，调用方据此回滚为 RUNNING，而不是以为暂停成功了。""" + handle = InlineExecutor().launch(_request(lambda ctx: "ok")) + assert handle.pause(timeout=0.01) is False + + +def test_resume_is_harmless_after_the_fact() -> None: + handle = InlineExecutor().launch(_request(lambda ctx: "ok")) + handle.resume() + assert handle.running is False + + +def test_stop_on_a_finished_inline_task_returns_its_real_outcome() -> None: + """已结束的任务应当返回真实结果，而不是落进"兜底放弃"分支。""" + handle = InlineExecutor().launch(_request(lambda ctx: 7)) + outcome = handle.stop(graceful_timeout=1.0) + assert outcome.ok + assert outcome.value == 7 + assert outcome.abandoned is False + + +def test_context_is_injected() -> None: + seen: dict[str, Any] = {} + + def work(ctx: Any) -> str: + seen["task_id"] = ctx.task_id + seen["token"] = ctx.token + return "ok" + + handle = InlineExecutor().launch(_request(work)) + assert seen["task_id"] == "t-inline" + assert seen["token"] is handle.token + + +def test_async_target_runs_to_completion_before_launch_returns() -> None: + """``async def`` 任务在内联执行器下同样是同步跑完的（本线程起一个事件循环）。""" + + async def work(ctx: Any) -> int: + await asyncio.sleep(0) + return 42 + + handle = InlineExecutor().launch(_request(work)) + assert handle.running is False + outcome = handle.wait() + assert outcome is not None + assert outcome.ok + assert outcome.value == 42 diff --git a/tests/unit/test_models.py b/tests/unit/test_models.py new file mode 100644 index 0000000..f515371 --- /dev/null +++ b/tests/unit/test_models.py @@ -0,0 +1,143 @@ +"""快照与正交维度模型的不变量。 + +重点覆盖那些"容易被合并成单一状态枚举"的维度：进度、失败原因、请求动作。 +""" + +from __future__ import annotations + +import dataclasses +from datetime import UTC, datetime + +import pytest + +from chongming_task.models import ( + AttemptRecord, + Failure, + FailureKind, + Progress, + RequestedAction, + TaskSnapshot, + TimeoutScope, +) +from chongming_task.states import TaskStatus + + +def test_progress_fraction_is_none_when_total_unknown() -> None: + """total 未知必须返回 None 而非 0.0——后者会被渲染成"尚未开始"。""" + assert Progress(done=5).fraction is None + assert Progress(done=0, total=0).fraction is None + assert Progress(done=25, total=100).fraction == 0.25 + + +def test_progress_rejects_done_beyond_total() -> None: + with pytest.raises(ValueError, match="不能超过"): + Progress(done=11, total=10) + + +def test_progress_rejects_negative_values() -> None: + with pytest.raises(ValueError, match="不能为负"): + Progress(done=-1) + with pytest.raises(ValueError, match="不能为负"): + Progress(total=-1) + + +def test_timeout_scope_maps_to_failure_kind() -> None: + """超时是失败原因而非生命周期状态，两者的对应关系必须稳定。""" + assert FailureKind.for_timeout(TimeoutScope.START) is FailureKind.TIMEOUT_START + assert FailureKind.for_timeout(TimeoutScope.ATTEMPT) is FailureKind.TIMEOUT_ATTEMPT + assert FailureKind.for_timeout(TimeoutScope.IDLE) is FailureKind.TIMEOUT_IDLE + + +def test_attempt_record_numbering_starts_at_one() -> None: + now = datetime(2026, 1, 1, tzinfo=UTC) + with pytest.raises(ValueError, match="从 1 开始"): + AttemptRecord(task_id="t1", attempt=0, started_at=now) + + +def test_snapshot_requires_identity() -> None: + with pytest.raises(ValueError, match="task_id"): + TaskSnapshot(task_id="", name="x") + with pytest.raises(ValueError, match="name"): + TaskSnapshot(task_id="t1", name="") + + +def test_snapshot_rejects_attempt_beyond_max_attempts() -> None: + with pytest.raises(ValueError, match="max_attempts"): + TaskSnapshot(task_id="t1", name="x", attempt=3, max_attempts=2) + + +def test_snapshot_is_frozen() -> None: + """快照不可变，才能安全跨线程/协程传递并直接作为持久化记录。""" + snap = TaskSnapshot(task_id="t1", name="x") + with pytest.raises(dataclasses.FrozenInstanceError): + snap.status = TaskStatus.RUNNING # type: ignore[misc] + + +def test_validate_flags_terminal_missing_ended_at() -> None: + snap = TaskSnapshot(task_id="t1", name="x", status=TaskStatus.SUCCEEDED) + assert any("ended_at" in problem for problem in snap.validate()) + + +def test_validate_flags_leftover_requested_action_on_terminal() -> None: + """终态残留请求会让 UI 永远显示"停止中"。""" + now = datetime(2026, 1, 1, tzinfo=UTC) + snap = TaskSnapshot( + task_id="t1", + name="x", + status=TaskStatus.STOPPED, + ended_at=now, + requested_action=RequestedAction.STOP, + ) + assert any("requested_action" in problem for problem in snap.validate()) + + +def test_retry_wait_may_carry_failure() -> None: + """RETRY_WAIT 必须能携带上次失败原因，否则界面无法回答"它为什么在等"。""" + snap = TaskSnapshot( + task_id="t1", + name="x", + status=TaskStatus.RETRY_WAIT, + failure=Failure(kind=FailureKind.EXCEPTION, message="boom"), + ) + assert snap.validate() == () + + +def test_retry_wait_without_failure_is_inconsistent() -> None: + snap = TaskSnapshot(task_id="t1", name="x", status=TaskStatus.RETRY_WAIT) + assert any("RETRY_WAIT" in problem for problem in snap.validate()) + + +def test_success_must_not_carry_failure() -> None: + now = datetime(2026, 1, 1, tzinfo=UTC) + snap = TaskSnapshot( + task_id="t1", + name="x", + status=TaskStatus.SUCCEEDED, + ended_at=now, + failure=Failure(kind=FailureKind.EXCEPTION, message="boom"), + ) + assert any("SUCCEEDED" in problem for problem in snap.validate()) + + +def test_consistent_terminal_snapshot_validates_clean() -> None: + now = datetime(2026, 1, 1, tzinfo=UTC) + snap = TaskSnapshot( + task_id="t1", + name="x", + status=TaskStatus.FAILED, + ended_at=now, + attempt=1, + max_attempts=1, + failure=Failure(kind=FailureKind.EXCEPTION, message="boom"), + ) + assert snap.validate() == () + assert snap.is_terminal + assert snap.is_revivable + + +def test_running_snapshot_requires_started_at_only_on_validate() -> None: + """时间线完整性刻意不放在 __post_init__，以免逐步构造快照时处处受限。""" + snap = TaskSnapshot(task_id="t1", name="x", status=TaskStatus.RUNNING) + assert any("started_at" in problem for problem in snap.validate()) + # 但构造本身不报错 + assert snap.status is TaskStatus.RUNNING diff --git a/tests/unit/test_pipeline.py b/tests/unit/test_pipeline.py new file mode 100644 index 0000000..68ffb70 --- /dev/null +++ b/tests/unit/test_pipeline.py @@ -0,0 +1,590 @@ +"""DAG 管道测试。 + +分三层： + +1. **定义期校验**——环、悬空依赖、未注册任务、quorum 都要在创建任何任务之前拒绝， + 否则存储里会留下一堆看起来正常的半成品任务； +2. **推进**——屏障、扇出、扇入、三种汇合语义； +3. **收场**——fail_fast 连坐、下游跳过、整体状态推导。 + +约定：节点 id 一律用 ``node_id=`` 显式给出。第一个位置参数是**已注册的任务名**， +其后才是传给任务的位置参数——这两者很容易看混，显式写出来既清楚又自成文档。 +""" + +from __future__ import annotations + +import threading +import time +from typing import Any + +import pytest + +from chongming_task.controller import Controller +from chongming_task.errors import ConfigurationError +from chongming_task.pipeline import ( + Pipeline, + PipelineStatus, + render_pipeline, +) +from chongming_task.policies import AggregationPolicy, JoinMode +from chongming_task.states import TaskStatus +from chongming_task.stores.memory import InMemoryTaskStore + +_WINDOW = 15.0 + + +# --------------------------------------------------------------------------- +# 1. 定义期校验 +# --------------------------------------------------------------------------- + + +def test_empty_pipeline_is_rejected() -> None: + with pytest.raises(ConfigurationError, match="没有任何节点"): + Pipeline("empty").validate() + + +def test_dangling_dependency_is_rejected() -> None: + pipeline = Pipeline("dangling") + pipeline.node("add", node_id="a", depends_on=["no-such-node"]) + with pytest.raises(ConfigurationError, match="不存在的节点"): + pipeline.validate() + + +def test_self_dependency_is_rejected() -> None: + pipeline = Pipeline("self") + pipeline.node("add", node_id="a", depends_on=["a"]) + with pytest.raises(ConfigurationError, match="依赖了自己"): + pipeline.validate() + + +def test_cycle_is_rejected() -> None: + """有环时必须显式报错。 + + 不检测的话调度器只会安静地什么都不做（所有节点都在等对方），表现为"管道卡住了" + ——那种故障最难查。 + """ + pipeline = Pipeline("cyclic") + first = pipeline.node("add", node_id="a") + second = pipeline.node("add", node_id="b", depends_on=[first]) + # 人为制造环：a 反过来依赖 b。 + object.__setattr__(pipeline.node_by_id("a"), "depends_on", ("b",)) + assert second.depends_on == ("a",) + + with pytest.raises(ConfigurationError, match="环形依赖"): + pipeline.validate() + + +def test_unregistered_task_is_rejected_before_creating_anything() -> None: + """控制面只接受任务名。写错的 DAG 不能留下半成品任务。""" + store = InMemoryTaskStore() + controller = Controller(store=store) + controller.register("add", _add) + + pipeline = Pipeline("bad") + pipeline.node("add", node_id="ok") + pipeline.node("nope", node_id="typo") + + with pytest.raises(ConfigurationError): + controller.submit_pipeline(pipeline) + assert store.query() == [], "校验失败前不该创建任何任务" + + +def test_quorum_settlement_is_rejected_with_an_explanation() -> None: + """诚实拒绝，而不是把 quorum 硬套到 DAG 上发明一个没人能预测的语义。""" + pipeline = Pipeline("q", aggregation=AggregationPolicy.of_quorum(2)) + pipeline.node("add") + with pytest.raises(ConfigurationError, match="不支持 quorum"): + pipeline.validate() + + +def test_duplicate_node_id_is_rejected() -> None: + pipeline = Pipeline("dup") + pipeline.node("add", node_id="same") + with pytest.raises(ConfigurationError, match="节点 id 重复"): + pipeline.node("add", node_id="same") + + +def test_same_task_can_appear_many_times_with_auto_ids() -> None: + """扇出靠"同名多次"表达，不该强迫调用方给每个节点起名。""" + pipeline = Pipeline("fanout") + pipeline.node("add") + pipeline.node("add") + pipeline.node("add") + assert [node.node_id for node in pipeline.nodes] == ["add", "add#2", "add#3"] + + +def test_positional_args_belong_to_the_task_not_the_node_id() -> None: + """位置参数是传给任务的，不是节点名——这正是把参数叫 ``node_id`` 的原因。""" + pipeline = Pipeline("args") + node = pipeline.node("add", 2, 3, node_id="sum") + assert node.node_id == "sum" + assert node.args == (2, 3) + + +def test_topological_order_puts_upstream_first() -> None: + pipeline = Pipeline("order") + last = pipeline.node("add", node_id="last") + first = pipeline.node("add", node_id="first") + pipeline.node("add", node_id="middle", depends_on=[first, last]) + + order = [node.node_id for node in pipeline.topological_order()] + assert order.index("first") < order.index("middle") + assert order.index("last") < order.index("middle") + + +# --------------------------------------------------------------------------- +# 2. 推进 +# --------------------------------------------------------------------------- + + +def test_a_linear_pipeline_runs_in_dependency_order() -> None: + controller = Controller(max_parallel=4) + controller.register("mark", _mark) + + pipeline = Pipeline("linear") + extract = pipeline.node("mark", node_id="extract") + transform = pipeline.node("mark", node_id="transform", depends_on=[extract]) + pipeline.node("mark", node_id="load", depends_on=[transform]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p1") + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + assert _finished_order(controller, "p1") == ["extract", "transform", "load"] + controller.close(timeout=_WINDOW) + + +def test_a_downstream_node_waits_in_blocked_until_its_upstream_finishes() -> None: + gate = threading.Event() + controller = Controller(max_parallel=4) + controller.register("wait_gate", _wait_gate) + controller.register("mark", _mark) + + pipeline = Pipeline("blocking") + upstream = pipeline.node("wait_gate", gate, node_id="upstream") + pipeline.node("mark", node_id="downstream", depends_on=[upstream]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p2") + try: + assert _wait_until(lambda: run.node_status("upstream") is TaskStatus.RUNNING) + time.sleep(0.1) + + assert run.node_status("downstream") is TaskStatus.BLOCKED, "上游没完成前必须停在 blocked" + assert run.status is PipelineStatus.RUNNING + + gate.set() + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +def test_fan_out_runs_siblings() -> None: + controller = Controller(max_parallel=8) + controller.register("mark", _mark) + + pipeline = Pipeline("fanout") + root = pipeline.node("mark", node_id="root") + for index in range(4): + pipeline.node("mark", node_id=f"shard{index}", depends_on=[root]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p3") + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + assert set(run.task_ids) == {"root", "shard0", "shard1", "shard2", "shard3"} + controller.close(timeout=_WINDOW) + + +def test_fan_in_with_all_succeeded_waits_for_every_upstream() -> None: + controller = Controller(max_parallel=8) + controller.register("mark", _mark) + + pipeline = Pipeline("fanin") + root = pipeline.node("mark", node_id="root") + left = pipeline.node("mark", node_id="left", depends_on=[root]) + right = pipeline.node("mark", node_id="right", depends_on=[root]) + pipeline.node("mark", node_id="join", depends_on=[left, right]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p4") + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + + order = _finished_order(controller, "p4") + assert order.index("join") > order.index("left") + assert order.index("join") > order.index("right") + controller.close(timeout=_WINDOW) + + +def test_any_succeeded_join_runs_after_the_first_success() -> None: + """先到先得：任一上游成功即可放行，不必等另一个失败的上游。""" + controller = Controller(max_parallel=8) + controller.register("mark", _mark) + controller.register("boom", _boom) + + pipeline = Pipeline("any") + good = pipeline.node("mark", node_id="good") + bad = pipeline.node("boom", node_id="bad") + winner = pipeline.node( + "mark", node_id="winner", depends_on=[good, bad], join=JoinMode.ANY_SUCCEEDED + ) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p5") + outcome = run.wait(timeout=_WINDOW) + + assert winner.join is JoinMode.ANY_SUCCEEDED + assert outcome.nodes["winner"] is TaskStatus.SUCCEEDED, "有一个上游成功就该放行" + assert outcome.status is PipelineStatus.FAILED, "另一个上游失败，整条管道仍失败" + controller.close(timeout=_WINDOW) + + +def test_any_succeeded_join_skips_when_nothing_succeeds() -> None: + controller = Controller(max_parallel=8) + controller.register("boom", _boom) + controller.register("mark", _mark) + + pipeline = Pipeline("any-none") + bad = pipeline.node("boom", node_id="bad") + pipeline.node("mark", node_id="winner", depends_on=[bad], join=JoinMode.ANY_SUCCEEDED) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p6") + outcome = run.wait(timeout=_WINDOW) + assert outcome.nodes["winner"] is TaskStatus.SKIPPED + assert outcome.status is PipelineStatus.FAILED + controller.close(timeout=_WINDOW) + + +def test_all_settled_join_runs_even_after_an_upstream_failure() -> None: + """清理/通知型节点：不管上游成败都要跑。""" + controller = Controller(max_parallel=8) + controller.register("boom", _boom) + controller.register("mark", _mark) + + pipeline = Pipeline("cleanup", aggregation=AggregationPolicy.all_settled()) + failed = pipeline.node("boom", node_id="work") + cleanup = pipeline.node( + "mark", node_id="cleanup", depends_on=[failed], join=JoinMode.ALL_SETTLED + ) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p7") + outcome = run.wait(timeout=_WINDOW) + assert cleanup.join is JoinMode.ALL_SETTLED + assert outcome.nodes["cleanup"] is TaskStatus.SUCCEEDED, "ALL_SETTLED 下游必须照跑" + assert outcome.status is PipelineStatus.FAILED, "上游失败仍让整条管道失败" + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 3. 收场 +# --------------------------------------------------------------------------- + + +def test_failed_upstream_skips_downstream_with_a_reason() -> None: + controller = Controller(max_parallel=8) + controller.register("boom", _boom) + controller.register("mark", _mark) + + pipeline = Pipeline("skip") + bad = pipeline.node("boom", node_id="bad") + pipeline.node("mark", node_id="after", depends_on=[bad]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p8") + outcome = run.wait(timeout=_WINDOW) + + assert outcome.nodes["after"] is TaskStatus.SKIPPED + snapshot = controller.get("p8:after") + assert snapshot is not None + assert snapshot.status is TaskStatus.SKIPPED + assert snapshot.failure is None, "跳过不是失败，不该带 failure" + controller.close(timeout=_WINDOW) + + +def test_fail_fast_cancels_independent_running_nodes() -> None: + """真正需要连坐的场景：另一个分支已经在跑，必须被取消。 + + 被连坐的必须落 ``CANCELLED`` 而不是 ``STOPPED``——"上游挂了"与"值班同学手动停的" + 是完全不同的事，混成一个状态就再也分不出来了。 + """ + gate = threading.Event() + controller = Controller(max_parallel=8) + controller.register("boom", _boom) + controller.register("wait_gate", _wait_gate) + + pipeline = Pipeline("cascade", aggregation=AggregationPolicy.fail_fast()) + pipeline.node("wait_gate", gate, node_id="independent") + pipeline.node("boom", node_id="bad") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p9") + try: + assert _wait_until(lambda: run.node_status("independent") is TaskStatus.RUNNING) + outcome = run.wait(timeout=_WINDOW) + + assert outcome.status is PipelineStatus.FAILED + assert outcome.nodes["independent"] is TaskStatus.CANCELLED, ( + "被连坐的应当是 cancelled，而不是 stopped" + ) + snapshot = controller.get("p9:independent") + assert snapshot is not None + assert snapshot.failure is not None + assert snapshot.failure.kind.value == "cancelled" + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +def test_fail_fast_never_settles_while_a_cancelled_node_is_still_running() -> None: + """``status`` 必须先看终态、再看失败标志。 + + 顺序反了，等待者会被提前唤醒，看到一个"已经失败、但还有任务在跑"的矛盾状态。 + """ + gate = threading.Event() + controller = Controller(max_parallel=8) + controller.register("boom", _boom) + controller.register("wait_gate", _wait_gate) + + pipeline = Pipeline("ordering", aggregation=AggregationPolicy.fail_fast()) + pipeline.node("wait_gate", gate, node_id="independent") + pipeline.node("boom", node_id="bad") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p10") + try: + assert _wait_until(lambda: run.node_status("independent") is TaskStatus.RUNNING) + outcome = run.wait(timeout=_WINDOW) + for node_id in outcome.nodes: + assert outcome.nodes[node_id] in { + TaskStatus.SUCCEEDED, + TaskStatus.FAILED, + TaskStatus.SKIPPED, + TaskStatus.CANCELLED, + TaskStatus.STOPPED, + TaskStatus.LOST, + }, f"{node_id} 在管道给出结论时还没落终态" + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +def test_all_settled_does_not_cancel_siblings() -> None: + gate = threading.Event() + controller = Controller(max_parallel=8) + controller.register("boom", _boom) + controller.register("wait_gate", _wait_gate) + + pipeline = Pipeline("settled", aggregation=AggregationPolicy.all_settled()) + pipeline.node("wait_gate", gate, node_id="independent") + pipeline.node("boom", node_id="bad") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p11") + try: + assert _wait_until(lambda: run.node_status("independent") is TaskStatus.RUNNING) + time.sleep(0.2) + assert run.node_status("independent") is TaskStatus.RUNNING, "all_settled 不该连坐" + + gate.set() + outcome = run.wait(timeout=_WINDOW) + assert outcome.nodes["independent"] is TaskStatus.SUCCEEDED + assert outcome.status is PipelineStatus.FAILED, "有人失败过，整体就是失败" + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +def test_pipeline_tasks_carry_the_run_id() -> None: + controller = Controller(max_parallel=4) + controller.register("mark", _mark) + pipeline = Pipeline("tagged") + pipeline.node("mark", node_id="one") + pipeline.node("mark", node_id="two") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="tagged-1") + run.wait(timeout=_WINDOW) + + for node_id, task_id in run.task_ids.items(): + snapshot = controller.get(task_id) + assert snapshot is not None + assert snapshot.pipeline_run_id == "tagged-1" + assert task_id == f"tagged-1:{node_id}" + controller.close(timeout=_WINDOW) + + +def test_cancelling_the_whole_pipeline() -> None: + gate = threading.Event() + controller = Controller(max_parallel=8) + controller.register("wait_gate", _wait_gate) + + pipeline = Pipeline("cancelme") + root = pipeline.node("wait_gate", gate, node_id="root") + pipeline.node("wait_gate", gate, node_id="child", depends_on=[root]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p12") + try: + assert _wait_until(lambda: run.node_status("root") is TaskStatus.RUNNING) + assert run.cancel(reason="运维取消") >= 1 + + outcome = run.wait(timeout=_WINDOW) + assert outcome.status is PipelineStatus.CANCELLED + assert outcome.nodes["root"] is TaskStatus.CANCELLED + assert outcome.nodes["child"] is TaskStatus.CANCELLED + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +def test_wait_timeout_does_not_change_anything() -> None: + gate = threading.Event() + controller = Controller(max_parallel=4) + controller.register("wait_gate", _wait_gate) + + pipeline = Pipeline("slow") + pipeline.node("wait_gate", gate, node_id="root") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p13") + try: + outcome = run.wait(timeout=0.2) + assert outcome.status is PipelineStatus.RUNNING + assert run.node_status("root") in {TaskStatus.STARTING, TaskStatus.RUNNING} + assert not run.settled + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +def test_pipeline_run_is_registered_on_the_controller() -> None: + controller = Controller(max_parallel=4) + controller.register("mark", _mark) + pipeline = Pipeline("reg") + pipeline.node("mark", node_id="only") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="reg-1") + run.wait(timeout=_WINDOW) + assert controller.get_pipeline("reg-1") is run + assert controller.get_pipeline("nope") is None + assert controller.pipelines() == (run,) + controller.close(timeout=_WINDOW) + + +def test_a_diamond_pipeline_works_end_to_end() -> None: + """完整形状：一个源 → 两个分支 → 汇合。""" + controller = Controller(max_parallel=8) + controller.register("mark", _mark) + + pipeline = Pipeline("diamond") + source = pipeline.node("mark", node_id="source") + left = pipeline.node("mark", node_id="left", depends_on=[source]) + right = pipeline.node("mark", node_id="right", depends_on=[source]) + pipeline.node("mark", node_id="sink", depends_on=[left, right]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p14") + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + + order = _finished_order(controller, "p14") + assert order[0] == "source" + assert order[-1] == "sink" + assert "sink" in render_pipeline(run) + controller.close(timeout=_WINDOW) + + +def test_two_runs_of_the_same_pipeline_are_independent() -> None: + controller = Controller(max_parallel=4) + controller.register("mark", _mark) + pipeline = Pipeline("reusable") + pipeline.node("mark", node_id="only") + + first = controller.submit_pipeline(pipeline, pipeline_run_id="r1") + second = controller.submit_pipeline(pipeline, pipeline_run_id="r2") + assert first.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + assert second.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + assert set(first.task_ids.values()).isdisjoint(second.task_ids.values()) + controller.close(timeout=_WINDOW) + + +def test_pipeline_node_args_reach_the_task() -> None: + controller = Controller(max_parallel=4) + controller.register("add", _add) + pipeline = Pipeline("args") + pipeline.node("add", 20, 22, node_id="sum") + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p15") + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + snapshot = controller.get("p15:sum") + assert snapshot is not None + assert snapshot.outcome == {"value": 42} + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 与并发额度协同 +# --------------------------------------------------------------------------- + + +def test_deferred_nodes_do_not_consume_concurrency_slots() -> None: + """等待上游的节点不该占额度——否则一条长管道会把额度全锁死。""" + gate = threading.Event() + controller = Controller(max_parallel=2) + controller.register("wait_gate", _wait_gate) + controller.register("mark", _mark) + + pipeline = Pipeline("slots") + root = pipeline.node("wait_gate", gate, node_id="root") + for index in range(5): + pipeline.node("mark", node_id=f"child{index}", depends_on=[root]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="p16") + try: + assert _wait_until(lambda: run.node_status("root") is TaskStatus.RUNNING) + time.sleep(0.1) + assert controller.active_count == 1, "只有 root 该占额度" + assert controller.queued_count == 0, "下游还在 blocked，不是 queued" + + gate.set() + assert run.wait(timeout=_WINDOW).status is PipelineStatus.SUCCEEDED + finally: + gate.set() + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 工具 +# --------------------------------------------------------------------------- + + +def _finished_order(controller: Controller, run_id: str) -> list[str]: + """按结束时间排出某个管道 run 里各节点的完成顺序。""" + snapshots = [ + item + for item in controller.list_tasks() + if item.pipeline_run_id == run_id and item.ended_at is not None + ] + snapshots.sort(key=lambda item: item.ended_at or item.created_at) # type: ignore[arg-type,return-value] + return [item.name.split("/", 1)[-1] for item in snapshots] + + +def _wait_until(predicate: Any, timeout: float = _WINDOW) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False + + +# --------------------------------------------------------------------------- +# 任务体 +# --------------------------------------------------------------------------- + + +def _mark(ctx: Any) -> str: + del ctx + return "ok" + + +def _add(ctx: Any, left: int, right: int) -> int: + del ctx + return left + right + + +def _boom(ctx: Any) -> str: + del ctx + raise RuntimeError("故意失败") + + +def _wait_gate(ctx: Any, gate: threading.Event) -> str: + while not gate.wait(0.01): + ctx.checkpoint() + ctx.checkpoint() + return "released" diff --git a/tests/unit/test_postgres_lazy.py b/tests/unit/test_postgres_lazy.py new file mode 100644 index 0000000..996b1f3 --- /dev/null +++ b/tests/unit/test_postgres_lazy.py @@ -0,0 +1,98 @@ +"""``PostgresTaskStore`` 的惰性导入契约——**不需要数据库**。 + +钉住两条承诺： + +1. 取到 ``chongming_task.PostgresTaskStore`` 之前，iss 不导入 psycopg——核心包保持 + 零第三方运行时依赖（``import chongming_task`` 不会因为没装驱动而变慢或失败）； +2. 没装可选依赖时，报错必须**可操作**：给出安装命令，而不是一个光秃秃的 + ``ModuleNotFoundError``。 + +真正的行为测试在 ``tests/integration/test_postgres_store.py``（需要 DSN）。 +""" + +from __future__ import annotations + +import os +import subprocess +import sys + +import pytest + +import chongming_task + + +def _run(script: str) -> subprocess.CompletedProcess[str]: + environment = {**os.environ} + return subprocess.run( + [sys.executable, "-c", script], + capture_output=True, + text=True, + env=environment, + timeout=60, + ) + + +def test_star_import_does_not_pull_the_driver() -> None: + """``from chongming_task import *`` 不该要求安装数据库驱动。""" + assert "PostgresTaskStore" not in chongming_task.__all__ + + script = ( + "import sys\n" + "from chongming_task import *\n" + "assert 'psycopg' not in sys.modules, '星号导入拉起了 psycopg'\n" + "assert 'PostgresTaskStore' not in dir(), '它不该出现在星号导入里'\n" + "print('ok')\n" + ) + completed = _run(script) + assert completed.returncode == 0, completed.stderr + assert completed.stdout.strip() == "ok" + + +def test_lazy_attribute_does_not_import_the_driver_until_used() -> None: + script = ( + "import sys\n" + "import chongming_task\n" + "assert 'psycopg' not in sys.modules, '导入核心包就拉起了 psycopg'\n" + "print('ok')\n" + ) + completed = _run(script) + assert completed.returncode == 0, completed.stderr + assert completed.stdout.strip() == "ok" + + +def test_missing_driver_gives_an_actionable_error() -> None: + """把 psycopg 屏蔽成"未安装"，取 ``PostgresTaskStore`` 必须报出安装指引。""" + script = ( + "import sys\n" + "sys.modules['psycopg'] = None\n" + "sys.modules['psycopg_pool'] = None\n" + "import chongming_task\n" + "try:\n" + " chongming_task.PostgresTaskStore\n" + "except ImportError as exc:\n" + " text = str(exc)\n" + " assert 'chongming-task[postgres]' in text, text\n" + " print('ok')\n" + "else:\n" + " raise SystemExit('没有抛出 ImportError')\n" + ) + completed = _run(script) + assert completed.returncode == 0, completed.stderr + assert completed.stdout.strip() == "ok" + + +def test_unknown_attribute_still_raises_attribute_error() -> None: + """``__getattr__`` 只放行已知的可选导出，其余照常抛 AttributeError。""" + with pytest.raises(AttributeError, match="no attribute"): + _ = chongming_task.NoSuchThing # type: ignore[attr-defined] + + +def test_postgres_store_resolves_when_the_extra_is_installed() -> None: + """装了可选依赖时必须能取到——并在 stores 包与顶层给出同一个类。""" + pytest.importorskip("psycopg") + + from chongming_task.stores import PostgresTaskStore as FromStores + from chongming_task.stores.postgres import PostgresTaskStore as FromModule + + assert chongming_task.PostgresTaskStore is FromModule + assert FromStores is FromModule diff --git a/tests/unit/test_process_executor.py b/tests/unit/test_process_executor.py new file mode 100644 index 0000000..1e3ed4f --- /dev/null +++ b/tests/unit/test_process_executor.py @@ -0,0 +1,390 @@ +"""进程执行器测试。 + +这些用例全部用**真实子进程**（``spawn``）跑：执行器的全部价值就在于子进程能不能被 +停住、能不能被强杀、父进程能不能看见它。用假的进程模型测不出这些。 + +三条约束写在前面，它们是这套设计的一部分而不是测试的毛病： + +* 目标函数必须是**模块级**的——spawn 靠"模块 + 限定名"在子进程里定位它； +* 所以子进程会重新 import 本模块（模块顶层不能有副作用）； +* 每个用例给足超时（5 秒），子进程数量控制在小规模，测试整体仍然是秒级。 +""" + +from __future__ import annotations + +import os +import threading +import time +from typing import Any + +import pytest + +from chongming_task.errors import ConfigurationError +from chongming_task.models import FailureKind +from chongming_task.runtime.executors.base import ExecutionRequest +from chongming_task.runtime.executors.process import ProcessExecutor +from chongming_task.testing import FakeClock + +_WINDOW = 10.0 + +#: 子进程启动本身就要几百毫秒（新解释器 + import），协作路径的窗口要比线程宽。 +_START = 15.0 + + +def _request(target: Any, *args: Any, **kwargs: Any) -> ExecutionRequest: + return ExecutionRequest( + task_id="t-proc", + name="t-proc", + target=target, + args=args, + kwargs=kwargs, + ) + + +# --------------------------------------------------------------------------- +# 子进程里跑的模块级函数（spawn 要求） +# --------------------------------------------------------------------------- + + +def _add(ctx: Any, left: int, right: int, *, scale: int = 1) -> int: + """两数相加。""" + del ctx + return (left + right) * scale + + +def _boom(ctx: Any) -> None: + """抛异常。""" + del ctx + raise ValueError("子进程里炸了") + + +def _describe(ctx: Any) -> dict[str, Any]: + """把注入进来的上下文信息带回来，验证它是真的被注入了。""" + return { + "task_id": ctx.task_id, + "name": ctx.name, + "attempt": ctx.attempt, + "pid": os.getpid(), + "has_token": ctx.token is not None, + } + + +def _beater(ctx: Any) -> str: + """持续 checkpoint，直到进程退出。""" + for _ in range(10_000): + ctx.checkpoint() + time.sleep(0.01) + return "never" + + +def _bounded_beater(ctx: Any) -> str: + """有限次 checkpoint 后正常返回——用于验证"暂停没生效时不能把它卡住"。""" + for _ in range(5): + ctx.checkpoint() + time.sleep(0.1) + return "done" + + +def _late_checkpointer(ctx: Any) -> str: + """先干一段**不 checkpoint** 的活，之后才正常推进。 + + 用来测暂停回滚：暂停请求会在它到达第一个检查点之前就超时。 + """ + time.sleep(0.5) + for _ in range(5): + ctx.checkpoint() + time.sleep(0.05) + return "done" + + +def _progress_reporter(ctx: Any) -> str: + """报进度，父进程应当收到。""" + for index in range(1, 4): + ctx.progress(index, total=3, unit="批") + time.sleep(0.02) + return "ok" + + +def _ignore_everything(ctx: Any) -> str: + """**不**调用 checkpoint 的死循环：只能靠强杀停下。""" + del ctx + end = time.monotonic() + 60 + while time.monotonic() < end: + pass + return "never" + + +def _abrupt_exit(ctx: Any) -> str: + """直接消失，连一句结论都不留。""" + del ctx + os._exit(3) + return "never" # pragma: no cover + + +def _unpicklable_result(ctx: Any) -> Any: + """返回值不可 pickle（线程执行器能接住它，进程执行器不行）。""" + del ctx + return lambda: None + + +def _explode_on_unpickle() -> None: # pragma: no cover - 只在子进程里被调用 + raise RuntimeError("反序列化时炸了") + + +class _CannotCrossTheBorder: + """能 pickle 出去、却在子进程里反序列化失败的对象。 + + ``__reduce__`` 只在**加载**时调用那个工厂函数，所以父进程的预检（只 dump）会放行 + ——正好用来测"子进程根本没跑起来"那条分支。 + """ + + def __reduce__(self) -> Any: + return (_explode_on_unpickle, ()) + + +# --------------------------------------------------------------------------- +# 载荷校验：把"子进程里神秘失败"变成一句能照做的话 +# --------------------------------------------------------------------------- + + +def test_a_lambda_target_is_rejected_with_an_actionable_error() -> None: + with pytest.raises(ConfigurationError, match="可 pickle"): + ProcessExecutor().launch(_request(lambda ctx: "nope")) + + +def test_a_closure_target_is_rejected() -> None: + captured = "closure" + + def inner(ctx: Any) -> str: + del ctx + return captured + + with pytest.raises(ConfigurationError, match="ThreadExecutor"): + ProcessExecutor().launch(_request(inner)) + + +def test_an_unpicklable_argument_is_rejected() -> None: + with pytest.raises(ConfigurationError, match="参数"): + ProcessExecutor().launch(_request(_add, threading.Lock(), 1)) + + +# --------------------------------------------------------------------------- +# 正常路径 +# --------------------------------------------------------------------------- + + +def test_target_runs_in_a_child_process_and_returns_its_value() -> None: + handle = ProcessExecutor().launch( + ExecutionRequest( + task_id="t-args", + name="t-args", + target=_add, + args=(2, 3), + kwargs={"scale": 10}, + ) + ) + outcome = handle.wait(timeout=_START) + + assert outcome is not None + assert outcome.ok + assert outcome.value == 50 + assert outcome.abandoned is False, "进程没有强杀这个词，结论是已知的" + + +def test_the_child_has_its_own_pid_and_gets_the_injected_context() -> None: + handle = ProcessExecutor().launch(_request(_describe)) + outcome = handle.wait(timeout=_START) + + assert outcome is not None and outcome.ok + payload = outcome.value + assert payload["task_id"] == "t-proc" + assert payload["name"] == "t-proc" + assert payload["has_token"] is True + assert payload["pid"] == handle.pid, "子进程里看到的 pid 就是句柄报出来的那个" + assert handle.pid != os.getpid(), "任务确实不在监督进程里跑" + + +def test_the_handle_knows_the_child_pid() -> None: + handle = ProcessExecutor().launch(_request(_add, 1, 1)) + assert handle.pid is not None + assert handle.running is True + assert handle.wait(timeout=_START) is not None + assert handle.running is False, "有结论之后就不该再报'在运行'" + + +def test_a_task_failure_crosses_the_boundary_as_a_structured_failure() -> None: + outcome = ProcessExecutor().launch(_request(_boom)).wait(timeout=_START) + + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.EXCEPTION + assert "子进程里炸了" in outcome.failure.message + assert outcome.failure.traceback, "子进程里的 traceback 必须带回来" + + +def test_an_unpicklable_result_is_reported_as_a_result_problem() -> None: + """返回值过不去，报的是"产出接不住"，不是"任务出错了"——两者的修法完全不同。""" + outcome = ProcessExecutor().launch(_request(_unpicklable_result)).wait(timeout=_START) + + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.RESULT_INVALID + assert outcome.failure.retryable is False, "同一个返回值重跑多少次都还是过不去" + + +# --------------------------------------------------------------------------- +# 心跳：idle 超时靠它，而它必须真的跨过进程边界 +# --------------------------------------------------------------------------- + + +def test_heartbeats_cross_the_process_boundary() -> None: + """把父进程的时钟推远，再等子进程的一次 heartbeat 把它拉回来。 + + 用假时钟是为了**确定性地**区分"子进程真的报了心跳"与"父进程自己刚记了一次"。 + """ + clock = FakeClock() + handle = ProcessExecutor(clock=clock).launch(_request(_beater)) + try: + assert handle.wait(timeout=0.2) is None, "子进程不该这么快结束" + clock.advance(300) + assert handle.heartbeat.idle_seconds() >= 300 + + deadline = time.monotonic() + _START + while handle.heartbeat.idle_seconds() >= 300 and time.monotonic() < deadline: + time.sleep(0.02) + assert handle.heartbeat.idle_seconds() == 0, "子进程的 checkpoint 应当刷新心跳" + finally: + handle.stop(graceful_timeout=_WINDOW) + + +def test_progress_reports_are_forwarded_to_the_parent() -> None: + seen: list[Any] = [] + request = ExecutionRequest( + task_id="t-progress", + name="t-progress", + target=_progress_reporter, + on_progress=seen.append, + ) + outcome = ProcessExecutor().launch(request).wait(timeout=_START) + + assert outcome is not None and outcome.ok + assert [item.done for item in seen] == [1, 2, 3] + assert seen[-1].unit == "批" + + +# --------------------------------------------------------------------------- +# 暂停：确认了才算，没确认就必须回滚 +# --------------------------------------------------------------------------- + + +def test_pause_suspends_the_child_and_resume_lets_it_finish() -> None: + handle = ProcessExecutor().launch(_request(_bounded_beater)) + assert handle.pause(timeout=_START) is True, "子进程应当在下一次 checkpoint 停下" + + # 若没真的暂停，这个任务 0.5 秒内就该跑完了。 + time.sleep(0.8) + assert handle.wait(timeout=0) is None, "被暂停的任务不该继续推进" + assert handle.running is True + + handle.resume() + outcome = handle.wait(timeout=_START) + assert outcome is not None and outcome.ok + assert outcome.value == "done" + + +def test_pause_rolls_back_when_the_child_never_reaches_a_checkpoint() -> None: + """暂停请求超时后必须**撤销**。 + + 不撤销的话，调用方已经按"暂停没生效"把状态回滚成 RUNNING，而子进程会在稍后的 + 某个检查点自己停住：那是一个没人知道、也没人会恢复的挂起——比暂停失败更糟。 + """ + handle = ProcessExecutor().launch(_request(_late_checkpointer)) + assert handle.wait(timeout=0.1) is None, "它还在那段不 checkpoint 的活里" + assert handle.pause(timeout=0.2) is False, "窗口内它压根没到检查点" + + outcome = handle.wait(timeout=_START) + assert outcome is not None, "回滚之后它必须继续跑完，而不是停在下一个检查点上" + assert outcome.ok, f"不该被自己撤销掉的暂停影响：{outcome.failure}" + + +# --------------------------------------------------------------------------- +# 停止：进程执行器存在的理由 +# --------------------------------------------------------------------------- + + +def test_cooperative_stop_ends_the_task_without_killing_it() -> None: + handle = ProcessExecutor().launch(_request(_beater)) + assert handle.wait(timeout=0.3) is None # 先让它跑起来 + + outcome = handle.stop(graceful_timeout=_WINDOW) + + assert not outcome.ok + assert outcome.abandoned is False, "它是自己收尾的，不是被放弃" + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + assert handle.running is False + + +def test_a_task_that_never_checks_the_token_is_actually_killed() -> None: + """**这是进程执行器唯一不可替代的能力。** 线程只能放弃等待并记 LOST； + 子进程会真的被停掉，结论因此是已知的。""" + handle = ProcessExecutor().launch(_request(_ignore_everything)) + pid = handle.pid + assert pid is not None + assert handle.wait(timeout=0.3) is None # 它连 checkpoint 都不调，只能强杀 + + outcome = handle.stop(graceful_timeout=0.3) + + assert not outcome.ok + assert outcome.abandoned is False, "任务确实停了，不是'不知道停没停'" + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + assert handle.running is False, "载体已经死了，不能谎报还在跑" + assert handle.process is not None and handle.process.exitcode is not None + + +def test_stop_is_also_delivered_through_the_handle_token() -> None: + """控制面有几条路径直接调 ``handle.token.request_stop()``，绕过了 ``stop()``。 + + 令牌不转发的话子进程永远收不到那次停止——表现出来就是"停止按钮没反应"。 + """ + handle = ProcessExecutor().launch(_request(_beater)) + assert handle.wait(timeout=0.3) is None + + handle.token.request_stop("走令牌这条路") + + outcome = handle.wait(timeout=_WINDOW) + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + + +# --------------------------------------------------------------------------- +# 崩溃隔离：父进程必须活着，并且如实说清楚发生了什么 +# --------------------------------------------------------------------------- + + +def test_an_abrupt_child_exit_is_reported_with_its_exit_code() -> None: + outcome = ProcessExecutor().launch(_request(_abrupt_exit)).wait(timeout=_START) + + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.NONZERO_EXIT + assert outcome.failure.exit_code == 3, "退出码是唯一能解释'它为什么没了'的事实" + assert "副作用" in outcome.failure.message, "结果未知必须说清楚，别让人放心重跑" + + +def test_a_child_that_never_boots_is_reported_distinctly() -> None: + """载荷在子进程里反序列化失败：任务**从未开始**，与"跑一半没了"是两回事。""" + handle = ProcessExecutor().launch(_request(_add, _CannotCrossTheBorder(), 1)) + outcome = handle.wait(timeout=_START) + + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.NONZERO_EXIT + assert "从未开始" in outcome.failure.message diff --git a/tests/unit/test_public_api.py b/tests/unit/test_public_api.py new file mode 100644 index 0000000..f86dba0 --- /dev/null +++ b/tests/unit/test_public_api.py @@ -0,0 +1,83 @@ +"""公开 API 的表面测试。 + +``chongming_task`` 的 ``__init__`` 是**嵌入其他应用**的入口，因此它本身的正确性 +值得一条用例：漏导出某个名字、或者导出顺序导致循环导入，都会让集成方第一步就失败。 +""" + +from __future__ import annotations + +import chongming_task + + +def test_version_is_exposed() -> None: + assert chongming_task.__version__ + + +def test_all_has_no_duplicates() -> None: + """重复导出会让"到底哪个生效"变得不确定。 + + 顺序**不在这里**断言：``__all__`` 的排序由 ruff 的 ``RUF022``（isort 风格， + 全大写常量优先）负责，那是 lint 的职责。写一条与 linter 规则不一致的断言， + 只会让两边互相打架。 + """ + names = chongming_task.__all__ + assert len(names) == len(set(names)) + + +def test_every_exported_name_resolves() -> None: + """``__all__`` 里写了却取不到，等于给集成方埋了一个 ImportError。""" + missing = [name for name in chongming_task.__all__ if not hasattr(chongming_task, name)] + assert missing == [] + + +def test_primary_integration_surface_is_importable() -> None: + """这一段就是 README 里"嵌入你自己的应用"的最小示例。""" + from chongming_task import ( # noqa: F401 - 只为验证可导入 + Controller, + InMemoryTaskStore, + SqliteTaskStore, + TaskStatus, + TaskStore, + run_task, + ) + + controller = Controller(max_parallel=2) + assert controller.registry.names() == () + assert TaskStatus.PENDING.value == "pending" + assert callable(run_task) + + +def test_store_implementations_satisfy_the_protocol() -> None: + """存储实现必须满足契约——第三方后端也照这条自证。""" + from chongming_task import InMemoryTaskStore, SqliteTaskStore, TaskStore + + assert isinstance(InMemoryTaskStore(), TaskStore) + + store = SqliteTaskStore(":memory:") + try: + assert isinstance(store, TaskStore) + finally: + # 不关掉连接会在解释器退出时触发 ResourceWarning；本项目的 pytest 配置 + # 把 warning 升级为 error，因此泄漏是会被抓住的。 + store.close() + + +def test_controller_accepts_a_store() -> None: + """存储已经接线。 + + 这条用例原先钉的是"尚未接线"——当 ``Controller(store=...)`` 真的开始工作时， + 它会自己失败并提示删掉。现在它钉的是接线本身。 + """ + from chongming_task import Controller, InMemoryTaskStore + + store = InMemoryTaskStore() + controller = Controller(store=store, max_parallel=2) + assert controller.store is store + assert controller.persist_failures == () + assert controller.owner + + +def test_private_helpers_are_not_exported() -> None: + """以下划线开头的实现细节不应出现在公开表面上。""" + leaked = [name for name in chongming_task.__all__ if name.startswith("_")] + assert leaked == [] diff --git a/tests/unit/test_run.py b/tests/unit/test_run.py new file mode 100644 index 0000000..1431efc --- /dev/null +++ b/tests/unit/test_run.py @@ -0,0 +1,404 @@ +"""单任务受控执行的测试。 + +覆盖三件用户最关心的事：**失败会怎样**（重试 / 超时 / 最终失败）、**我能不能干预** +（停止 / 暂停 / 恢复），以及**结果是否诚实**（不配合的任务落 LOST 而不是"已停止"）。 +""" + +from __future__ import annotations + +import threading +import time +from typing import Any + +import pytest + +from chongming_task.errors import ConfigurationError +from chongming_task.models import FailureKind +from chongming_task.policies import FixedBackoff, RetryPolicy, Timeout +from chongming_task.run import TaskRun, run_task +from chongming_task.runtime.executors.inline import InlineExecutor +from chongming_task.states import TaskStatus + +_WINDOW = 10.0 + + +def _wait_for_status(run: TaskRun, status: TaskStatus, timeout: float = _WINDOW) -> bool: + """等状态就位。 + + 比 ``sleep`` 可靠：``start()`` 是异步的，"任务已经开始执行"与"状态机已迁移到 + RUNNING"之间存在极短窗口。 + """ + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if run.status is status: + return True + time.sleep(0.01) + return False + + +# --------------------------------------------------------------------------- +# 成功路径 +# --------------------------------------------------------------------------- + + +def test_successful_run_returns_value_and_succeeds() -> None: + outcome = run_task(lambda ctx, a, b: a + b, 2, 3, task_id="t-ok", name="加法") + assert outcome.ok + assert outcome.status is TaskStatus.SUCCEEDED + assert outcome.value == 5 + assert outcome.failure is None + assert outcome.snapshot.ended_at is not None + + +def test_transition_history_is_complete_and_ordered() -> None: + """状态历史只追加，且每条都带触发源——这是可审计性的基础。""" + run = TaskRun(lambda ctx: 1, task_id="t-hist", name="demo") + run.start() + assert run.wait(timeout=_WINDOW).ok + + assert [t.to_status for t in run.transitions] == [ + TaskStatus.QUEUED, + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.SUCCEEDED, + ] + assert all(t.trigger for t in run.transitions) + assert run.transitions[-1].task_id == "t-hist" + + +def test_exception_is_captured_with_traceback() -> None: + def boom(ctx: Any) -> None: + raise ValueError("炸了") + + outcome = run_task(boom, task_id="t-boom") + assert outcome.status is TaskStatus.FAILED + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.EXCEPTION + assert outcome.failure.traceback is not None + assert "boom" in outcome.failure.traceback + + +# --------------------------------------------------------------------------- +# 重试 +# --------------------------------------------------------------------------- + + +def test_retries_until_success_and_counts_attempts() -> None: + attempts: list[int] = [] + + def flaky(ctx: Any, fail_times: int) -> str: + attempts.append(1) + if len(attempts) <= fail_times: + raise RuntimeError("暂时性故障") + return "ok" + + outcome = run_task( + flaky, + 2, + retry=RetryPolicy(max_attempts=3, backoff=FixedBackoff(0.0)), + task_id="t-flaky", + ) + assert outcome.ok + assert outcome.snapshot.attempt == 3 + assert len(attempts) == 3 + + +def test_retry_exhaustion_lands_failed() -> None: + def always_fails(ctx: Any) -> None: + raise RuntimeError("一直失败") + + outcome = run_task( + always_fails, + retry=RetryPolicy(max_attempts=2, backoff=FixedBackoff(0.0)), + task_id="t-always", + ) + assert outcome.status is TaskStatus.FAILED + assert outcome.snapshot.attempt == 2 + assert outcome.failure is not None + + +def test_retry_wait_is_visible_in_history() -> None: + seen: list[int] = [] + + def flaky(ctx: Any) -> str: + if not seen: + seen.append(1) + raise RuntimeError("第一次失败") + return "ok" + + run = TaskRun( + flaky, + retry=RetryPolicy(max_attempts=2, backoff=FixedBackoff(0.0)), + task_id="t-wait", + ) + run.start() + assert run.wait(timeout=_WINDOW).ok + assert TaskStatus.RETRY_WAIT in [t.to_status for t in run.transitions] + + +# --------------------------------------------------------------------------- +# 超时 +# --------------------------------------------------------------------------- + + +def test_per_attempt_timeout_stops_a_slow_task() -> None: + def slow(ctx: Any) -> str: + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.01) + return "永远跑不完" + + outcome = run_task(slow, timeout=Timeout(per_attempt=0.3), task_id="t-slow") + assert outcome.status is TaskStatus.FAILED + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.TIMEOUT_ATTEMPT + + +def test_idle_timeout_is_skipped_while_paused() -> None: + """被暂停的任务不推进心跳，若照常判 idle 超时，暂停就会被误判成卡死。""" + started = threading.Event() + + def work(ctx: Any) -> str: + started.set() + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.02) + return "done" + + run = TaskRun( + work, + timeout=Timeout(idle=0.2), + task_id="t-idle-paused", + graceful_stop=2.0, + ) + run.start() + assert started.wait(_WINDOW) + assert _wait_for_status(run, TaskStatus.RUNNING) + assert run.pause(timeout=_WINDOW) is True + + # 暂停期间超过 idle 阈值，但不应被判超时。 + time.sleep(0.5) + assert run.status is TaskStatus.PAUSED + + run.resume() + run.stop(reason="测试结束") + assert run.wait(timeout=_WINDOW).status is TaskStatus.STOPPED + + +# --------------------------------------------------------------------------- +# 停止 +# --------------------------------------------------------------------------- + + +def test_stop_lands_stopped_and_is_auditable() -> None: + started = threading.Event() + + def work(ctx: Any) -> str: + started.set() + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "done" + + run = TaskRun(work, task_id="t-stop") + run.start() + assert started.wait(_WINDOW) + assert _wait_for_status(run, TaskStatus.RUNNING) + + run.stop(reason="发布窗口") + outcome = run.wait(timeout=_WINDOW) + assert outcome.status is TaskStatus.STOPPED + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + assert outcome.failure.retryable is False, "取消不是失败，不能重试" + + stopping = [t for t in run.transitions if t.to_status is TaskStatus.STOPPING] + assert stopping, "应当能看到 STOPPING 这个中间态" + assert stopping[0].reason == "发布窗口" + + +def test_uncooperative_task_is_reported_as_lost_not_stopped() -> None: + """线程停不掉时结果**未知**：任务可能仍在后台产生副作用。 + + 记成"已停止"会让运维误判资源已释放，因此必须落 LOST。 + """ + + def stubborn(ctx: Any) -> str: + for _ in range(500): + time.sleep(0.01) + return "ignored" + + run = TaskRun(stubborn, task_id="t-stubborn", graceful_stop=0.2) + run.start() + time.sleep(0.3) + run.stop(reason="不配合") + + outcome = run.wait(timeout=_WINDOW) + assert outcome.status is TaskStatus.LOST + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.LOST + + +def test_stop_before_start_lands_stopped_without_burning_an_attempt() -> None: + """都还没跑就要求停止，不该白白消耗一次尝试。""" + run = TaskRun(lambda ctx: "ok", task_id="t-pre-stop") + run.stop(reason="还没跑就不要了") + run.start() + outcome = run.wait(timeout=_WINDOW) + assert outcome.status is TaskStatus.STOPPED + assert outcome.snapshot.attempt == 0 + assert all(t.to_status is not TaskStatus.RUNNING for t in run.transitions) + + +# --------------------------------------------------------------------------- +# 暂停与恢复 +# --------------------------------------------------------------------------- + + +def test_pause_freezes_progress_and_resume_continues() -> None: + started = threading.Event() + progress: list[int] = [] + + def work(ctx: Any) -> str: + started.set() + for index in range(1_000_000): + ctx.checkpoint() + progress.append(index) + time.sleep(0.005) + return "done" + + run = TaskRun(work, task_id="t-pause") + run.start() + assert started.wait(_WINDOW) + assert _wait_for_status(run, TaskStatus.RUNNING) + + assert run.pause(timeout=_WINDOW) is True + assert run.status is TaskStatus.PAUSED + frozen = len(progress) + time.sleep(0.15) + assert len(progress) == frozen, "暂停生效后不应继续推进" + + run.resume() + assert run.status is TaskStatus.RUNNING, "就地暂停的任务恢复后回到 RUNNING" + time.sleep(0.15) + assert len(progress) > frozen, "恢复后应继续推进" + + run.stop() + assert run.wait(timeout=_WINDOW).status is TaskStatus.STOPPED + + +def test_pause_returns_false_when_task_never_checkpoints() -> None: + """暂停没生效就必须如实回滚为 RUNNING，不能假装已暂停。""" + + def work(ctx: Any) -> str: + for _ in range(500): + time.sleep(0.01) + return "ignored" + + run = TaskRun(work, task_id="t-nopause", graceful_stop=0.2) + run.start() + assert _wait_for_status(run, TaskStatus.RUNNING) + + assert run.pause(timeout=0.2) is False + assert run.status is TaskStatus.RUNNING + assert any(t.to_status is TaskStatus.PAUSING for t in run.transitions) + assert any( + t.to_status is TaskStatus.RUNNING and t.trigger == "runtime:pause_aborted" + for t in run.transitions + ), "回滚应当留下 PAUSE_ABORTED 记录" + + run.stop() + run.wait(timeout=_WINDOW) + + +def test_pause_on_a_finished_run_is_rejected() -> None: + run = TaskRun(lambda ctx: 1, task_id="t-pause-late") + run.start() + assert run.wait(timeout=_WINDOW).ok + assert run.pause(timeout=0.01) is False + assert run.status is TaskStatus.SUCCEEDED + + +# --------------------------------------------------------------------------- +# wait 的契约 +# --------------------------------------------------------------------------- + + +def test_wait_timeout_raises_and_does_not_change_state() -> None: + """把"我没等到"当成"任务失败"是最常见的误用，因此这里只抛异常。""" + release = threading.Event() + + def work(ctx: Any) -> str: + release.wait(_WINDOW) + return "ok" + + run = TaskRun(work, task_id="t-wait") + run.start() + with pytest.raises(TimeoutError, match="状态未被改变"): + run.wait(timeout=0.2) + assert run.status in (TaskStatus.STARTING, TaskStatus.RUNNING) + + release.set() + assert run.wait(timeout=_WINDOW).ok + + +def test_wait_before_start_is_an_error() -> None: + run = TaskRun(lambda ctx: 1, task_id="t-not-started") + with pytest.raises(RuntimeError, match="start"): + run.wait() + + +def test_starting_twice_is_an_error() -> None: + run = TaskRun(lambda ctx: 1, task_id="t-twice") + run.start() + run.wait(timeout=_WINDOW) + with pytest.raises(RuntimeError, match="已经启动过"): + run.start() + + +# --------------------------------------------------------------------------- +# 执行器可替换 +# --------------------------------------------------------------------------- + + +def test_executor_is_pluggable() -> None: + run = TaskRun( + lambda ctx: "inline", + task_id="t-inline", + executor=InlineExecutor(), + ) + run.start() + outcome = run.wait(timeout=_WINDOW) + assert outcome.ok + assert outcome.value == "inline" + assert outcome.snapshot.executor == "inline" + + +class _RefusingExecutor: + """拉不起来任何任务，只会抛错——进程执行器的"目标不可 pickle"就长这样。""" + + name = "refusing" + + def launch(self, request: Any) -> Any: + del request + raise ConfigurationError("目标不可 pickle：这类任务请改用线程执行器") + + +def test_a_launch_failure_is_recorded_as_the_tasks_own_failure() -> None: + """执行器拉不起来时，必须如实记在**任务**上，而不是让异常逃进监督线程。 + + 逃进去的后果有两条：任务先停在 ``starting``，最后由兜底说成"监督循环内部错误" + ——那等于把一个任务侧的问题指成框架的 bug；而调用方拿到的诊断里也看不到真正 + 该改的东西。 + """ + outcome = run_task( + lambda ctx: "never", + task_id="t-refused", + executor=_RefusingExecutor(), + ) + + assert outcome.status is TaskStatus.FAILED + assert outcome.failure is not None + assert "不可 pickle" in outcome.failure.message + assert "监督循环内部错误" not in outcome.failure.message, "别把任务侧的问题说成框架的" + assert outcome.failure.retryable is False, "配置类错误重试多少次都是同一个结果" diff --git a/tests/unit/test_state_machine.py b/tests/unit/test_state_machine.py new file mode 100644 index 0000000..01819d8 --- /dev/null +++ b/tests/unit/test_state_machine.py @@ -0,0 +1,413 @@ +"""状态机行为测试。 + +覆盖三类东西： + +1. 转移表被**真正强制执行**，而不是只写在文档里； +2. 控制动词语义（stop / pause / resume / retry / reset / force）与设计文档一致； +3. 审计记录与版本号的不变量。 +""" + +from __future__ import annotations + +import pytest + +from chongming_task.core.state_machine import StateMachine, Trigger +from chongming_task.errors import AttemptsExhausted, InvalidTransition +from chongming_task.models import Failure, FailureKind, Progress, RequestedAction +from chongming_task.states import TaskStatus +from chongming_task.testing import FakeClock + + +@pytest.fixture +def clock() -> FakeClock: + return FakeClock() + + +@pytest.fixture +def machine(clock: FakeClock) -> StateMachine: + return StateMachine.create("t1", "抽取", max_attempts=3, clock=clock) + + +def _run_to_running(machine: StateMachine, executor: str = "inline") -> None: + machine.mark_queued() + machine.mark_starting(executor=executor) + machine.mark_running() + + +# --------------------------------------------------------------------------- +# 创建与转移表强制 +# --------------------------------------------------------------------------- + + +def test_create_starts_pending(machine: StateMachine, clock: FakeClock) -> None: + assert machine.status is TaskStatus.PENDING + assert machine.snapshot.created_at == clock.now + assert machine.snapshot.attempt == 0 + assert machine.history == () + + +def test_illegal_transition_raises_and_leaves_state_untouched( + machine: StateMachine, +) -> None: + before = machine.snapshot + with pytest.raises(InvalidTransition): + machine.mark_running() + assert machine.snapshot is before + assert machine.history == () + + +def test_invalid_transition_message_lists_allowed_targets(machine: StateMachine) -> None: + with pytest.raises(InvalidTransition) as info: + machine.mark_succeeded() + assert info.value.current is TaskStatus.PENDING + assert info.value.requested is TaskStatus.SUCCEEDED + assert "非法状态转移" in str(info.value) + assert "queued" in str(info.value) + + +def test_happy_path_pending_to_succeeded(machine: StateMachine, clock: FakeClock) -> None: + _run_to_running(machine, executor="thread") + record = machine.mark_succeeded(outcome={"rows": 10}) + + assert record.from_status is TaskStatus.RUNNING + assert record.to_status is TaskStatus.SUCCEEDED + assert machine.snapshot.outcome == {"rows": 10} + assert machine.snapshot.ended_at == clock.now + assert machine.snapshot.attempt == 1 + assert machine.snapshot.validate() == () + assert [t.to_status for t in machine.history] == [ + TaskStatus.QUEUED, + TaskStatus.STARTING, + TaskStatus.RUNNING, + TaskStatus.SUCCEEDED, + ] + + +def test_status_version_increases_by_one_per_transition(machine: StateMachine) -> None: + _run_to_running(machine) + assert [t.version for t in machine.history] == [1, 2, 3] + assert machine.snapshot.status_version == 3 + + +def test_terminal_state_cannot_transition_at_all(machine: StateMachine) -> None: + _run_to_running(machine) + machine.mark_succeeded() + for action in ( + machine.mark_queued, + machine.mark_blocked, + machine.mark_running, + machine.mark_cancelled, + machine.mark_lost, + machine.mark_skipped, + ): + with pytest.raises(InvalidTransition): + action() + assert machine.status is TaskStatus.SUCCEEDED + + +# --------------------------------------------------------------------------- +# 高频旁路更新 +# --------------------------------------------------------------------------- + + +def test_heartbeat_and_progress_do_not_touch_status_version( + machine: StateMachine, clock: FakeClock +) -> None: + """心跳是高频写入；若参与乐观并发，会与状态写入持续冲突。""" + machine.mark_queued() + before = machine.snapshot.status_version + clock.advance(5) + machine.heartbeat() + machine.update_progress(Progress(done=1, total=4)) + assert machine.snapshot.status_version == before + assert machine.snapshot.last_heartbeat_at == clock.now + assert machine.snapshot.progress is not None + assert machine.snapshot.progress.done == 1 + + +def test_transitions_use_injected_clock(machine: StateMachine, clock: FakeClock) -> None: + machine.mark_queued() + queued_at = machine.snapshot.queued_at + clock.advance(30) + machine.mark_starting(executor="inline") + assert machine.history[-1].at == clock.now + assert queued_at != machine.history[-1].at + + +# --------------------------------------------------------------------------- +# 重试预算 +# --------------------------------------------------------------------------- + + +def test_attempt_is_consumed_only_when_starting(machine: StateMachine) -> None: + """预算只在"真正要跑"的那一刻消费。""" + machine.mark_queued() + assert machine.snapshot.attempt == 0 + machine.mark_starting(executor="inline") + assert machine.snapshot.attempt == 1 + + +def test_retry_revives_from_failed_and_keeps_history(clock: FakeClock) -> None: + machine = StateMachine.create("t1", "x", max_attempts=3, clock=clock) + _run_to_running(machine) + machine.mark_failed(Failure(kind=FailureKind.EXCEPTION, message="boom")) + + record = machine.retry() + assert record.trigger == Trigger.RETRY_REQUESTED + assert machine.status is TaskStatus.QUEUED + assert machine.snapshot.failure is None + assert machine.snapshot.ended_at is None + assert machine.snapshot.attempt == 1 # 历史不被改写 + machine.mark_starting(executor="inline") + assert machine.snapshot.attempt == 2 + + +def test_retry_rejected_when_budget_exhausted(clock: FakeClock) -> None: + machine = StateMachine.create("t1", "x", max_attempts=1, clock=clock) + _run_to_running(machine) + machine.mark_failed(Failure(kind=FailureKind.EXCEPTION, message="boom")) + with pytest.raises(AttemptsExhausted): + machine.retry() + + +def test_forced_retry_raises_the_budget(clock: FakeClock) -> None: + """force=True 是"我知道它已失败够多次了，再试一次"。""" + machine = StateMachine.create("t1", "x", max_attempts=1, clock=clock) + _run_to_running(machine) + machine.mark_failed(Failure(kind=FailureKind.EXCEPTION, message="boom")) + + machine.retry(force=True) + assert machine.snapshot.max_attempts == 2 + machine.mark_starting(executor="inline") + assert machine.snapshot.attempt == 2 + + +def test_retry_rejected_from_non_revivable_terminal(clock: FakeClock) -> None: + """SUCCEEDED / SKIPPED 不可复活：前者重跑无意义，后者是策略决定的结果。""" + machine = StateMachine.create("t1", "x", clock=clock) + _run_to_running(machine) + machine.mark_succeeded() + assert not machine.can_retry + with pytest.raises(InvalidTransition): + machine.retry() + + +def test_retry_is_not_the_path_for_retry_wait(clock: FakeClock) -> None: + """RETRY_WAIT 已安排好重试，不应再被 retry() 接受，否则会双倍消费预算。""" + machine = StateMachine.create("t1", "x", max_attempts=3, clock=clock) + _run_to_running(machine) + machine.mark_retry_wait(Failure(kind=FailureKind.EXCEPTION, message="boom")) + with pytest.raises(InvalidTransition): + machine.retry() + + +# --------------------------------------------------------------------------- +# 停止语义 +# --------------------------------------------------------------------------- + + +def test_stop_on_unstarted_task_lands_stopped_without_buffer( + machine: StateMachine, +) -> None: + """还没跑起来的任务应当立即落 STOPPED；显示"停止中"是虚假信息。""" + record = machine.request_stop(actor="ops") + assert record.to_status is TaskStatus.STOPPED + assert machine.snapshot.requested_action is None + assert machine.snapshot.is_terminal + + +def test_stop_on_running_task_goes_through_stopping(machine: StateMachine) -> None: + _run_to_running(machine) + record = machine.request_stop(actor="ops", reason="发布窗口") + assert record.to_status is TaskStatus.STOPPING + assert machine.snapshot.requested_action is RequestedAction.STOP + assert not machine.snapshot.is_terminal + + final = machine.mark_stopped() + assert final.to_status is TaskStatus.STOPPED + assert machine.snapshot.requested_action is None + assert machine.snapshot.validate() == () + + +def test_forced_stop_uses_distinct_trigger(machine: StateMachine) -> None: + _run_to_running(machine, executor="thread") + machine.request_stop(actor="ops") + record = machine.mark_stopped(forced=True, reason="优雅窗口超时") + assert record.trigger == Trigger.STOP_FORCED + assert not record.is_override + + +def test_task_may_still_succeed_after_stop_requested(machine: StateMachine) -> None: + """停止请求发出后任务恰好跑完，是合法结果，不能判为失败。""" + _run_to_running(machine) + machine.request_stop(actor="ops") + record = machine.mark_succeeded() + assert record.to_status is TaskStatus.SUCCEEDED + + +def test_stop_capability_is_false_on_terminal(machine: StateMachine) -> None: + _run_to_running(machine) + machine.mark_succeeded() + assert not machine.can_stop + with pytest.raises(InvalidTransition): + machine.request_stop(actor="ops") + + +# --------------------------------------------------------------------------- +# 暂停语义 +# --------------------------------------------------------------------------- + + +def test_pause_on_unstarted_task_lands_paused_directly(machine: StateMachine) -> None: + """排队中的任务本就静止，pause() 应立即落 PAUSED，不该显示"暂停中"。""" + machine.mark_queued() + record = machine.request_pause(actor="ops") + assert record.to_status is TaskStatus.PAUSED + assert machine.status is TaskStatus.PAUSED + assert machine.snapshot.requested_action is None + assert machine.can_resume + + +def test_pause_while_pausing_is_not_short_circuited(machine: StateMachine) -> None: + """已在 PAUSING 的任务不能被"加急"成 PAUSED：它还没到达检查点。""" + _run_to_running(machine) + machine.request_pause() + assert machine.status is TaskStatus.PAUSING + with pytest.raises(InvalidTransition): + machine.request_pause() + assert machine.status is TaskStatus.PAUSING + + +def test_starting_always_records_started_at(machine: StateMachine) -> None: + """任何到达 RUNNING 的路径都必须带有 started_at（含暂停撤销路径）。""" + machine.mark_queued() + machine.mark_starting(executor="inline") + assert machine.snapshot.started_at is not None + machine.request_pause() + machine.abort_pause(reason="未到达检查点") + assert machine.status is TaskStatus.RUNNING + assert machine.snapshot.validate() == () + + +def test_pause_and_resume_round_trip(machine: StateMachine) -> None: + _run_to_running(machine) + machine.request_pause(actor="ops") + assert machine.snapshot.requested_action is RequestedAction.PAUSE + + machine.mark_paused() + assert machine.status is TaskStatus.PAUSED + + machine.resume() + assert machine.status is TaskStatus.RUNNING + assert machine.snapshot.requested_action is None + + +def test_inplace_resume_returns_to_running_not_queued(machine: StateMachine) -> None: + """就地暂停的任务始终占着执行槽，调用栈还在内存里，恢复就是回到 RUNNING。 + + 我们没有办法把任意的 Python 调用栈换出内存再换回来，所以"暂停后重新排队" + 对就地暂停并不成立——那会导致任务状态与实际执行情况脱节。 + """ + _run_to_running(machine) + machine.request_pause() + machine.mark_paused() + machine.resume() + assert machine.status is TaskStatus.RUNNING + assert machine.snapshot.queued_at is None or machine.snapshot.started_at is not None + + +def test_resume_of_a_never_started_pause_returns_to_queued(machine: StateMachine) -> None: + """排队期间被暂停的任务从未启动过，恢复必须重新排队。""" + machine.mark_queued() + assert machine.request_pause().to_status is TaskStatus.PAUSED + assert machine.snapshot.attempt == 0 + + machine.resume() + assert machine.status is TaskStatus.QUEUED + + +def test_resume_of_a_retry_wait_pause_returns_to_queued(clock: FakeClock) -> None: + machine = StateMachine.create("t1", "x", max_attempts=3, clock=clock) + _run_to_running(machine) + machine.mark_retry_wait(Failure(kind=FailureKind.EXCEPTION, message="boom")) + + assert machine.request_pause().to_status is TaskStatus.PAUSED + machine.resume() + assert machine.status is TaskStatus.QUEUED + + +def test_aborted_pause_reverts_to_running(machine: StateMachine) -> None: + """暂停没能生效时必须如实回滚，而不是假装已暂停、让用户以为资源已释放。""" + _run_to_running(machine, executor="thread") + machine.request_pause() + record = machine.abort_pause(reason="任务未调用 checkpoint") + assert record.to_status is TaskStatus.RUNNING + assert record.trigger == Trigger.PAUSE_ABORTED + assert machine.snapshot.requested_action is None + + +# --------------------------------------------------------------------------- +# 人工干预 +# --------------------------------------------------------------------------- + + +def test_force_transition_requires_audit_fields(machine: StateMachine) -> None: + with pytest.raises(ValueError, match="actor"): + machine.force_transition(TaskStatus.SUCCEEDED, actor=" ", reason="ok") + with pytest.raises(ValueError, match="reason"): + machine.force_transition(TaskStatus.SUCCEEDED, actor="ops", reason="") + + +def test_force_transition_bypasses_table_and_marks_override(machine: StateMachine) -> None: + """PENDING -> SUCCEEDED 在转移表中非法，人工改写必须能通过，但要留痕。""" + record = machine.force_transition( + TaskStatus.SUCCEEDED, actor="ops@example", reason="已人工核验产物" + ) + assert record.to_status is TaskStatus.SUCCEEDED + assert record.is_override + assert record.actor == "ops@example" + assert machine.snapshot.overridden + assert machine.snapshot.validate() == () + + +def test_reset_returns_to_pending_and_clears_attempt_state(clock: FakeClock) -> None: + machine = StateMachine.create("t1", "x", max_attempts=2, clock=clock) + _run_to_running(machine) + machine.mark_failed(Failure(kind=FailureKind.EXCEPTION, message="boom")) + + machine.reset(actor="ops", reason="重新跑一遍") + snapshot = machine.snapshot + assert snapshot.status is TaskStatus.PENDING + assert snapshot.attempt == 0 + assert snapshot.failure is None + assert snapshot.ended_at is None + assert snapshot.started_at is None + assert not snapshot.overridden + + +# --------------------------------------------------------------------------- +# 能力预检（供控制面提前置灰按钮） +# --------------------------------------------------------------------------- + + +def test_capability_prechecks_match_actions(machine: StateMachine) -> None: + assert machine.can_stop + assert not machine.can_pause + assert not machine.can_resume + + _run_to_running(machine) + assert machine.can_stop + assert machine.can_pause + assert not machine.can_resume + + machine.request_pause() + machine.mark_paused() + assert machine.can_resume + assert machine.can_stop + + +def test_lost_is_terminal_but_revivable(machine: StateMachine) -> None: + _run_to_running(machine, executor="thread") + machine.mark_lost(reason="心跳超时") + assert machine.snapshot.is_terminal + assert machine.can_retry diff --git a/tests/unit/test_states.py b/tests/unit/test_states.py new file mode 100644 index 0000000..0c6f0d4 --- /dev/null +++ b/tests/unit/test_states.py @@ -0,0 +1,135 @@ +"""状态表的结构性不变量。 + +这些测试只约束 :mod:`chongming_task.states` 的数据本身，不依赖状态机实现。 +它们的作用是：让"转移表是唯一真相"成为可执行的断言，而不是文档里的承诺。 +""" + +from __future__ import annotations + +import pytest + +from chongming_task.states import ( + NON_TERMINAL, + REVIVABLE, + TERMINAL, + TRANSITIONS, + TaskStatus, + allowed_from, + can_transition, + is_terminal, +) + + +def test_transition_table_is_exhaustive() -> None: + """每个规范状态都必须有表项，否则状态机会在运行期 KeyError。""" + assert set(TRANSITIONS) == set(TaskStatus) + + +def test_alias_does_not_add_a_table_key() -> None: + """``FINISHED`` 只是别名，不应在表中占据独立条目。""" + assert len(TRANSITIONS) == len(set(TaskStatus)) + + +def test_terminal_set_is_derived_from_the_table() -> None: + """终态必须由表推导，避免"表"与"集合"两处维护而不一致。""" + derived = frozenset(status for status, nxt in TRANSITIONS.items() if not nxt) + assert derived == TERMINAL + + +def test_terminal_and_non_terminal_partition_the_enum() -> None: + partition = TERMINAL | NON_TERMINAL + assert not (TERMINAL & NON_TERMINAL) + assert partition == set(TaskStatus) + + +@pytest.mark.parametrize("status", sorted(TERMINAL)) +def test_terminal_states_have_no_outgoing_transitions(status: TaskStatus) -> None: + assert allowed_from(status) == frozenset() + for target in TaskStatus: + assert not can_transition(status, target) + + +@pytest.mark.parametrize("status", sorted(NON_TERMINAL)) +def test_non_terminal_states_have_outgoing_transitions(status: TaskStatus) -> None: + """非终态若无出边，任务会永久卡死。""" + assert allowed_from(status), f"{status} 必须有至少一条出边" + + +def test_no_self_loops() -> None: + """自环会掩盖状态机缺陷，并让 status_version 无意义地增长。""" + for source, targets in TRANSITIONS.items(): + assert source not in targets, f"{source} 存在自环" + + +def test_all_targets_are_known_statuses() -> None: + known = set(TaskStatus) + for targets in TRANSITIONS.values(): + assert targets <= known + + +@pytest.mark.parametrize("start", sorted(NON_TERMINAL)) +def test_every_non_terminal_state_can_reach_a_terminal_state(start: TaskStatus) -> None: + """不存在只能内部互跳、永远收敛不了的死角。""" + seen: set[TaskStatus] = set() + stack: list[TaskStatus] = [start] + while stack: + current = stack.pop() + if current in TERMINAL: + return + if current in seen: + continue + seen.add(current) + stack.extend(allowed_from(current)) + pytest.fail(f"{start} 无法到达任何终态") + + +# --------------------------------------------------------------------------- +# 以下用例锁定设计决策本身，防止后续重构悄悄改变语义 +# --------------------------------------------------------------------------- + + +def test_paused_can_resume_either_in_place_or_back_to_the_queue() -> None: + """恢复目标取决于暂停发生在哪一层： + + * 就地暂停（经 ``PAUSING``）→ ``RUNNING``，槽位从未释放； + * 未启动即暂停 → ``QUEUED``，重新排队。 + """ + assert can_transition(TaskStatus.PAUSED, TaskStatus.RUNNING) + assert can_transition(TaskStatus.PAUSED, TaskStatus.QUEUED) + + +def test_stopping_may_still_succeed() -> None: + """停止请求发出后任务恰好自然结束，是合法结果，不能判为失败。""" + assert can_transition(TaskStatus.STOPPING, TaskStatus.SUCCEEDED) + + +def test_pending_cannot_jump_straight_to_running() -> None: + """PENDING 必须经过 QUEUED/STARTING，不允许绕过并发槽与执行器启动。""" + assert not can_transition(TaskStatus.PENDING, TaskStatus.RUNNING) + + +def test_running_cannot_return_to_queued_directly() -> None: + """RUNNING 回落 QUEUED 只能经由 PAUSED，否则会绕过暂停语义。""" + assert not can_transition(TaskStatus.RUNNING, TaskStatus.QUEUED) + + +def test_revivable_is_a_subset_of_terminal() -> None: + assert REVIVABLE <= TERMINAL + + +def test_success_and_skip_are_not_revivable() -> None: + """成功重跑没有意义；SKIPPED 是策略决定的结果，不应被 retry 复活。""" + assert TaskStatus.SUCCEEDED not in REVIVABLE + assert TaskStatus.SKIPPED not in REVIVABLE + + +def test_legacy_finished_alias() -> None: + assert TaskStatus.FINISHED is TaskStatus.SUCCEEDED + assert "finished" not in {s.value for s in TaskStatus} + + +def test_helpers_agree_with_the_table() -> None: + assert is_terminal(TaskStatus.SUCCEEDED) + assert not is_terminal(TaskStatus.RUNNING) + assert can_transition(TaskStatus.RUNNING, TaskStatus.STOPPING) + assert not can_transition(TaskStatus.SUCCEEDED, TaskStatus.RUNNING) diff --git a/tests/unit/test_task_context.py b/tests/unit/test_task_context.py new file mode 100644 index 0000000..53a81cf --- /dev/null +++ b/tests/unit/test_task_context.py @@ -0,0 +1,439 @@ +"""``@task`` 装饰器与 ``TaskContext`` 的能力测试。 + +三块： + +1. **``@task``**——元数据从装饰器来；不产生全局状态（导入无副作用）； +2. **进度**——进快照、进存储、**不碰版本号**； +3. **上游数据**——管道节点之间真正能传值，且失败的上游不传 ``None``。 +""" + +from __future__ import annotations + +import logging +import time +from typing import Any + +import pytest + +from chongming_task.controller import Controller, TaskRegistry +from chongming_task.decorators import get_task_spec, task +from chongming_task.errors import ConfigurationError +from chongming_task.pipeline import Pipeline +from chongming_task.policies import RetryPolicy, Timeout +from chongming_task.states import TaskStatus +from chongming_task.stores.memory import InMemoryTaskStore + +_WINDOW = 15.0 + + +# --------------------------------------------------------------------------- +# 1. @task +# --------------------------------------------------------------------------- + + +@task +def bare(ctx: Any) -> str: + """裸装饰器。""" + del ctx + return "bare" + + +@task(tags={"io", "nightly"}, retry=RetryPolicy(max_attempts=4), timeout=Timeout(per_attempt=9.0)) +def decorated(ctx: Any) -> str: + """带元数据的任务。 + + 第二行不该出现在描述里。 + """ + del ctx + return "decorated" + + +@task(name="renamed") +def original_name(ctx: Any) -> str: + del ctx + return "renamed" + + +def undecorated(ctx: Any) -> str: + del ctx + return "nope" + + +def test_bare_decorator_works() -> None: + """``@task`` 不带括号也太自然，不支持会让人反复踩空。""" + spec = get_task_spec(bare) + assert spec is not None + assert spec.name == "bare" + assert spec.description == "裸装饰器。" + + +def test_decorator_captures_metadata() -> None: + spec = get_task_spec(decorated) + assert spec is not None + assert spec.tags == frozenset({"io", "nightly"}) + assert spec.retry is not None + assert spec.retry.max_attempts == 4 + assert spec.timeout is not None + assert spec.timeout.per_attempt == 9.0 + + +def test_description_defaults_to_the_first_docstring_line() -> None: + spec = get_task_spec(decorated) + assert spec is not None + assert spec.description == "带元数据的任务。", "只取第一行" + assert "第二行" not in spec.description + + +def test_name_override() -> None: + spec = get_task_spec(original_name) + assert spec is not None + assert spec.name == "renamed" + + +def test_undecorated_function_has_no_spec() -> None: + assert get_task_spec(undecorated) is None + + +def test_decorating_does_not_create_global_state() -> None: + """规格只挂在函数对象上。 + + 全局注册表意味着 **import 即注册**，而 import 可能发生在任何地方（包括测试收集 + 阶段），那种隐式行为极难排查。 + """ + registry = TaskRegistry() + assert registry.names() == () + assert get_task_spec(bare) is not None, "规格依然取得到，但没有登记进任何注册表" + + +def test_registry_registers_decorated_functions() -> None: + registry = TaskRegistry() + registration = registry.register_decorated(decorated) + assert registration.name == "decorated" + assert registration.tags == frozenset({"io", "nightly"}) + assert registration.retry.max_attempts == 4 + + +def test_registry_rejects_an_undecorated_function() -> None: + registry = TaskRegistry() + with pytest.raises(ConfigurationError, match="没有用 @task 装饰过"): + registry.register_decorated(undecorated) + + +def test_register_module_picks_up_every_decorated_function() -> None: + import sys + + module = sys.modules[__name__] + registry = TaskRegistry() + registered = registry.register_module(module) + + names = {item.name for item in registered} + assert {"bare", "decorated", "renamed"} <= names + assert "undecorated" not in names + + +def test_register_module_rejects_a_module_with_nothing_decorated() -> None: + import json + + registry = TaskRegistry() + with pytest.raises(ConfigurationError, match="没有用 @task 装饰过的函数"): + registry.register_module(json) + + +def test_controller_registers_a_module_end_to_end() -> None: + import sys + + controller = Controller(max_parallel=2) + controller.register_module(sys.modules[__name__]) + run = controller.submit("bare") + outcome = controller.wait(run.snapshot.task_id, timeout=_WINDOW) + assert outcome.ok + assert outcome.value == "bare" + controller.close(timeout=_WINDOW) + + +def test_decorated_retry_actually_applies() -> None: + """装饰器上写的重试次数必须真的生效，而不只是躺在元数据里。""" + store = InMemoryTaskStore() + controller = Controller(store=store, max_parallel=2) + registry = TaskRegistry() + registry.register( + "flaky", + _flaky, + retry=RetryPolicy(max_attempts=3, backoff=_no_backoff()), + ) + controller.register("flaky", _flaky, retry=RetryPolicy(max_attempts=3, backoff=_no_backoff())) + run = controller.submit("flaky") + outcome = controller.wait(run.snapshot.task_id, timeout=_WINDOW) + assert outcome.ok, "第三次应成功" + snapshot = store.get(run.snapshot.task_id) + assert snapshot is not None + assert snapshot.attempt == 3 + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 2. TaskContext：进度与日志 +# --------------------------------------------------------------------------- + + +def test_progress_reaches_the_snapshot_without_bumping_the_version() -> None: + """进度若参与乐观并发，会与状态写入持续冲突。""" + controller = Controller(max_parallel=2) + controller.register("reporter", _reporter) + run = controller.submit("reporter") + outcome = controller.wait(run.snapshot.task_id, timeout=_WINDOW) + assert outcome.ok + + snapshot = outcome.snapshot + assert snapshot.progress is not None + assert snapshot.progress.done == 3 + assert snapshot.progress.total == 3 + assert snapshot.progress.unit == "rows" + # 状态迁移只有 4 次（queued/starting/running/succeeded），进度不该挤进去。 + assert snapshot.status_version == 4 + + +def test_progress_is_persisted_and_visible_in_queries() -> None: + """进度要经 touch 旁路落到存储，否则看板的 PROGRESS 列永远是空的。""" + store = InMemoryTaskStore() + controller = Controller(store=store, max_parallel=2, command_poll_interval=0.05) + controller.register("slow_reporter", _slow_reporter) + run = controller.submit("slow_reporter") + + assert _wait_until( + lambda: (store.get(run.snapshot.task_id) or _stub()).progress is not None, + timeout=_WINDOW, + ) + persisted = store.get(run.snapshot.task_id) + assert persisted is not None + assert persisted.progress is not None + assert persisted.progress.done >= 1 + + controller.stop(run.snapshot.task_id) + controller.wait_all(timeout=_WINDOW) + controller.close(timeout=_WINDOW) + + +def test_progress_on_an_unsupported_context_is_silently_dropped() -> None: + """没有回写通道时静默丢弃，而不是抛异常。 + + 直接构造上下文（不经过执行器）是测试与工具里的常见用法，不该因此报错。 + """ + from chongming_task.runtime.cancellation import CancellationToken + from chongming_task.runtime.context import RuntimeContext + from chongming_task.runtime.heartbeat import Heartbeat + + context = RuntimeContext( + task_id="t", + name="t", + attempt=1, + token=CancellationToken(), + heartbeat=Heartbeat(), + ) + context.progress(1, total=2) + context.log("hello") + assert list(context.logs) == ["hello"] + + +def test_log_records_into_the_buffer_and_the_stdlib_logger( + caplog: pytest.LogCaptureFixture, +) -> None: + # caplog 默认只捕获 WARNING 及以上，而 INFO 才是我们最常用的级别。 + caplog.set_level(logging.INFO, logger="chongming_task.task") + + controller = Controller(max_parallel=2) + controller.register("talker", _talker) + run = controller.submit("talker") + outcome = controller.wait(run.snapshot.task_id, timeout=_WINDOW) + assert outcome.ok + + assert outcome.value == "done" + messages = [record.message for record in caplog.records] + assert "hello" in messages, "INFO 日志应当走到标准库 logging" + assert "careful" in messages + + record = next(r for r in caplog.records if r.message == "hello") + assert record.task_id == run.snapshot.task_id + assert record.task_name == "talker" + assert record.attempt == 1 + + +def test_task_context_is_the_runtime_context() -> None: + """公开名字与内部名字是**同一个类**。 + + 任务作者只面对一个 ``ctx``；两个类会立刻带来"我该用哪个"的困惑。 + """ + from chongming_task import RuntimeContext, TaskContext + + assert TaskContext is RuntimeContext + + +def test_checkpoint_still_works_alongside_progress() -> None: + controller = Controller(max_parallel=2) + controller.register("stopme", _stopme) + run = controller.submit("stopme") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + controller.stop(run.snapshot.task_id, reason="收工") + outcome = controller.wait(run.snapshot.task_id, timeout=_WINDOW) + assert outcome.status is TaskStatus.STOPPED + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 3. 上游数据（管道传值） +# --------------------------------------------------------------------------- + + +def test_downstream_receives_upstream_values() -> None: + controller = Controller(max_parallel=4) + controller.register("produce", _produce) + controller.register("consume", _consume) + + pipeline = Pipeline("data") + source = pipeline.node("produce", node_id="source") + pipeline.node("consume", node_id="sink", depends_on=[source]) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="d1") + outcome = run.wait(timeout=_WINDOW) + assert outcome.status.value == "succeeded" + assert outcome.nodes["sink"] is TaskStatus.SUCCEEDED + + snapshot = controller.get("d1:sink") + assert snapshot is not None + assert snapshot.outcome == {"value": 42}, "下游应当拿到上游的返回值 42" + controller.close(timeout=_WINDOW) + + +def test_a_failed_upstream_contributes_no_value() -> None: + """失败的上游不塞 ``None``：那会让下游以为"上游返回了 None"。""" + controller = Controller(max_parallel=4) + controller.register("boom", _boom) + controller.register("inspect", _inspect) + + pipeline = Pipeline("missing") + bad = pipeline.node("boom", node_id="bad") + pipeline.node("inspect", node_id="probe", depends_on=[bad], join=_all_settled()) + + run = controller.submit_pipeline(pipeline, pipeline_run_id="d2") + run.wait(timeout=_WINDOW) + + snapshot = controller.get("d2:probe") + assert snapshot is not None + assert snapshot.outcome == {"value": []}, "上游没有值，ctx.upstream 必须是空的" + controller.close(timeout=_WINDOW) + + +def test_input_can_be_read_with_a_default() -> None: + controller = Controller() + controller.register("use_default", _use_default) + run = controller.submit("use_default") + outcome = controller.wait(run.snapshot.task_id, timeout=_WINDOW) + assert outcome.value == "fallback" + controller.close(timeout=_WINDOW) + + +def test_inputs_cannot_be_injected_after_the_task_started() -> None: + controller = Controller(max_parallel=2) + controller.register("stopme", _stopme) + run = controller.submit("stopme") + assert _wait_until(lambda: run.status is TaskStatus.RUNNING) + + with pytest.raises(RuntimeError, match="已经开始执行"): + run.set_inputs({"x": 1}) + + controller.stop(run.snapshot.task_id) + controller.wait_all(timeout=_WINDOW) + controller.close(timeout=_WINDOW) + + +# --------------------------------------------------------------------------- +# 任务体 +# --------------------------------------------------------------------------- + + +def _stub() -> Any: + from chongming_task.models import TaskSnapshot + + return TaskSnapshot(task_id="stub", name="stub") + + +def _no_backoff() -> Any: + from chongming_task.policies import FixedBackoff + + return FixedBackoff(0.0) + + +def _flaky(ctx: Any) -> str: + state = getattr(_flaky, "_count", 0) + 1 + _flaky._count = state # type: ignore[attr-defined] + if state < 3: + raise RuntimeError(f"第 {state} 次失败") + return "ok" + + +def _reporter(ctx: Any) -> str: + for index in range(1, 4): + ctx.checkpoint() + ctx.progress(index, total=3, unit="rows") + return "reported" + + +def _slow_reporter(ctx: Any) -> str: + for index in range(1, 200): + ctx.checkpoint() + ctx.progress(index, total=200, unit="rows") + time.sleep(0.01) + return "done" + + +def _talker(ctx: Any) -> str: + ctx.log("hello") + ctx.warning("careful") + return "done" + + +def _stopme(ctx: Any) -> str: + for _ in range(1_000_000): + ctx.checkpoint() + time.sleep(0.005) + return "never" + + +def _produce(ctx: Any) -> int: + del ctx + return 42 + + +def _consume(ctx: Any) -> int: + value = ctx.input("source") + assert value == 42, f"预期 42，收到 {value!r}" + return int(value) + + +def _inspect(ctx: Any) -> list[str]: + return sorted(ctx.upstream) + + +def _use_default(ctx: Any) -> str: + return str(ctx.input("nobody", "fallback")) + + +def _boom(ctx: Any) -> str: + del ctx + raise RuntimeError("故意失败") + + +def _all_settled() -> Any: + from chongming_task.policies import JoinMode + + return JoinMode.ALL_SETTLED + + +def _wait_until(predicate: Any, timeout: float = _WINDOW) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.01) + return False diff --git a/tests/unit/test_thread_executor.py b/tests/unit/test_thread_executor.py new file mode 100644 index 0000000..dc81ead --- /dev/null +++ b/tests/unit/test_thread_executor.py @@ -0,0 +1,410 @@ +"""线程执行器测试。 + +这些用例全部使用**真实线程与真实时间**：执行器的全部价值就在于线程能不能被 +停下、能不能被暂停、放弃之后进程还能不能退出。用假的线程模型测不出这些。 + +为避免偶发失败，协作路径给足超时（5 秒），而"不配合"的用例刻意把任务做成 +有限时长（约 5 秒后自行结束），这样即使被放弃，残留的守护线程也不会长期占用资源。 +""" + +from __future__ import annotations + +import asyncio +import threading +import time +from typing import Any + +import pytest + +from chongming_task.models import FailureKind +from chongming_task.runtime.executors.base import ExecutionRequest +from chongming_task.runtime.executors.thread import ThreadExecutor +from chongming_task.testing import FakeClock + +_COOPERATIVE_WINDOW = 5.0 +_ABANDON_WINDOW = 0.2 + + +def _request(task_id: str, target: Any, *args: Any) -> ExecutionRequest: + return ExecutionRequest(task_id=task_id, name=task_id, target=target, args=args) + + +# --------------------------------------------------------------------------- +# 上下文注入契约 +# --------------------------------------------------------------------------- + + +def test_context_is_injected_with_the_handle_token_and_metadata() -> None: + """令牌必须是句柄那一个，否则控制面会停错对象。 + + 这也是"令牌由执行器注入、而非被任务闭包捕获"的直接体现——令牌每次尝试才诞生， + 闭包捕获到的只可能是上一次尝试留下的那一个。 + """ + seen: dict[str, Any] = {} + + def work(ctx: Any) -> str: + seen["token"] = ctx.token + seen["task_id"] = ctx.task_id + seen["name"] = ctx.name + seen["attempt"] = ctx.attempt + return "ok" + + handle = ThreadExecutor().launch( + ExecutionRequest(task_id="t-ctx", name="抽取", target=work, attempt=2) + ) + assert handle.wait(timeout=_COOPERATIVE_WINDOW) is not None + assert seen["token"] is handle.token + assert seen["task_id"] == "t-ctx" + assert seen["name"] == "抽取" + assert seen["attempt"] == 2 + + +def test_arguments_are_forwarded_after_the_context() -> None: + def work(ctx: Any, left: int, right: int, *, scale: int = 1) -> int: + del ctx + return (left + right) * scale + + handle = ThreadExecutor().launch( + ExecutionRequest( + task_id="t-args", + name="t-args", + target=work, + args=(2, 3), + kwargs={"scale": 10}, + ) + ) + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert outcome.value == 50 + + +def test_checkpoint_refreshes_the_heartbeat() -> None: + """checkpoint 就是"我还在推进"的信号，心跳绑在它上面是有意的。""" + clock = FakeClock() + observed: list[float] = [] + + def work(ctx: Any) -> str: + for _ in range(3): + clock.advance(30) + observed.append(ctx.heartbeat.idle_seconds()) + ctx.checkpoint() + return "ok" + + handle = ThreadExecutor(clock=clock).launch(_request("t-beat", work)) + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert outcome.ok + assert observed == [30.0, 30.0, 30.0], "每次 checkpoint 都应把静默时长清零" + assert handle.heartbeat.idle_seconds() == 0.0 + + +# --------------------------------------------------------------------------- +# async 任务 +# --------------------------------------------------------------------------- + + +def test_async_target_is_actually_awaited() -> None: + """``async def`` 任务必须真的被执行完，而不是把协程对象当返回值。 + + 后者曾经是静默的：没有存储时它看起来"成功"了，返回值却是个 ``coroutine``； + 有存储时才会以"返回值不可序列化"失败。 + """ + + async def work(ctx: Any) -> str: + await asyncio.sleep(0) + return "done" + + handle = ThreadExecutor().launch(_request("t-async", work)) + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert outcome.ok + assert outcome.value == "done" + + +def test_async_target_cooperates_with_stop() -> None: + """协作式停止对 async 任务同样有效：``await ctx.acheckpoint()``。""" + + async def work(ctx: Any) -> str: + for _ in range(10_000): + await ctx.acheckpoint() + await asyncio.sleep(0.005) + return "never" + + handle = ThreadExecutor().launch(_request("t-async-stop", work)) + outcome = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW) + assert not outcome.ok, "停止请求之后不该报成功" + assert not outcome.abandoned, "它会到达检查点，应当优雅收尾而不是被放弃" + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + + +def test_async_target_failure_is_reported_across_the_loop() -> None: + """协程里抛的异常照常转成结构化失败，不能被事件循环吞掉。""" + + async def work(ctx: Any) -> str: + await asyncio.sleep(0) + raise ValueError("协程里炸了") + + handle = ThreadExecutor().launch(_request("t-async-boom", work)) + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.EXCEPTION + assert "协程里炸了" in outcome.failure.message + + +# --------------------------------------------------------------------------- +# 正常路径 +# --------------------------------------------------------------------------- + + +def test_launch_returns_immediately_without_waiting_for_the_task() -> None: + """调度器契约：launch 必须立刻返回，否则调度循环会被单个任务卡住。""" + release = threading.Event() + + def work(ctx: Any) -> str: + del ctx + release.wait(5) + return "done" + + handle = ThreadExecutor().launch(_request("t-async-launch", work)) + try: + assert handle.running is True + assert handle.wait(timeout=0.05) is None, "任务尚未结束，wait 应返回 None" + finally: + release.set() + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert outcome.ok + assert outcome.value == "done" + + +def test_successful_task_returns_its_value() -> None: + handle = ThreadExecutor().launch(_request("t-ok", lambda ctx: 6 * 7)) + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert outcome.ok + assert outcome.value == 42 + assert handle.running is False + + +def test_failing_task_reports_exception_with_traceback() -> None: + def boom(ctx: Any) -> None: + del ctx + raise ValueError("坏掉了") + + handle = ThreadExecutor().launch(_request("t-boom", boom)) + outcome = handle.wait(timeout=_COOPERATIVE_WINDOW) + assert outcome is not None + assert not outcome.ok + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.EXCEPTION + assert "ValueError" in outcome.failure.message + assert outcome.failure.traceback is not None + assert "boom" in outcome.failure.traceback + assert outcome.failure.retryable is True + + +# --------------------------------------------------------------------------- +# 协作式停止 +# --------------------------------------------------------------------------- + + +def test_stop_completes_cooperatively_when_task_checks_the_token() -> None: + started = threading.Event() + steps: list[int] = [] + + def work(ctx: Any) -> str: + started.set() + for index in range(100_000): + ctx.checkpoint() + steps.append(index) + time.sleep(0.001) + return "ran to completion" + + handle = ThreadExecutor().launch(_request("t-coop", work)) + assert started.wait(_COOPERATIVE_WINDOW) + + outcome = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW, reason="发布窗口") + assert outcome.abandoned is False + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + assert outcome.failure.retryable is False, "取消不是失败，绝不能被重试" + assert handle.running is False + assert len(steps) < 100_000, "任务应当在收到停止请求后提前退出" + + +def test_cancellation_is_not_swallowed_by_a_broad_except_in_the_task_body() -> None: + """这正是 ``TaskCancelled`` 继承 ``BaseException`` 的理由。 + + 若它继承 ``Exception``，下面这种极常见的写法会把停止信号吃掉，任务永远停不掉。 + """ + started = threading.Event() + swallowed: list[BaseException] = [] + + def work(ctx: Any) -> str: + started.set() + try: + while True: + ctx.checkpoint() + time.sleep(0.001) + except Exception as exc: # 用户代码里最常见的写法 + swallowed.append(exc) + return "signal was swallowed" + + handle = ThreadExecutor().launch(_request("t-swallow", work)) + assert started.wait(_COOPERATIVE_WINDOW) + + outcome = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW) + assert swallowed == [], "取消信号被 except Exception 吞掉了" + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + + +def test_stop_is_idempotent() -> None: + def work(ctx: Any) -> str: + del ctx + return "quick" + + handle = ThreadExecutor().launch(_request("t-idem", work)) + first = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW) + second = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW) + assert first.ok + assert second.ok + assert handle.running is False + + +# --------------------------------------------------------------------------- +# 协作式暂停（D6 定案的语义，端到端验证） +# --------------------------------------------------------------------------- + + +def test_pause_suspends_a_running_task_and_resume_continues_it() -> None: + started = threading.Event() + progress: list[int] = [] + + def work(ctx: Any) -> str: + started.set() + for index in range(100_000): + ctx.checkpoint() + progress.append(index) + time.sleep(0.001) + return "ran to completion" + + handle = ThreadExecutor().launch(_request("t-pause", work)) + assert started.wait(_COOPERATIVE_WINDOW) + + assert handle.pause(timeout=_COOPERATIVE_WINDOW) is True + frozen = len(progress) + assert frozen > 0, "暂停前任务应当已经推进过" + time.sleep(0.05) + assert len(progress) == frozen, "暂停生效后任务不应继续推进" + + handle.resume() + time.sleep(0.05) + assert len(progress) > frozen, "恢复后任务应继续推进" + + outcome = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW) + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + + +def test_pause_reports_false_when_task_never_reaches_a_checkpoint() -> None: + """这正是"暂停没生效必须回滚为 RUNNING"的依据：如实返回 False。""" + started = threading.Event() + + def work(ctx: Any) -> str: + del ctx + started.set() + for _ in range(500): + time.sleep(0.01) + return "ignored" + + handle = ThreadExecutor().launch(_request("t-nopause", work)) + assert started.wait(_COOPERATIVE_WINDOW) + assert handle.pause(timeout=0.2) is False + handle.stop(graceful_timeout=_ABANDON_WINDOW) + + +def test_stop_wakes_a_paused_task() -> None: + """暂停中的任务必须能被停止，否则暂停就成了不可撤销的挂起。""" + started = threading.Event() + + def work(ctx: Any) -> str: + started.set() + for _ in range(100_000): + ctx.checkpoint() + time.sleep(0.001) + return "ran to completion" + + handle = ThreadExecutor().launch(_request("t-pause-stop", work)) + assert started.wait(_COOPERATIVE_WINDOW) + assert handle.pause(timeout=_COOPERATIVE_WINDOW) is True + + outcome = handle.stop(graceful_timeout=_COOPERATIVE_WINDOW, reason="运维介入") + assert outcome.abandoned is False + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.CANCELLED + assert "运维介入" in outcome.failure.message + + +# --------------------------------------------------------------------------- +# 无法停止：放弃 +# --------------------------------------------------------------------------- + + +def _uncooperative(ctx: Any) -> str: + """刻意不检查令牌；约 5 秒后自行结束，避免残留线程长期占用资源。""" + del ctx + for _ in range(500): + time.sleep(0.01) + return "ignored the stop request" + + +def test_uncooperative_task_is_abandoned_and_reported_as_lost() -> None: + """线程无法强杀：必须如实上报，不能假装已停止。""" + handle = ThreadExecutor().launch(_request("t-stubborn", _uncooperative)) + outcome = handle.stop(graceful_timeout=_ABANDON_WINDOW) + assert outcome.abandoned is True + assert outcome.ok is False + assert outcome.failure is not None + assert outcome.failure.kind is FailureKind.LOST + assert "无法强制终止" in outcome.failure.message + + +def test_abandoned_task_still_reports_running() -> None: + """它确实还在跑。谎报为 False 会让上层以为资源已经释放。""" + handle = ThreadExecutor().launch(_request("t-still-running", _uncooperative)) + handle.stop(graceful_timeout=_ABANDON_WINDOW) + assert handle.running is True + + +def test_abandoned_outcome_is_sticky() -> None: + """已记为 LOST 的结论不应被后台线程后来的返回悄悄改写，否则审计自相矛盾。""" + handle = ThreadExecutor().launch(_request("t-sticky", _uncooperative)) + first = handle.stop(graceful_timeout=_ABANDON_WINDOW) + assert handle.wait(timeout=_ABANDON_WINDOW) is first + assert handle.stop(graceful_timeout=_ABANDON_WINDOW) is first + + +def test_worker_threads_are_daemon_so_an_abandoned_task_cannot_block_shutdown() -> None: + """``ThreadPoolExecutor`` 的工作线程会在解释器退出时被 join，一个不响应取消 + 的任务会让整个进程无法退出。这里用守护线程正是为了避开这一点。""" + handle = ThreadExecutor().launch(_request("t-daemon", _uncooperative)) + try: + assert handle.thread.daemon is True + assert handle.thread.name == "task-t-daemon" + finally: + handle.stop(graceful_timeout=_ABANDON_WINDOW) + + +@pytest.mark.parametrize("graceful_timeout", [0.0, 0.05]) +def test_zero_or_tiny_graceful_window_still_yields_a_conclusion( + graceful_timeout: float, +) -> None: + """优雅窗口再短也必须给出结论，不能抛异常或永远阻塞。""" + handle = ThreadExecutor().launch(_request(f"t-window-{graceful_timeout}", _uncooperative)) + outcome = handle.stop(graceful_timeout=graceful_timeout) + assert outcome.abandoned is True + assert outcome.failure is not None diff --git a/tests/unit/test_timeouts.py b/tests/unit/test_timeouts.py new file mode 100644 index 0000000..b671790 --- /dev/null +++ b/tests/unit/test_timeouts.py @@ -0,0 +1,154 @@ +"""心跳与超时判定的测试。 + +全部基于 ``FakeClock``：不 sleep、不依赖真实时间，因此既快又不会有偶发失败。 +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import pytest + +from chongming_task.models import FailureKind, TimeoutScope +from chongming_task.policies import Timeout +from chongming_task.runtime.heartbeat import Heartbeat +from chongming_task.runtime.timeouts import ( + Deadline, + is_expired, + next_deadline, + remaining_seconds, +) +from chongming_task.testing import FakeClock + +_T0 = datetime(2026, 1, 1, tzinfo=UTC) + + +# --------------------------------------------------------------------------- +# 心跳 +# --------------------------------------------------------------------------- + + +def test_heartbeat_starts_at_construction() -> None: + clock = FakeClock(_T0) + beat = Heartbeat(clock) + assert beat.last_beat == _T0 + assert beat.idle_seconds() == 0.0 + + +def test_heartbeat_advances_only_on_beat() -> None: + """心跳由 checkpoint 驱动，不额外起线程；没 beat 就不该自己变新。""" + clock = FakeClock(_T0) + beat = Heartbeat(clock) + clock.advance(30) + assert beat.idle_seconds() == 30.0 + beat.beat() + assert beat.last_beat == clock.now + assert beat.idle_seconds() == 0.0 + + +def test_heartbeat_staleness_boundary_is_inclusive() -> None: + clock = FakeClock(_T0) + beat = Heartbeat(clock) + clock.advance(59.999) + assert not beat.is_stale(60) + clock.advance(0.001) + assert beat.is_stale(60) + + +def test_heartbeat_rejects_non_positive_timeout() -> None: + beat = Heartbeat(FakeClock(_T0)) + with pytest.raises(ValueError, match="必须为正数"): + beat.is_stale(0) + + +# --------------------------------------------------------------------------- +# 超时判定 +# --------------------------------------------------------------------------- + + +def test_no_timeout_configured_yields_none() -> None: + assert next_deadline(Timeout()) is None + + +def test_start_scope_applies_only_before_started() -> None: + timeout = Timeout(start=30) + pending = next_deadline(timeout, submitted_at=_T0) + assert pending is not None + assert pending.scope is TimeoutScope.START + assert pending.at == _T0 + timedelta(seconds=30) + + # 一旦开始执行，start 超时即失效。 + assert next_deadline(timeout, submitted_at=_T0, started_at=_T0) is None + + +def test_start_scope_needs_submitted_at() -> None: + assert next_deadline(Timeout(start=30)) is None + + +def test_attempt_scope_measures_from_started_at() -> None: + deadline = next_deadline(Timeout(per_attempt=600), started_at=_T0) + assert deadline is not None + assert deadline.scope is TimeoutScope.ATTEMPT + assert deadline.at == _T0 + timedelta(seconds=600) + + +def test_idle_scope_follows_the_latest_heartbeat() -> None: + """心跳推进后，idle 截止点必须随之顺延——否则长任务会被误杀。""" + timeout = Timeout(idle=60) + first = next_deadline(timeout, started_at=_T0, last_beat_at=_T0) + assert first is not None + assert first.at == _T0 + timedelta(seconds=60) + + later = _T0 + timedelta(seconds=45) + second = next_deadline(timeout, started_at=_T0, last_beat_at=later) + assert second is not None + assert second.at == later + timedelta(seconds=60) + + +def test_idle_scope_requires_the_task_to_have_started() -> None: + """任务还没跑起来就谈不上"卡住"，否则排队时长会被误判成静默超时。""" + assert next_deadline(Timeout(idle=60), last_beat_at=_T0) is None + + +def test_before_start_only_the_start_scope_applies() -> None: + timeout = Timeout(start=10, per_attempt=600, idle=60) + deadline = next_deadline(timeout, submitted_at=_T0, last_beat_at=_T0) + assert deadline is not None + assert deadline.scope is TimeoutScope.START + assert deadline.at == _T0 + timedelta(seconds=10) + + +def test_after_start_attempt_and_idle_compete_and_earliest_wins() -> None: + """开始执行后只剩两档竞争：宽松的一档不能掩盖已经到期的一档。""" + timeout = Timeout(per_attempt=600, idle=60) + + fresh_beat = next_deadline(timeout, started_at=_T0, last_beat_at=_T0) + assert fresh_beat is not None + assert fresh_beat.scope is TimeoutScope.IDLE + + # 心跳落后很久时，attempt 反而先到期。 + late_beat = _T0 + timedelta(seconds=590) + stale_beat = next_deadline(timeout, started_at=_T0, last_beat_at=late_beat) + assert stale_beat is not None + assert stale_beat.scope is TimeoutScope.ATTEMPT + assert stale_beat.at == _T0 + timedelta(seconds=600) + + +def test_deadline_maps_to_failure_kind() -> None: + """超时是失败原因，不是生命周期状态；映射必须稳定。""" + assert Deadline(TimeoutScope.START, _T0).kind is FailureKind.TIMEOUT_START + assert Deadline(TimeoutScope.ATTEMPT, _T0).kind is FailureKind.TIMEOUT_ATTEMPT + assert Deadline(TimeoutScope.IDLE, _T0).kind is FailureKind.TIMEOUT_IDLE + + +def test_is_expired_boundary_is_inclusive() -> None: + deadline = Deadline(TimeoutScope.ATTEMPT, _T0) + assert not is_expired(deadline, now=_T0 - timedelta(microseconds=1)) + assert is_expired(deadline, now=_T0) + + +def test_remaining_seconds_goes_negative_after_expiry() -> None: + """调度器要把负数当成"立刻到期"，而不是交给 sleep——sleep(-1) 会抛异常。""" + deadline = Deadline(TimeoutScope.ATTEMPT, _T0) + assert remaining_seconds(deadline, now=_T0 - timedelta(seconds=5)) == 5.0 + assert remaining_seconds(deadline, now=_T0 + timedelta(seconds=5)) == -5.0 diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000..7adfdd4 --- /dev/null +++ b/uv.lock @@ -0,0 +1,596 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" +resolution-markers = [ + "python_full_version >= '3.15'", + "python_full_version < '3.15'", +] + +[[package]] +name = "ast-serialize" +version = "0.11.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/54/1e/4f6082cdd6e5a29093513e9a3eabc5ed1c5331a9a84386b2fece80a00a48/ast_serialize-0.11.2.tar.gz", hash = "sha256:976a5bd75845d22f4b52905ddf53ab669ef1b14dba7735f5512841a2ef2b5450", size = 954387, upload-time = "2026-09-13T18:48:55.673Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a3/2e/beec3364eef4b01793a676d8cd16e9014c42044a5505000ceae3955e33fa/ast_serialize-0.11.2-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:f6a8dfc5ab204a706f6e5d39c6f77c18c27ef084fa2081803a64a9160ce89277", size = 897089, upload-time = "2026-09-13T18:47:22.69Z" }, + { url = "https://files.pythonhosted.org/packages/6f/d7/ef56443df2891c6ba2c4019c2cb3dcaf97c9948da6d963068e04e8dac6ea/ast_serialize-0.11.2-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:cb073bfa15742699d408ac50f60878383b5665ae1791d1b6799ea6f08633cd77", size = 1235218, upload-time = "2026-09-13T18:47:24.541Z" }, + { url = "https://files.pythonhosted.org/packages/42/8d/cff58d17ba1d0272ff0b7ab5d3bdfcf8f47317eb0f47c001d394bffebf95/ast_serialize-0.11.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1d6ad94edbe93bf1dabc06c9f37d55b898fdabc456aa6d7ced5e23c14f795f32", size = 1216399, upload-time = "2026-09-13T18:47:26.202Z" }, + { url = "https://files.pythonhosted.org/packages/de/d2/a1da7675af5f42335c36e4da6d86ef4fd7168cead18de81df0a2d6faeb1a/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:40b2801cf2221bd922d9f69d2f0ebc373c3db47207315d525b2d87fa161a2af4", size = 1282064, upload-time = "2026-09-13T18:47:27.787Z" }, + { url = "https://files.pythonhosted.org/packages/97/89/5a400a13b2c9c0152ebb5ad45408a3fe5e4e60e325d3ac4e5cf6e915a0cc/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fd666cebd6ab3b3c0fd348a6202c26e18a401ee34293c3804d3472266bc146f6", size = 1285864, upload-time = "2026-09-13T18:47:29.667Z" }, + { url = "https://files.pythonhosted.org/packages/02/b8/80a381c70fd49f0316fb0383c4f9e4c13e81b010b64889bd45898ce8f5f4/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0d01f61352c96370febf6c0dbd488dee9183a731fb2702170da9163ae317cded", size = 1554755, upload-time = "2026-09-13T18:47:31.257Z" }, + { url = "https://files.pythonhosted.org/packages/90/97/dcaa34a32d2db789221c125b3eb10feb5089715fe53d9874d627afc26231/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a0fd40c668b0fa19b8fdb61d9e63d547e2e19cfbfe053a51ef0b6c37070298a8", size = 1301807, upload-time = "2026-09-13T18:47:32.714Z" }, + { url = "https://files.pythonhosted.org/packages/5c/9a/84a22420cb312642d7d31547c644d09a3d101418c6d6b9ef2ec30735cf11/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:efa819d7c14c8e4153dcd84671826331538be7cbe460383fc6386f5eea5bd234", size = 1301941, upload-time = "2026-09-13T18:47:34.418Z" }, + { url = "https://files.pythonhosted.org/packages/20/8a/aa5f3dcf1aed9678c25982f40d366004e3c0cac47bc0c240f6b837dcbb1f/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:a9ffa8a197a721f07a352d0be6185f5b3e6f9aaebfdb66169ed652108531ae3b", size = 1307910, upload-time = "2026-09-13T18:47:36.253Z" }, + { url = "https://files.pythonhosted.org/packages/78/79/91a5102797fe3dc992171382d8579bcb33cbd1424b864ad3117ac43fb3fe/ast_serialize-0.11.2-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:00119a8fb8c1dc0f1fab023f4d8071fa49e3b0208ee54d589fd463c16ab0124e", size = 1356258, upload-time = "2026-09-13T18:47:37.984Z" }, + { url = "https://files.pythonhosted.org/packages/51/52/54eeef9918e187ced417c4363eecea66975314cd5b9c91759eef7f7b714b/ast_serialize-0.11.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:0de02520c11391a026e62987a9aa2c3c2ff01545155059ddf0c4bdf2c5ecbe9f", size = 1459057, upload-time = "2026-09-13T18:47:39.891Z" }, + { url = "https://files.pythonhosted.org/packages/e4/cb/fd84b52b15d42f2423319cffd1fb7f1e9df5d5198e69ab0b449c450254cf/ast_serialize-0.11.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:b4e4558956b6a0fb35e18fba58f7d1810b1f2c0e6b52352572cd5dfb6b4ef33a", size = 1562447, upload-time = "2026-09-13T18:47:41.727Z" }, + { url = "https://files.pythonhosted.org/packages/be/92/9fb34f2e64b84a63cca92fb86bd0847b995a63b67477f44c20502fb60352/ast_serialize-0.11.2-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:6061a54f39e82a9f2cbcb9c268fc441890e4818a6636473caa4f4063254e0750", size = 1556423, upload-time = "2026-09-13T18:47:43.357Z" }, + { url = "https://files.pythonhosted.org/packages/75/0f/c43c44449e7ebc4e83ebd48750088fb06234622faa2d62d2a6dc8970d2a3/ast_serialize-0.11.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:85fbb01e83967a126d71f679f2b9528ef0912cb0854aa1a4657314c34e255b57", size = 1687156, upload-time = "2026-09-13T18:47:44.995Z" }, + { url = "https://files.pythonhosted.org/packages/2b/a7/9e520f4a79b639da9ee20c1e747c3d739329e902fc55ac38065f25419f56/ast_serialize-0.11.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:7aaaffc32905159774a107d3cf33dad59bd41b7a0d1bc9885532186753ee7439", size = 1481008, upload-time = "2026-09-13T18:47:46.602Z" }, + { url = "https://files.pythonhosted.org/packages/48/a8/bdd3989f19de09cffcd8179c131f6741a5a8619705fc75b09541ff61530b/ast_serialize-0.11.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:08eda88a0f290a36c38cab33df8bf7e35eb95bc802ca5beb2c8fcda471a7d10c", size = 1501597, upload-time = "2026-09-13T18:47:48.265Z" }, + { url = "https://files.pythonhosted.org/packages/a5/8a/ca2dce2950875a4ef1d7c298196f803b0adcdb7c15ed0cecc71d84bccd70/ast_serialize-0.11.2-cp314-cp314t-win32.whl", hash = "sha256:76cc294246e60a914326b4ca88c6a5ea89c064906614aaf1537ce82f09e9449f", size = 1119503, upload-time = "2026-09-13T18:47:49.896Z" }, + { url = "https://files.pythonhosted.org/packages/5a/12/3f38e3613d07c46f9f81c5b1352748c6552397cc52825502e2c6ae44c6ea/ast_serialize-0.11.2-cp314-cp314t-win_amd64.whl", hash = "sha256:43b51e6ebe6549bf21416c3c78ee886147b80875a87cc6f69e303dde0d75be0b", size = 1156828, upload-time = "2026-09-13T18:47:51.454Z" }, + { url = "https://files.pythonhosted.org/packages/22/90/f89a4f67428a261daafdb69a0d0132c27933268702d1ba47e0b61c51aff1/ast_serialize-0.11.2-cp314-cp314t-win_arm64.whl", hash = "sha256:8df32ad4ff7843734a6c2f067ee974f6d3109ee5a2c3e1a9d2f79347bd282a9a", size = 1128298, upload-time = "2026-09-13T18:47:53.008Z" }, + { url = "https://files.pythonhosted.org/packages/0b/55/a1962188abf0e62d84d55892bb044347e434711763b9a1d4ad867a70c1be/ast_serialize-0.11.2-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:ab924ba260efd7509492f272d4e236d24564033f20c005d7c63a107c6a76fc85", size = 1235457, upload-time = "2026-09-13T18:47:54.554Z" }, + { url = "https://files.pythonhosted.org/packages/2a/ad/439c2959150718446af76fbe2f4000f35eba9869ef8564f3d9a3d0b1c370/ast_serialize-0.11.2-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:a586be418eb70a9f1396cea29ddac8f4b9bf277fb73ea2340db31e218bc00f32", size = 1215705, upload-time = "2026-09-13T18:47:56.178Z" }, + { url = "https://files.pythonhosted.org/packages/6d/d8/2c6542fc3e7c56a0a25d8d12d034d5a2d2e1900e292567b1c1dca8e83124/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8532f20916fa3189d4d785ef2a62d93c4d651ec9c5bffda66d2fc36898351f34", size = 1282530, upload-time = "2026-09-13T18:47:57.619Z" }, + { url = "https://files.pythonhosted.org/packages/fa/ad/6f6755cd0842db46c3b10b1e4735f14aad78d71dea4753eb46933101711b/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ee732ae167e686d1d3c00f98d7d82b23138304694f0441b14d7ddf9c0f8a921c", size = 1287792, upload-time = "2026-09-13T18:47:59.227Z" }, + { url = "https://files.pythonhosted.org/packages/03/40/5da672f5dd23fb7dc0c884c97711e56a3540f2fe3c4355a81f8beb385911/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:75a1c7f46b9c19fc0ae01ca6fd076301628faa2ed7a8edbd55c6353c483946a3", size = 1557971, upload-time = "2026-09-13T18:48:00.96Z" }, + { url = "https://files.pythonhosted.org/packages/df/c7/2bb25684f697801eb72866fdb94ed5edbff3867ce878b0e542a4a5b9dab9/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2fdf31a0bb85ea2575cc91669f005e6647d2efed491231c4dc1497bc9a5b3aa6", size = 1303230, upload-time = "2026-09-13T18:48:02.337Z" }, + { url = "https://files.pythonhosted.org/packages/d8/85/754681846f26e0ff1da729b1ffe3171e93c22f0aa6ec3cea5b14e3703846/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3b78e6fdef3b06c86ed263e1962fee5a7b9d2d158e738b212d13b2c605ee12f5", size = 1302271, upload-time = "2026-09-13T18:48:03.915Z" }, + { url = "https://files.pythonhosted.org/packages/fb/dc/f5521d8cb44b69095c3982ae3658a12c403e0efa19e51aeb9c8a79dff60c/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:8d62a47714c8bc432b9fabcc29989c815c5da17327d35151f2fd0d85c2a7a5ff", size = 1309529, upload-time = "2026-09-13T18:48:05.562Z" }, + { url = "https://files.pythonhosted.org/packages/73/0d/649182c7fd7c4f782279bed514de2dd67e48a5afecb605a098d64fdc01fd/ast_serialize-0.11.2-cp315-abi3.abi3t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8a5ffa70e76191dcf240d3c43e20c93b3bfd26f54d89148c762d57837f5bcd2c", size = 1356869, upload-time = "2026-09-13T18:48:07.534Z" }, + { url = "https://files.pythonhosted.org/packages/bf/cc/aff4d84c16afa742d13a75384127c7d24594dc8c304f0558a15924fd51af/ast_serialize-0.11.2-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:bfbe47a3a7c368f28836e78b2440a3643ac0ec4c67d9fe53588e1448f0a3d35d", size = 1460006, upload-time = "2026-09-13T18:48:09.162Z" }, + { url = "https://files.pythonhosted.org/packages/94/a7/891cbec2e5e0d7159196159d3ff0646622f3120ff4576c839ac2dd56c719/ast_serialize-0.11.2-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:7f1823275b246f9c7d373be6879e4eec09686948895d4ad083f4b27fd7e4da70", size = 1562935, upload-time = "2026-09-13T18:48:10.978Z" }, + { url = "https://files.pythonhosted.org/packages/45/c4/2c8c4498340ea9aff87a9fd408309aa25d56dd51d7bbddfdb46a3c31424a/ast_serialize-0.11.2-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:57c0f5cb0021a5beb1e5e4d6e840ae2f23a28909703ef4d256a144cc1ad3d437", size = 1557109, upload-time = "2026-09-13T18:48:12.616Z" }, + { url = "https://files.pythonhosted.org/packages/0d/8b/c5d4e5226fa18885fe17f949aee3ab1aeb8389c384d946ec1b7c9489cc94/ast_serialize-0.11.2-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:cd320a5c4f1f2742af97eea22954f776379175c5ef2504801e9a155f2ff9a4d7", size = 1691603, upload-time = "2026-09-13T18:48:14.293Z" }, + { url = "https://files.pythonhosted.org/packages/73/d6/1d2ca472586f9e3416a289a22f36eeb6dd6f47d77b1a4aba358405babbc7/ast_serialize-0.11.2-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:13b13afe32e845c86a573497729e1b7ddeb26c572c78bf50ece51da23b8fad5e", size = 1483053, upload-time = "2026-09-13T18:48:15.789Z" }, + { url = "https://files.pythonhosted.org/packages/0e/16/d3703a7c1e3c76b144ac9349a54d3926d0749918a8dc13a66cede208b8ec/ast_serialize-0.11.2-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:9d80a81ec84660422579bdb8e789f656a794b48c7a1ae1261f6bd8bc1897d17d", size = 1502499, upload-time = "2026-09-13T18:48:17.405Z" }, + { url = "https://files.pythonhosted.org/packages/2b/e4/d974e55c2e247ef26ed1df01c74940583db9a5b3a8bcaad5732c6e2047fb/ast_serialize-0.11.2-cp315-abi3.abi3t-win32.whl", hash = "sha256:af8c003ce721b0099dd55cef4ba733500fc3054ea0cc8565d8957aaf7cccdeb4", size = 1119739, upload-time = "2026-09-13T18:48:19.005Z" }, + { url = "https://files.pythonhosted.org/packages/0d/00/d229443488e095054d5e0c0cc20689a2633b899d735849ff1b2c8e4f0cbf/ast_serialize-0.11.2-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:554d117cb916d8032d85007c654d179efbbfd446174c048062778136a922944f", size = 1158602, upload-time = "2026-09-13T18:48:20.524Z" }, + { url = "https://files.pythonhosted.org/packages/11/75/389fc1a6cfa0c4b2ce522f47d8401329d8bb11732e516d46465960fef1d9/ast_serialize-0.11.2-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:d60515335750d431e462af6e722bb55720a5e7827192777bddfd9c4376065a4d", size = 1128842, upload-time = "2026-09-13T18:48:22.052Z" }, + { url = "https://files.pythonhosted.org/packages/45/7d/c4f36898f19c728d091cdfdf960c9488e8d82bbe2e49ba13c05f43907a5d/ast_serialize-0.11.2-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:89499a439955931281986e97ca4dd3c064bf0d2e0027c0017344eb86667733a1", size = 897204, upload-time = "2026-09-13T18:48:23.695Z" }, + { url = "https://files.pythonhosted.org/packages/b1/54/f67120006fc73a55b6d057d4662d061fbb4eceafce3047c76ca8b382eb11/ast_serialize-0.11.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:daadf1c3e0224621607ffe16f1379e4bd372271ed2e1db8a67878f0bab3ef7e4", size = 1240734, upload-time = "2026-09-13T18:48:25.287Z" }, + { url = "https://files.pythonhosted.org/packages/9a/7e/8f2ab68bddbe58a66fbbaad87beeae3e7d7edddb17263d1fc423936cf34d/ast_serialize-0.11.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1844ed9a487fb3de7325c52ddb33f2918b66b65cd54d3f8d83d23785ffe99fa4", size = 1228053, upload-time = "2026-09-13T18:48:26.788Z" }, + { url = "https://files.pythonhosted.org/packages/d2/1b/8a69ab68f4c1603819f0481d756abdd8caf27cec7f1d77caa71007ebe997/ast_serialize-0.11.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b17869f4ba261a5fa468a753328a548f4dbaf74b4eadae9e28aff66df7f1425b", size = 1292542, upload-time = "2026-09-13T18:48:28.295Z" }, + { url = "https://files.pythonhosted.org/packages/d1/ce/872f2e00f0467c289e483f0a34543463347243a2d0632748d89fcee5e0dc/ast_serialize-0.11.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:feb16d9c2a720e0120c58dd5d6e7b3c7c86b43249b60a3bc212bcb8fa031e2dd", size = 1294791, upload-time = "2026-09-13T18:48:29.969Z" }, + { url = "https://files.pythonhosted.org/packages/3a/82/36277c12af861c64b375c316135d8feffe3f400568463a8d2b2de4c2c4fb/ast_serialize-0.11.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f3109fe4805384effc8d0f8e41fbf875aa8f389af91b4348c1cfb60ea6e4cb82", size = 1567583, upload-time = "2026-09-13T18:48:31.85Z" }, + { url = "https://files.pythonhosted.org/packages/b0/d7/ec643df91cea8bcbcb4e8011d6a8b08e5119b84f9554879f3e3c786d29d1/ast_serialize-0.11.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:abdb3e49ba053c3486ac1263bee9f16cc9a4a8abd9f8c90bfc21e3669f3ad9d1", size = 1312878, upload-time = "2026-09-13T18:48:33.495Z" }, + { url = "https://files.pythonhosted.org/packages/04/6f/4c992cd7841ba589fefb14ddc9aff2f6db7f2a615d4074f9ad04115b5ce0/ast_serialize-0.11.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a7004ba572f09be34342ccb98dcd4bad5707d3d81adc8cb4c3f685d2a2c51bbc", size = 1312642, upload-time = "2026-09-13T18:48:35.294Z" }, + { url = "https://files.pythonhosted.org/packages/d5/e3/22aaa209c231a83cfea004fd67dee7a7a54da3f169c6c460b14b96887385/ast_serialize-0.11.2-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:59c25f47524efa052971b860e128b1add0c94ede7dd16b2962952c85c3582365", size = 1319776, upload-time = "2026-09-13T18:48:36.866Z" }, + { url = "https://files.pythonhosted.org/packages/c1/f7/d4685fb54d10108ce44d3bc893ef670854d61645d47ed96d73524db90c23/ast_serialize-0.11.2-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f3a367e0e05ed2d1b747ceb07aa728a8c204cc008b589127e9bd4f40053d7575", size = 1365324, upload-time = "2026-09-13T18:48:38.412Z" }, + { url = "https://files.pythonhosted.org/packages/42/3a/250643ffad02bda520c50a9a5f02a5d43259a06f34ce393c91761d134d7e/ast_serialize-0.11.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:00bbf1f6669f813b48925b759f7ae4591067d456d443924055cab386e7e0a719", size = 1467653, upload-time = "2026-09-13T18:48:40.348Z" }, + { url = "https://files.pythonhosted.org/packages/c7/da/af66a646b9b7f8fdec95ce83fc7b1fe538b06864bc79bd554ac4fae2e6ea/ast_serialize-0.11.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:ec1c20f89c3e0d83576e3c06f79375ce936266591fe0d5fd969914af3185cbaa", size = 1571914, upload-time = "2026-09-13T18:48:41.968Z" }, + { url = "https://files.pythonhosted.org/packages/34/82/77a9714564b9e8800087a8afec41527c65c39e49282baae2ac847b9c1c6a/ast_serialize-0.11.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:c58bb119b73657fdc5569692f316e1e25ca114bd62f7782eb527c6be438ba3a9", size = 1569862, upload-time = "2026-09-13T18:48:43.701Z" }, + { url = "https://files.pythonhosted.org/packages/65/06/fa77b52f46b9bd6dcd8ff2b880e3781f8c1a316bb1342bc3de92907c6f96/ast_serialize-0.11.2-cp39-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:f739e0b601be7300c5697a2573d9200bd1db74b34ab111ef9537b9d5dcd7f106", size = 1699020, upload-time = "2026-09-13T18:48:45.261Z" }, + { url = "https://files.pythonhosted.org/packages/e1/09/239c83153c7e0798e5867d6909cb06f53dccfef02f6999c8e2e21ecb98c3/ast_serialize-0.11.2-cp39-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:cae5addfbb54cc1d47fe947ef9138e9d83849ed1cbc72b819cf36d96a2315b07", size = 1492869, upload-time = "2026-09-13T18:48:46.922Z" }, + { url = "https://files.pythonhosted.org/packages/2f/eb/6108fb9a43fc7ab5529856e38e33c6e3e064fbfe375fdcbb208c7cd5438d/ast_serialize-0.11.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2fa3be25f7f5351b1b39c9f8a52779b2dbf21199efbae564b4746422e8edca4e", size = 1511621, upload-time = "2026-09-13T18:48:48.667Z" }, + { url = "https://files.pythonhosted.org/packages/8a/82/60367e58ef346a41ebc90d3f28593c1b8f5c2cb5314c7b2bbd98910ee131/ast_serialize-0.11.2-cp39-abi3-win32.whl", hash = "sha256:d70556a2f9230a44c99a655774cde823f056efc34466eabfb4085f0cb1ea9f99", size = 1125873, upload-time = "2026-09-13T18:48:50.661Z" }, + { url = "https://files.pythonhosted.org/packages/23/bf/b419c3205ce1143ba7c69baef4f0ba43c14d8712113bf34f9e0d27d609be/ast_serialize-0.11.2-cp39-abi3-win_amd64.whl", hash = "sha256:b9065dd23131a23b41f5bab3bf4e9b3c350a3fe8e36e8200eded9b729fcea484", size = 1165434, upload-time = "2026-09-13T18:48:52.169Z" }, + { url = "https://files.pythonhosted.org/packages/91/a7/c8bbb2173f7a7131b3b2412035b2d814ab5ef2ce9799bd06f07c451640e4/ast_serialize-0.11.2-cp39-abi3-win_arm64.whl", hash = "sha256:dab599cbdcb7b45b18c41fad746645580b3a24357082b7f0e8921cd373804f27", size = 1136031, upload-time = "2026-09-13T18:48:54.04Z" }, +] + +[[package]] +name = "chongming-task" +version = "0.1.0" +source = { editable = "." } + +[package.optional-dependencies] +postgres = [ + { name = "psycopg", extra = ["binary"] }, + { name = "psycopg-pool" }, +] + +[package.dev-dependencies] +dev = [ + { name = "hypothesis" }, + { name = "mypy" }, + { name = "psycopg", extra = ["binary"] }, + { name = "psycopg-pool" }, + { name = "pytest" }, + { name = "ruff" }, +] + +[package.metadata] +requires-dist = [ + { name = "psycopg", extras = ["binary"], marker = "extra == 'postgres'", specifier = ">=3.2" }, + { name = "psycopg-pool", marker = "extra == 'postgres'", specifier = ">=3.2" }, +] +provides-extras = ["postgres"] + +[package.metadata.requires-dev] +dev = [ + { name = "hypothesis", specifier = ">=6.100" }, + { name = "mypy", specifier = ">=1.11" }, + { name = "psycopg", extras = ["binary"], specifier = ">=3.2" }, + { name = "psycopg-pool", specifier = ">=3.2" }, + { name = "pytest", specifier = ">=8.0" }, + { name = "ruff", specifier = ">=0.6" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "hypothesis" +version = "6.168.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "sortedcontainers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/09/b7/13118bbc45d6d8b9d04e2de779e2a4ff23145ea39efa692b994fb874ca72/hypothesis-6.168.3.tar.gz", hash = "sha256:a43388f9067678fef6e13bdff325b6cfa6961a590498bb37f7ff31589c83bc75", size = 511022, upload-time = "2026-09-28T05:20:58.499Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2b/cd/746a2fc1e5e5ba34f07ea6ee1ad07525a3dde5ff4836db3a7980af49a891/hypothesis-6.168.3-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:d20972ca134e652e9928ecd200966a8a2857adc2812c1d74b12a872e5cd503eb", size = 791536, upload-time = "2026-09-28T05:18:57.162Z" }, + { url = "https://files.pythonhosted.org/packages/c3/0f/7a158e377b69556c8e12c25c8fd811d0103e54c24a12dab1f3b9819f202e/hypothesis-6.168.3-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:eafbec09d3e87d13d8242411d1f5868b5e879f1bd6d95e233e9ff80c527bea1c", size = 787313, upload-time = "2026-09-28T05:20:18.122Z" }, + { url = "https://files.pythonhosted.org/packages/45/c6/7df9104ee359e8fcd77791781c47e70794964ca69673665e0de2a6590f4c/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:73c5627497968cc62d140e9fde1ea21e12cac7b64dc419fea8286cd34ff1ad4e", size = 1124036, upload-time = "2026-09-28T05:20:04.372Z" }, + { url = "https://files.pythonhosted.org/packages/92/0f/8a61715404a73b9a82cb1f976803428d603cdea976a9c30266c72ad6a7ce/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:29dc56e6dc6eeb0aeb08ac463f847279bde2336c4786faf7ee0af4b93cd031a7", size = 1147875, upload-time = "2026-09-28T05:19:15.495Z" }, + { url = "https://files.pythonhosted.org/packages/e4/3f/2b16e95cc3b9a069afa23830a012aefe377ce4a457a227e5a5026eb827ff/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:753bb501f8560d2e321ed3b62596b4496c56f15668b0a0669231ccc3e6c4e80d", size = 1149489, upload-time = "2026-09-28T05:19:18.886Z" }, + { url = "https://files.pythonhosted.org/packages/f6/e7/d7cf6dc068bb02732b2a6e38a35a0dcb7f2137608198fc79d740f69d197c/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:71ab606c472449cb872ef2a7acaec679ee4f651b7f0a477a04ebc85d8933ef8c", size = 1191992, upload-time = "2026-09-28T05:19:21.83Z" }, + { url = "https://files.pythonhosted.org/packages/5f/26/f1e5b25dec15998e8375d722825ec3dae1075cbe7cb1f74221b2312a2e33/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:26928956c54748e4dfa587333741ab750246123b93484955646ff316cca27eef", size = 1169911, upload-time = "2026-09-28T05:19:27.996Z" }, + { url = "https://files.pythonhosted.org/packages/03/36/901234a49147e6bf8a5b9e54b775706a223c2b52befea1106ea2aaf15d48/hypothesis-6.168.3-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:0bdfc53c041b61c854fa3761735736b991bc2bddafee45a361cfe4d43027c1ef", size = 1129406, upload-time = "2026-09-28T05:18:42.632Z" }, + { url = "https://files.pythonhosted.org/packages/55/c1/5fc9ff91ec9fba6bb527833386c66812524b5cc75fe7d4a4f9122ee1c420/hypothesis-6.168.3-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:650528e2b1e2a45e95c2df624d4b9364b8ac5a027ba84e1eea2bc5398dd01dcd", size = 1160399, upload-time = "2026-09-28T05:20:23.911Z" }, + { url = "https://files.pythonhosted.org/packages/9b/d2/49ad1ef5c55547c8abfd0fc571b3493806dda530fd5cb3d8263dbad86a72/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:209dc54cdb1b4d6d7020ad8d09e44c49756361b7183adacea4d6f5705085b595", size = 1299906, upload-time = "2026-09-28T05:18:58.417Z" }, + { url = "https://files.pythonhosted.org/packages/0d/5f/f98a26094c4f462c4215807bed0f9a6b5508b27a3c329bcba1bc8adc7ac7/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:af8ca98cd11dc7f9427bb90483abcd9688d4df2e8e63893a30a2423b027ebb11", size = 1425538, upload-time = "2026-09-28T05:20:49.074Z" }, + { url = "https://files.pythonhosted.org/packages/d0/b8/c5b4406e3a6591e51a42f85469351d1d824e0a54b84167aec9b2353759fe/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:f8122cfdd0bc0ba843063effa22ac310bdebdb3a1319bf1e63f14baa119c72ab", size = 1377084, upload-time = "2026-09-28T05:20:06.618Z" }, + { url = "https://files.pythonhosted.org/packages/e0/01/0d84a8ea469d024c15f602799c0b8410fbe03d99fdb0370449b33c3c916f/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:4bedbb379eab34f792af7ee9a05aae04e9c08bbb52e0f90f5a2110d4fe4b2fbd", size = 1281162, upload-time = "2026-09-28T05:19:42.768Z" }, + { url = "https://files.pythonhosted.org/packages/22/b5/ea6435038de1a795f005cb603531e8ccc053febc50180f3ab55583c9c60e/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:5a953115b9f5c95133ab2d04efffeec96e5658c3207923dca7285f7db3e6bbef", size = 1300433, upload-time = "2026-09-28T05:18:55.634Z" }, + { url = "https://files.pythonhosted.org/packages/6c/78/fa6c77d9f64b6d592e69e582d83cbce7d7e56897faa4762885a2123b0166/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6173558e676ad25ed1e20507fa4024a0d816dd90f715f77006e5a28f19109026", size = 1336273, upload-time = "2026-09-28T05:19:08.273Z" }, + { url = "https://files.pythonhosted.org/packages/f1/56/4acd0d778bab818c9ea336fcba768bf43bc70db7f84cc373579298df4641/hypothesis-6.168.3-cp310-abi3-win32.whl", hash = "sha256:dc66390fb12d80585aa9222bf538ce8b7aa22cf5d118250647355a1c9e8f62f4", size = 678211, upload-time = "2026-09-28T05:19:23.355Z" }, + { url = "https://files.pythonhosted.org/packages/b0/db/0a02b146ad1c30f9716362f2e1a379dfd6b0f7be03ac1e22c9da24c9e2a9/hypothesis-6.168.3-cp310-abi3-win_amd64.whl", hash = "sha256:92325b276360fe86c5bf71a568c0d53a6d140b0de36dcf029f9164a17803bb24", size = 684906, upload-time = "2026-09-28T05:18:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/a3/c0/958deaf726848f96f52250740bf39f13f476b068e22f73e2b358eaa07532/hypothesis-6.168.3-cp310-abi3-win_arm64.whl", hash = "sha256:3cf6f1eeaf41cd8d60cf1f88fde905ca1dd77c906929a507d6ac7f66f2ccba2a", size = 683292, upload-time = "2026-09-28T05:19:57.404Z" }, + { url = "https://files.pythonhosted.org/packages/f4/ab/443208c2988a24eaafbfbd6791b897ad9a8a1dd7f870d5b455d7d15a28ad/hypothesis-6.168.3-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:9029775dc2e25e3e8e596c4306926f0079158353631c7c52c285bbc5c8073c2b", size = 792239, upload-time = "2026-09-28T05:19:59.144Z" }, + { url = "https://files.pythonhosted.org/packages/cf/7b/619c1a76c52de2d3cfd68a2f9876c09464d63097db5bf86a28b62dc37bb1/hypothesis-6.168.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0e1d91530cefdb9e0b6467c203eb509c61a11d70480a37db59cb53a2f9913102", size = 788113, upload-time = "2026-09-28T05:19:17.145Z" }, + { url = "https://files.pythonhosted.org/packages/ea/dd/33841e8833105f850214a486d83740ba52041892ca94e687a05829c0ce83/hypothesis-6.168.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0df00cbe8133aa220308d11fa28e3add996cf5102a39b279c1d711015fd9114b", size = 1124241, upload-time = "2026-09-28T05:18:37.589Z" }, + { url = "https://files.pythonhosted.org/packages/4b/43/c84a7900feced6a420e824ab61d316e4f59a6887d991bc143ecc294a09ed/hypothesis-6.168.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd7f75a2e23288ee82ee965a952473d9c5447c2cbc1afc94be09d2402201774e", size = 1170406, upload-time = "2026-09-28T05:19:46.365Z" }, + { url = "https://files.pythonhosted.org/packages/47/5d/6e2a32b9c3020af7a06b00fe446a2c5155ae76ac468c0b68d7188e524002/hypothesis-6.168.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:bc2b1c37631f2231dd0d077225aa5908fc2bd34a4c3faf475053b0bc2eb24c95", size = 1300339, upload-time = "2026-09-28T05:20:31.968Z" }, + { url = "https://files.pythonhosted.org/packages/f9/78/19f3c9eb9a5c215b9fbde57be18ef7c1c78afa509b460d4e8f86026d741a/hypothesis-6.168.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:73960a58f6efc8cbc57d8ad0647955f7c5b25f3d562e3eaa57afa1c69894f7aa", size = 1336484, upload-time = "2026-09-28T05:19:02.841Z" }, + { url = "https://files.pythonhosted.org/packages/88/bb/d46c4519e989e022667705d8a56e83a09ad4ea50eae07ca4bb0f4708dd23/hypothesis-6.168.3-cp311-cp311-win_amd64.whl", hash = "sha256:3f1122223759acc0fe5301c1ae505d762ee61af91db7b191a791ccdbcd965462", size = 684646, upload-time = "2026-09-28T05:20:00.842Z" }, + { url = "https://files.pythonhosted.org/packages/7a/56/44a17266bcf1add5fba7c3df5c3920396a5e924d917fb0534a6f9e2b9b9a/hypothesis-6.168.3-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:bff12e036b67abc5ded682f724f4d7afaff5223190270a0763695f218c079068", size = 793313, upload-time = "2026-09-28T05:20:22.045Z" }, + { url = "https://files.pythonhosted.org/packages/d8/18/1a25b11e9133cb54c02ee6cf99a61296ad461aabc7aae3399e1d0ff1f095/hypothesis-6.168.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c73c6188056e6dc110260e39d141c69c3d02f23e3ba1d8610c7a8b6f2510ec96", size = 784855, upload-time = "2026-09-28T05:19:34.496Z" }, + { url = "https://files.pythonhosted.org/packages/8b/da/728d3210f0b0089337f37206a100d31ef546d1f99839e709f0a9ca443f8e/hypothesis-6.168.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bb1063cb794765097b8c0add598bd34a904f44dc15db81275dd1f0ed1ac567ed", size = 1123043, upload-time = "2026-09-28T05:19:05.468Z" }, + { url = "https://files.pythonhosted.org/packages/53/86/bfb9ec344a8b3ef2fa6f59bb3d30c5b3a7d86d594107670539c34ee53d1b/hypothesis-6.168.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9b2d47f5d9060b049039bef0b267e88480f6468f95ee394e514e84a81ccdc5f5", size = 1169121, upload-time = "2026-09-28T05:20:40.512Z" }, + { url = "https://files.pythonhosted.org/packages/3b/e0/5a76dbafbcaf8fff075f09377cddefa17289ef05dbf51878dde88385b966/hypothesis-6.168.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8367f623a98cbf90f33fd2a43b100671b152b5975bc343a578290945f52f7018", size = 1298749, upload-time = "2026-09-28T05:19:32.883Z" }, + { url = "https://files.pythonhosted.org/packages/28/15/0e65af96b353f2c4bdc283c53ed0980e5ac5009afe97ef3b23c77b364664/hypothesis-6.168.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fd81241f4cd76fb18d481e87b9688b30a0cb5c30841730513323ba1c7aba1837", size = 1335271, upload-time = "2026-09-28T05:20:54.005Z" }, + { url = "https://files.pythonhosted.org/packages/f5/c5/37eab2155664aa544df321f6d3940b89c6702d79a72fdc98d7b5a2286f8c/hypothesis-6.168.3-cp312-cp312-win_amd64.whl", hash = "sha256:377438de53afb94347d9845b7d90db5905c6d2a8d612108e9938e0f80503bb90", size = 682211, upload-time = "2026-09-28T05:18:51.206Z" }, + { url = "https://files.pythonhosted.org/packages/0c/a2/6787da846d929e52fc3344d803299c45782dbeac287528af08380e984bf8/hypothesis-6.168.3-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1b230a850de63334c16654a34a2d547e0179d36b9071d4439b3e7237f6d077e7", size = 793254, upload-time = "2026-09-28T05:19:36.294Z" }, + { url = "https://files.pythonhosted.org/packages/89/ba/2893f5ca42501f4d562cba3229c8994c3a3e5fac66ef60c1e0e0b5220a5d/hypothesis-6.168.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d63b0226cd3e0d8bdd97c3384b22a21934ed4d53246c1c93575dada616672499", size = 784699, upload-time = "2026-09-28T05:18:54.194Z" }, + { url = "https://files.pythonhosted.org/packages/b7/aa/7d7349daf75b71f6f35876f8de115e974c5c04d600e86df1b2779b0883ab/hypothesis-6.168.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0369f5df055f96e117ab12e5f249668ff144731ab5280bc7a205fdf81f990b89", size = 1123016, upload-time = "2026-09-28T05:20:36.372Z" }, + { url = "https://files.pythonhosted.org/packages/00/f0/7774e1ea072708ea46cb25c4aeee5f9978b6c271764809069e8a6789858e/hypothesis-6.168.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f076bcd0f77fdcdb7797826c879099573d03a02e228ebe649ea917081b962ac0", size = 1168989, upload-time = "2026-09-28T05:20:08.44Z" }, + { url = "https://files.pythonhosted.org/packages/a7/7d/113992abed9efbd7944e3a496a382da58152dce126fddf6419ff31a0a57d/hypothesis-6.168.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:b823ba1fcec8da730f29316d010b06d3f7e0c3828dcf91020e24c55e7d24652a", size = 1298626, upload-time = "2026-09-28T05:19:40.986Z" }, + { url = "https://files.pythonhosted.org/packages/d5/65/3659fa5e733027e5b37a27486e57f4053535e40853d23501c422bb275043/hypothesis-6.168.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dd2849c269d674e4618590f3b48d443bd4c06b5aef3d8d869086c2e6d213d248", size = 1335184, upload-time = "2026-09-28T05:20:20.087Z" }, + { url = "https://files.pythonhosted.org/packages/b4/6c/35aab2221b5ea65125340f236e1a765a9e9f3b28d89a389ea0033fed29f7/hypothesis-6.168.3-cp313-cp313-win_amd64.whl", hash = "sha256:3ef7d26f5789e691401d5f87eafed9bd2763f0dbe47af6d6b66012a509404766", size = 682173, upload-time = "2026-09-28T05:19:12.762Z" }, + { url = "https://files.pythonhosted.org/packages/6e/79/27b0cb56ff5d2bc92458bf6fcebdb1f0dc01f57f043a178e9f9d74498169/hypothesis-6.168.3-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:e2d4c68729a13df9af4998d2652cfb5d541c5609b88c306880a5dfb284938ec2", size = 793287, upload-time = "2026-09-28T05:20:38.522Z" }, + { url = "https://files.pythonhosted.org/packages/e8/bd/5342f95c3bc36586777ef8cb8eba87ac7acf0184210327fb0e6d8654e295/hypothesis-6.168.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b345f818083ec99966a43ca4f7b38feb62c6920ce28572bc2bde4948a8b7eaba", size = 784857, upload-time = "2026-09-28T05:20:42.609Z" }, + { url = "https://files.pythonhosted.org/packages/4a/5f/fc774241518a0588d362680a3084275058aab86d6bc02a2c61e1073142d0/hypothesis-6.168.3-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0608c610fc002978fc5de0f471770d8817e9455cb8649a47983c9f8bccfc1897", size = 1123330, upload-time = "2026-09-28T05:19:29.735Z" }, + { url = "https://files.pythonhosted.org/packages/b0/e6/131f16775a3dca5f4fe27f0d6ad9a9851600098a54a872d163f6ebf0b664/hypothesis-6.168.3-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9dcf6448b1ecc37f2b23f2d1b3ddfc9ff6b6910614c15a642dc82f419b96037b", size = 1169178, upload-time = "2026-09-28T05:19:53.436Z" }, + { url = "https://files.pythonhosted.org/packages/06/61/c9f5bff8b73321c9fd12f2d69666c6e4861b97b60f5f24fdd6ac293007fd/hypothesis-6.168.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ae4f9f094041dcce5119ebd7bab71062743ab02b6e654d056b370beda78c19e2", size = 1299127, upload-time = "2026-09-28T05:20:14.486Z" }, + { url = "https://files.pythonhosted.org/packages/e4/6d/d4618f8ab12dd76c4d58ea052252759aa2bf0c4e71f4b502ba5d577e0b9f/hypothesis-6.168.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:d479985fe73af97badfb72dc6d20c6a353e486a36f6d065f600026e0cf954a86", size = 1335390, upload-time = "2026-09-28T05:20:10.326Z" }, + { url = "https://files.pythonhosted.org/packages/95/b6/727161e17cc297df1aeea945de05c532033b66751625bec192bc5b5ff707/hypothesis-6.168.3-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:785e2c45f8c08e274b4bf1ccae97f1a4e09407e9a68e80790cfab1d17a4a45fa", size = 624291, upload-time = "2026-09-28T05:18:36.376Z" }, + { url = "https://files.pythonhosted.org/packages/77/23/2f9b506392a0b8712440bcb781abc5713be9ffac1303b6d10f3669050882/hypothesis-6.168.3-cp314-cp314-win_amd64.whl", hash = "sha256:320920b1e3dae8611eee8a03d063cf2187446f2a17c38cfb8a7fc1466f71eee2", size = 682064, upload-time = "2026-09-28T05:20:16.336Z" }, + { url = "https://files.pythonhosted.org/packages/50/93/efabfd95eb2b69c0c9fa1e1c83c8290aa2715d46baeb5a7764faf09c133a/hypothesis-6.168.3-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:35380baa981108a7f60c4eab71e46acd8d8f58440520346a1a6aba06dca7e074", size = 791875, upload-time = "2026-09-28T05:19:37.789Z" }, + { url = "https://files.pythonhosted.org/packages/1b/fc/2a0ada1623a9a33048bbf9f00a7c92513398c6b8865cba5efb854262ce30/hypothesis-6.168.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f0aaaed00438fa6673d856aaee12a4a8afbde82a9f3c5ff487cacfb064239afe", size = 783412, upload-time = "2026-09-28T05:18:48.266Z" }, + { url = "https://files.pythonhosted.org/packages/41/31/73c615d37eeb12da0209555612c32ced628d860267c23cfd2691c6207aea/hypothesis-6.168.3-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f27e6df1576bf838e7f484d4ae5cba92114497ca30afb917056bf1ea33e95675", size = 1121662, upload-time = "2026-09-28T05:19:26.225Z" }, + { url = "https://files.pythonhosted.org/packages/92/54/1893344a3b8bbf83fa7c9bb7e96f6836580213854f46671cb749f7b4706c/hypothesis-6.168.3-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3bc85014577982ec6d2e266edc5cd6e7a7d3674c648791ca983c67a52f89a0c4", size = 1167761, upload-time = "2026-09-28T05:18:49.85Z" }, + { url = "https://files.pythonhosted.org/packages/26/97/a61d82968febd25a0f08ffbbfabfaeead66f68fc5fa73f3c9ece1c36fc8f/hypothesis-6.168.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ccfc29505aa1cdcc254cb9cd701fd0811fef5480addd97d4127a00df023410f7", size = 1297309, upload-time = "2026-09-28T05:19:51.711Z" }, + { url = "https://files.pythonhosted.org/packages/bc/01/fe8e4cf6d39d02f4efadaa351427fa13722086cfc54a2da3579741c5177a/hypothesis-6.168.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32d0699566aaa93f9e97a44705d7164386f1e91de78d277f53bada35e78bcc96", size = 1334260, upload-time = "2026-09-28T05:19:00.121Z" }, + { url = "https://files.pythonhosted.org/packages/39/6a/09177ebc62f4778f94379ecade6a51299d3d8c4f4b75ba636bf3a0202364/hypothesis-6.168.3-cp314-cp314t-win_amd64.whl", hash = "sha256:28d88fa174ecbd4ecbd7bb290f06d0db084a3971c3f511ae2830b65b5e25500f", size = 681992, upload-time = "2026-09-28T05:19:48.043Z" }, + { url = "https://files.pythonhosted.org/packages/f9/f6/890bf33d608cd63348d3146a3ca83362fbf09e589c5e5230a00c403070f4/hypothesis-6.168.3-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:61f5782d807b1e6aa5c9beef1054cb2037e7d1add48a64972cd6ee447778781f", size = 791231, upload-time = "2026-09-28T05:18:43.998Z" }, + { url = "https://files.pythonhosted.org/packages/03/dd/fc36b204f8aa7437c676576d9437f0422de46aa8a729e6cd5bbb0224e759/hypothesis-6.168.3-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:7aacf3cf40c7ce8f9e4924d5b57bc0b068beafdfed2347bcf160a28b4cfbba7b", size = 783171, upload-time = "2026-09-28T05:19:20.353Z" }, + { url = "https://files.pythonhosted.org/packages/5c/d2/3cd087d577db67c404991d7723e64d7cd75570cdc82a0f7329aed6f2086c/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:01768a03a30dc54df7fe457c0b34016c84d598fa00d5965ededab93ba4eb3408", size = 1121152, upload-time = "2026-09-28T05:18:39.095Z" }, + { url = "https://files.pythonhosted.org/packages/42/6f/a05a68cc66e7a22701c50bf94fb9ddeb36f4f5d0de3cc130066709eda621/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1a8a4ffc6c6e6e577f2bfbcfebf7cffbb310283ba2532c729570a0a752cebaaa", size = 1144069, upload-time = "2026-09-28T05:20:44.705Z" }, + { url = "https://files.pythonhosted.org/packages/4a/f4/fdd7a093fb2fb3a0ce24256c92ad5bf67936f4669bd06d5ceae4298150ec/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b987d73eba95183a7e59cca6d1925c588aa4307922d9852cc2b4d282a2ae4128", size = 1146692, upload-time = "2026-09-28T05:20:25.83Z" }, + { url = "https://files.pythonhosted.org/packages/f1/c8/6dbd4377e935505ae4fc8ee4c7b18c69994c4bbaca015447c470218bdbee/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d4569c39bd97d9573e946429ed676f3b55a7c8ed80920addd67d384a47d59b38", size = 1189480, upload-time = "2026-09-28T05:20:46.734Z" }, + { url = "https://files.pythonhosted.org/packages/ba/bf/ff288b496b690000d2686dcfa7f67855e4c5a6dddb464ed8e4880be83bb2/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6042b8707a4b25bbbfe20b110258fb7549a68951e5525608a8ce06091039e9fc", size = 1167109, upload-time = "2026-09-28T05:19:24.758Z" }, + { url = "https://files.pythonhosted.org/packages/d9/d1/3fee2bc29fc747fabbb27e174515592e390809ca465d3bbbabbcfa3235a8/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:f413629de94d38a7a2ad259697a6752526143cb49c2e2d7bdba19381c80693f6", size = 1126823, upload-time = "2026-09-28T05:18:52.636Z" }, + { url = "https://files.pythonhosted.org/packages/58/6a/585294392fa6a6d9719446a042a777ba98ecca265d9e48cbedc10c88df0f/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f071737e4e775bebba07e319eb645a880d1e0186b4d24bad23255e849d86d483", size = 1155773, upload-time = "2026-09-28T05:20:51.268Z" }, + { url = "https://files.pythonhosted.org/packages/cd/4d/fa283ff79996debf1ff08593e01f8b773eb8211b19b3d6d5d491317a65bf/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:f59a3912858d0609c26054aa1c474847c797937f5e0e1c77f0d3f08032e50f09", size = 1296671, upload-time = "2026-09-28T05:18:46.937Z" }, + { url = "https://files.pythonhosted.org/packages/42/05/98f9c2f628da5afabb6c3e8df5d2d299d2d50bf306b14a6e26d3954c4951/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:f09c05a23a8025dd5cad07c2ed299a46778e72d49ff0dd5bf353bcc0f7dc1580", size = 1422001, upload-time = "2026-09-28T05:19:04.188Z" }, + { url = "https://files.pythonhosted.org/packages/b8/c9/f2109ade29a7ec284d55bca328d784743e8b13e321576fd34ee7e65f99af/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:44ace770bda3a0301739fc5a413c764de790c739df1f1a7218dd049f8594d9f5", size = 1374182, upload-time = "2026-09-28T05:20:27.764Z" }, + { url = "https://files.pythonhosted.org/packages/ce/6d/e05d5f72441564a3bebc71fa155deafd0fd3b015d6014ec8a00edf6e42bb/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:03b131043608f94a2578a2a896a1a72092acb7079815eef5b8513b08447e0e62", size = 1278402, upload-time = "2026-09-28T05:20:56.332Z" }, + { url = "https://files.pythonhosted.org/packages/96/d7/6988a7f1f69c5c530687dce03a32c157e2d878e3a2e32af9269ce016b78a/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:e9784aca26eddfe99b03a0292320db742cc8a74200ef864fdce949f526f973cd", size = 1297770, upload-time = "2026-09-28T05:19:11.387Z" }, + { url = "https://files.pythonhosted.org/packages/a2/28/5bb82b60b836bd2329e1fe01ad94efc2b14dfa806f8e50ed14b795d78770/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:2b52ac363096232bebc2add117e9178d91f4248f4cdb919fd1026b5f86a4bb16", size = 1333977, upload-time = "2026-09-28T05:20:02.483Z" }, + { url = "https://files.pythonhosted.org/packages/8e/7d/d419841b8f65481ea1a50c4ba36f2670d48b8c7a51e4e8586089564cf7f6/hypothesis-6.168.3-cp315-abi3.abi3t-win32.whl", hash = "sha256:d28e3a6b511a74ce37df5274b51f36c2b274fea7365e00b16f0c81e22acd5957", size = 675394, upload-time = "2026-09-28T05:19:55.237Z" }, + { url = "https://files.pythonhosted.org/packages/d8/d2/1de6a2ad100e44817f2e9a8e8ba3eeaf4769621f4f87f2d4966c4971fcc4/hypothesis-6.168.3-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:7b9638789548361a57d984f56619ac694a914c328181d911271618409261ff4a", size = 681699, upload-time = "2026-09-28T05:19:09.957Z" }, + { url = "https://files.pythonhosted.org/packages/c0/79/be3370fd02734d6b1d950183ae9224580d19fa8d356b95348d60f1e3eeb7/hypothesis-6.168.3-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:65d78e4357ec48ed2c67825f06740ee3599be4cfe770a6092bed07108d679ac5", size = 679849, upload-time = "2026-09-28T05:19:06.884Z" }, + { url = "https://files.pythonhosted.org/packages/3f/0e/27960db1d45a94497e1681fb9d7b9140b44da4e655ba20f0d82f95d10230/hypothesis-6.168.3-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:101531b4ccf7fa12965a6b295d10b64c2f8a8fab61e1f9b18c55335d8fb02575", size = 793135, upload-time = "2026-09-28T05:19:14.056Z" }, + { url = "https://files.pythonhosted.org/packages/9a/cb/be08379286c1925fcd651b92ed835f470a83b859b5c484b3431da9d851f5/hypothesis-6.168.3-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:41e0120814de5c3a6b58d8cb27cb11b8873ab33fba130cd09855a27f5a12a1ca", size = 788995, upload-time = "2026-09-28T05:20:34.164Z" }, + { url = "https://files.pythonhosted.org/packages/38/7b/6d885d139b2ed76e2f4908cf10eae56acc73bf05cbef286357c4559c9a04/hypothesis-6.168.3-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9d120009d145697909f2f0c532f2d568558ddd6990707f8cba6576bfb9770696", size = 1124974, upload-time = "2026-09-28T05:19:49.903Z" }, + { url = "https://files.pythonhosted.org/packages/05/cc/b302ae809b48254830cae57801d0af0b66783df515b5dd56b9f21c8dfa86/hypothesis-6.168.3-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:818b3d09bba60ef90463e47a4bc87275dba1e8aeb184d3102888440ca7c7837d", size = 1171967, upload-time = "2026-09-28T05:20:12.295Z" }, + { url = "https://files.pythonhosted.org/packages/2c/ab/51031080a6600f5b07adf5f9dfa117345c2adda07a5c24bda523e70bd1e9/hypothesis-6.168.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:9e3c36eba80e089ecfa28ec08049723223cb41d3cfe4d9988169cf98440e402b", size = 685641, upload-time = "2026-09-28T05:19:44.609Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "librt" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/04/f5/9dc696772d241814bacac7880bac32f2930b5a6ebc1f85317b83161a011c/librt-0.16.0.tar.gz", hash = "sha256:ac38d6d8d66bf3d744148dbbc0b8e193e195a51e364ed55e224631f5721891fc", size = 219839, upload-time = "2026-09-29T00:55:32.891Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ba/0e/b04fd2d76b6a78a02286938251b2c119fb36ddece92c76cb4886ef7d44b8/librt-0.16.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:fe4372c52d4849096c6cc1cda2817d293ec51440c890474ed59ef38d46556f18", size = 148058, upload-time = "2026-09-29T00:44:40.903Z" }, + { url = "https://files.pythonhosted.org/packages/a3/fc/e1ed24aadb32d38603c0c07f579d739eb4d90e64e90d525b024e4a999b4d/librt-0.16.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c72c5295a84bd249526da9bdca38f2e176d15c31c13bb0063c5053f4ca023421", size = 153073, upload-time = "2026-09-29T00:44:42.165Z" }, + { url = "https://files.pythonhosted.org/packages/be/38/133ab655409420e0c34a9a9836a7f4e5ae75fba758d7fe177099130d58ba/librt-0.16.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:be56ba9c884143495b517f23fe794ae367d58cd89ea0fdd6d437e3c024a87f9f", size = 493052, upload-time = "2026-09-29T00:44:44.043Z" }, + { url = "https://files.pythonhosted.org/packages/94/42/aaa663bc9421f5bf7bbbe9d3316d7219749698ec7688ac3b519770a89c04/librt-0.16.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:ef46c1a29ffb8c72e882e22618ec618778eacd0578fb22c6e7cf9c11d15f357b", size = 485506, upload-time = "2026-09-29T00:44:45.479Z" }, + { url = "https://files.pythonhosted.org/packages/7b/59/83a6eb5087ba8b45b083859e2085256963e06d4c2531944cdd962670bfb5/librt-0.16.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d3c94211ee0c4f8d649ec06b7c115c0ec4eadb873a0e3154ca15cef3f814b071", size = 498708, upload-time = "2026-09-29T00:44:46.916Z" }, + { url = "https://files.pythonhosted.org/packages/a0/e4/15a8d62b2bc12935080ec7277e2e0b9b233047781eb784a51d3e25af669a/librt-0.16.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:94aed6a8308818b91677957d1bd03188869cd7aeb23c5dba7912a6c0402f7602", size = 515981, upload-time = "2026-09-29T00:44:48.316Z" }, + { url = "https://files.pythonhosted.org/packages/46/4f/287cc282ac5dd815204c0779e79401bb0a568c3a1335fb319bf0838fb566/librt-0.16.0-cp311-cp311-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:349c0bcb87ebd07481b6ff781e25cdc699723dbe2212e57dabb27f7a13b7b87d", size = 508670, upload-time = "2026-09-29T00:44:49.726Z" }, + { url = "https://files.pythonhosted.org/packages/bf/5b/ab7292add898ade0d2b5b75d32a923aae315861671aa5c39f5491309f7ae/librt-0.16.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:001bfd59a7d45b17e3e75f2a8c6405280b35e7b84471792778e718c4f368950e", size = 530393, upload-time = "2026-09-29T00:44:51.356Z" }, + { url = "https://files.pythonhosted.org/packages/7a/9e/8e6bcf027d4bcbc16a4e864b3d158a1d3e5f55ffa53af0238020cad89271/librt-0.16.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:28e038895b998d7a0c7798922ce8a1dc157675df5cf1c9ef0aca809ed804b7a1", size = 534302, upload-time = "2026-09-29T00:44:52.893Z" }, + { url = "https://files.pythonhosted.org/packages/96/a4/8c5ab4f3b3a8e905ff8305d76976a2e666f52fe049edd398aed2402beb36/librt-0.16.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:0dbe4096a7ecc00fa835d24510ad8545a4efef738dac96e0e63516783ccde905", size = 536150, upload-time = "2026-09-29T00:44:54.421Z" }, + { url = "https://files.pythonhosted.org/packages/ea/5e/7c4bc267e3cd4b8bd288de9f62d61fff203829c9cc649ea2aa12e9218cf0/librt-0.16.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5cd5b092441053364af968ea12084692cb9d4a22f3ce9524e377880bf028761e", size = 514316, upload-time = "2026-09-29T00:44:55.998Z" }, + { url = "https://files.pythonhosted.org/packages/e0/75/c8bc6b50f43c73819334233f5a3259c81216742cef56d2140c3dd377c3f0/librt-0.16.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3ddeb3c9dedb461bb457c6c7d9aa7fbf35329da313d1a7543d00c8d0f3473c96", size = 557731, upload-time = "2026-09-29T00:44:57.581Z" }, + { url = "https://files.pythonhosted.org/packages/93/fb/e241f91bc8de6c7ac1c6a20ea6dd381bc52312b44d711ffcd97140db1e57/librt-0.16.0-cp311-cp311-win32.whl", hash = "sha256:e05108e0849966f53a8d2d3112a7af881d0efaa479bc735bba91108f9f2350a7", size = 104922, upload-time = "2026-09-29T00:44:59.078Z" }, + { url = "https://files.pythonhosted.org/packages/d1/38/c892383d2bb1a6a80b2ec8ede84970a265a792c56614a06282bf3008517c/librt-0.16.0-cp311-cp311-win_amd64.whl", hash = "sha256:5f49cff01bd608ef7d97104cb035c75455e79c2d70bf4a506cf773338ac1860d", size = 125093, upload-time = "2026-09-29T00:49:01.621Z" }, + { url = "https://files.pythonhosted.org/packages/e4/56/8bbde1ed0b21480163617648f638fdaec9374cf5e9c846ebcee34ec02a42/librt-0.16.0-cp311-cp311-win_arm64.whl", hash = "sha256:1d28ae980ae2218f9c5b95d191e947296f918c9bf0b400d467a9430275bbe678", size = 115910, upload-time = "2026-09-29T00:52:40.131Z" }, + { url = "https://files.pythonhosted.org/packages/ad/76/bbdaeb87b7c47b5c7343e90222b9bfa4e4a8a83f647e02933ad0129225b1/librt-0.16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fe52bf4641069e7978a14253b036cb9002def1926317e710f2e249f8a8c47742", size = 151028, upload-time = "2026-09-29T00:52:41.508Z" }, + { url = "https://files.pythonhosted.org/packages/fd/0c/ab8ed3dab0085931aec4a792c7eaac8dc6c5ff4691fda3a5360d9d8a9cd2/librt-0.16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5bcc2c4726ced915b00de0c9856a4eeabfb3fddb93e10e0b8f735b7709358b6d", size = 155283, upload-time = "2026-09-29T00:52:42.873Z" }, + { url = "https://files.pythonhosted.org/packages/eb/36/494e79d460c80c1f030661e8287c9eca5e1ad652dc2b2180b6cd42abce0a/librt-0.16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ff7baa55f8e7c69851419e50a666015d02a74198716fd45c0125a2112e0a389f", size = 503185, upload-time = "2026-09-29T00:52:44.638Z" }, + { url = "https://files.pythonhosted.org/packages/9b/34/a8464038dd9db6e4381fa2b6eb73dc9a50888d77102c4c139304ac35cddc/librt-0.16.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:b95d5d92ab83d39e760a52091bb1baba664f3a2351e39b1e16801e5747c2f0e9", size = 496738, upload-time = "2026-09-29T00:52:46.441Z" }, + { url = "https://files.pythonhosted.org/packages/6a/53/e0e5e334ef0c6ed27039d323819368b9ef6712be87d55ee2bf9799398afd/librt-0.16.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b6d085d70bce51d43c5c7c36d63490770180d8779e71c49305c87b4213918de7", size = 513818, upload-time = "2026-09-29T00:52:48.068Z" }, + { url = "https://files.pythonhosted.org/packages/ad/f7/7ce72cbf19d0addd05090b339152fd0548a02562c2866a603e6e3b3da2df/librt-0.16.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:36e53948e99bbe3ffea257124cfcae1cfb01831555c9a9c903c9f9a72db7fd07", size = 531952, upload-time = "2026-09-29T00:52:49.816Z" }, + { url = "https://files.pythonhosted.org/packages/83/22/0b1bcb6a8e723c8b4fd60dfc8ae8ec6461c54073fbc8685efeb8d900d407/librt-0.16.0-cp312-cp312-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:54d11f726aae9df5a6ffbbf0a03a52449bbac84a53ef03669cb41cdfd4ae41bf", size = 524465, upload-time = "2026-09-29T00:52:51.354Z" }, + { url = "https://files.pythonhosted.org/packages/64/2e/e9c23b8b9df1813da1be205deca9606beb7ddd033972246cd426d05374a0/librt-0.16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4323193ac0cd025f85af531df8ba91bf24d1973b401697347a6282e8fd3fcf5e", size = 543191, upload-time = "2026-09-29T00:52:53.284Z" }, + { url = "https://files.pythonhosted.org/packages/bc/e5/6a8b21b342c03ed7e230fa3afbfd2edc58e6e87ef1f0d11fa2b9a748c252/librt-0.16.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e42f8e098b9c5396fefa05fb1cc7e33b0e08fc51da106b5de4a45fd22aac6743", size = 546979, upload-time = "2026-09-29T00:52:54.932Z" }, + { url = "https://files.pythonhosted.org/packages/84/9e/b5129023eced1be01e01c22757f53be551d463b1bb7264f787927404c1d6/librt-0.16.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:39ec1d5a14e37baf1450a6cabf03fe552340808bf1ad9d71824ab90117716459", size = 555317, upload-time = "2026-09-29T00:52:56.869Z" }, + { url = "https://files.pythonhosted.org/packages/71/89/28bba5938c725fe91f06bf93f7fa6c6b150229df53a87b454b0d5c2a796e/librt-0.16.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d1aabe3925cbb4a08d15b7b20ba4011b53019da0c4173a25155139b7b1baed65", size = 535846, upload-time = "2026-09-29T00:52:58.475Z" }, + { url = "https://files.pythonhosted.org/packages/22/92/63773026614f888c5d4e370e395ce42ca604b89f70b3acdfedbf94851b80/librt-0.16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:300c3ffdc459f4a779a8411ecb188e3ac0b1ff3a3a7b099642555dedae06c69b", size = 573700, upload-time = "2026-09-29T00:53:00.127Z" }, + { url = "https://files.pythonhosted.org/packages/66/8f/347d4677eefb57cd9f8e01d95a1dcee9a91b4e44e664173c32ff6f3752e5/librt-0.16.0-cp312-cp312-win32.whl", hash = "sha256:c17194318e4c0c0348b36f36c2ec7534436fe0a4c15582403162a4f08c80797a", size = 106158, upload-time = "2026-09-29T00:53:02.031Z" }, + { url = "https://files.pythonhosted.org/packages/f0/2c/5193dc81127cd5ddfad031391b046bf32dda219b38463ab872407ca30646/librt-0.16.0-cp312-cp312-win_amd64.whl", hash = "sha256:25a58a19ea8d83b68209f04912df765e9260635ef77646542ed4b4abe6bc7940", size = 126176, upload-time = "2026-09-29T00:53:03.445Z" }, + { url = "https://files.pythonhosted.org/packages/ff/3d/9668a400c8dd81d162eba38b33fa49fa6205f1a64493570a13fbb815c3ee/librt-0.16.0-cp312-cp312-win_arm64.whl", hash = "sha256:f7be7cf555bc30ec12622e9447299cc4a9b8ff307548b634794353db0c2065dc", size = 116572, upload-time = "2026-09-29T00:53:04.815Z" }, + { url = "https://files.pythonhosted.org/packages/46/cd/ae5e0e9dba45d1399aa04a5395bcc0bead40d9fa06dc903634a7b4d7473d/librt-0.16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c5e6144e68b577f157519f2ba88ca20e3ed61c29b00e5cdfa76cd2d45acf059a", size = 151036, upload-time = "2026-09-29T00:53:06.284Z" }, + { url = "https://files.pythonhosted.org/packages/41/5a/48a16e323c5f9447a94cce7b59babf60fa04e62c3365ecf060c77ed8b320/librt-0.16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:33f41443a1f4e1f099331b3d8120e409fbff84b9760bc1cc9ea496f37ddaa5cc", size = 155211, upload-time = "2026-09-29T00:53:07.72Z" }, + { url = "https://files.pythonhosted.org/packages/3f/29/0f59299eb4251a409b2e690ad4b7d9f8a676db829d7817ec961f32b44f7e/librt-0.16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7e510b7770bee609617a3374a96548eb114cae048023e3f049ee449e7ff2db32", size = 502587, upload-time = "2026-09-29T00:53:09.191Z" }, + { url = "https://files.pythonhosted.org/packages/de/ba/d6fb4ef8d1537c396079d72289f16be7cd35a366e5065c51253fea2760b6/librt-0.16.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:efc49c462d4516b8a58b00b490078fa64689fd1fe66970cc190131d7afb8027e", size = 496146, upload-time = "2026-09-29T00:53:11.016Z" }, + { url = "https://files.pythonhosted.org/packages/52/fc/8c50dd4d7cc97c0ee8f252c8a3104980f234391cf1519b554e8b9de08b60/librt-0.16.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:92caf82ebef5e12d21c72242b70d1e92536f1711cf2a727a4c276de4b4469087", size = 513336, upload-time = "2026-09-29T00:53:12.655Z" }, + { url = "https://files.pythonhosted.org/packages/a9/59/16c409c56f708eda2db9a0553662845d45e3871c77d70a240dae3f3bdc56/librt-0.16.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:17bac7f7a16b328fff77e440287693eb017abde913595b5827ebccbc21ecd8a6", size = 531689, upload-time = "2026-09-29T00:53:14.39Z" }, + { url = "https://files.pythonhosted.org/packages/88/82/d34772a6c29d1446dcca6e64d74062efd508523ab351aa16625a4689d5cc/librt-0.16.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5b976054553670829985ed767feb78fb6bcede0175327c4844dd5c281c1be659", size = 524537, upload-time = "2026-09-29T00:53:16.064Z" }, + { url = "https://files.pythonhosted.org/packages/77/8f/24c5631313746131ccee53bc91fdc8374f9cf25e0082a1fee9c93bb98acc/librt-0.16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0058f9d68721094105917254c72ac0569117bb7b13b9769cf45d26d89f9d21cd", size = 543231, upload-time = "2026-09-29T00:53:17.939Z" }, + { url = "https://files.pythonhosted.org/packages/f2/cb/5f8e0d41dbd8b499c2265e939c31acc9ba59845565bf99539ad1c06aebcf/librt-0.16.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:30b7beaf3f4487b7d8adef1f158b49067cb4d5a19fa7a3bf31a4e7a820e435c5", size = 546499, upload-time = "2026-09-29T00:53:19.666Z" }, + { url = "https://files.pythonhosted.org/packages/be/61/063052de441d1385f59cea4223f184bf9e5d125de1ae3239b490aa1e486e/librt-0.16.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:468df902df016a06eb0e40b0747dc8d14e47d7a38b18b63b1fb167d85cb94d63", size = 555261, upload-time = "2026-09-29T00:53:21.292Z" }, + { url = "https://files.pythonhosted.org/packages/14/11/a2ada0529372268d6401afa9d457a095b68cd7753532b6f7f33049a19b43/librt-0.16.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:aea7b1f2b125dad5de85f049136651bff256c883c65e6b9209b2da0a1ac3cdef", size = 536029, upload-time = "2026-09-29T00:53:23.07Z" }, + { url = "https://files.pythonhosted.org/packages/23/9d/5bb6d38853382986dca702fc7e06c8256d30f5fa0676d882773b744610dc/librt-0.16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a8afb6557920860b7a3a596eb804cf37e09e7cf8a803db2478c202acc72d8c2e", size = 573748, upload-time = "2026-09-29T00:53:24.696Z" }, + { url = "https://files.pythonhosted.org/packages/99/f6/0025cde35ff7f607684dc775a2b2d732cce2561cec050a6ee1fb2e1fc6fe/librt-0.16.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:77c7a2b4fe2c1369e0d5aa1cade26740a7b14be32fbc9a5535d617d20065c39d", size = 79861, upload-time = "2026-09-29T00:53:26.089Z" }, + { url = "https://files.pythonhosted.org/packages/93/93/303b8592909bd583f83f02818ffbea3f7647ca1e22b1cd5465d04b8145fe/librt-0.16.0-cp313-cp313-win32.whl", hash = "sha256:02d89c813d5ff74b17df72d3a34819d132cd168e56b81bf755b809bd9e46b8c4", size = 106236, upload-time = "2026-09-29T00:53:27.43Z" }, + { url = "https://files.pythonhosted.org/packages/cf/24/80bbb463c60ed18e29cb26aba386ddb76609580e4e7160710e550587018c/librt-0.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:14ed6ebe3e4f85f326d7920011ad30ff49ed9334e62cf88caef9ba973d9e3a92", size = 126180, upload-time = "2026-09-29T00:53:28.7Z" }, + { url = "https://files.pythonhosted.org/packages/43/80/b1a6fbdd7da825cdd55c71aa81eb6cfa82c513c360774152eacb50b4a771/librt-0.16.0-cp313-cp313-win_arm64.whl", hash = "sha256:83d4041a3d9b2fd053a8a4e1f22878b3e5833e2712956382d5c048d791454e91", size = 116557, upload-time = "2026-09-29T00:53:30.012Z" }, + { url = "https://files.pythonhosted.org/packages/1e/93/9e0cf7da129a93c3dc7f45bc3cd4a660f2aaa995aa8a6f95c2583ef41239/librt-0.16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:931a0bb0fcac88f263e269e46eb30ba8e21402cd3c62ca40cb97034c0693fab1", size = 149844, upload-time = "2026-09-29T00:53:31.391Z" }, + { url = "https://files.pythonhosted.org/packages/8f/26/8a90d2a8f2b2e471bb486b7aec117b8ae622715ed6c39853aec48ea20073/librt-0.16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1bc17e54e5305f8d40b7ca203671ff5a9e59c1d0f8ea0f625dcca53a3984de11", size = 154111, upload-time = "2026-09-29T00:53:32.718Z" }, + { url = "https://files.pythonhosted.org/packages/35/ce/67abb46258da4d3e42ff5b141db6f38c59357bef84c7979f183b22f924f1/librt-0.16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:877698bf6bca5721d8be345f2fe09778e40ecadea8b58c73075f2b1a53666bf2", size = 494234, upload-time = "2026-09-29T00:53:34.466Z" }, + { url = "https://files.pythonhosted.org/packages/12/f9/ea7162414a16f8f1bbd3b493ad6d22b926c5471916e078350bdbca8c4e5f/librt-0.16.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:5981c011b306781ce561e18e14230a14524a3d8109b97553666c942c18f31a96", size = 491173, upload-time = "2026-09-29T00:53:36.124Z" }, + { url = "https://files.pythonhosted.org/packages/b1/09/9b3e869060dd33f9989b80ba4fea306f6db8cecefcdfe6d7346ef0603f65/librt-0.16.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:afced3dfc17cd805ecf7a3d77996a71cf5f2c75aa66eb0c21a9930f4fc992f86", size = 505547, upload-time = "2026-09-29T00:53:37.686Z" }, + { url = "https://files.pythonhosted.org/packages/50/07/79007d2165f649ea93e08c0962d1d62c70af9cde77965255095bf9d96f9a/librt-0.16.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ca8052401c55d7511dda6760719fda7618067e83535d7d0010096d216c34b667", size = 523057, upload-time = "2026-09-29T00:53:39.301Z" }, + { url = "https://files.pythonhosted.org/packages/61/0c/8fbaff66d0ba376d8864653f5acce1569bae27e89648671f26f7eca67ab8/librt-0.16.0-cp314-cp314-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1e511762a074005bb0aa569166779834e75e438370226930d0ce1866d4b6a33b", size = 515158, upload-time = "2026-09-29T00:53:41.012Z" }, + { url = "https://files.pythonhosted.org/packages/df/2e/23ff0dece76f07a4124413a57682efa0bbeb5765ac0122bc0955513f82ca/librt-0.16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f1e8591bd8a5a628cd7f07954c6a1592359a878bf032957a8e9057a41d644311", size = 534183, upload-time = "2026-09-29T00:53:42.451Z" }, + { url = "https://files.pythonhosted.org/packages/26/c4/e11dea21d9a29486eba78887380374189d472734fa32cc50cb37ca44d3d0/librt-0.16.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4aaefb4ba6c07e1aeebb2795c8958148f1d6f9af3b555b53d23d766edb6d67a", size = 540663, upload-time = "2026-09-29T00:53:44.272Z" }, + { url = "https://files.pythonhosted.org/packages/44/75/e873ae158a8b7f5359be33e7fd6c1fbe02d9a78a3e89a77de6b0e837f476/librt-0.16.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:d92db7a0f6aee44f1baee94750457e8d2d1c6ccea41842de6268d34e8dc7eddd", size = 545908, upload-time = "2026-09-29T00:53:45.812Z" }, + { url = "https://files.pythonhosted.org/packages/ba/36/8939d3f6a93e11bd9592e6fe28d2b44f1c2dc4bed6e22e72359a91e18ffb/librt-0.16.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:378dfaffb38e59c24a87cde5713cd865d51ff7383fa12947f3907f306ea1ca55", size = 523302, upload-time = "2026-09-29T00:53:47.596Z" }, + { url = "https://files.pythonhosted.org/packages/71/14/35309f44a077f0f42ade0e2e7cd88c0cea760c661af205c01ea90d3c0e1f/librt-0.16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:3e0c39bdc85370422e8b637be76eb1fd07d30967551b03e62267dd156f553152", size = 565911, upload-time = "2026-09-29T00:53:49.278Z" }, + { url = "https://files.pythonhosted.org/packages/78/0c/df6255b94967f3159ebc46f08d8e783c12da6ee269ddb74b2efed63c640c/librt-0.16.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:1b384b90ab79a7bc30b566895809a636e0666f21f3cf12b54823d025b7e83839", size = 79119, upload-time = "2026-09-29T00:53:50.938Z" }, + { url = "https://files.pythonhosted.org/packages/6b/44/d30d5a5461378c9d33f36736c6791c3b4c4ba4b1ffe0da7350aedcb2c9a0/librt-0.16.0-cp314-cp314-win32.whl", hash = "sha256:52327da75a94012e7f932f913d20d3876bed3c102be00e6c3e8600ff7bdd58a7", size = 100140, upload-time = "2026-09-29T00:53:52.205Z" }, + { url = "https://files.pythonhosted.org/packages/c2/98/769712f356a1e897df3581bb0c3100375d054a000de26099360ea65b5111/librt-0.16.0-cp314-cp314-win_amd64.whl", hash = "sha256:3f0b8114c44b2ac06ff5dacd08e07e8e807ff4f46083f2a1602685122559be41", size = 120464, upload-time = "2026-09-29T00:53:53.495Z" }, + { url = "https://files.pythonhosted.org/packages/bf/d3/ae2abccc8bdc8b063c1613a77686e17b74e9d3d60cfe6f12c63fa2821b9f/librt-0.16.0-cp314-cp314-win_arm64.whl", hash = "sha256:8caf96a4ef8fb27d0ac0d1ad8337d26a240acd4a02fe4345d0a8f264753e8f99", size = 110889, upload-time = "2026-09-29T00:53:54.817Z" }, + { url = "https://files.pythonhosted.org/packages/e4/26/0737d4be058dd6376eade7dd8b380d4b869b6394cee929393a5a431c45bb/librt-0.16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:953107e2f68d0f3512c48f898b0dbf0ce5cc52bba0f318d847c985dc555ee4cc", size = 159838, upload-time = "2026-09-29T00:53:56.22Z" }, + { url = "https://files.pythonhosted.org/packages/01/96/9bc96531d7c620e9949af904470f02e3fa8f35129ab8e8f281c51eaa3788/librt-0.16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ad37d5b9abd49c9a655dcda7ea52a8a752884062ef1ee71ae17c2f2a0f81fe6a", size = 162018, upload-time = "2026-09-29T00:53:57.532Z" }, + { url = "https://files.pythonhosted.org/packages/0e/fa/b0289dcb186eb3f97221ba00da5f4bd3ba7fa5e99752d48aa8d336615334/librt-0.16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c4aa329c17bd1aaea4f6e89335d8ccd494b3a5830b6654462273e50e11023f0", size = 703983, upload-time = "2026-09-29T00:53:59.024Z" }, + { url = "https://files.pythonhosted.org/packages/d9/ab/05ebbbde7530fc5eeb58fbd1581522a9f64f132fa703c4c595eed6e14760/librt-0.16.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:0ead24d2562a49473dddd9efef8581f020007eb0054389c3ee3ffad38b1ca4c9", size = 683847, upload-time = "2026-09-29T00:54:00.671Z" }, + { url = "https://files.pythonhosted.org/packages/a3/75/f52aeecd4dbadbddf80725ba7de126d8bd5d0eb66247eae17a81ed90dd4d/librt-0.16.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:02118f56a9c36ddd07dfd9b919d9ecc117ba20a90987d56aa4c429fa34509188", size = 698045, upload-time = "2026-09-29T00:54:02.247Z" }, + { url = "https://files.pythonhosted.org/packages/e6/55/fa277a835cd6eb42380591ceb85f48c5b4d2b2d7e2cb9869e1e17d24d237/librt-0.16.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4e29522c62e28595ff7e324c6834ade51127707f0e255b18d1c1cf03d39c1048", size = 724640, upload-time = "2026-09-29T00:54:04.078Z" }, + { url = "https://files.pythonhosted.org/packages/25/e4/2cf64354f3fde8ebd591b3b48f96510bee24ac5f7d1e567adbbe210abdbd/librt-0.16.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3ff4b2367926b69c6215635902cccb04048e73094e9862900d27cb2c6bbff143", size = 731748, upload-time = "2026-09-29T00:54:05.741Z" }, + { url = "https://files.pythonhosted.org/packages/f8/c9/c180af3e94e01aa529fa93d7733fec2abc47f222345400cf21d5481d5f8a/librt-0.16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c6f1b27bf1632a7e016af9f145f82be95e1edd7721a646505c21059257cb5a04", size = 754960, upload-time = "2026-09-29T00:54:07.38Z" }, + { url = "https://files.pythonhosted.org/packages/95/d6/01073aa78c58f356b10d9c57b3fe9abb143df338142bcd316df417b89db0/librt-0.16.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:5696d7f52e7b37217cb3a8f92c744fe835942602fdd4c1a8bc4741d3bfdce15e", size = 747000, upload-time = "2026-09-29T00:54:09.06Z" }, + { url = "https://files.pythonhosted.org/packages/f7/d8/1de3783908658d697a8cfc00582f61299ffba7796d7260c112e4700b1109/librt-0.16.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:6072e92dd876ff6ceeb6cf371e35e51f479349837391341f479b08df4564242b", size = 749569, upload-time = "2026-09-29T00:54:10.702Z" }, + { url = "https://files.pythonhosted.org/packages/b7/32/e817f66c96d6caa8bb8435ff93c4c220624d59efc506be59ab98fcd01d0c/librt-0.16.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:39ca4f2f2fe05de8e63493da592d84311adabe5bef52b193851981da9816b302", size = 729483, upload-time = "2026-09-29T00:54:12.25Z" }, + { url = "https://files.pythonhosted.org/packages/7c/c9/23992ccd2b9d22798fdd0f61353183a47414e651da782ad83eb680f50833/librt-0.16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f9807485a908f00355820f18e91e045ffdcdc5adb68aaec40a1e2b88c5f7bba1", size = 776584, upload-time = "2026-09-29T00:54:13.837Z" }, + { url = "https://files.pythonhosted.org/packages/67/3b/e8af957f08e6e2e8e566b099e43d2748418aa565df6ee1d9d3331209fdfe/librt-0.16.0-cp314-cp314t-win32.whl", hash = "sha256:94be5cb7bca4df6201f4183e9e4fa2086c655283d20b38cd84500a69057575a7", size = 104402, upload-time = "2026-09-29T00:54:15.568Z" }, + { url = "https://files.pythonhosted.org/packages/ca/1c/946e6443d7cd32347a086043395e421e52fd603c9163d2ec970ceab8eed6/librt-0.16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d46ca272b251d033dd4527b0dec5f261a28a52bd5fa0f99c117b0a1f8588cc2d", size = 126227, upload-time = "2026-09-29T00:54:17.165Z" }, + { url = "https://files.pythonhosted.org/packages/c2/a3/bc4f9959d3c62bcbf9e5fd3470a8bbd8bf33224ed4c25d7fba173201b8ac/librt-0.16.0-cp314-cp314t-win_arm64.whl", hash = "sha256:b9d6d4b14e92d876f8026b54c20c445f36425214c1081dc76f74e40db386b82b", size = 116493, upload-time = "2026-09-29T00:54:18.567Z" }, + { url = "https://files.pythonhosted.org/packages/b6/4b/10fdb42dfab4c1533e1570e686b18e86ff4328b406361b39fb3016667638/librt-0.16.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:6fe436af2eaf630474f491af5d032cbe45f93fcff5c3b9fe4ab194a7255b20ff", size = 149829, upload-time = "2026-09-29T00:54:19.933Z" }, + { url = "https://files.pythonhosted.org/packages/8e/30/a90ca13f1d3d91af1680000a4038536907018fbd51764767287a05d28b8b/librt-0.16.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:8ff5d26c529336be9bd7ae04483235d77778ee7d6444a95353102b542601ce81", size = 154522, upload-time = "2026-09-29T00:54:21.306Z" }, + { url = "https://files.pythonhosted.org/packages/5c/dd/bcf364eacfa070bb1fc88d503111ae7177914197ba1fa717c748926e7930/librt-0.16.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:909d8e3c1faee44cb762b1c519ff8613dcc5ceae5c99987a00917b5a31fd1d6a", size = 497593, upload-time = "2026-09-29T00:54:22.798Z" }, + { url = "https://files.pythonhosted.org/packages/18/c1/2c4e81e347bdabfe8346bfc6dc37ae5e154d61c88e30848ec0606025a5e0/librt-0.16.0-cp315-cp315-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:6d4a64283ee61824b5790de882bc68e2d9d7a5143537cb7a966f7354f71646d4", size = 480459, upload-time = "2026-09-29T00:54:24.364Z" }, + { url = "https://files.pythonhosted.org/packages/ab/d7/fef2a3cb8400701be496f6e459876f650b3451f407e30cb243de571ae615/librt-0.16.0-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5810ba811297fdf37a1531a57667cb8ace0842013ca8606bf9eb7c24cf4be154", size = 507814, upload-time = "2026-09-29T00:54:25.997Z" }, + { url = "https://files.pythonhosted.org/packages/e1/6f/53762927a32e9dc9eb1d1c1f3528da281290c86d96929a8af652f671266e/librt-0.16.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e56aaf8c167548dc8e5d6f3bd0f48dcdd299a23c73be3f744aab79d99e9c7f5d", size = 525114, upload-time = "2026-09-29T00:54:27.747Z" }, + { url = "https://files.pythonhosted.org/packages/6c/67/0b9d031f303c4e8c691a9a8ef9d272f13b530c11cc563b819df621b6a348/librt-0.16.0-cp315-cp315-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8f36c58e33b304b525c6c9c5076399c6ebf1109e17b9051a05a407b091b9215b", size = 520465, upload-time = "2026-09-29T00:54:29.331Z" }, + { url = "https://files.pythonhosted.org/packages/ae/d5/2056a3a85864e882eb17a203a10ddb26fa748bc9718ec67e79059ab46cae/librt-0.16.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:242e00b3d4fa37c3d3c1ca5f5c9adb7d909ddb1eac9c41f2787320d00caa0af2", size = 537257, upload-time = "2026-09-29T00:54:30.915Z" }, + { url = "https://files.pythonhosted.org/packages/54/57/e0d79790c163cbc0909e209a6f62e30bb713b647f905a176cdc64848fd4d/librt-0.16.0-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:0253721561787b8df8443eb347b7a6461015354e5bdd37ee38a41fef220d2bb0", size = 527442, upload-time = "2026-09-29T00:54:32.478Z" }, + { url = "https://files.pythonhosted.org/packages/aa/50/1c0c95aba7af51f4752ea34fbf2eb79b36e7cae3a72536248e8c735de735/librt-0.16.0-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:3e483a8d69ede8067db70c0e83007423b6925de6fd53afed01d66160f2e9398c", size = 548139, upload-time = "2026-09-29T00:54:34.183Z" }, + { url = "https://files.pythonhosted.org/packages/98/91/a8a43dd5138d4f55f88846b8f0c85454a4fad69952cebfcff9948f831290/librt-0.16.0-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:69ba927445cfaaffb4081003ef5224c55a5c2ab67ef956f416ef744916e44121", size = 529708, upload-time = "2026-09-29T00:54:35.761Z" }, + { url = "https://files.pythonhosted.org/packages/2d/41/d5226881ab2b7c20d9d587b37bdd4a0ec8775a96d00ca87ac9f385587db4/librt-0.16.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:d6a365f2ab45a984d0e00eee0dd17f599ceab8cadab6ea07b6111c8132fc0e42", size = 567781, upload-time = "2026-09-29T00:54:37.409Z" }, + { url = "https://files.pythonhosted.org/packages/bb/bf/2345ba57a626e8c78c4ddcc724636a8df5a593fe46c1ee76bbf477e32b0d/librt-0.16.0-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:f01f3805f2dae4781c0c34b440e31740d082950bdaf89a6f601ad589a28af57a", size = 79177, upload-time = "2026-09-29T00:54:38.792Z" }, + { url = "https://files.pythonhosted.org/packages/1e/40/99e77936cc9207f629b077bf7cbc5e2f0827cddd7c29792f04ef881bd2c3/librt-0.16.0-cp315-cp315-win32.whl", hash = "sha256:b0e3e721c75d2e79a76d4422c79d7ba705fe1bbafec907037fe7a657a480a0e3", size = 100127, upload-time = "2026-09-29T00:54:40.251Z" }, + { url = "https://files.pythonhosted.org/packages/56/1e/801fe26bc622061b9dfd010e166d94142cb774b733217e6d98d1c0cf2638/librt-0.16.0-cp315-cp315-win_amd64.whl", hash = "sha256:bc02954b1295de798bbdb0b4e2d8a28c2117de8b5c73dcbeb27dc32572dfb971", size = 120480, upload-time = "2026-09-29T00:54:41.722Z" }, + { url = "https://files.pythonhosted.org/packages/bb/a9/d533983055bd36e112627384c2c038845d1df882540b8bcefb566475640b/librt-0.16.0-cp315-cp315-win_arm64.whl", hash = "sha256:c5db585d43449a5f54303d4b2774e45e1babd975cfe1630a3d708c0b80c3e560", size = 110915, upload-time = "2026-09-29T00:54:43.052Z" }, + { url = "https://files.pythonhosted.org/packages/55/fe/d62238fa9c653b0e0613290349467cb65711b5800e8e474f45e942a0ad95/librt-0.16.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:f06c689cb14afd9b612727553a5ec5a40febf113ca41c4413a2b0b334285884b", size = 159714, upload-time = "2026-09-29T00:54:44.497Z" }, + { url = "https://files.pythonhosted.org/packages/f3/ac/f31efe7818700be72ba4f9af8a80fa67c39808c8d26dacc55dc1f6f35172/librt-0.16.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:13b4e8aba90b0b1c82474e9844aa9ffe7ad3faa484350e1da64cb8188d903134", size = 162086, upload-time = "2026-09-29T00:54:45.968Z" }, + { url = "https://files.pythonhosted.org/packages/d9/16/4d7487bf86a9d7e8e18f37ba5538789233ea693637379b75701bd35ec9de/librt-0.16.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a269c46ae327d8e6f8c1f85f7516cb52c0fa48127565a1105a4f4a05ff2a0b4", size = 712727, upload-time = "2026-09-29T00:54:47.486Z" }, + { url = "https://files.pythonhosted.org/packages/77/74/50cd550ccc1a517b9ed62347625c01ddf8c4afad66490312677c011458f6/librt-0.16.0-cp315-cp315t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:a33e0dae1f8592146a4764d54ce842b278732d21a84e17c3bbe6b1bc158a2248", size = 681106, upload-time = "2026-09-29T00:54:49.126Z" }, + { url = "https://files.pythonhosted.org/packages/df/5d/7293f712975ee6fdd2251411fd9ef1c62bc99c7b83ecbe62dbc999b1fab7/librt-0.16.0-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:47ada6ea32636492c61aa8ad27ae3b9404bfe7a97e3ba946d1984236cc741da0", size = 705921, upload-time = "2026-09-29T00:54:50.905Z" }, + { url = "https://files.pythonhosted.org/packages/67/f7/8aab946f11d59d1bece9ffc65d994b200c789a8ca5a95c17e17e609f912a/librt-0.16.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c43bd6e642d8a248c114327f98dd25ac5a7cb5aa168ef02f0559b91874df16b8", size = 731786, upload-time = "2026-09-29T00:54:52.584Z" }, + { url = "https://files.pythonhosted.org/packages/43/76/1c42ab31e7cb8384ebf6d3af607213c495222474f4940443ae7639ab7685/librt-0.16.0-cp315-cp315t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c3d1bb7841a816ace6449bb26d3f9560dbfa20e71c568d23f0f62bf1e68f50b1", size = 744964, upload-time = "2026-09-29T00:54:54.217Z" }, + { url = "https://files.pythonhosted.org/packages/6f/2e/4b19982d933d2dfced671e840b219e4c1cd3f507df6e6dabb51bbd4e3850/librt-0.16.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:3931f7a3db322e7f44e02a280e3949326ce9579ad388ee8d691dc7c76da9fb70", size = 765305, upload-time = "2026-09-29T00:54:55.815Z" }, + { url = "https://files.pythonhosted.org/packages/b8/1b/e872583de2dcb3ac7746e7a2321aeeb168274f3952a87dc66e05ddb29faf/librt-0.16.0-cp315-cp315t-musllinux_1_2_i686.whl", hash = "sha256:f4462528b6000afe8f16907b5c7c2553abf1df005ba5140e6eb394541c3624c3", size = 744959, upload-time = "2026-09-29T00:54:57.532Z" }, + { url = "https://files.pythonhosted.org/packages/10/de/a18c6bcfb297af2674233e90b3c3661c3a0af0f1434a0c966d2ef708a835/librt-0.16.0-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:80039ba9b6a7d5f1a0175a4cca6bbefead87bd854c80abad1cb30afe47a830db", size = 758138, upload-time = "2026-09-29T00:54:59.484Z" }, + { url = "https://files.pythonhosted.org/packages/22/1c/0df1d732539c297bb1a093e1fe3204d2faf76a9022cf4e1d1c2fce059790/librt-0.16.0-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:7a1d272724b581bb6bc769dfdafed6da2ecc9886ba2450311de55a4ac2e1e9cd", size = 744268, upload-time = "2026-09-29T00:55:01.179Z" }, + { url = "https://files.pythonhosted.org/packages/a2/f7/ccaf31331f20c91a5bd9bd48ffc3f743f9c81bb5720cc7b2a720ca204f0a/librt-0.16.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:fbe4fb8c5445f7496d7f7f6bb0807875d09d47e6771ffa175fb2df2895fb86ba", size = 785783, upload-time = "2026-09-29T00:55:02.938Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ea/7421d9e6db894cd6cd788b604b3394a163ded6b942052b6f96c23ccf08e1/librt-0.16.0-cp315-cp315t-win32.whl", hash = "sha256:375bfe6b572a8f6cfc398709356046173bf27e64c4c5edaf5f7062f051fb4bf9", size = 104355, upload-time = "2026-09-29T00:55:04.533Z" }, + { url = "https://files.pythonhosted.org/packages/85/6d/7c31a506eb847bc58aeb2402d58e17605e821f68ab5df5e66fe274a6df41/librt-0.16.0-cp315-cp315t-win_amd64.whl", hash = "sha256:bd3150023d3dc2bc70f3784e59ffa1140d56ddba3d8125b3d6f9f85221279bfc", size = 126134, upload-time = "2026-09-29T00:55:06.017Z" }, + { url = "https://files.pythonhosted.org/packages/36/69/7a5d10ac409c4da0355e054a14371871da9b5557fcc42772cd00181c6cce/librt-0.16.0-cp315-cp315t-win_arm64.whl", hash = "sha256:8ceafb70f2a4f0826f11031942e59c0728fd98da112dc346d4352bde1e486866", size = 116391, upload-time = "2026-09-29T00:55:07.484Z" }, +] + +[[package]] +name = "mypy" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "ast-serialize" }, + { name = "librt", marker = "platform_python_implementation != 'PyPy'" }, + { name = "mypy-extensions" }, + { name = "pathspec" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/34/4e/64300736cf0a0373a27b94a91b664ee7382e36f77b0621bae6381da3e180/mypy-2.4.0.tar.gz", hash = "sha256:77bdaebd452f43fcfc4cc3ba94352a3ea537cd01e3f2d0879f48673d2ec00d6e", size = 4064425, upload-time = "2026-10-01T20:40:39.229Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/26/67/e53965e67d23d48e78dd603aa7525a8d77e7818c23954cc217ccde5cd5c0/mypy-2.4.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5d20e6c7c35fcbf2a0ebdd0eaeacfbc243009dfd33ab7822d54e213912e6dbbd", size = 14315745, upload-time = "2026-10-01T20:39:25.03Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5d/a588ccd57625822d8484d7aff4e9c70f88d042a91c2ec84324a2bc14b3f3/mypy-2.4.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3011537be6cf1de4511c0255a324362a812b58184bbe61e15f59c8b31033bd74", size = 14465652, upload-time = "2026-10-01T20:38:57.19Z" }, + { url = "https://files.pythonhosted.org/packages/05/af/b4978a5566cb829ca563dc08e3ad8d944f116078786ad54ba3dff2bcb64d/mypy-2.4.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a96b07a49b7b1d025ce59c1b3acbcf24bead9a83da4523c4a6bde1bb94e7a0e1", size = 15419180, upload-time = "2026-10-01T20:40:27.808Z" }, + { url = "https://files.pythonhosted.org/packages/b2/8c/52c71f31c54d0a749d832cc73b933c6e38c52d9b270d5503e031c73723dc/mypy-2.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:7c4f8f8d1d1c0e2832d8ee7113dd08f6df6c7aad9e863fcbed9f25832be0b8c4", size = 15669212, upload-time = "2026-10-01T20:39:20.443Z" }, + { url = "https://files.pythonhosted.org/packages/e8/55/ba63121494404fd7d3787ec176ccd02820d6ec125ac4e7c4c0b79f2cc6c7/mypy-2.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:ba05652540bf12828e52abae807b024b09ca144ff4f75e2450a81d69c376425b", size = 11299572, upload-time = "2026-10-01T20:40:03.627Z" }, + { url = "https://files.pythonhosted.org/packages/9c/9c/385c81b5c0d3a2917f077c4da660d0493f9521ccfd4b058e33b860f72d02/mypy-2.4.0-cp311-cp311-win_arm64.whl", hash = "sha256:6306086b87cf7f8a29aa618d9fd9bffb56c59247166b9660fdb54d86d7714ecd", size = 10869751, upload-time = "2026-10-01T20:40:25.483Z" }, + { url = "https://files.pythonhosted.org/packages/68/ed/e5d7cf4017e74a1c1e1c4058ce8f614fc1e3e7606564f47166e22bfc9f95/mypy-2.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e05ff2925d8b37ad26c80c1b9dc43ae5d455da2df1e23c24c095a6425917c57e", size = 14378593, upload-time = "2026-10-01T20:38:48.222Z" }, + { url = "https://files.pythonhosted.org/packages/30/7d/12d994886a922f0f1997becc9c6625198d61eeb5962558a886af7dd38d54/mypy-2.4.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:29243242cf72582b65f9582ad9e56e8cb281566ed3519f4cd70bb8b9f2977e90", size = 14464697, upload-time = "2026-10-01T20:39:11.059Z" }, + { url = "https://files.pythonhosted.org/packages/f3/9e/bcc9af755425ad17790bf11d73c2ea7592914cf309ee1340997670f9d57a/mypy-2.4.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:29eb0b9427a6b11b992e452f6cceb8af724f4dceb47e779d0b35e405e996ea5e", size = 15552257, upload-time = "2026-10-01T20:39:56.269Z" }, + { url = "https://files.pythonhosted.org/packages/af/0c/3343fc4525d6f00d75ad17a93a9f052d5163641cb8911840d4a12cb59ff2/mypy-2.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e3ebe2f72a2a1156065a9851570ffbf50c0a93cdccadef9c6e05c508a4fd10b1", size = 15800443, upload-time = "2026-10-01T20:38:54.815Z" }, + { url = "https://files.pythonhosted.org/packages/8f/0e/69aac6b8159da7c53e6115a2be1bf48e85503402cadbf506dd708e8b2ad7/mypy-2.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:236e0d68f6941992b0811128e652590f590db444ab29ad8f1324765b9298b946", size = 11381574, upload-time = "2026-10-01T20:39:49.406Z" }, + { url = "https://files.pythonhosted.org/packages/6e/d3/d32ce4feb5993eec09d2024bef16cedc9b93701b1446b86092f08b24491b/mypy-2.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:82d0f94c8587ccb472622ee7795280aaa38a06640d5f45b3f16909d6dd86a989", size = 10944729, upload-time = "2026-10-01T20:40:15.203Z" }, + { url = "https://files.pythonhosted.org/packages/44/f2/eb15183c97c69d7cbfac990a6efd33a19ecfd97dab9e714c742fa78a784f/mypy-2.4.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7da85fbcff6dac1abcc636707bed38b45598131fb7a605d9719c70b5cc733af8", size = 14213395, upload-time = "2026-10-01T20:40:21.178Z" }, + { url = "https://files.pythonhosted.org/packages/8c/b5/ba91b6ff65e4d6b6ff53b2b3b3ac5f1babf0c7c27d0b43a0196b1c967926/mypy-2.4.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4209da39d85cf240f762af622d8180fcdfcb4727d021f44ade62d613a1a43324", size = 14373009, upload-time = "2026-10-01T20:39:58.86Z" }, + { url = "https://files.pythonhosted.org/packages/d5/c4/484275efc935c0003e55e4e8a33e4b8e99528ee956c12256ece4708f903f/mypy-2.4.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6be721bd4bd57576193653b75b4af3461c9d0bf7dd8b528f782e9be210dc75bb", size = 15465557, upload-time = "2026-10-01T20:39:01.794Z" }, + { url = "https://files.pythonhosted.org/packages/50/30/66eb6fdd0875e3c9025a02f0bb0ea2e524b74274b658c37fde0068c4939d/mypy-2.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e1fde197ae65be856a034a91b70ed747a16562ca69577785f06c661548424bf1", size = 15728673, upload-time = "2026-10-01T20:38:43.724Z" }, + { url = "https://files.pythonhosted.org/packages/58/bc/2aa98fd7f49c42dba8e9c065886fadffdd00f3c654cff3f2a7103a797ac8/mypy-2.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:295ecf2e57542cd836ca537486951289678c8c7d1ee6ad74ebe29b2168a003cf", size = 11389803, upload-time = "2026-10-01T20:39:37.547Z" }, + { url = "https://files.pythonhosted.org/packages/49/41/17b60df2d946792ef6af43b89351f5c9ddabc69053206b2304945572a744/mypy-2.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:bc378bdad4e9f12b5bd96466083d1e71acf00594ec9c7b2bdb5e02816f77f303", size = 10951060, upload-time = "2026-10-01T20:39:04.015Z" }, + { url = "https://files.pythonhosted.org/packages/e0/66/924be0b653372ed31ad5c48e26044cc00840e591943a56e61621cf05b60e/mypy-2.4.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:058165f564ccf559c68c70fec2091fca5891110480210c22594635e3f6683437", size = 14234230, upload-time = "2026-10-01T20:39:44.535Z" }, + { url = "https://files.pythonhosted.org/packages/56/39/c4f176880a4177123576de6cec6309feea8f42fca2bf2f6584e88054f656/mypy-2.4.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9fa247e02b505a45a2775f69df38d360d197e3790bc60f717595db9eda358b6e", size = 14392944, upload-time = "2026-10-01T20:40:05.78Z" }, + { url = "https://files.pythonhosted.org/packages/bc/1c/26e16977e25ef2494a74f8ffc872a76ec2c3aa43c3156057cbdf88f352c5/mypy-2.4.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:20e9a5cd875837520c43db98dea0b6d0c2197833d95c30127d8f570fb9b1f00b", size = 15441463, upload-time = "2026-10-01T20:40:32.412Z" }, + { url = "https://files.pythonhosted.org/packages/31/9c/9e4b049f0ecefbfd6817ca2a16eea55dd74a07950268edc6cffd29ccfe98/mypy-2.4.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:9f03a7828cca2b0adcd6662aee8f2711ff8830e1027641fdea3ab0b787483566", size = 15705406, upload-time = "2026-10-01T20:40:34.844Z" }, + { url = "https://files.pythonhosted.org/packages/4a/5e/e861b5f6c5ef9ee6cd24683aed1edecf82a26dbd536049f9b7850b586267/mypy-2.4.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:9279488933040b638c0ab739084c0ca100efeea6db581bf5d7628d8e89de53fe", size = 7897523, upload-time = "2026-10-01T20:39:39.861Z" }, + { url = "https://files.pythonhosted.org/packages/3a/87/61ffee58b25a956532a6006fae84918a2de849ed25f397492b2e815fe7b3/mypy-2.4.0-cp314-cp314-win_amd64.whl", hash = "sha256:2106b55105ba5ea9be4f53a24517fc5fa927ff1585edc9bc1a975abb72caef89", size = 11576345, upload-time = "2026-10-01T20:39:41.553Z" }, + { url = "https://files.pythonhosted.org/packages/a1/88/a331c20698971c2ce8d1c30f317fd61b5be13a85b1f053e12dfa22ac6568/mypy-2.4.0-cp314-cp314-win_arm64.whl", hash = "sha256:528c8744b8b5e3ecb8774f86af38d2376216816e9908317ad055f3c9c2d74799", size = 11172062, upload-time = "2026-10-01T20:38:59.534Z" }, + { url = "https://files.pythonhosted.org/packages/47/c0/4f7daa73270dced8e86c6f4a911c68082d03a84e6c1ec9bd916028d66131/mypy-2.4.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0bb95cf34899e4619c61ab0a8667804e139e580b30d5df12af2102dfe44d0c97", size = 15711208, upload-time = "2026-10-01T20:39:30.597Z" }, + { url = "https://files.pythonhosted.org/packages/2f/05/f1afa303c678be24cf7a266d38fb24b3de4599a024c2f9ba0d5905a3efa3/mypy-2.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:86d616fe84c6eab8026f8c50ab5bcb90db780d2ccd233d971e34e92bede9b359", size = 15935442, upload-time = "2026-10-01T20:39:22.781Z" }, + { url = "https://files.pythonhosted.org/packages/9d/d6/6a1a45459b63716e0d035f4892a926d3054f0a8cbfa02551d228a9946083/mypy-2.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a3f86fd1313dd69d013e265f1fdcd12ea7a9d606f9875b2a3db946cd334555f3", size = 17240044, upload-time = "2026-10-01T20:39:32.95Z" }, + { url = "https://files.pythonhosted.org/packages/87/85/ae33bee66c13f98d421964d87bf0888be941063bc75c1204a4cf142cf1cc/mypy-2.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:720434d48542ecfe84d32d287b727569d3fc8f5769acd39051130e490a5c295c", size = 17545856, upload-time = "2026-10-01T20:39:08.827Z" }, + { url = "https://files.pythonhosted.org/packages/df/de/eeff209b65c3e267818d79333bb9f058da6e4e73761461bd94748a2eb628/mypy-2.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:a6e851b82c0661f69f1630fc16172c68787a6a9cf0991e7c6437d60976cdcd76", size = 12369342, upload-time = "2026-10-01T20:40:10.516Z" }, + { url = "https://files.pythonhosted.org/packages/2b/43/e62d8d5c1dd737aa248968302ff7d6eabf997c3301773ed8bcb64932ca77/mypy-2.4.0-cp314-cp314t-win_arm64.whl", hash = "sha256:3bd0e340f0ebe65c548210f53be3fd8192e83964760caf0c28bef368e68b0d37", size = 11849099, upload-time = "2026-10-01T20:39:51.834Z" }, + { url = "https://files.pythonhosted.org/packages/2e/5f/335b8980055118dc131355155883fb676bb2161a0d97e08658e479c776fb/mypy-2.4.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:afa89837d9be67e0cadfa33bca3bb7efdda98c3b07e74dc3b635ebfb1c8a926a", size = 14241302, upload-time = "2026-10-01T20:39:13.227Z" }, + { url = "https://files.pythonhosted.org/packages/18/37/1482fdc49332b145828912b15f16eee0c4ca70a8bce0f6514ece79b14680/mypy-2.4.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fb443e81057896132d3642d6be219e6efd158691ac7883e3ba8fcb469865f05d", size = 14408200, upload-time = "2026-10-01T20:38:52.576Z" }, + { url = "https://files.pythonhosted.org/packages/04/09/dce2e8f6c1b31053c430ef6963f6f7a38ccb49b90c5e01e37ed0129b7d5a/mypy-2.4.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7f38f57d344f8b6accb40e01c3d83cfc590498231724d16c07ffb7940f157818", size = 15444409, upload-time = "2026-10-01T20:39:18.1Z" }, + { url = "https://files.pythonhosted.org/packages/43/c5/91b68306da4cd280cb15be35dbb5ffc343cabd67c4b121b6625bf2d13177/mypy-2.4.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e76172710bd4e5eeae061abfd68347e5264632e02778be61784671ae3a2132f5", size = 15731186, upload-time = "2026-10-01T20:40:23.426Z" }, + { url = "https://files.pythonhosted.org/packages/64/71/2d0340182a8f27352fb1e25531355951108b506097c433937e9eec452ffd/mypy-2.4.0-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:f83353e47ab520bf6fd4df8f5897d9fe081211f2fbc4b7d37736a3e3c166cbcf", size = 7897838, upload-time = "2026-10-01T20:38:50.779Z" }, + { url = "https://files.pythonhosted.org/packages/3e/08/32703c117e134c02efa2a82705bb40cee91eaedea10b6530e20eba651b1f/mypy-2.4.0-cp315-cp315-win_amd64.whl", hash = "sha256:970b221ed5842213d98e3c480c08f795ace4b1f81fb21e1b126bd0476bce1c34", size = 11575700, upload-time = "2026-10-01T20:39:53.982Z" }, + { url = "https://files.pythonhosted.org/packages/ab/08/08bb269feafdaad031046ee2d771528a64467e6a180f962afc28c4a3ccd8/mypy-2.4.0-cp315-cp315-win_arm64.whl", hash = "sha256:502b94b0b331f7dafe32fd6b151797ddbb4f32385b362e722c783a025e5954a3", size = 11172908, upload-time = "2026-10-01T20:40:12.91Z" }, + { url = "https://files.pythonhosted.org/packages/fb/3f/c5c92626006ca92c7686adfbece47fc6a0daa0e54753952a9ad13ce0561c/mypy-2.4.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:c9de622fd397495695d0598ddc789222bfcfec9d7c9ec3a1e385c855e3bc5e01", size = 15705970, upload-time = "2026-10-01T20:40:17.79Z" }, + { url = "https://files.pythonhosted.org/packages/10/f3/863365f7997a76a5a1dd42d7902ab05afa4124e8419089cbca1ce2554db1/mypy-2.4.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9f459f0b4f0596d9d51fe7716b404b35287b99e77da98a7af90a65dd5fd61141", size = 15942234, upload-time = "2026-10-01T20:40:37.243Z" }, + { url = "https://files.pythonhosted.org/packages/f6/30/2f45b1f425a2c95dbe1a3f4d076bfd42b76e9615dba5906230703b14e4eb/mypy-2.4.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f9b028548b3af480e2b1ed8df14ccaac86f99c9f600d1580770ab7ba3dcd40f0", size = 17218364, upload-time = "2026-10-01T20:39:06.497Z" }, + { url = "https://files.pythonhosted.org/packages/43/8b/5b2bbfc69e84800b78fa2dba16d995b003f93b573558e412c5490d6e6c37/mypy-2.4.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:cb734b2668c1f40d07ce093bbeb4407e9527c67901627b0e1679825be3f09975", size = 17557934, upload-time = "2026-10-01T20:38:46.077Z" }, + { url = "https://files.pythonhosted.org/packages/df/c0/1a5dc601c22041a7bbfe1ff71cf097fbbd4fb49930867c6789baf5102106/mypy-2.4.0-cp315-cp315t-win_amd64.whl", hash = "sha256:172e30b8fea631fe310f0c665477f52d9ea40bb4e99e0c81dc30118563b13710", size = 12350955, upload-time = "2026-10-01T20:40:01.52Z" }, + { url = "https://files.pythonhosted.org/packages/1c/bc/697e9e26fc2a86c094ad67ee1e419971b7f01ded66ee66231b09e9f3e12e/mypy-2.4.0-cp315-cp315t-win_arm64.whl", hash = "sha256:5786ef987b3767e51aaa53f20aec104c0252b42ecda7aef8e8b4cbae279b05c5", size = 11842738, upload-time = "2026-10-01T20:40:30.026Z" }, + { url = "https://files.pythonhosted.org/packages/81/12/46ae8670c98a3cd0286ca5645c2f918f8f6be65edfed81b916010619f668/mypy-2.4.0-py3-none-any.whl", hash = "sha256:d01c5d26a352acc6d5cf3128225477e1e8465e8d3029d4c345807fbf7f3cf093", size = 2800734, upload-time = "2026-10-01T20:39:26.837Z" }, +] + +[[package]] +name = "mypy-extensions" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, +] + +[[package]] +name = "packaging" +version = "26.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, +] + +[[package]] +name = "pathspec" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "psycopg" +version = "3.3.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, + { name = "tzdata", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/26/3ea4ca5eaea1c0debcdf7ee7c1613fbe721dc27a03c461c0817ffd8a0601/psycopg-3.3.6.tar.gz", hash = "sha256:c081f2250df751a943036e42db6df4571c66cd0aabe8291a7a506512b12007d2", size = 168171, upload-time = "2026-09-18T13:22:55.152Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4e/de/748bd7609c71cae5d737f0ba9192f19329f70180ecda8fff3cac02c5abe3/psycopg-3.3.6-py3-none-any.whl", hash = "sha256:a1db9f7148b06a28606767efaca51fa6f9398c5c0a3810519be69d7000bdb631", size = 215490, upload-time = "2026-09-18T13:15:29.374Z" }, +] + +[package.optional-dependencies] +binary = [ + { name = "psycopg-binary", marker = "implementation_name != 'pypy'" }, +] + +[[package]] +name = "psycopg-binary" +version = "3.3.6" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/70/86/b71166048974d49c6d136b2ed1c0e5bec0b974d8c4de5cbce7e86a9e412a/psycopg_binary-3.3.6-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:be4f9b3c9338ac5dd217c5847e21521b396c8117f78dc420d495a5c49bbef874", size = 4728002, upload-time = "2026-09-18T13:16:53.393Z" }, + { url = "https://files.pythonhosted.org/packages/12/1d/1e06c0de7ed5aed898acb87544eac6ef0bc7d752a67ec6e5d6b835e9b40c/psycopg_binary-3.3.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f0535693ce476a722b718b002d5d2c27d47e71ca945276ac194409c98e74c492", size = 4775403, upload-time = "2026-09-18T13:16:58.939Z" }, + { url = "https://files.pythonhosted.org/packages/84/02/2ffcbc43f8e4bbc38e5286a22013bcac01898d13cd38325f60dd5428a8af/psycopg_binary-3.3.6-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:3c9e663b2e800e3218994cf948c11bcc2844e6491b34aa80d089baf6531827bf", size = 5594145, upload-time = "2026-09-18T13:17:08.515Z" }, + { url = "https://files.pythonhosted.org/packages/e1/25/031dae2c7d2e7e77dcf5b1962c1e0684fa548d7af0ff6707b6b5e6054ca7/psycopg_binary-3.3.6-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a2e44a342d2aee40508e28a563d8961c39d9bbd8cae36d8578f0a3c6658aab0f", size = 5267985, upload-time = "2026-09-18T13:17:16.24Z" }, + { url = "https://files.pythonhosted.org/packages/8c/e5/94c89ada3c003a4d858178f3bba49a35e0297ef2aad659b80eb5e380e690/psycopg_binary-3.3.6-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f598f19fa9a91540b5cee17932ffd227b7b53a481605bcc4573c0eafa647300", size = 6860986, upload-time = "2026-09-18T13:17:23.348Z" }, + { url = "https://files.pythonhosted.org/packages/9d/a0/81bf499d095adee8413bd19822a6872fbfa21663ec78014a68d83a8db83c/psycopg_binary-3.3.6-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6ff05561e4a067d35507dc5c90f1deb2ec1c9703ac5cccc1bc26e08a197f9c5a", size = 5106702, upload-time = "2026-09-18T13:17:28.847Z" }, + { url = "https://files.pythonhosted.org/packages/00/75/99d56da64c27bd985fd82c6ecbf7976b724ac638fdd1654ef995323a1a26/psycopg_binary-3.3.6-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:566dd827f17728efdf7d88a5b066f815170f6fdad13967ae952842d90e6aaa9f", size = 4627090, upload-time = "2026-09-18T13:17:36.668Z" }, + { url = "https://files.pythonhosted.org/packages/3e/0c/0222171d11233332c6a24b1cef1578215f0ffddf3642eb8dd8c4448ad69f/psycopg_binary-3.3.6-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:9b2f11794e017ce340934e35de46181c46ef71ec75ea3d85dd75cd836761c01e", size = 4320353, upload-time = "2026-09-18T13:17:42.526Z" }, + { url = "https://files.pythonhosted.org/packages/62/6f/e1cc2a28dd1228c67c969ba6fd37cd8726b312e2ff51380f847ddb38ccde/psycopg_binary-3.3.6-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:910ace140e3e7b7596898d083f37a8fe90c5c40684252ad4e682364b2cd3deba", size = 4047253, upload-time = "2026-09-18T13:17:47.068Z" }, + { url = "https://files.pythonhosted.org/packages/d8/fd/38b64790ce7a515b1dbd2bab3d119637a858aeb22c380cf4859bc4ce0e42/psycopg_binary-3.3.6-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37e517c146b185f9c0c6e8d0a0ebbdeeeb67896af28466e032bc810d0c7dc7a7", size = 4353208, upload-time = "2026-09-18T13:17:52.41Z" }, + { url = "https://files.pythonhosted.org/packages/f7/dc/45386530ceb2a8c789a226de9b9b34eca8fccf1feba2e4ef68a6aca50c56/psycopg_binary-3.3.6-cp311-cp311-win_amd64.whl", hash = "sha256:c7f92daa0d2a1c76f07264abddf8cbabd30152a2f09c3270e50f0c7efdf5dcac", size = 3675638, upload-time = "2026-09-18T13:17:58.112Z" }, + { url = "https://files.pythonhosted.org/packages/e6/01/2cdd1824e58b4467ee0b9498664cd28c42d8794db6b1e35b6bcb834f0044/psycopg_binary-3.3.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3f84dab25e0385692ee13274c68678377e0b1a70ab9d14e56264cbf61f60c62d", size = 4707086, upload-time = "2026-09-18T13:18:05.138Z" }, + { url = "https://files.pythonhosted.org/packages/f6/76/de9948ac06895261c84d5b9fbe283d8f3c5bc9f070691b8d9eaa1b51e322/psycopg_binary-3.3.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:612382ac3ed13651c7fa44b5fee9fbf7baaa2ddbc6f500391672682c5f1df9e0", size = 4769607, upload-time = "2026-09-18T13:18:12.83Z" }, + { url = "https://files.pythonhosted.org/packages/76/a9/72436c9915ee4905964689e7f0e182ce7767cc0a0390b3ce703be8177625/psycopg_binary-3.3.6-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:366db6e97e66b37211475f20c4c1324a2dc0dd825e46d4e87f9d599304d276f9", size = 5554134, upload-time = "2026-09-18T13:18:21.175Z" }, + { url = "https://files.pythonhosted.org/packages/0a/42/948bb3d2617795093512613fd96ba380e922992c7908fbc073858147d196/psycopg_binary-3.3.6-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1679a1cb93fbe5a6d1fd58d82cbddcc6fcb8c61446ba7cae6eb2a7b19bc585de", size = 5235723, upload-time = "2026-09-18T13:18:27.071Z" }, + { url = "https://files.pythonhosted.org/packages/99/47/93e823ff1b0088400703410939c9bda3e63ed9c850b3ee088e8769f4c10b/psycopg_binary-3.3.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:37d40450659401600e6d043ff586c89a71a69f33cbb8bcdba6cdb2569beecdbe", size = 6833587, upload-time = "2026-09-18T13:18:33.794Z" }, + { url = "https://files.pythonhosted.org/packages/5e/2d/ecc69c847795aa704041a9f5667a6b0938a088cf1853636d762a6938e493/psycopg_binary-3.3.6-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a5165300324efd5a772c48a88ab3a928513ab3979fca76553e62ee815f7b2b9c", size = 5070013, upload-time = "2026-09-18T13:18:39.628Z" }, + { url = "https://files.pythonhosted.org/packages/92/36/6126f0dac21713dcae91404f2a76da18598a6252339a8c669c46370d43b2/psycopg_binary-3.3.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d636338c8f21b0df2f84657b00bc34f9313f826ef93f1155bc743607e4a0c5eb", size = 4597367, upload-time = "2026-09-18T13:18:45.023Z" }, + { url = "https://files.pythonhosted.org/packages/4d/29/7ecfc04243b46c89ffd49924e9c5634ea904ef96c7d0f37e4073623584c1/psycopg_binary-3.3.6-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:a4ee3bdd5468a725f2a4d9aab8a74b6d0279f768c8b5d3aeb102c5307ff3d59c", size = 4275419, upload-time = "2026-09-18T13:18:49.299Z" }, + { url = "https://files.pythonhosted.org/packages/6e/90/2f46d2e0de79706ac170df0a3637fe63c4498fc04f131f6049520b78b806/psycopg_binary-3.3.6-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:289aadd6a00e151203c081f708348ec89f1e483c9b510ef4ac3981f847f01f79", size = 4007358, upload-time = "2026-09-18T13:18:53.944Z" }, + { url = "https://files.pythonhosted.org/packages/03/48/6744e91291b751a8cf12d63d719977974bb94c84ceba913e7ddb2e478e51/psycopg_binary-3.3.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f21d057f3e5f5491067e5b292498073b73847d48799b099803fef100775fcc52", size = 4320156, upload-time = "2026-09-18T13:18:59.258Z" }, + { url = "https://files.pythonhosted.org/packages/1a/9b/94ff7fce53a64d5b286e2ec454e0a025cf3d6e6b4a9189bef16aa5de98b2/psycopg_binary-3.3.6-cp312-cp312-win_amd64.whl", hash = "sha256:e23a66a763fbe83fcc210bc77c27e5a5ea380ebf091c06f34d8561b695e5a40f", size = 3658864, upload-time = "2026-09-18T13:19:06.503Z" }, + { url = "https://files.pythonhosted.org/packages/b4/c3/c072584b69ad44a747b448cfc9766fecb8aae56e372a017e2ef668790057/psycopg_binary-3.3.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5ad8f35e67cc16d1fad1fa8c88972dc9b3a3141ea67897399904edab96a301b6", size = 4712284, upload-time = "2026-09-18T13:19:13.451Z" }, + { url = "https://files.pythonhosted.org/packages/0a/b9/4283b785339e8e2318d03048994b093d650ea6289fabaa806b765dc0d449/psycopg_binary-3.3.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:373704aea331d3f3e3402c125a1543f5875e2986ebb54f97d1647942161f803f", size = 4772031, upload-time = "2026-09-18T13:19:18.524Z" }, + { url = "https://files.pythonhosted.org/packages/6f/72/7a1321d359246769fff1affffbd0132785a28f7f63c18524c15a502398f4/psycopg_binary-3.3.6-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:b82491019b884d62318b5f30706c3d7e6d4e5a6cb7eabcb3edc0c1b0fdaceae9", size = 5556392, upload-time = "2026-09-18T13:19:24.418Z" }, + { url = "https://files.pythonhosted.org/packages/de/b0/c6f8a0585a5dacbea74e130bcfc66629390e8f5bbc79d2a8e806e8952150/psycopg_binary-3.3.6-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cec5ea900390897d0b46130f60bc2883bf19c314f9044235217c8be88b0ef269", size = 5237855, upload-time = "2026-09-18T13:19:31.257Z" }, + { url = "https://files.pythonhosted.org/packages/e2/fc/c3a7a8bbef7e945ec584ac61d460a612363ea398511cd0e220242b1d69f1/psycopg_binary-3.3.6-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:98c02090d88f2ebc0ec1e8da538f77d225ce0fffecf372aa39262e62a1b054ef", size = 6833856, upload-time = "2026-09-18T13:19:43.622Z" }, + { url = "https://files.pythonhosted.org/packages/a9/f2/8e80b921db728ebb68fc105bd7c4277f908210ad755bd6481d5ea7add740/psycopg_binary-3.3.6-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ee2c4728c691245e24501fcd7a97b5b381236b9985bc445bba88cdce7d1b5784", size = 5070730, upload-time = "2026-09-18T13:19:49.968Z" }, + { url = "https://files.pythonhosted.org/packages/54/6a/5b313e0c5348244f0e973aff3258bf86766656256d5ece8d541a53e35b4a/psycopg_binary-3.3.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f19cc87343eaa55255e76b31259a570072ac95d6ae82c92dd34b97691f5e49dc", size = 4598089, upload-time = "2026-09-18T13:19:56.426Z" }, + { url = "https://files.pythonhosted.org/packages/32/e9/db7f76ec24bf6699e92bf604e5c4bae10664a681a8999ef42aa0faf0f2c6/psycopg_binary-3.3.6-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:fdccb3a0e184b03e9baa673b15a809cf36c339c85dbda0ebc25a698846dfbee8", size = 4278481, upload-time = "2026-09-18T13:20:04.681Z" }, + { url = "https://files.pythonhosted.org/packages/61/83/72c67013656f4d6b547caabffb193e91d57e63f90eefdcc6d045c400e97d/psycopg_binary-3.3.6-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:9892188bb15e5803beb51afe8a25add6b56be391a53058e8bca03b74e1e6bf22", size = 4009229, upload-time = "2026-09-18T13:20:11.905Z" }, + { url = "https://files.pythonhosted.org/packages/82/35/5e4500df2c999eb0faed8b184e6958b834172128274f06167a5deef4c19c/psycopg_binary-3.3.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3af90f92769d8cc10f94515ee7a0aef36ea85ca733a0ce22858f6e0953f41138", size = 4321467, upload-time = "2026-09-18T13:20:17.949Z" }, + { url = "https://files.pythonhosted.org/packages/55/7f/e350e1cf498ba2565c3f87b12f429d2012eb86b76c2b3845a19ee5fbb4d6/psycopg_binary-3.3.6-cp313-cp313-win_amd64.whl", hash = "sha256:0ebfad5d131de9f892ae9e70cc7616207768b6714b66a52d4612b8ceaf78b372", size = 3658179, upload-time = "2026-09-18T13:20:22.691Z" }, + { url = "https://files.pythonhosted.org/packages/6d/b9/60711317c284a442511644ea7185b56ebe627606d6741e732cd16108c47b/psycopg_binary-3.3.6-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:b3f75dee0f9afafabe4edc52c4842f1e1878ed2069bd05b22d6fe961e97e4dba", size = 4720512, upload-time = "2026-09-18T13:20:29.278Z" }, + { url = "https://files.pythonhosted.org/packages/63/da/28befc84454cbc6374550de7746f591f8fe1b6165c1fce249652cc8291c4/psycopg_binary-3.3.6-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:5927b7ba63153cd8e9862987290a2b783a5c590daf2a4ef981700cc3569166d4", size = 4782318, upload-time = "2026-09-18T13:20:35.401Z" }, + { url = "https://files.pythonhosted.org/packages/a4/8a/0d21c2c833cdc0d4244c77e858e0ed37fa2abec2623be4fd686f617109ce/psycopg_binary-3.3.6-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:0bf08b749cc144f33b44a91b78e3f71c60eb07963746a0df5a100b36ce3d7475", size = 5567460, upload-time = "2026-09-18T13:20:41.902Z" }, + { url = "https://files.pythonhosted.org/packages/49/6d/7692d0d4e656b6cc9868d8acc2e3b42f17a0db4a625400a6d093cb0533a1/psycopg_binary-3.3.6-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:31cd942c23f613276b81a6e6598cefa12960058b0f46e1e874b540c793f6aca5", size = 5246902, upload-time = "2026-09-18T13:20:47.661Z" }, + { url = "https://files.pythonhosted.org/packages/d4/c1/b8a1f18fb1b7558a17f57f7cb3fc8bc93189feea2958925950b3acb15743/psycopg_binary-3.3.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4690cf67738f0e0e49a32aeec99bf0e4595cc2b4f1af984a4345394b1dcff91a", size = 6847192, upload-time = "2026-09-18T13:20:56.874Z" }, + { url = "https://files.pythonhosted.org/packages/a5/76/404f33519167c65cca88ec4998776f1dbebccc301ee977f0e62c47fb0826/psycopg_binary-3.3.6-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ad1c785e784cfd87e8436c6b7702f2d321fc39601bbaf29bc63a41a867091638", size = 5079573, upload-time = "2026-09-18T13:21:04.155Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d9/79e8fbc8f37262a415f3550f0bcc5f98037442bf3d12ef6cbae2056655ae/psycopg_binary-3.3.6-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:79a2a1c3449f6c3409427078ed1cec10de79f3023cb5f2504f0597d350ad46c7", size = 4613633, upload-time = "2026-09-18T13:21:10.664Z" }, + { url = "https://files.pythonhosted.org/packages/d4/47/96225db74be7d2ce04b3a58678b53cda610225055edf5faa775c9f501d8b/psycopg_binary-3.3.6-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:86147cb5d140341c3363fb5bacce31f8d5543902a46699d3c536b101bbceaf9e", size = 4293375, upload-time = "2026-09-18T13:21:16.027Z" }, + { url = "https://files.pythonhosted.org/packages/2a/d2/18e9c779a5efd565250329adaf529ecc2b8b2ed5be5cb0f6ccee208cbfd9/psycopg_binary-3.3.6-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:7308c93cf0b19bbaf8e6ff0a6ad50d3c442385739245fe15a8d593bf841734a6", size = 4019883, upload-time = "2026-09-18T13:21:21.587Z" }, + { url = "https://files.pythonhosted.org/packages/ef/28/0cc654afc6c2cda982767f5679d3646b30b1ec86545bdaa9402202d6776c/psycopg_binary-3.3.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:05a83ac9fd52b9bca7cb5ab04b3691163170bd16f53defa27216ea3aa07ee781", size = 4332607, upload-time = "2026-09-18T13:21:27.63Z" }, + { url = "https://files.pythonhosted.org/packages/f1/3e/0a753a74fbd7aef120f286c016e09d3cc3f1daf7688f4a145d27281260b2/psycopg_binary-3.3.6-cp314-cp314-win_amd64.whl", hash = "sha256:1fbd30e537dab22cafdf080608f10148fe2a5f3a61294ddb5113caac8a623840", size = 3755671, upload-time = "2026-09-18T13:21:33.855Z" }, + { url = "https://files.pythonhosted.org/packages/0e/b1/a372b9c02aea50148e71c9853e19efca8fa5ae2010a8e27243b9b8f790c0/psycopg_binary-3.3.6-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:bf8c8481d026b85dd70c5fa7dde85b2333aed0b32a2602bcd38a900cbd78a49c", size = 4719571, upload-time = "2026-09-18T13:21:41.437Z" }, + { url = "https://files.pythonhosted.org/packages/65/7c/811e3828c6b82e2f10c6c9cdd963cfc66f3e024026e5a69ac18530bad984/psycopg_binary-3.3.6-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:b599defe9190b17e9907c8b4d114c181e702c87efcd1b8a0ad40971cdcc4634a", size = 4781230, upload-time = "2026-09-18T13:21:49.516Z" }, + { url = "https://files.pythonhosted.org/packages/3e/15/9a784eed813ea9e97c294af3ead63d02b7b203502c66380336c50065e441/psycopg_binary-3.3.6-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:b8ece331509f7a975b90501f41e83ad905e4141753fedf3f2711b2bc70a8efbc", size = 5566111, upload-time = "2026-09-18T13:21:58.089Z" }, + { url = "https://files.pythonhosted.org/packages/68/16/47194e002007c27337b11e49bf459c4b19727463f9aff2e1a90917bcc806/psycopg_binary-3.3.6-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c61617eaae0112ca154da87ffb99b73af2c74067acac28dfb9a4455b019dff2e", size = 5249963, upload-time = "2026-09-18T13:22:06.695Z" }, + { url = "https://files.pythonhosted.org/packages/53/84/5dcf9f310b11f0675cd860c6b2c70f58ce61798a3ee3f6f962b53fa358ca/psycopg_binary-3.3.6-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c6d19cb4999d03231e8730a5f66c8f5068bc3b532677eb39dab0f600bff3e312", size = 6847925, upload-time = "2026-09-18T13:22:13.088Z" }, + { url = "https://files.pythonhosted.org/packages/f3/06/1957a06dc22963c418c27b284929579de84f29c37ad1abe6dc6ee9e8cf25/psycopg_binary-3.3.6-cp315-cp315-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8cbb54454dbf1bbf2ff08dd7693e8d94ac94b1a20f70f4b3b813d52ecb5cbc1", size = 5087720, upload-time = "2026-09-18T13:22:17.959Z" }, + { url = "https://files.pythonhosted.org/packages/21/43/ac07d042bae99b57bf123bb473632f29af544008094da0ffd285ab8011e2/psycopg_binary-3.3.6-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:dc75da5a20951049f7b773145f998f69d181adad9c58a0ff36e0cf1d73c10e10", size = 4613412, upload-time = "2026-09-18T13:22:26.719Z" }, + { url = "https://files.pythonhosted.org/packages/aa/b1/019156fbeafcefb4cccc9d109de4699493bceb8313c7545c8349e089dfbc/psycopg_binary-3.3.6-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:955e3dd94da361e052d2e49acf591017158dc8f8ed2c8a42c2e3943403c39dc2", size = 4292618, upload-time = "2026-09-18T13:22:33.042Z" }, + { url = "https://files.pythonhosted.org/packages/5d/0f/62113dc6b1df65983a1f2fc816c04b1edfa22f2ae9d4abee74ed267f4a96/psycopg_binary-3.3.6-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:c7753871eb57e6a5f4646f6168590c6653073dea5e9e720b201c8875332df4c8", size = 4027121, upload-time = "2026-09-18T13:22:38.334Z" }, + { url = "https://files.pythonhosted.org/packages/5d/d5/cf0cbd1ea5a7d8167fe2c6953efde19101f7b193bd61a23e6d622ad6854c/psycopg_binary-3.3.6-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:303732e798fe6729f8e12021b9c96107df8e95ecec4dd487c67b98ec2a59435e", size = 4336388, upload-time = "2026-09-18T13:22:45.576Z" }, + { url = "https://files.pythonhosted.org/packages/98/33/e2a5b36edf8aa422f6fa4b894756eb33dc93b36df5f65121280bb8b929c4/psycopg_binary-3.3.6-cp315-cp315-win_amd64.whl", hash = "sha256:2f122603f36050937982abf9668d8bc4769a79f7c93a65013b1c49f1cab7b56b", size = 3756154, upload-time = "2026-09-18T13:22:51.283Z" }, +] + +[[package]] +name = "psycopg-pool" +version = "3.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/74/5e/c0664b968b102ff68b811d999c728546c48d5c1eec03e3bbaf88c0cb4472/psycopg_pool-3.3.3.tar.gz", hash = "sha256:df87b5d9d0ad7db37f6cdad4fa8ce113d250f5997f6db38e9a99192fb67f9e1d", size = 32006, upload-time = "2026-09-22T15:53:24.947Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5d/b4/452c6607a0f479465cd8a9b0d9956919fcb150050c1f83f9f11e6b8ee8dc/psycopg_pool-3.3.3-py3-none-any.whl", hash = "sha256:9b9cd6a4fcec47a410f7e82d408540e7f77b478509e91b44c1a5457a13e5ff37", size = 40304, upload-time = "2026-09-22T15:53:23.712Z" }, +] + +[[package]] +name = "pygments" +version = "2.21.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/49/2e/ced460408999b33da6b31b0021b0f37d329e202d4169aeb164493778f25b/pygments-2.21.0.tar.gz", hash = "sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c", size = 5005329, upload-time = "2026-08-17T08:02:48.824Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/46/17f022dd3e953bf20a04a028a21ec746d942f8d2af30fa0f124fa0e6a684/pygments-2.21.0-py3-none-any.whl", hash = "sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", size = 1250147, upload-time = "2026-08-17T08:02:44.912Z" }, +] + +[[package]] +name = "pytest" +version = "9.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, +] + +[[package]] +name = "ruff" +version = "0.16.10" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c4/49/23802c45f093eb14bde54b141d2b2f058edfa63a06db7beed047308cc08f/ruff-0.16.10.tar.gz", hash = "sha256:eff4728c4eaae93f0955cd264d24b2ab348e74bf59986ccf282ba6dc16b3b017", size = 4958724, upload-time = "2026-10-01T18:03:21.697Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2f/21/ebce22e1d90cdb2cd691026b9c6e9bec6499f0b396089481755b5d49efff/ruff-0.16.10-py3-none-linux_armv6l.whl", hash = "sha256:488b0fe3f3574210e5cf80d9f59b9e3ab17a127a8155de3f307b392589cfb511", size = 10095557, upload-time = "2026-10-01T18:02:36.072Z" }, + { url = "https://files.pythonhosted.org/packages/cb/98/a54de85876a8b2612bfa0d84c7b9abfb39c6a3354aee7800b09c1649b9e3/ruff-0.16.10-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:e748ff95c934c4e978783b8e687bc174e7bd84e8ad24e3243e1ecfcda5e0282d", size = 10340577, upload-time = "2026-10-01T18:02:39.258Z" }, + { url = "https://files.pythonhosted.org/packages/9f/16/1a5a4a2657effe4806110f2b907313802f1367fcdbb29e8122766e407fab/ruff-0.16.10-py3-none-macosx_11_0_arm64.whl", hash = "sha256:3031a4a2e8e7b8a46f70be45f198c35a11ece509a94b80334d8d397a33c67550", size = 9774282, upload-time = "2026-10-01T18:02:41.79Z" }, + { url = "https://files.pythonhosted.org/packages/6e/fb/470085af734da396e68cd80fb0a3e7c109a459716ae59588c0f6fab8a17d/ruff-0.16.10-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:494401c86df4c4c25f69b9419605d944467ee98c42fb6ad405ef4fa40b8fb67d", size = 9920895, upload-time = "2026-10-01T18:02:44.458Z" }, + { url = "https://files.pythonhosted.org/packages/57/de/f10cffe4f88a37ea6615ec460f01bf76f0bc1477737a35d9ee61a630a78b/ruff-0.16.10-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d203abc0ff2b773ee33d00ab8df0bb67046fbc7c07b119332f08b9b345cf8221", size = 9902707, upload-time = "2026-10-01T18:02:47.041Z" }, + { url = "https://files.pythonhosted.org/packages/ff/44/3fdcedf83ae60ef837dd239170e480dce606a9142cfa2db911afdf855fd9/ruff-0.16.10-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bd83d1235a5258d318477bc5b576303974cbdef5df0c01a1bff14efcc23bd12a", size = 10619287, upload-time = "2026-10-01T18:02:49.485Z" }, + { url = "https://files.pythonhosted.org/packages/c1/62/02e76a5574002153618eafbb70e159468addc72a5d2c488e65cbb4639d2d/ruff-0.16.10-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:bc2610fb269fa56dd8a68669ae470fa6272902668c0fc2ebc3aa112b2633d5b8", size = 11339942, upload-time = "2026-10-01T18:02:52.008Z" }, + { url = "https://files.pythonhosted.org/packages/6b/c4/cde27d47ad8d4126c587e608c67c46e7feac11763f606d261a1bc8a489e6/ruff-0.16.10-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3e70175e29cc94c26ea296c80e470180b744b7419026898e58f520c6ab32578e", size = 10934316, upload-time = "2026-10-01T18:02:54.811Z" }, + { url = "https://files.pythonhosted.org/packages/e4/03/17234145f302a645a123e8c3bb2411ecf4669fbc350de3b0d803230a1729/ruff-0.16.10-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f33f43a864a8483eebd160e713336c8bab02c934feaff0a33cf5ccb41546d09a", size = 10387968, upload-time = "2026-10-01T18:02:57.497Z" }, + { url = "https://files.pythonhosted.org/packages/71/29/2493af60240ee7644b38a4b821f5f9c3b5a4fa3178770fe1d0c685217395/ruff-0.16.10-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:1dfc6f0088149fb6a362c1c446bcbb3fd2157b3852fe2fa68409276eab9ad9b3", size = 10537906, upload-time = "2026-10-01T18:03:00.006Z" }, + { url = "https://files.pythonhosted.org/packages/2e/9a/f56b28f3b143bb9e518c34fb89ab191b626bca8b70dbf8af0d2e2d473572/ruff-0.16.10-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:6553498afc35f580f036030795810b9e6bcea31604b0fd9e8d352795473042e3", size = 10012938, upload-time = "2026-10-01T18:03:03.006Z" }, + { url = "https://files.pythonhosted.org/packages/c2/c4/fca37362848ea4d4d80712df13632e645e7c7cfb1bdedf140699ac7a090b/ruff-0.16.10-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:a3b8471dea115d37f123882be852bed13403746d3a76c11de4a19ec5f9ff5a03", size = 9897945, upload-time = "2026-10-01T18:03:05.818Z" }, + { url = "https://files.pythonhosted.org/packages/7d/c7/e0bc57664d6e0c61fd22f620af260fe9662d7e1ddb320ea7f160e76ef165/ruff-0.16.10-py3-none-musllinux_1_2_i686.whl", hash = "sha256:92e59a70bcbd9d3a5483656da906ec28edfdacfce00afd99edb8b4e9d15644be", size = 10333134, upload-time = "2026-10-01T18:03:08.25Z" }, + { url = "https://files.pythonhosted.org/packages/21/aa/5c9f3b68737c0e4a33a1db5dfab44f7b786d96ca91a7233112ddab1e6dc2/ruff-0.16.10-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:7ae7375f803b5520dc9f546bed7e3a0acb70b91812e9bb4b19927de22f25b77d", size = 10741702, upload-time = "2026-10-01T18:03:10.775Z" }, + { url = "https://files.pythonhosted.org/packages/78/fa/0f9c2020dc316c53d983be011720b8157cc052b97e3f4dfa7db6d0f880a6/ruff-0.16.10-py3-none-win32.whl", hash = "sha256:2a12e01cb9156c10c466f63b46eaae5ecea28dfbd21b5836353ae498e7d1349a", size = 10139176, upload-time = "2026-10-01T18:03:13.267Z" }, + { url = "https://files.pythonhosted.org/packages/99/29/cfb0df9448d4d4ad48c2de029ada9ebd71baa6da983a6c77ee6c6cd0fe82/ruff-0.16.10-py3-none-win_amd64.whl", hash = "sha256:97f2015c92aa97105b0eab19eb5d224884399281cfc5da86a92db4ab5e7fb2ca", size = 10584734, upload-time = "2026-10-01T18:03:16.006Z" }, + { url = "https://files.pythonhosted.org/packages/fc/05/c16957eb287c3fc062e032619a25868d93d408a844b725bcf514f7a378ff/ruff-0.16.10-py3-none-win_arm64.whl", hash = "sha256:25a65fe998c4e6861ec079ada5826a2fc605e6cbccbe9dcd7fac1f54e791621b", size = 10366440, upload-time = "2026-10-01T18:03:19.04Z" }, +] + +[[package]] +name = "sortedcontainers" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594, upload-time = "2021-05-16T22:03:42.897Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d9/68/f1b440335057bfce71b6e50a9d09445aa2ecbd08359a337976627b8409e7/tzdata-2026.5.tar.gz", hash = "sha256:8cc73c0a0bfca7dbfa59235d60b2eff82231dee33f53d206db1acd9173cfc0a7", size = 200404, upload-time = "2026-10-03T09:23:14.143Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/94/21/1e5995a1c920cce14e4bffae20c665ec10e7ed03ab25e006cd741092b718/tzdata-2026.5-py2.py3-none-any.whl", hash = "sha256:b683bd1b6659ddcd810ff02ad09ba821d4bf1065072805063eb35c49617905ac", size = 347996, upload-time = "2026-10-03T09:23:12.535Z" }, +]