feat: 新增管理后台前端项目及安全加固
Some checks failed
CI / Lint & TypeCheck (push) Has been cancelled
CI / Unit Tests (push) Has been cancelled
CI / Build Frontend (push) Has been cancelled
CI / Rust Check (push) Has been cancelled
CI / Security Scan (push) Has been cancelled
CI / E2E Tests (push) Has been cancelled
Some checks failed
CI / Lint & TypeCheck (push) Has been cancelled
CI / Unit Tests (push) Has been cancelled
CI / Build Frontend (push) Has been cancelled
CI / Rust Check (push) Has been cancelled
CI / Security Scan (push) Has been cancelled
CI / E2E Tests (push) Has been cancelled
refactor(saas): 重构认证中间件与限流策略
- 登录限流调整为5次/分钟/IP
- 注册限流调整为3次/小时/IP
- GET请求不计入限流
fix(saas): 修复调度器时间戳处理
- 使用NOW()替代文本时间戳
- 兼容TEXT和TIMESTAMPTZ列类型
feat(saas): 实现环境变量插值
- 支持${ENV_VAR}语法解析
- 数据库密码支持环境变量注入
chore: 新增前端管理界面
- 基于React+Ant Design Pro
- 包含路由守卫/错误边界
- 对接58个API端点
docs: 更新安全加固文档
- 新增密钥管理规范
- 记录P0安全项审计结果
- 补充TLS终止说明
test: 完善配置解析单元测试
- 新增环境变量插值测试用例
This commit is contained in:
iven
173
docs/knowledge-base/axum-dashmap-deadlock.md
Normal file
173
docs/knowledge-base/axum-dashmap-deadlock.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Axum + DashMap 运行时死锁：根因分析与修复
|
||||
|
||||
> 2026-03-30 | 严重级别: P0 | 影响范围: SaaS 后端所有 protected route
|
||||
|
||||
---
|
||||
|
||||
## 问题描述
|
||||
|
||||
Admin V2 管理后台（Ant Design Pro SPA）浏览器访问后端时，protected route 的 handler 链进入后永不返回。后端冻结，health 端点也无响应，必须 kill 进程重启。
|
||||
|
||||
**关键特征**:
|
||||
- curl 单请求测试一切正常（包括 abort 模拟）
|
||||
- 浏览器访问必然触发冻结
|
||||
- TimeoutLayer（15s）无法触发超时
|
||||
- health 端点（public route，不经过 auth 中间件）正常返回
|
||||
- protected routes（经过 auth → rate_limit → request_id → api_version 中间件链）全部卡死
|
||||
|
||||
---
|
||||
|
||||
## 根本原因
|
||||
|
||||
### 直接原因：DashMap `RefMut` 跨 `.await` 持有
|
||||
|
||||
`rate_limit_middleware`（[middleware.rs](../../crates/zclaw-saas/src/middleware.rs)）中，`DashMap::entry().or_insert_with()` 返回的 `RefMut` 持有 `parking_lot::RwLockWriteGuard`，跨越了 `next.run(req).await`。
|
||||
|
||||
```rust
|
||||
// ❌ 原始代码（有 bug）
|
||||
let mut entries = state.rate_limit_entries.entry(key).or_insert_with(Vec::new);
|
||||
entries.retain(|&time| time > window_start);
|
||||
entries.push(now);
|
||||
|
||||
next.run(req).await // ← RefMut 仍持有 parking_lot shard 写锁！
|
||||
```
|
||||
|
||||
### 死锁机制
|
||||
|
||||
```
|
||||
时间线:
|
||||
T0: Worker Thread 1 → 请求 A 获取 DashMap shard X 写锁 → await DB 查询 → yield
|
||||
T1: Worker Thread 2 → 请求 B 尝试获取 shard X 写锁 → parking_lot 阻塞 → Thread 2 冻结
|
||||
T2: Worker Thread 3 → 请求 C 尝试获取 shard X 写锁 → parking_lot 阻塞 → Thread 3 冻结
|
||||
...
|
||||
TN: 所有 Worker Thread 都被 parking_lot 阻塞 → Tokio 运行时无法调度任何 task
|
||||
→ TimeoutLayer 的 timeout future 无法被 poll → 超时机制失效
|
||||
→ 运行时全局死锁
|
||||
```
|
||||
|
||||
### 为什么 curl 不触发
|
||||
|
||||
curl 每次发送单个请求，请求完成后连接关闭。没有并发争抢同一 DashMap shard 的场景。
|
||||
|
||||
### 为什么浏览器触发
|
||||
|
||||
浏览器对同一域名默认保持 6 个并发连接（HTTP/1.1 RFC 规范）。SPA 页面加载时同时发起多个 API 请求（dashboard stats、logs、models 等），这些请求都使用同一 JWT token → 同一 account_id → 同一 DashMap shard key。
|
||||
|
||||
---
|
||||
|
||||
## 解决方案
|
||||
|
||||
### 核心修复：作用域块限定 DashMap 锁
|
||||
|
||||
将 DashMap 操作限定在独立的作用域块 `{ ... }` 内，确保 `RefMut`（持有 parking_lot 锁）在 `next.run(req).await` 之前 drop：
|
||||
|
||||
```rust
|
||||
// ✅ 修复后代码
|
||||
let blocked = {
|
||||
let mut entries = state.rate_limit_entries.entry(key).or_insert_with(Vec::new);
|
||||
entries.retain(|&time| time > window_start);
|
||||
|
||||
if entries.len() >= rate_limit {
|
||||
true
|
||||
} else {
|
||||
entries.push(now);
|
||||
false
|
||||
}
|
||||
}; // ← RefMut 在此处 drop，释放 parking_lot shard 锁
|
||||
|
||||
if blocked {
|
||||
return SaasError::RateLimited(...).into_response();
|
||||
}
|
||||
|
||||
next.run(req).await // 安全：无同步锁持有
|
||||
```
|
||||
|
||||
### 附加修复
|
||||
|
||||
| 修复项 | 文件 | 说明 |
|
||||
|--------|------|------|
|
||||
| ConnectInfo 提取 | [main.rs](../../crates/zclaw-saas/src/main.rs) | `into_make_service()` → `into_make_service_with_connect_info::<SocketAddr>()`，修复 login handler 的 IP 提取 |
|
||||
| 移除诊断中间件 | [main.rs](../../crates/zclaw-saas/src/main.rs) | 移除 `close_connection_middleware` 和 `diag_middleware` |
|
||||
| AbortSignal 链路 | [admin-v2/src/services/](../../admin-v2/src/services/) | 44 个 service 方法 + 17 个 useQuery 传入 signal，页面切换时自动取消请求 |
|
||||
| Vite proxy 超时 | [admin-v2/vite.config.ts](../../admin-v2/vite.config.ts) | proxy timeout 30s + configure hook |
|
||||
| PG 连接池调优 | [db.rs](../../crates/zclaw-saas/src/db.rs) | max=20, min=2, acquire=5s, idle=180s |
|
||||
|
||||
---
|
||||
|
||||
## 验证结果
|
||||
|
||||
| 测试项 | 结果 |
|
||||
|--------|------|
|
||||
| Health check | 200, ~0.2s |
|
||||
| Login + JWT 获取 | 200, token 正常 |
|
||||
| Dashboard API | 200, ~0.23s |
|
||||
| Logs API | 200, ~0.22s |
|
||||
| 8 并发同端点 | 全部 200, <0.25s |
|
||||
| 10 并发混合端点 | 全部 200 |
|
||||
| 并发后 health check | 200, 正常 |
|
||||
|
||||
---
|
||||
|
||||
## 防范模式
|
||||
|
||||
### Rust async 中同步锁使用规则
|
||||
|
||||
**绝对禁止**：在 `.await` 点持有任何同步锁（`std::sync::Mutex`、`parking_lot::Mutex`、`parking_lot::RwLock`、`DashMap::Ref`/`RefMut`）。
|
||||
|
||||
```rust
|
||||
// ❌ 危险：同步锁跨 await
|
||||
let guard = some_sync_lock.lock();
|
||||
do_async_work().await; // guard 仍持有！
|
||||
|
||||
// ✅ 安全：作用域块确保锁释放
|
||||
let result = {
|
||||
let guard = some_sync_lock.lock();
|
||||
compute_something(&guard)
|
||||
}; // guard drop
|
||||
do_async_work(result).await; // 安全
|
||||
```
|
||||
|
||||
### 何时用 tokio::sync vs parking_lot
|
||||
|
||||
| 场景 | 推荐 | 原因 |
|
||||
|------|------|------|
|
||||
| 短暂持有，无 .await | `parking_lot` / `DashMap` | 性能更好，无 async 开销 |
|
||||
| 需要跨 .await 持有 | `tokio::sync::Mutex` / `RwLock` | guard 是 Send，yield 友好 |
|
||||
| 高并发读多写少 | `DashMap`（但注意作用域） | 分片锁，读不阻塞 |
|
||||
| 需要跨 .await 持有 + 高并发 | `tokio::sync::RwLock` 或消息传递 | 避免锁竞争 |
|
||||
|
||||
### 代码审计清单
|
||||
|
||||
审计 `crates/zclaw-saas/src/` 下所有 DashMap 使用：
|
||||
|
||||
| 文件 | 使用位置 | 状态 |
|
||||
|------|----------|------|
|
||||
| `middleware.rs` | `rate_limit_entries.entry()` | ✅ 已修复，作用域块限定 |
|
||||
| `state.rs` | `rate_limit_entries.retain()` (cleanup) | ✅ 安全，同步函数无 .await |
|
||||
| `state.rs` | `role_permissions_cache` 定义 | ✅ 安全，只在 handlers.rs 中使用 |
|
||||
| `auth/handlers.rs` | `get_role_permissions()` 中 `.get()` | ✅ 安全，guard 在 .await 前 drop（但模式脆弱，建议后续重构） |
|
||||
|
||||
---
|
||||
|
||||
## 外部参考
|
||||
|
||||
- [DashMap Issue #79: Deadlock in async threads](https://github.com/xacrimon/dashmap/issues/79) — DashMap 维护者确认的 canonical issue
|
||||
- [Beware of the DashMap Deadlock](https://www.gnunicorn.org/writings/beware-of-the-dashmap-deadlock/) — 深入分析 DashMap 死锁机制
|
||||
- [How to Prevent Async Deadlocks in Rust](https://savannahar68.medium.com/rust-deadlock-do-not-hold-blocking-locks-over-await-1628bf12c6d9) — 同步锁跨 await 的通用模式
|
||||
- [Turso: How to deadlock Tokio with just a single mutex](https://turso.tech/blog/how-to-deadlock-tokio-application-in-rust-with-just-a-single-mutex) — 单锁即可死锁运行时的最小复现
|
||||
- [Hyper Issue #2366: Server stops accepting connections](https://github.com/hyperium/hyper/issues/2366) — Hyper 层面的连接累积问题
|
||||
- [Axum Discussion #1094: Detect connection closed](https://github.com/tokio-rs/axum/discussions/1094) — handler 中检测客户端断开
|
||||
|
||||
---
|
||||
|
||||
## 相关文件
|
||||
|
||||
| 文件 | 角色 |
|
||||
|------|------|
|
||||
| [crates/zclaw-saas/src/middleware.rs](../../crates/zclaw-saas/src/middleware.rs) | rate_limit_middleware — 修复点 |
|
||||
| [crates/zclaw-saas/src/auth/mod.rs](../../crates/zclaw-saas/src/auth/mod.rs) | auth_middleware — 中间件链上层 |
|
||||
| [crates/zclaw-saas/src/state.rs](../../crates/zclaw-saas/src/state.rs) | AppState 定义，含 DashMap 字段 |
|
||||
| [crates/zclaw-saas/src/main.rs](../../crates/zclaw-saas/src/main.rs) | 路由构建、TimeoutLayer、ConnectInfo |
|
||||
| [crates/zclaw-saas/src/db.rs](../../crates/zclaw-saas/src/db.rs) | PG 连接池配置 |
|
||||
| [admin-v2/src/services/request.ts](../../admin-v2/src/services/request.ts) | AbortSignal 传递工具函数 |
|
||||
| [admin-v2/vite.config.ts](../../admin-v2/vite.config.ts) | Vite 代理超时配置 |
|
||||
Reference in New Issue
Block a user