34047007@qq.com
a6cd99a4ca
OncoLit: a multi-tenant oncology literature search, feed, and collaboration platform. Built with FastAPI + Vue 3 + PostgreSQL. Includes PubMed pipeline, drug approvals, AI summaries, and systematic review tools.
169 lines
9.4 KiB
Markdown
169 lines
9.4 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Development Commands
|
||
|
||
```bash
|
||
# Start dev environment (backend + frontend)
|
||
cd backend && python scripts/seed_data.py && uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
|
||
cd frontend && npm install && npm run dev
|
||
|
||
# Run tests (611 tests, ~3min)
|
||
cd backend && python -m pytest tests/ -v --no-cov
|
||
|
||
# Build frontend (type-check + prod bundle, ~5s)
|
||
cd frontend && npm run build
|
||
|
||
# Database migrations (Alembic)
|
||
cd backend && python -m alembic -c alembic/alembic.ini upgrade head # 应用迁移
|
||
cd backend && python -m alembic -c alembic/alembic.ini downgrade -1 # 回滚一步
|
||
cd backend && python -m alembic -c alembic/alembic.ini history # 查看历史
|
||
cd backend && python -m alembic -c alembic/alembic.ini current # 查看当前版本
|
||
# 自动生成迁移（修改模型后执行）:
|
||
cd backend && python -m alembic -c alembic/alembic.ini revision --autogenerate -m "变更说明"
|
||
|
||
# Docker deployment
|
||
docker compose up -d # development（自动运行迁移）
|
||
docker compose -f docker-compose.prod.yml up -d # production（自动运行迁移）
|
||
|
||
# PubMed pipeline (real data)
|
||
curl -X POST localhost:8000/api/v1/admin/pipeline/run -H "Authorization: Bearer $(admin_token)"
|
||
# FTP bulk import
|
||
cd backend && python scripts/pubmed_baseline.py --dir /path/to/pubmed/baseline/ --demo
|
||
# 批量刷新被引次数（PubMed elink，免费，3 req/s。全库约N分钟/N秒）
|
||
curl -X POST localhost:8000/api/v1/admin/pipeline/refresh-citations -H "Authorization: Bearer $(admin_token)"
|
||
curl -X POST "localhost:8000/api/v1/admin/pipeline/refresh-citations?limit=500" -H "Authorization: Bearer $(admin_token)"
|
||
```
|
||
|
||
## Architecture Overview
|
||
|
||
### Database Migrations (Alembic)
|
||
|
||
Alembic 管理数据库 schema 版本。初始迁移 `alembic/versions/e56e408b2208_initial_schema.py` 包含所有 32 个模型的建表语句。
|
||
|
||
**工作流：**
|
||
1. 修改 Python 模型（增/删/改字段）
|
||
2. 运行 `alembic revision --autogenerate -m "描述"` 自动生成迁移脚本
|
||
3. 检查生成的脚本，确认无误
|
||
4. 运行 `alembic upgrade head` 应用到数据库
|
||
|
||
**部署：** Docker Compose 启动时会自动执行 `alembic upgrade head`。
|
||
|
||
**模型修改后务必同步文档：** 每次通过 migration 增/删/改字段后，必须同步更新 `docs/03-数据库设计.md` 中对应表的 `CREATE TABLE` 定义和索引，保持与模型代码一致。
|
||
|
||
### Multi-Tenant Isolation
|
||
|
||
Two levels: `ContextVar` (application) + PostgreSQL RLS (database, production only). Every tenant-scoped request sets `tenant_ctx` via `get_current_user` (not middleware — avoids connection pool races). The JWT access token carries `tid` (tenant_id) and `is_superuser`. Admin routes (`/admin/*`) enforce `require_superuser` via router-level dependency — demo users get 403.
|
||
|
||
### Middleware Stack (inside→out)
|
||
|
||
`CORS → RateLimit → PerformanceMonitor → SecurityHeaders → CSRFProtection → CSRFCookie → Trace`
|
||
|
||
Rate limiter caches plan quotas 5 minutes. CSRF exempts Bearer tokens and `/auth/*`, `/public/*`, `/captcha/*`.
|
||
|
||
### Backend Layer Pattern
|
||
|
||
```
|
||
api/v1/ → schemas/ → services/ → models/ (DB)
|
||
↑
|
||
core/ (security, permissions, tenant_ctx, middleware)
|
||
```
|
||
|
||
Services never import from api/. Models never import from services/. `core/plans.py` defines Free/Pro/Team/Enterprise feature matrix — use `get_tenant_plan()` to check feature access.
|
||
|
||
### Database
|
||
|
||
39 SQLAlchemy 2.0 models. All `TIMESTAMPTZ` columns use `server_default=func.now()`. Pure date fields (pub_date, approval_date) use `DATE`.
|
||
|
||
`user_feed` is the only table needing partitioning — `PARTITION BY RANGE (pushed_at)` monthly.
|
||
|
||
### Feed Engine
|
||
|
||
`generate_feeds_for_literature()`: new article → match tags against active subscriptions → compute priority (must_read/recommended/related) → insert `user_feed`. Called after every PubMed pipeline run.
|
||
|
||
### Frontend Route Architecture
|
||
|
||
Three layouts: `PublicLayout` (no auth, no sidebar), `AuthLayout` (centered card), `AppLayout` (sidebar + header). Admin uses its own `AdminLayout`. All pages lazy-loaded via `() => import(...)`. Auth guard in `router.beforeEach()` checks `auth.isAuthenticated`.
|
||
|
||
### PubMed Pipeline
|
||
|
||
Two modes: API (`pubmed_api.py` using NCBI E-utilities, 3 req/s) and FTP (`pubmed_baseline.py` for bulk). Both filter articles by MeSH tags defined in `config/specialties/oncology.yaml`. Tagging is bidirectional: MeSH UI → `global_tags` lookup → `global_literature_tags` INSERT.
|
||
|
||
**两阶段召回策略：**
|
||
1. **MAJR 高精度** — `ONCOLOGY_SEARCH_QUERIES`，`[MAJR]` 限定 MeSH Major Topic，仅 indexed 记录
|
||
2. **Title/Abstract 高召回** — `BROAD_ONCOLOGY_QUERIES`，关键词覆盖 in-process + publisher 记录
|
||
|
||
`_run_pipeline()` 是共享核心（`pubmed_api.py:498`），通过 `use_majr`/`use_broad` 标志控制执行集合。标题/抽象查询通过 `seen_pmids` 集合自动去重。已存在的 PMID 通过 `_update_lit_from_article()` 原地覆盖更新。
|
||
|
||
### ARQ Scheduled Tasks (`backend/app/tasks/worker.py`)
|
||
|
||
定时任务使用 ARQ (Redis-backed)，所有时间均为 **UTC**。
|
||
|
||
| 任务 | 函数 | Cron (UTC) | 北京时间 | 说明 |
|
||
|---|---|---|---|---|
|
||
| 每日精搜 | `daily_pubmed_pipeline` | `03:07` 每天 | 11:07 | MAJR MeSH 高精度搜索，20 篇/query |
|
||
| 每周宽搜 | `weekly_broad_pipeline` | `03:37` 周日 | 11:37 | Title/Abstract 覆盖 in-process + publisher |
|
||
| 引用更新 | `daily_citation_update` | `05:13` 每天 | 13:13 | 刷新最近文献被引次数 |
|
||
| 摘要邮件 | `daily_digest_task` | `22:30` 每天 | 06:30 (次日) | 每日摘要推送 |
|
||
|
||
`weekly_broad_pipeline` 与 `daily_pubmed_pipeline` 错开 30 分钟执行，避免 PubMed API 限速竞争。
|
||
|
||
手动触发：
|
||
```bash
|
||
# 全量运行（MAJR + 宽搜）
|
||
curl -X POST localhost:8000/api/v1/admin/pipeline/run -H "Authorization: Bearer $(admin_token)"
|
||
# 仅宽搜
|
||
curl -X POST "localhost:8000/api/v1/admin/pipeline/run?mode=broad" -H "Authorization: Bearer $(admin_token)"
|
||
```
|
||
|
||
### Citation Counts (PubMed elink)
|
||
|
||
每条文献入库时**自动查询被引次数**（`pubmed_api.py:fetch_citedby_counts()`，通过 NCBI elink 免费接口，200 篇/req）。每日定时任务 `daily_citation_update` 刷新最近文献的被引。管理后台也支持手动全量刷新。
|
||
|
||
**技术要点：**
|
||
- 接口：`elink.fcgi?dbfrom=pubmed&linkname=pubmed_pubmed_citedin&id=PMID1,PMID2,...`
|
||
- 限速：无 API Key 3 req/s，有 key 10 req/s
|
||
- 批量 200 篇/次，全库 1 万篇约需 15-20 分钟免费模式
|
||
- 更新 `global_literature.cited_by_count` + `updated_at`（所以 🔄 徽标会联动）
|
||
- 新文献引用数可能为 0（刚发表，随后每次定时刷新自动更新）
|
||
|
||
**NLM 行为要点：**
|
||
- `DateRevised` **不可靠**——NLM 明确说不应依赖它判断 revision（如 MeSH 年更大批改记录但不一定赋值）。判 new/revised 的唯一稳妥方式是 PMID 撞库
|
||
- `_update_lit_from_article()` 对已存在 PMID 直接覆盖所有字段，不做"是否变了"的检查
|
||
- 每年 11 月中–12 月中是 MeSH 年更期，indexed 记录暂停灌入，只发 in-process/publisher。此时宽搜补充尤为重要
|
||
|
||
### PMC OA TDM (Full Text)
|
||
|
||
为未来深度学习 NLP（药物-靶点关系抽取、临床试验结构化提取）做数据准备。
|
||
|
||
**PMC OA 覆盖：** 约 15% 的 PubMed 文献有 PMC 全文，但高影响力期刊 OA 比例远高于此，对 oncology 领域数据挖掘足够。
|
||
|
||
**数据流：** PubMed efetch 已返回 `pmc_id` + `is_oa` → `pmc_oa.py` 用 OA Service API 下载 XML → `jats_parser.py` 解析成结构化 sections。
|
||
|
||
**存储策略：** 存 parsed sections JSON（`full_text_sections` 字段），不存原始 XML。sections 结构：
|
||
```json
|
||
{"sections": [{"heading": "Introduction", "text": "..."}, {"heading": "Methods", "text": "..."}],
|
||
"chemicals": [{"name": "Trastuzumab", "registry_number": "..."}],
|
||
"references": [{"pmid": 12345, "title": "..."}],
|
||
"tables": [{"caption": "Table 1", "data": "..."}]}
|
||
```
|
||
|
||
**实现（Phase 1）：** `app/services/pmc_oa.py` — OA Service API 客户端；`app/services/jats_parser.py` — JATS XML 解析；`GlobalLiterature.full_text_sections` JSON 字段（Alembic 迁移）；在 `_run_pipeline()` 中新增 OA 全文抓取阶段。Phase 2 回填现有 OA 文献。Phase 3 按需上 NLP 模型。
|
||
|
||
**技术要点：**
|
||
- API：`https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=PMCID` → XML 响应
|
||
- PMCID ↔ PMID 映射已在 PubMed efetch 阶段完成（`pmc_id` 字段）
|
||
- JATS XML 按 `<sec>` 标签组织，`<sec-title>` + 段落文本即可提取结构化 sections
|
||
- `<chemical>` 标签含 Registry Number，可对接 PubChem
|
||
- 限速：共用 NCBI E-utilities 池（3 req/s 无 key，10 req/s 有 key）
|
||
- 与 `_run_pipeline()` 集成：在打标 + Feed 生成后，对 `is_oa=True` 且无 `full_text_sections` 的文献执行
|
||
|
||
### Key Design Decisions
|
||
|
||
- **No multi-specialty in one deployment.** One codebase, one YAML config, one Docker stack per specialty. Oncology is the first.
|
||
- **Personal → Team upgrade is zero-data-migration.** Personal user's `tenant_id` stays the same; only `is_personal` flips and `plan_type` changes.
|
||
- **SQLite in dev, PostgreSQL in prod.** SQLAlchemy generic types enable this.
|
||
- **Dev mode password reset** returns the reset link directly in API response (no SMTP needed).
|
||
- **`user["sub"]` is a string.** Always convert to `uuid.UUID()` before passing to SQLAlchemy queries.
|