chore: add Gitea service to docker-compose, fix gitignore
CI / backend (push) Canceled after 0s
CI / frontend (push) Canceled after 0s

- Gitea (postgres-backed) on ports 3000/2222
- Exclude .claude/ dir and *.db from git
- Update CLAUDE.md with Gitea remote info
This commit is contained in:
34047007@qq.com
2026-07-27 08:13:53 +08:00
parent a6cd99a4ca
commit 036494ae35
3 changed files with 47 additions and 13 deletions
+17 -12
View File
@@ -29,7 +29,7 @@ docker compose -f docker-compose.prod.yml up -d # production(自动运行迁
# PubMed pipeline (real data)
curl -X POST localhost:8000/api/v1/admin/pipeline/run -H "Authorization: Bearer $(admin_token)"
# FTP bulk import
# FTP baseline import (first-time full data)
cd backend && python scripts/pubmed_baseline.py --dir /path/to/pubmed/baseline/ --demo
# 批量刷新被引次数(PubMed elink,免费,3 req/s。全库约N分钟/N秒)
curl -X POST localhost:8000/api/v1/admin/pipeline/refresh-citations -H "Authorization: Bearer $(admin_token)"
@@ -88,13 +88,17 @@ Three layouts: `PublicLayout` (no auth, no sidebar), `AuthLayout` (centered card
### PubMed Pipeline
Two modes: API (`pubmed_api.py` using NCBI E-utilities, 3 req/s) and FTP (`pubmed_baseline.py` for bulk). Both filter articles by MeSH tags defined in `config/specialties/oncology.yaml`. Tagging is bidirectional: MeSH UI → `global_tags` lookup → `global_literature_tags` INSERT.
**FTP 每日更新文件为主数据源**2026-07-25 决策)。取代旧 E-utilities API 多路搜索策略。
**两阶段召回策略:**
1. **MAJR 高精度**`ONCOLOGY_SEARCH_QUERIES``[MAJR]` 限定 MeSH Major Topic,仅 indexed 记录
2. **Title/Abstract 高召回**`BROAD_ONCOLOGY_QUERIES`,关键词覆盖 in-process + publisher 记录
日常运行:
1. **FTP 下载** `pubmed25updateNNNN.xml.gz`(每日 ~10MB,含全部新增/修改/删除记录
2. **解析**`_extract_article()`(复用 `pubmed_baseline.py` 的 lxml 解析器)
3. **过滤**`_is_oncology()` 按 MeSH 肿瘤科筛选(复用 `pubmed_baseline.py`
4. **处理**`<PubmedArticle>``_update_lit_from_article()` upsert`<DeleteCitation>` → 标记 `retracted=True`
5. **打标**: MeSH UI → `global_tags` lookup → `global_literature_tags` INSERT
6. **检查点**`pipeline_runs.processed_date` 记录已处理的 EDAT 日期
`_run_pipeline()` 是共享核心(`pubmed_api.py:498`),通过 `use_majr`/`use_broad` 标志控制执行集合。标题/抽象查询通过 `seen_pmids` 集合自动去重。已存在的 PMID 通过 `_update_lit_from_article()` 原地覆盖更新
降级:FTP 不可用时,回退到 `pubmed_api.py`NCBI E-utilities3 req/s)的 `reldate=1&datetype=edat` 查询
### ARQ Scheduled Tasks (`backend/app/tasks/worker.py`)
@@ -102,19 +106,19 @@ Two modes: API (`pubmed_api.py` using NCBI E-utilities, 3 req/s) and FTP (`pubme
| 任务 | 函数 | Cron (UTC) | 北京时间 | 说明 |
|---|---|---|---|---|
| 每日精搜 | `daily_pubmed_pipeline` | `03:07` 每天 | 11:07 | MAJR MeSH 高精度搜索,20 篇/query |
| 每周宽搜 | `weekly_broad_pipeline` | `03:37` 周日 | 11:37 | Title/Abstract 覆盖 in-process + publisher |
| 每日 FTP 增量 | `daily_ftp_update` | `03:07` 每天 | 11:07 | FTP 下载更新文件,处理新增/修改/删除(取代精搜+宽搜+retagger |
| 引用更新 | `daily_citation_update` | `05:13` 每天 | 13:13 | 刷新最近文献被引次数 |
| 摘要邮件 | `daily_digest_task` | `22:30` 每天 | 06:30 (次日) | 每日摘要推送 |
`weekly_broad_pipeline``daily_pubmed_pipeline` 错开 30 分钟执行,避免 PubMed API 限速竞争
> **旧任务已移除:** `daily_pubmed_pipeline`MAJR 精搜)、`weekly_broad_pipeline`Title/Abstract 宽搜)和 `mesh_retagger.py`(回查)已被 `daily_ftp_update` 完全取代。FTP 每日更新文件包含所有新/改/删记录,覆盖 MeSH in-process→medline 过渡,不再需要多路搜索和回查
手动触发:
```bash
# 全量运行(MAJR + 宽搜)
# FTP 增量更新
curl -X POST localhost:8000/api/v1/admin/pipeline/run -H "Authorization: Bearer $(admin_token)"
# 仅宽搜
curl -X POST "localhost:8000/api/v1/admin/pipeline/run?mode=broad" -H "Authorization: Bearer $(admin_token)"
# 批量刷新被引次数
curl -X POST localhost:8000/api/v1/admin/pipeline/refresh-citations -H "Authorization: Bearer $(admin_token)"
curl -X POST "localhost:8000/api/v1/admin/pipeline/refresh-citations?limit=500" -H "Authorization: Bearer $(admin_token)"
```
### Citation Counts (PubMed elink)
@@ -166,3 +170,4 @@ curl -X POST "localhost:8000/api/v1/admin/pipeline/run?mode=broad" -H "Authoriza
- **SQLite in dev, PostgreSQL in prod.** SQLAlchemy generic types enable this.
- **Dev mode password reset** returns the reset link directly in API response (no SMTP needed).
- **`user["sub"]` is a string.** Always convert to `uuid.UUID()` before passing to SQLAlchemy queries.
- **搜索功能必须与 PubMed 完全一致。** 这是硬性要求,不是"未来优化"。所有 PubMed 字段标签必须全量支持,已存储数据的立即接通搜索路径,缺失数据的补充 XML 抽取和存储。不允许任何字段退化到纯文本搜索。