feat(config): 将服务配置全面迁移至数据库
本次提交完成了一项重要的架构重构,将所有外部服务的API凭证管理从环境变量迁移到了中心化的数据库配置中。
主要变更:
1. **统一配置源**:
- `data-persistence-service` 现已提供 `/api/v1/configs/data_sources` 端点,用于统一管理数据源配置。
- 所有配置(LLM 和数据源)现在都通过数据库的 `system_config` 表进行管理,实现了“单一事实源”。
2. **增强服务韧性**:
- 重构了 `finnhub-`, `tushare-`, `alphavantage-provider-service`。
- 这些服务在启动时不再强制要求 API Key。
- 引入了动态配置轮询器 (`config_poller`),服务现在可以定期从数据库获取最新配置。
- 实现了“降级模式”:当配置缺失时,服务会进入 `Degraded` 状态并暂停处理消息,而不是直接崩溃。配置恢复后,服务会自动回到 `Active` 状态。
- `/health` 端点现在能准确反映服务的真实运行状态。
3. **前端易用性提升**:
- 您在 `/config` 页面上增加了“数据源配置”面板,允许用户通过 UI 动态更新所有 API Token。
4. **部署简化**:
- 从 `docker-compose.yml` 中移除了所有已废弃的 `_API_KEY` 环境变量,消除了启动时的警告。
这项重构显著提升了系统的可维护性、健壮性和用户体验,为未来的功能扩展奠定了坚实的基础。
This commit is contained in:
parent
a1e4b265ba
commit
60e6c8f61b
@ -67,7 +67,8 @@ services:
|
|||||||
ports:
|
ports:
|
||||||
- "13001:3001"
|
- "13001:3001"
|
||||||
depends_on:
|
depends_on:
|
||||||
- api-gateway
|
api-gateway:
|
||||||
|
condition: service_healthy
|
||||||
networks:
|
networks:
|
||||||
- app-network
|
- app-network
|
||||||
|
|
||||||
@ -95,6 +96,11 @@ services:
|
|||||||
- yfinance-provider-service
|
- yfinance-provider-service
|
||||||
networks:
|
networks:
|
||||||
- app-network
|
- app-network
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "curl -fsS http://localhost:4000/health >/dev/null || exit 1"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 12
|
||||||
|
|
||||||
alphavantage-provider-service:
|
alphavantage-provider-service:
|
||||||
build:
|
build:
|
||||||
@ -105,7 +111,6 @@ services:
|
|||||||
SERVER_PORT: 8000
|
SERVER_PORT: 8000
|
||||||
NATS_ADDR: nats://nats:4222
|
NATS_ADDR: nats://nats:4222
|
||||||
DATA_PERSISTENCE_SERVICE_URL: http://data-persistence-service:3000/api/v1
|
DATA_PERSISTENCE_SERVICE_URL: http://data-persistence-service:3000/api/v1
|
||||||
ALPHAVANTAGE_API_KEY: ${ALPHAVANTAGE_API_KEY}
|
|
||||||
RUST_LOG: info,axum=info
|
RUST_LOG: info,axum=info
|
||||||
RUST_BACKTRACE: "1"
|
RUST_BACKTRACE: "1"
|
||||||
depends_on:
|
depends_on:
|
||||||
@ -129,8 +134,6 @@ services:
|
|||||||
NATS_ADDR: nats://nats:4222
|
NATS_ADDR: nats://nats:4222
|
||||||
DATA_PERSISTENCE_SERVICE_URL: http://data-persistence-service:3000/api/v1
|
DATA_PERSISTENCE_SERVICE_URL: http://data-persistence-service:3000/api/v1
|
||||||
TUSHARE_API_URL: http://api.waditu.com
|
TUSHARE_API_URL: http://api.waditu.com
|
||||||
# Please provide your Tushare token via .env
|
|
||||||
TUSHARE_API_TOKEN: ${TUSHARE_API_TOKEN}
|
|
||||||
RUST_LOG: info,axum=info
|
RUST_LOG: info,axum=info
|
||||||
RUST_BACKTRACE: "1"
|
RUST_BACKTRACE: "1"
|
||||||
depends_on:
|
depends_on:
|
||||||
@ -154,8 +157,6 @@ services:
|
|||||||
NATS_ADDR: nats://nats:4222
|
NATS_ADDR: nats://nats:4222
|
||||||
DATA_PERSISTENCE_SERVICE_URL: http://data-persistence-service:3000/api/v1
|
DATA_PERSISTENCE_SERVICE_URL: http://data-persistence-service:3000/api/v1
|
||||||
FINNHUB_API_URL: https://finnhub.io/api/v1
|
FINNHUB_API_URL: https://finnhub.io/api/v1
|
||||||
# Please provide your Finnhub token in .env file
|
|
||||||
FINNHUB_API_KEY: ${FINNHUB_API_KEY}
|
|
||||||
RUST_LOG: info,axum=info
|
RUST_LOG: info,axum=info
|
||||||
RUST_BACKTRACE: "1"
|
RUST_BACKTRACE: "1"
|
||||||
depends_on:
|
depends_on:
|
||||||
|
|||||||
@ -29,8 +29,8 @@ owner: '@lv'
|
|||||||
```
|
```
|
||||||
+-------------+ +------------------+ +---------------------------+
|
+-------------+ +------------------+ +---------------------------+
|
||||||
| | HTTP | | | |
|
| | HTTP | | | |
|
||||||
| 前端 |----->| API 网关 |----->| 消息总线 |
|
| 前端 |----->| API 网关 |----->| 消息总线 (NATS) |
|
||||||
| (Next.js) | | (Rust) | | (例如 RabbitMQ, NATS) |
|
| (Next.js) | | (Rust) | | |
|
||||||
| | | | | |
|
| | | | | |
|
||||||
+-------------+ +-------+----------+ +-------------+-------------+
|
+-------------+ +-------+----------+ +-------------+-------------+
|
||||||
| |
|
| |
|
||||||
@ -69,7 +69,7 @@ owner: '@lv'
|
|||||||
- 为所有其他内部微服务提供稳定、统一的数据库读写 HTTP 接口。
|
- 为所有其他内部微服务提供稳定、统一的数据库读写 HTTP 接口。
|
||||||
|
|
||||||
- **消息总线 (Message Bus)**:
|
- **消息总线 (Message Bus)**:
|
||||||
- 整个系统的神经中枢,负责所有服务间的异步通信。
|
- 整个系统的神经中枢,负责所有服务间的异步通信。当前选用 **NATS** 作为具体实现。
|
||||||
|
|
||||||
## 3. `SystemModule` 核心规范
|
## 3. `SystemModule` 核心规范
|
||||||
|
|
||||||
|
|||||||
@ -0,0 +1,98 @@
|
|||||||
|
# 任务文档:配置管理重构——统一API凭证管理
|
||||||
|
|
||||||
|
- **状态**: Active
|
||||||
|
- **创建日期**: 2025-11-17
|
||||||
|
- **负责人**: @AI-Assistant
|
||||||
|
- **审查人**: @lv
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 背景与目标
|
||||||
|
|
||||||
|
### 1.1. 当前问题
|
||||||
|
|
||||||
|
当前系统对外部服务(如 Tushare, Finnhub)API Token 的管理方式存在两个主要问题:
|
||||||
|
|
||||||
|
1. **配置方式分裂**:
|
||||||
|
- **敏感凭证 (API Tokens)**: 通过启动时的**环境变量**注入。这种方式虽然安全,但缺乏灵活性,每次修改都需要重新部署或重启服务。
|
||||||
|
- **业务逻辑配置 (AI模型选择等)**: 通过**数据库**统一管理,并支持UI动态调整。
|
||||||
|
- 这种分裂的管理模式增加了系统的运维复杂性,与我们追求的“单一事实源”架构理念不符。
|
||||||
|
|
||||||
|
2. **服务韧性不足**:
|
||||||
|
- 依赖环境变量的服务采取“快速失败” (Fail-Fast) 策略。如果启动时未提供有效的 API Token,服务会立即崩溃退出。
|
||||||
|
- 这种模式虽然能尽早暴露问题,但在一个动态的、持续运行的系统中显得过于“僵硬”。我们期望的行为是:服务在缺少非核心配置时,应能进入一个“降级”状态,待配置就绪后再自动恢复工作,而不是直接停止运行。
|
||||||
|
|
||||||
|
### 1.2. 改造目标
|
||||||
|
|
||||||
|
本次重构旨在将所有外部服务的 API Token 配置,从环境变量迁移到数据库中,实现与业务逻辑配置的统一管理。具体目标如下:
|
||||||
|
|
||||||
|
- **统一配置源**: 将 `system_config` 数据库表作为所有可变配置(包括API Tokens)的唯一事实源。
|
||||||
|
- **提升易用性**: 允许用户通过前端UI界面,集中管理和更新所有数据源的 API Token。
|
||||||
|
- **增强服务韧性**: 改造数据提供商服务,使其在缺少 API Token 时不会崩溃,而是进入“降级模式”,并能在 Token 被提供后自动恢复正常工作。
|
||||||
|
- **简化部署**: 移除对多个环境变量的依赖,使服务的部署和运维过程更加简洁。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 实施方案
|
||||||
|
|
||||||
|
本次改造将遵循“后端 -> 服务 -> 前端”的顺序分层实施,确保每一步都有坚实的基础。
|
||||||
|
|
||||||
|
### 2.1. 数据模型与持久化层
|
||||||
|
|
||||||
|
我们将通过复用 `system_config` 表中现有的 `(config_key, config_value)` 存储模式,来扩展配置管理的能力,使其能够安全地存储和检索数据源的配置。
|
||||||
|
|
||||||
|
1. **定义数据结构**: 在 `common-contracts` 共享库中,定义一个清晰的、用于描述数据源配置的 `DataSourceConfig` 结构体。它将包含 `provider_id`, `api_token`, `api_url` 等字段。
|
||||||
|
2. **复用现有表结构**: 我们将向 `system_config` 表中插入一条新的记录,其 `config_key` 固定为 `"data_sources"`,并将所有数据源的配置集合(一个 `HashMap<String, DataSourceConfig>`)序列化后存入该记录的 `config_value` 字段中。
|
||||||
|
3. **扩展API**: 在 `data-persistence-service` 中增加新的 HTTP API 端点,用于对数据源配置进行增、删、改、查(CRUD)操作。例如:
|
||||||
|
- `GET /api/v1/configs/data-sources`: 获取所有数据源的配置列表。
|
||||||
|
- `PUT /api/v1/configs/data-sources`: 创建或更新所有数据源的配置。
|
||||||
|
|
||||||
|
### 2.2. 微服务改造:引入“降级与恢复”模式
|
||||||
|
|
||||||
|
这是本次重构的核心。所有依赖外部 API Token 的数据提供商服务 (`finnhub`, `tushare`, `alphavantage`) 都将进行如下改造:
|
||||||
|
|
||||||
|
1. **移除启动时检查**: 删除 `config.rs` 中检查环境变量并导致程序崩溃的逻辑。
|
||||||
|
2. **引入内部状态机**: 每个服务内部将维护一个状态(例如 `State<ServiceOperationalStatus>`),包含 `Active` 和 `Degraded(reason: String)` 两种状态。
|
||||||
|
3. **动态配置加载**: 服务将不再从环境变量读取 Token,而是在内部启动一个**后台任务**(轮询器),该任务会:
|
||||||
|
- 在服务启动时,以及之后每隔一段时间(例如 60 秒),调用 `data-persistence-service` 的新 API 来获取自己的配置。
|
||||||
|
- 如果成功获取到有效的 Token,则更新服务内部的 API 客户端,并将服务状态设置为 `Active`。此时,服务正常订阅和处理来自 NATS 的消息。
|
||||||
|
- 如果未能获取 Token(或 Token 为空),则将服务状态设置为 `Degraded`,并附上原因(如 "API Token not configured")。在这种状态下,服务**不会**订阅 NATS 消息队列,避免接收无法处理的任务。
|
||||||
|
4. **更新健康检查**: 服务的 `/health` 端点将反映其内部状态。当处于 `Degraded` 状态时,健康检查接口应返回相应的状态码和信息,以便监控系统能够清晰地了解服务当前是否可用。
|
||||||
|
|
||||||
|
### 2.3. 前端UI实现
|
||||||
|
|
||||||
|
为了让用户能够方便地管理这些配置,我们将在前端进行如下调整:
|
||||||
|
|
||||||
|
1. **创建新UI组件**: 在 `/config` 页面,新增一个名为“数据源配置”的管理面板。
|
||||||
|
2. **功能实现**: 该面板将提供一个表单或列表,允许用户:
|
||||||
|
- 查看当前所有数据源(Tushare, Finnhub 等)的配置状态。
|
||||||
|
- 为每个数据源输入或更新其 API Token。
|
||||||
|
- 保存更改。点击保存后,前端将调用 `data-persistence-service` 的新 API,将更新后的配置持久化到数据库中。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 详细任务清单
|
||||||
|
|
||||||
|
### 第一阶段:后端基础
|
||||||
|
|
||||||
|
- [x] ~~**任务 BE-1**: 在 `common-contracts` 中定义 `DataSourceConfig` 和 `DataSourceProvider` 等共享数据结构。~~
|
||||||
|
- [x] ~~**任务 BE-3**: 在 `data-persistence-service` 中实现对数据源配置的 CRUD 业务逻辑。~~
|
||||||
|
- [x] ~~**任务 BE-4**: 在 `data-persistence-service` 中暴露 `GET /api/v1/configs/data-sources` 和 `PUT /api/v1/configs/data-sources` 这两个 API 端点。~~
|
||||||
|
|
||||||
|
### 第二阶段:微服务改造
|
||||||
|
|
||||||
|
- [x] ~~**任务 SVC-1**: **(Finnhub)** 重构 `finnhub-provider-service`:~~
|
||||||
|
- [x] ~~移除 `config.rs` 中的 `FINNHUB_API_KEY` 环境变量加载逻辑。~~
|
||||||
|
- [x] ~~实现内部状态机 (`Active`/`Degraded`) 和动态配置轮询器。~~
|
||||||
|
- [x] ~~修改 `/health` 端点以反映内部状态。~~
|
||||||
|
- [x] ~~调整 NATS 消息订阅逻辑,只在 `Active` 状态下进行订阅。~~
|
||||||
|
- [x] ~~**任务 SVC-2**: **(Tushare)** 以 `finnhub-provider-service` 为模板,对 `tushare-provider-service` 进行相同的重构。~~
|
||||||
|
- [x] ~~**任务 SVC-3**: **(Alphavantage)** 以 `finnhub-provider-service` 为模板,对 `alphavantage-provider-service` 进行相同的重构。~~
|
||||||
|
- [x] ~~**任务 SVC-4**: **(审查)** 审查 `report-generator-service` 的 LLM 配置加载逻辑,确保其与新的动态配置模式在设计理念上保持一致。~~
|
||||||
|
|
||||||
|
### 第三阶段:前端实现
|
||||||
|
|
||||||
|
- [x] **任务 FE-1**: 在 `/config` 页面设计并实现“数据源配置”UI 组件。
|
||||||
|
- [x] **任务 FE-2**: 实现 `useApi.ts` 中用于获取和更新数据源配置的 hooks。
|
||||||
|
- [x] **任务 FE-3**: 将 UI 组件与 API hooks 连接,完成前端的完整功能。
|
||||||
|
- [x] **任务 FE-4**: 调整 `/llm-config` 页面,使其在UI/UX风格上与新的“数据源配置”面板保持一致性。
|
||||||
@ -0,0 +1,111 @@
|
|||||||
|
---
|
||||||
|
status: "Active"
|
||||||
|
date: "2025-11-17"
|
||||||
|
author: "AI 助手"
|
||||||
|
---
|
||||||
|
|
||||||
|
# 设计文档:分析模块编排器
|
||||||
|
|
||||||
|
## 1. 概述与目标
|
||||||
|
|
||||||
|
### 1.1. 问题陈述
|
||||||
|
|
||||||
|
我们当前基于 Rust 的后端缺少执行智能、多步骤财务分析所需的核心业务逻辑。尽管旧的 Python 系统拥有一个功能性的分析框架,但在初次重构过程中,这部分逻辑并未被迁移。本应承载此逻辑的 `report-generator-service` 服务目前仅包含一个无法正常工作的占位符实现。此外,前端配置页面缺少从零开始创建或管理分析模块的用户界面,这导致了一个“先有鸡还是先有蛋”的困境——系统中不存在任何可供配置的默认模块。
|
||||||
|
|
||||||
|
### 1.2. 目标
|
||||||
|
|
||||||
|
本任务旨在我们的 Rust 微服务架构中,设计并实现一个健壮的、可配置的**分析模块编排器(Analysis Module Orchestrator)**。该系统将复刻并改进旧 Python 系统的逻辑,以支持完全通过配置(提示词和依赖关系)来创建、管理和执行复杂的、具备依赖关系感知能力的分析工作流。
|
||||||
|
|
||||||
|
为达成此目标,需要完成以下任务:
|
||||||
|
1. 在前端为分析模块管理实现一个完整的 CRUD (创建、读取、更新、删除) 操作界面。
|
||||||
|
2. 在 `report-generator-service` 中实现一个健壮的后端编排器,使其能够基于模块依赖关系构成的有向无环图 (DAG) 来执行分析工作流。
|
||||||
|
3. 通过 `api-gateway` 和 NATS 消息总线整合前端与后端服务,以打造无缝的端到端用户体验。
|
||||||
|
4. 实现一个数据播种(Data Seeding)机制,以确保系统在首次启动时能够预加载一套默认的分析模块。
|
||||||
|
|
||||||
|
## 2. 系统架构与数据流
|
||||||
|
|
||||||
|
本次实现将涉及四个关键服务和一个消息总线:`前端`、`API 网关`、`数据持久化服务` 和 `报告生成服务`。
|
||||||
|
|
||||||
|
### 2.1. 高层数据流
|
||||||
|
|
||||||
|
1. **配置流程**:
|
||||||
|
* **用户** 在 **前端** 配置页面上进行交互,以创建或更新分析模块。
|
||||||
|
* **前端** 向 **API 网关** 发送 `PUT /api/v1/configs/analysis_modules` 请求。
|
||||||
|
* **API 网关** 将这些请求代理至 **数据持久化服务**,由其将配置保存到数据库的 `system_config` 表中。
|
||||||
|
|
||||||
|
2. **执行流程**:
|
||||||
|
* **用户** 在 **前端** 为特定的股票代码触发一次分析运行。
|
||||||
|
* **前端** 向 **API 网关** 发送 `POST /api/v1/analysis-requests/{symbol}` 请求。
|
||||||
|
* **API 网关** 验证请求,并向 **NATS 消息总线** 的一个新主题发布一条 `GenerateReportCommand` 消息。随后,它会立即向前端返回一个带有请求ID的 `202 Accepted` 响应。
|
||||||
|
* **报告生成服务** 订阅 `GenerateReportCommand` 主题,接收到消息后,启动编排工作流。
|
||||||
|
* **报告生成服务** 从 **数据持久化服务** 获取所需的分析模块配置。
|
||||||
|
* 服务执行分析,为每个模块调用 LLM API,并通过 **数据持久化服务** 将结果持久化存回数据库。
|
||||||
|
|
||||||
|
## 3. 前端实施计划 (`/config` 页面)
|
||||||
|
|
||||||
|
我们将修改 `frontend/src/app/config/page.tsx` 文件,为分析模块提供完整的 CRUD 用户体验。
|
||||||
|
|
||||||
|
- **创建 (Create)**: 添加一个“新增模块”按钮。点击后,将显示一个表单,用于输入:
|
||||||
|
- **模块 ID**: 一个唯一的、机器可读的字符串 (例如, `fundamental_analysis`)。
|
||||||
|
- **模块名称**: 一个人类可读的显示名称 (例如, "基本面分析")。
|
||||||
|
- **读取 (Read)**: 页面将为每个已存在的分析模块渲染一个卡片,展示其当前配置。
|
||||||
|
- **更新 (Update)**: 每个模块卡片将包含以下可编辑字段:
|
||||||
|
- **LLM Provider**: 一个下拉菜单,其选项从 `llm_providers` 配置中动态填充。
|
||||||
|
- **Model**: 一个级联下拉菜单,显示所选 Provider 下可用的模型。
|
||||||
|
- **提示词模板**: 一个用于编写 Prompt 的大文本区域。
|
||||||
|
- **依赖关系**: 一个包含所有其他模块ID的复选框列表,允许用户定义模块间的依赖。
|
||||||
|
- **删除 (Delete)**: 每个模块卡片将有一个带有确认对话框的“删除”按钮。
|
||||||
|
|
||||||
|
## 4. 后端实施计划
|
||||||
|
|
||||||
|
### 4.1. `data-persistence-service`
|
||||||
|
|
||||||
|
- **数据播种 (关键任务)**: 实现一次性的启动逻辑。
|
||||||
|
1. 在服务启动时,检查 `system_config` 表中是否存在键为 `analysis_modules` 的记录。
|
||||||
|
2. 如果记录**不存在**,则从磁盘读取旧的 `config/analysis-config.json` 文件。
|
||||||
|
3. 解析文件内容,并将其作为 `analysis_modules` 的值插入数据库。
|
||||||
|
4. 此机制确保系统在首次部署时,即被预置一套默认且功能完备的分析模块。
|
||||||
|
- **API**: 无需变更。现有的 `GET /configs/analysis_modules` 和 `PUT /configs/analysis_modules` 端点已能满足需求。
|
||||||
|
|
||||||
|
### 4.2. `api-gateway`
|
||||||
|
|
||||||
|
- **新端点**: 创建一个新的端点 `POST /api/v1/analysis-requests/{symbol}`。
|
||||||
|
- **逻辑**:
|
||||||
|
1. 此端点不应执行任何重度计算任务。
|
||||||
|
2. 它将从路径中接收一个股票 `symbol`。
|
||||||
|
3. 它将生成一个唯一的 `request_id` (例如, UUID)。
|
||||||
|
4. 它将构建一条包含 `symbol` 和 `request_id` 的 `GenerateReportCommand` 消息。
|
||||||
|
5. 它将此消息发布到一个专用的 NATS 主题 (例如, `analysis.commands.generate_report`)。
|
||||||
|
6. 它将立即返回一个 `202 Accepted` 状态码,并在响应体中包含 `request_id`。
|
||||||
|
|
||||||
|
### 4.3. `report-generator-service` (核心任务)
|
||||||
|
|
||||||
|
此服务需要进行最主要的开发工作。所有逻辑将在 `worker.rs` 文件中实现。
|
||||||
|
|
||||||
|
1. **消息消费者**: 服务将订阅 `analysis.commands.generate_report` NATS 主题。一旦收到 `GenerateReportCommand` 消息,即触发 `run_report_generation_workflow` 工作流。
|
||||||
|
|
||||||
|
2. **编排逻辑 (`run_report_generation_workflow`)**:
|
||||||
|
* **获取配置**: 从 `data-persistence-service` 获取完整的 `AnalysisModulesConfig`。
|
||||||
|
* **构建依赖图**: 根据模块配置,在内存中构建一个有向图。强烈推荐使用 `petgraph` crate 来完成此任务。
|
||||||
|
* **拓扑排序**: 对该图执行拓扑排序,以获得一个线性的执行顺序。该算法**必须**包含循环检测功能,以便在配置错误时能够优雅地处理,并记录错误日志。
|
||||||
|
* **顺序执行**: 遍历排序后的模块列表。对每个模块:
|
||||||
|
* **构建上下文**: 收集其所有直接依赖模块的文本输出(这些模块已保证被提前执行)。
|
||||||
|
* **渲染提示词**: 使用 `Tera` 模板引擎,将依赖模块的输出以及其他所需数据(如公司名称、财务数据)注入到当前模块的 `prompt_template` 中。
|
||||||
|
* **执行 LLM 调用**: 通过 `LlmClient` 调用相应的 LLM API。
|
||||||
|
* **持久化结果**: 成功生成内容后,调用 `data-persistence-service` 将输出文本保存,并与 `symbol` 和 `module_id` 关联。同时,将结果保存在本地,以供工作流中的后续模块使用。
|
||||||
|
|
||||||
|
3. **补全缺失逻辑**:
|
||||||
|
* 实现 `// TODO` 中关于持久化结果的部分。
|
||||||
|
* 将 `financial_data` 占位符替换为从 `data-persistence-service` 获取并格式化后的真实财务数据。
|
||||||
|
|
||||||
|
## 5. 未来工作:向 "Deep Research" 模块演进
|
||||||
|
|
||||||
|
如前所述,初始实现将依赖 LLM 的内部知识来完成“新闻”或“市场情绪”等分析。这是一个为快速实现功能而刻意选择的短期策略。
|
||||||
|
|
||||||
|
长期愿景是用一个 `Deep Research` 模块来取代这种模式。该模块将作为一个智能的数据预处理器。届时,编排器将不再注入简单的文本,而是触发 Deep Research 模块,后者将:
|
||||||
|
1. 理解目标分析模块(如 `news_analysis`)的数据需求。
|
||||||
|
2. 查询内部数据源(例如,数据库中的 `news` 表)以查找相关信息。
|
||||||
|
3. 对检索到的数据执行多步推理或摘要。
|
||||||
|
4. 为最终的分析模块提示词提供一个高质量、经过浓缩的数据包。
|
||||||
|
|
||||||
|
这一演进将使我们的系统从“提示词驱动”转变为“数据驱动”,从而显著提升分析结果的可靠性、可控性和准确性。
|
||||||
45
frontend/src/app/api/configs/data_sources/route.ts
Normal file
45
frontend/src/app/api/configs/data_sources/route.ts
Normal file
@ -0,0 +1,45 @@
|
|||||||
|
const BACKEND_BASE = process.env.BACKEND_INTERNAL_URL || process.env.NEXT_PUBLIC_BACKEND_URL;
|
||||||
|
|
||||||
|
export async function GET() {
|
||||||
|
if (!BACKEND_BASE) {
|
||||||
|
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${BACKEND_BASE}/configs/data_sources`, {
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
cache: 'no-store',
|
||||||
|
});
|
||||||
|
const text = await resp.text();
|
||||||
|
return new Response(text, {
|
||||||
|
status: resp.status,
|
||||||
|
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
||||||
|
});
|
||||||
|
} catch (e: any) {
|
||||||
|
const errorBody = JSON.stringify({ message: e?.message || '连接后端失败' });
|
||||||
|
return new Response(errorBody, { status: 502, headers: { 'Content-Type': 'application/json' } });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function PUT(req: Request) {
|
||||||
|
if (!BACKEND_BASE) {
|
||||||
|
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
||||||
|
}
|
||||||
|
const body = await req.text();
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${BACKEND_BASE}/configs/data_sources`, {
|
||||||
|
method: 'PUT',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body,
|
||||||
|
});
|
||||||
|
const text = await resp.text();
|
||||||
|
return new Response(text, {
|
||||||
|
status: resp.status,
|
||||||
|
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
||||||
|
});
|
||||||
|
} catch (e: any) {
|
||||||
|
const errorBody = JSON.stringify({ message: e?.message || '连接后端失败' });
|
||||||
|
return new Response(errorBody, { status: 502, headers: { 'Content-Type': 'application/json' } });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@ -4,6 +4,7 @@ export async function GET() {
|
|||||||
if (!BACKEND_BASE) {
|
if (!BACKEND_BASE) {
|
||||||
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
||||||
}
|
}
|
||||||
|
try {
|
||||||
const resp = await fetch(`${BACKEND_BASE}/configs/llm_providers`, {
|
const resp = await fetch(`${BACKEND_BASE}/configs/llm_providers`, {
|
||||||
headers: { 'Content-Type': 'application/json' },
|
headers: { 'Content-Type': 'application/json' },
|
||||||
cache: 'no-store',
|
cache: 'no-store',
|
||||||
@ -13,6 +14,10 @@ export async function GET() {
|
|||||||
status: resp.status,
|
status: resp.status,
|
||||||
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
||||||
});
|
});
|
||||||
|
} catch (e: any) {
|
||||||
|
const errorBody = JSON.stringify({ message: e?.message || '连接后端失败' });
|
||||||
|
return new Response(errorBody, { status: 502, headers: { 'Content-Type': 'application/json' } });
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function PUT(req: Request) {
|
export async function PUT(req: Request) {
|
||||||
@ -20,6 +25,7 @@ export async function PUT(req: Request) {
|
|||||||
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
||||||
}
|
}
|
||||||
const body = await req.text();
|
const body = await req.text();
|
||||||
|
try {
|
||||||
const resp = await fetch(`${BACKEND_BASE}/configs/llm_providers`, {
|
const resp = await fetch(`${BACKEND_BASE}/configs/llm_providers`, {
|
||||||
method: 'PUT',
|
method: 'PUT',
|
||||||
headers: { 'Content-Type': 'application/json' },
|
headers: { 'Content-Type': 'application/json' },
|
||||||
@ -30,5 +36,9 @@ export async function PUT(req: Request) {
|
|||||||
status: resp.status,
|
status: resp.status,
|
||||||
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
||||||
});
|
});
|
||||||
|
} catch (e: any) {
|
||||||
|
const errorBody = JSON.stringify({ message: e?.message || '连接后端失败' });
|
||||||
|
return new Response(errorBody, { status: 502, headers: { 'Content-Type': 'application/json' } });
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -1,13 +1,18 @@
|
|||||||
const BACKEND_BASE = process.env.NEXT_PUBLIC_BACKEND_URL;
|
const BACKEND_BASE = process.env.BACKEND_INTERNAL_URL || process.env.NEXT_PUBLIC_BACKEND_URL;
|
||||||
|
|
||||||
export async function GET(
|
export async function GET(
|
||||||
_req: Request,
|
_req: Request,
|
||||||
context: { params: Promise<{ provider_id: string }> }
|
context: any
|
||||||
) {
|
) {
|
||||||
if (!BACKEND_BASE) {
|
if (!BACKEND_BASE) {
|
||||||
return new Response('NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
return new Response('BACKEND_INTERNAL_URL/NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
||||||
|
}
|
||||||
|
const raw = context?.params;
|
||||||
|
const params = raw && typeof raw.then === 'function' ? await raw : raw;
|
||||||
|
const provider_id = params?.provider_id as string | undefined;
|
||||||
|
if (!provider_id) {
|
||||||
|
return new Response('provider_id 缺失', { status: 400 });
|
||||||
}
|
}
|
||||||
const { provider_id } = await context.params;
|
|
||||||
const target = `${BACKEND_BASE}/discover-models/${encodeURIComponent(provider_id)}`;
|
const target = `${BACKEND_BASE}/discover-models/${encodeURIComponent(provider_id)}`;
|
||||||
const resp = await fetch(target, {
|
const resp = await fetch(target, {
|
||||||
headers: { 'Content-Type': 'application/json' },
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
|||||||
26
frontend/src/app/api/discover-models/route.ts
Normal file
26
frontend/src/app/api/discover-models/route.ts
Normal file
@ -0,0 +1,26 @@
|
|||||||
|
const BACKEND_BASE = process.env.BACKEND_INTERNAL_URL || process.env.NEXT_PUBLIC_BACKEND_URL;
|
||||||
|
|
||||||
|
export async function POST(req: Request) {
|
||||||
|
if (!BACKEND_BASE) {
|
||||||
|
return new Response('BACKEND_INTERNAL_URL/NEXT_PUBLIC_BACKEND_URL 未配置', { status: 500 });
|
||||||
|
}
|
||||||
|
const body = await req.text();
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${BACKEND_BASE}/discover-models`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body,
|
||||||
|
cache: 'no-store',
|
||||||
|
});
|
||||||
|
const text = await resp.text();
|
||||||
|
return new Response(text, {
|
||||||
|
status: resp.status,
|
||||||
|
headers: { 'Content-Type': resp.headers.get('Content-Type') || 'application/json' },
|
||||||
|
});
|
||||||
|
} catch (e: any) {
|
||||||
|
const errorBody = JSON.stringify({ message: e?.message || '连接后端失败' });
|
||||||
|
return new Response(errorBody, { status: 502, headers: { 'Content-Type': 'application/json' } });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -7,6 +7,7 @@ import {
|
|||||||
LlmProvidersConfig,
|
LlmProvidersConfig,
|
||||||
AnalysisModulesConfig,
|
AnalysisModulesConfig,
|
||||||
FinancialConfigResponse,
|
FinancialConfigResponse,
|
||||||
|
DataSourcesConfig,
|
||||||
} from "@/types";
|
} from "@/types";
|
||||||
import { useEffect, useState } from "react";
|
import { useEffect, useState } from "react";
|
||||||
// Execution-step types not used currently; keep API minimal and explicit
|
// Execution-step types not used currently; keep API minimal and explicit
|
||||||
@ -335,6 +336,15 @@ export async function discoverProviderModels(providerId: string) {
|
|||||||
return res.json();
|
return res.json();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function discoverProviderModelsPreview(apiBaseUrl: string, apiKey: string) {
|
||||||
|
const res = await fetch(`/api/discover-models`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ api_base_url: apiBaseUrl, api_key: apiKey }),
|
||||||
|
});
|
||||||
|
return res.json();
|
||||||
|
}
|
||||||
|
|
||||||
// --- Analysis Modules Config Hooks ---
|
// --- Analysis Modules Config Hooks ---
|
||||||
|
|
||||||
export function useAnalysisModules() {
|
export function useAnalysisModules() {
|
||||||
@ -353,3 +363,22 @@ export async function updateAnalysisModules(payload: AnalysisModulesConfig) {
|
|||||||
}
|
}
|
||||||
return res.json() as Promise<AnalysisModulesConfig>;
|
return res.json() as Promise<AnalysisModulesConfig>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- Data Sources Config Hooks ---
|
||||||
|
|
||||||
|
export function useDataSourcesConfig() {
|
||||||
|
return useSWR<DataSourcesConfig>('/api/configs/data_sources', fetcher);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function updateDataSourcesConfig(payload: DataSourcesConfig) {
|
||||||
|
const res = await fetch('/api/configs/data_sources', {
|
||||||
|
method: 'PUT',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify(payload),
|
||||||
|
});
|
||||||
|
if (!res.ok) {
|
||||||
|
const text = await res.text().catch(() => '');
|
||||||
|
throw new Error(text || `HTTP ${res.status}`);
|
||||||
|
}
|
||||||
|
return res.json() as Promise<DataSourcesConfig>;
|
||||||
|
}
|
||||||
|
|||||||
@ -497,4 +497,19 @@ export interface AnalysisModuleConfig {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** 分析模块配置集合:键为 module_id(如 bull_case) */
|
/** 分析模块配置集合:键为 module_id(如 bull_case) */
|
||||||
export type AnalysisModulesConfig = Record<string, AnalysisModuleConfig>;
|
export type AnalysisModulesConfig = Record<string, AnalysisModuleConfig>;
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// 数据源配置类型(与后端 common-contracts 配置保持结构一致)
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
export type DataSourceProvider = 'tushare' | 'finnhub' | 'alphavantage' | 'yfinance';
|
||||||
|
|
||||||
|
export interface DataSourceConfig {
|
||||||
|
provider: DataSourceProvider;
|
||||||
|
api_key?: string | null;
|
||||||
|
api_url?: string | null;
|
||||||
|
enabled: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type DataSourcesConfig = Record<string, DataSourceConfig>;
|
||||||
@ -6,7 +6,7 @@ use axum::{
|
|||||||
Router,
|
Router,
|
||||||
};
|
};
|
||||||
use common_contracts::observability::{HealthStatus, ServiceStatus, TaskProgress};
|
use common_contracts::observability::{HealthStatus, ServiceStatus, TaskProgress};
|
||||||
use crate::state::AppState;
|
use crate::state::{AppState, ServiceOperationalStatus};
|
||||||
|
|
||||||
pub fn create_router(app_state: AppState) -> Router {
|
pub fn create_router(app_state: AppState) -> Router {
|
||||||
Router::new()
|
Router::new()
|
||||||
@ -17,14 +17,22 @@ pub fn create_router(app_state: AppState) -> Router {
|
|||||||
|
|
||||||
/// [GET /health]
|
/// [GET /health]
|
||||||
/// Provides the current health status of the module.
|
/// Provides the current health status of the module.
|
||||||
async fn health_check(State(_state): State<AppState>) -> Json<HealthStatus> {
|
async fn health_check(State(state): State<AppState>) -> Json<HealthStatus> {
|
||||||
let mut details = HashMap::new();
|
let mut details = HashMap::new();
|
||||||
// In a real scenario, we would check connections to the message bus, etc.
|
let operational_status = state.status.read().await;
|
||||||
details.insert("message_bus_connection".to_string(), "ok".to_string());
|
|
||||||
|
let (service_status, reason) = match &*operational_status {
|
||||||
|
ServiceOperationalStatus::Active => (ServiceStatus::Ok, "ok".to_string()),
|
||||||
|
ServiceOperationalStatus::Degraded { reason } => {
|
||||||
|
(ServiceStatus::Degraded, reason.clone())
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
details.insert("operational_status".to_string(), reason);
|
||||||
|
|
||||||
let status = HealthStatus {
|
let status = HealthStatus {
|
||||||
module_id: "alphavantage-provider-service".to_string(),
|
module_id: "alphavantage-provider-service".to_string(),
|
||||||
status: ServiceStatus::Ok,
|
status: service_status,
|
||||||
version: env!("CARGO_PKG_VERSION").to_string(),
|
version: env!("CARGO_PKG_VERSION").to_string(),
|
||||||
details,
|
details,
|
||||||
};
|
};
|
||||||
|
|||||||
@ -1,12 +1,12 @@
|
|||||||
use secrecy::SecretString;
|
use secrecy::SecretString;
|
||||||
use serde::Deserialize;
|
use serde::Deserialize;
|
||||||
|
|
||||||
#[derive(Deserialize, Debug)]
|
#[derive(Debug, Deserialize, Clone)]
|
||||||
pub struct AppConfig {
|
pub struct AppConfig {
|
||||||
pub server_port: u16,
|
pub server_port: u16,
|
||||||
pub nats_addr: String,
|
pub nats_addr: String,
|
||||||
pub alphavantage_api_key: SecretString,
|
|
||||||
pub data_persistence_service_url: String,
|
pub data_persistence_service_url: String,
|
||||||
|
pub alphavantage_api_key: Option<SecretString>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppConfig {
|
impl AppConfig {
|
||||||
@ -15,6 +15,14 @@ impl AppConfig {
|
|||||||
.add_source(config::Environment::default().separator("__"))
|
.add_source(config::Environment::default().separator("__"))
|
||||||
.build()?;
|
.build()?;
|
||||||
|
|
||||||
config.try_deserialize()
|
let cfg: AppConfig = config.try_deserialize()?;
|
||||||
|
|
||||||
|
if cfg.data_persistence_service_url.trim().is_empty() {
|
||||||
|
return Err(config::ConfigError::Message(
|
||||||
|
"DATA_PERSISTENCE_SERVICE_URL must not be empty".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(cfg)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
56
services/alphavantage-provider-service/src/config_poller.rs
Normal file
56
services/alphavantage-provider-service/src/config_poller.rs
Normal file
@ -0,0 +1,56 @@
|
|||||||
|
use crate::error::Result;
|
||||||
|
use crate::state::AppState;
|
||||||
|
use common_contracts::config_models::{DataSourceConfig, DataSourceProvider};
|
||||||
|
use secrecy::SecretString;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::time::Duration;
|
||||||
|
use tracing::{error, info, instrument};
|
||||||
|
|
||||||
|
const POLLING_INTERVAL_SECONDS: u64 = 60;
|
||||||
|
|
||||||
|
#[instrument(skip(state))]
|
||||||
|
pub async fn run_config_poller(state: AppState) {
|
||||||
|
info!("Starting configuration poller...");
|
||||||
|
let mut interval = tokio::time::interval(Duration::from_secs(POLLING_INTERVAL_SECONDS));
|
||||||
|
interval.tick().await; // Initial tick happens immediately
|
||||||
|
|
||||||
|
loop {
|
||||||
|
if let Err(e) = poll_and_update_config(&state).await {
|
||||||
|
error!("Failed to poll and update config: {:?}", e);
|
||||||
|
}
|
||||||
|
interval.tick().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn poll_and_update_config(state: &AppState) -> Result<()> {
|
||||||
|
info!("Polling for data source configurations...");
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
let url = format!(
|
||||||
|
"{}/configs/data_sources",
|
||||||
|
state.config.data_persistence_service_url
|
||||||
|
);
|
||||||
|
|
||||||
|
let response = client.get(&url).send().await?;
|
||||||
|
response.error_for_status_ref()?;
|
||||||
|
|
||||||
|
let configs: HashMap<String, DataSourceConfig> = response.json().await?;
|
||||||
|
|
||||||
|
let alphavantage_config = configs.values().find(|cfg| {
|
||||||
|
matches!(cfg.provider, DataSourceProvider::Alphavantage) && cfg.enabled
|
||||||
|
});
|
||||||
|
|
||||||
|
if let Some(config) = alphavantage_config {
|
||||||
|
if let Some(api_key) = &config.api_key {
|
||||||
|
state.update_provider(Some(SecretString::from(api_key.clone()))).await;
|
||||||
|
info!("Successfully updated Alphavantage provider with new configuration.");
|
||||||
|
} else {
|
||||||
|
state.update_provider(None).await;
|
||||||
|
info!("Alphavantage provider is enabled but API key is missing. Service is degraded.");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
state.update_provider(None).await;
|
||||||
|
info!("No enabled Alphavantage configuration found. Service is degraded.");
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@ -1,40 +1,41 @@
|
|||||||
|
use anyhow::anyhow;
|
||||||
|
use reqwest::Error as ReqwestError;
|
||||||
use thiserror::Error;
|
use thiserror::Error;
|
||||||
|
|
||||||
pub type Result<T> = std::result::Result<T, AppError>;
|
|
||||||
|
|
||||||
#[derive(Error, Debug)]
|
#[derive(Error, Debug)]
|
||||||
pub enum AppError {
|
pub enum AppError {
|
||||||
#[error("Configuration error: {0}")]
|
#[error("Configuration error: {0}")]
|
||||||
Configuration(String),
|
Configuration(String),
|
||||||
|
|
||||||
#[error("Message bus error: {0}")]
|
|
||||||
MessageBus(#[from] async_nats::Error),
|
|
||||||
|
|
||||||
#[error("Message bus publish error: {0}")]
|
|
||||||
MessageBusPublish(#[from] async_nats::PublishError),
|
|
||||||
|
|
||||||
#[error("Message bus subscribe error: {0}")]
|
|
||||||
MessageBusSubscribe(String),
|
|
||||||
|
|
||||||
#[error("Message bus connect error: {0}")]
|
|
||||||
MessageBusConnect(String),
|
|
||||||
|
|
||||||
#[error("HTTP request to another service failed: {0}")]
|
|
||||||
ServiceRequest(#[from] reqwest::Error),
|
|
||||||
|
|
||||||
#[error("Data parsing error: {0}")]
|
#[error("Data parsing error: {0}")]
|
||||||
DataParsing(#[from] anyhow::Error),
|
DataParsing(#[from] anyhow::Error),
|
||||||
|
|
||||||
|
#[error("Internal error: {0}")]
|
||||||
|
Internal(String),
|
||||||
|
|
||||||
|
#[error("Provider not available: {0}")]
|
||||||
|
ProviderNotAvailable(String),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Reqwest(#[from] ReqwestError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Nats(#[from] async_nats::Error),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsSubscribe(#[from] async_nats::client::SubscribeError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsUnsubscribe(#[from] async_nats::UnsubscribeError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsPublish(#[from] async_nats::error::Error<async_nats::client::PublishErrorKind>),
|
||||||
}
|
}
|
||||||
|
|
||||||
// 手动实现针对 async-nats 泛型错误类型的 From 转换
|
impl From<config::ConfigError> for AppError {
|
||||||
impl From<async_nats::error::Error<async_nats::ConnectErrorKind>> for AppError {
|
fn from(e: config::ConfigError) -> Self {
|
||||||
fn from(err: async_nats::error::Error<async_nats::ConnectErrorKind>) -> Self {
|
AppError::Configuration(e.to_string())
|
||||||
AppError::MessageBusConnect(err.to_string())
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl From<async_nats::SubscribeError> for AppError {
|
pub type Result<T, E = AppError> = std::result::Result<T, E>;
|
||||||
fn from(err: async_nats::SubscribeError) -> Self {
|
|
||||||
AppError::MessageBusSubscribe(err.to_string())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@ -7,6 +7,7 @@ mod persistence;
|
|||||||
mod state;
|
mod state;
|
||||||
mod worker;
|
mod worker;
|
||||||
mod av_client;
|
mod av_client;
|
||||||
|
mod config_poller;
|
||||||
|
|
||||||
use crate::config::AppConfig;
|
use crate::config::AppConfig;
|
||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
@ -29,6 +30,9 @@ async fn main() -> Result<()> {
|
|||||||
// Initialize application state
|
// Initialize application state
|
||||||
let app_state = AppState::new(config)?;
|
let app_state = AppState::new(config)?;
|
||||||
|
|
||||||
|
// --- Start the config poller ---
|
||||||
|
tokio::spawn(config_poller::run_config_poller(app_state.clone()));
|
||||||
|
|
||||||
// Create the Axum router
|
// Create the Axum router
|
||||||
let app = api::create_router(app_state.clone());
|
let app = api::create_router(app_state.clone());
|
||||||
|
|
||||||
|
|||||||
@ -1,26 +1,57 @@
|
|||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
use crate::state::AppState;
|
use crate::state::{AppState, ServiceOperationalStatus};
|
||||||
use common_contracts::messages::FetchCompanyDataCommand;
|
use common_contracts::messages::FetchCompanyDataCommand;
|
||||||
use futures_util::StreamExt;
|
use futures_util::StreamExt;
|
||||||
use tracing::{error, info};
|
use std::time::Duration;
|
||||||
|
use tracing::{error, info, warn};
|
||||||
|
|
||||||
const SUBJECT_NAME: &str = "data_fetch_commands";
|
const SUBJECT_NAME: &str = "data_fetch_commands";
|
||||||
|
|
||||||
pub async fn run(state: AppState) -> Result<()> {
|
pub async fn run(state: AppState) -> Result<()> {
|
||||||
info!("Starting NATS message consumer...");
|
info!("Starting NATS message consumer...");
|
||||||
|
|
||||||
let client = async_nats::connect(&state.config.nats_addr).await?;
|
loop {
|
||||||
info!("Connected to NATS.");
|
let status = state.status.read().await.clone();
|
||||||
|
if let ServiceOperationalStatus::Degraded { reason } = status {
|
||||||
|
warn!(
|
||||||
|
"Service is in degraded state (reason: {}). Pausing message consumption for 30s.",
|
||||||
|
reason
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_secs(30)).await;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
// This is a simple subscriber. For production, consider JetStream for durability.
|
info!("Service is Active. Connecting to NATS...");
|
||||||
|
match async_nats::connect(&state.config.nats_addr).await {
|
||||||
|
Ok(client) => {
|
||||||
|
info!("Successfully connected to NATS.");
|
||||||
|
if let Err(e) = subscribe_and_process(state.clone(), client).await {
|
||||||
|
error!("NATS subscription error: {}. Reconnecting in 10s...", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
error!("Failed to connect to NATS: {}. Retrying in 10s...", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn subscribe_and_process(state: AppState, client: async_nats::Client) -> Result<()> {
|
||||||
let mut subscriber = client.subscribe(SUBJECT_NAME.to_string()).await?;
|
let mut subscriber = client.subscribe(SUBJECT_NAME.to_string()).await?;
|
||||||
|
|
||||||
info!(
|
info!(
|
||||||
"Consumer started, waiting for messages on subject '{}'",
|
"Consumer started, waiting for messages on subject '{}'",
|
||||||
SUBJECT_NAME
|
SUBJECT_NAME
|
||||||
);
|
);
|
||||||
|
|
||||||
while let Some(message) = subscriber.next().await {
|
while let Some(message) = subscriber.next().await {
|
||||||
|
let current_status = state.status.read().await.clone();
|
||||||
|
if matches!(current_status, ServiceOperationalStatus::Degraded {..}) {
|
||||||
|
warn!("Service became degraded. Disconnecting from NATS and pausing consumption.");
|
||||||
|
subscriber.unsubscribe().await?;
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
info!("Received NATS message.");
|
info!("Received NATS message.");
|
||||||
let state_clone = state.clone();
|
let state_clone = state.clone();
|
||||||
let publisher_clone = client.clone();
|
let publisher_clone = client.clone();
|
||||||
@ -42,6 +73,5 @@ pub async fn run(state: AppState) -> Result<()> {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,23 +1,74 @@
|
|||||||
use std::sync::Arc;
|
use crate::av_client::AvClient;
|
||||||
|
use crate::config::AppConfig;
|
||||||
use common_contracts::observability::TaskProgress;
|
use common_contracts::observability::TaskProgress;
|
||||||
use dashmap::DashMap;
|
use dashmap::DashMap;
|
||||||
|
use secrecy::{ExposeSecret, SecretString};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::RwLock;
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
use crate::config::AppConfig;
|
|
||||||
use crate::error::Result;
|
|
||||||
|
|
||||||
pub type TaskStore = Arc<DashMap<Uuid, TaskProgress>>;
|
#[derive(Clone, Debug)]
|
||||||
|
pub enum ServiceOperationalStatus {
|
||||||
|
Active,
|
||||||
|
Degraded { reason: String },
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct AppState {
|
pub struct AppState {
|
||||||
|
pub tasks: Arc<DashMap<Uuid, TaskProgress>>,
|
||||||
pub config: Arc<AppConfig>,
|
pub config: Arc<AppConfig>,
|
||||||
pub tasks: TaskStore,
|
pub status: Arc<RwLock<ServiceOperationalStatus>>,
|
||||||
|
av_provider: Arc<RwLock<Option<Arc<AvClient>>>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppState {
|
impl AppState {
|
||||||
pub fn new(config: AppConfig) -> Result<Self> {
|
pub fn new(config: AppConfig) -> Result<Self, anyhow::Error> {
|
||||||
|
let initial_status = if config.alphavantage_api_key.is_some() {
|
||||||
|
ServiceOperationalStatus::Degraded { reason: "Initializing provider, waiting for config poller to connect.".to_string() }
|
||||||
|
} else {
|
||||||
|
ServiceOperationalStatus::Degraded { reason: "Alphavantage API Key is not configured.".to_string() }
|
||||||
|
};
|
||||||
|
|
||||||
Ok(Self {
|
Ok(Self {
|
||||||
config: Arc::new(config),
|
|
||||||
tasks: Arc::new(DashMap::new()),
|
tasks: Arc::new(DashMap::new()),
|
||||||
|
config: Arc::new(config),
|
||||||
|
status: Arc::new(RwLock::new(initial_status)),
|
||||||
|
av_provider: Arc::new(RwLock::new(None)),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub async fn get_provider(&self) -> Option<Arc<AvClient>> {
|
||||||
|
self.av_provider.read().await.clone()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn update_provider(&self, api_key: Option<SecretString>) {
|
||||||
|
let mut provider_guard = self.av_provider.write().await;
|
||||||
|
let mut status_guard = self.status.write().await;
|
||||||
|
|
||||||
|
if let Some(key) = api_key {
|
||||||
|
let mcp_endpoint = format!(
|
||||||
|
"https://mcp.alphavantage.co/mcp?apikey={}",
|
||||||
|
key.expose_secret()
|
||||||
|
);
|
||||||
|
match AvClient::connect(&mcp_endpoint).await {
|
||||||
|
Ok(new_provider) => {
|
||||||
|
*provider_guard = Some(Arc::new(new_provider));
|
||||||
|
*status_guard = ServiceOperationalStatus::Active;
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
*provider_guard = None;
|
||||||
|
*status_guard = ServiceOperationalStatus::Degraded {
|
||||||
|
reason: format!("Failed to connect to Alphavantage: {}", e),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
*provider_guard = None;
|
||||||
|
*status_guard = ServiceOperationalStatus::Degraded {
|
||||||
|
reason: "Alphavantage API Key is not configured.".to_string(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub type TaskStore = DashMap<Uuid, TaskProgress>;
|
||||||
|
|||||||
@ -1,4 +1,4 @@
|
|||||||
use crate::error::Result;
|
use crate::error::{Result, AppError};
|
||||||
use crate::mapping::{CombinedFinancials, parse_company_profile, parse_financials, parse_realtime_quote};
|
use crate::mapping::{CombinedFinancials, parse_company_profile, parse_financials, parse_realtime_quote};
|
||||||
use crate::persistence::PersistenceClient;
|
use crate::persistence::PersistenceClient;
|
||||||
use crate::state::{AppState, TaskStore};
|
use crate::state::{AppState, TaskStore};
|
||||||
@ -30,9 +30,21 @@ pub async fn handle_fetch_command(
|
|||||||
};
|
};
|
||||||
state.tasks.insert(command.request_id, task);
|
state.tasks.insert(command.request_id, task);
|
||||||
|
|
||||||
let api_key = state.config.alphavantage_api_key.expose_secret();
|
let client = match state.get_provider().await {
|
||||||
let mcp_endpoint = format!("https://mcp.alphavantage.co/mcp?apikey={}", api_key);
|
Some(p) => p,
|
||||||
let client = Arc::new(AvClient::connect(&mcp_endpoint).await?);
|
None => {
|
||||||
|
let reason = "Execution failed: Alphavantage provider is not available (misconfigured).".to_string();
|
||||||
|
error!("{}", reason);
|
||||||
|
update_task_progress(
|
||||||
|
&state.tasks,
|
||||||
|
command.request_id,
|
||||||
|
100,
|
||||||
|
&reason,
|
||||||
|
).await;
|
||||||
|
return Err(AppError::ProviderNotAvailable(reason));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
let persistence_client =
|
let persistence_client =
|
||||||
PersistenceClient::new(state.config.data_persistence_service_url.clone());
|
PersistenceClient::new(state.config.data_persistence_service_url.clone());
|
||||||
let symbol = command.symbol.clone();
|
let symbol = command.symbol.clone();
|
||||||
|
|||||||
1
services/api-gateway/Cargo.lock
generated
1
services/api-gateway/Cargo.lock
generated
@ -2997,6 +2997,7 @@ dependencies = [
|
|||||||
"tower",
|
"tower",
|
||||||
"tower-layer",
|
"tower-layer",
|
||||||
"tower-service",
|
"tower-service",
|
||||||
|
"tracing",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
|
|||||||
@ -7,7 +7,7 @@ edition = "2024"
|
|||||||
# Web Service
|
# Web Service
|
||||||
axum = "0.8.7"
|
axum = "0.8.7"
|
||||||
tokio = { version = "1", features = ["full"] }
|
tokio = { version = "1", features = ["full"] }
|
||||||
tower-http = { version = "0.6.6", features = ["cors"] }
|
tower-http = { version = "0.6.6", features = ["cors", "trace"] }
|
||||||
|
|
||||||
# Shared Contracts
|
# Shared Contracts
|
||||||
common-contracts = { path = "../common-contracts" }
|
common-contracts = { path = "../common-contracts" }
|
||||||
|
|||||||
@ -3,20 +3,12 @@ FROM rust:1.90 as builder
|
|||||||
|
|
||||||
WORKDIR /usr/src/app
|
WORKDIR /usr/src/app
|
||||||
|
|
||||||
# Pre-build dependencies to leverage Docker layer caching
|
# Deterministic dependency caching without shipping a stub binary
|
||||||
COPY ./services/common-contracts /usr/src/app/services/common-contracts
|
COPY ./services/common-contracts /usr/src/app/services/common-contracts
|
||||||
COPY ./services/api-gateway/Cargo.toml ./services/api-gateway/Cargo.lock* ./services/api-gateway/
|
COPY ./services/api-gateway/Cargo.toml ./services/api-gateway/Cargo.lock* ./services/api-gateway/
|
||||||
|
|
||||||
WORKDIR /usr/src/app/services/api-gateway
|
WORKDIR /usr/src/app/services/api-gateway
|
||||||
RUN mkdir -p src && \
|
# Copy the full source code and build the final binary
|
||||||
echo "fn main() {}" > src/main.rs && \
|
|
||||||
cargo build --release --bin api-gateway
|
|
||||||
|
|
||||||
# Copy the full source code
|
|
||||||
COPY ./services/api-gateway /usr/src/app/services/api-gateway
|
COPY ./services/api-gateway /usr/src/app/services/api-gateway
|
||||||
|
|
||||||
# Build the application
|
|
||||||
WORKDIR /usr/src/app/services/api-gateway
|
|
||||||
RUN cargo build --release --bin api-gateway
|
RUN cargo build --release --bin api-gateway
|
||||||
|
|
||||||
# 2. Runtime Stage
|
# 2. Runtime Stage
|
||||||
@ -25,7 +17,10 @@ FROM debian:bookworm-slim
|
|||||||
# Set timezone
|
# Set timezone
|
||||||
ENV TZ=Asia/Shanghai
|
ENV TZ=Asia/Shanghai
|
||||||
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
|
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates libssl3 && rm -rf /var/lib/apt/lists/*
|
# Install minimal runtime deps:
|
||||||
|
# - ca-certificates/libssl3: TLS support for outbound HTTPS
|
||||||
|
# - curl: required for container healthcheck defined in docker-compose.yml
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates libssl3 curl && rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Copy the built binary from the builder stage
|
# Copy the built binary from the builder stage
|
||||||
COPY --from=builder /usr/src/app/services/api-gateway/target/release/api-gateway /usr/local/bin/
|
COPY --from=builder /usr/src/app/services/api-gateway/target/release/api-gateway /usr/local/bin/
|
||||||
|
|||||||
@ -46,8 +46,10 @@ fn create_v1_router() -> Router<AppState> {
|
|||||||
// --- New Config Routes ---
|
// --- New Config Routes ---
|
||||||
.route("/configs/llm_providers", get(get_llm_providers_config).put(update_llm_providers_config))
|
.route("/configs/llm_providers", get(get_llm_providers_config).put(update_llm_providers_config))
|
||||||
.route("/configs/analysis_modules", get(get_analysis_modules_config).put(update_analysis_modules_config))
|
.route("/configs/analysis_modules", get(get_analysis_modules_config).put(update_analysis_modules_config))
|
||||||
// --- New Discover Route ---
|
.route("/configs/data_sources", get(get_data_sources_config).put(update_data_sources_config))
|
||||||
|
// --- New Discover Routes ---
|
||||||
.route("/discover-models/{provider_id}", get(discover_models))
|
.route("/discover-models/{provider_id}", get(discover_models))
|
||||||
|
.route("/discover-models", post(discover_models_preview))
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Health & Stateless Tasks ---
|
// --- Health & Stateless Tasks ---
|
||||||
@ -159,7 +161,7 @@ async fn get_task_progress(
|
|||||||
|
|
||||||
// --- Config API Handlers (Proxy to data-persistence-service) ---
|
// --- Config API Handlers (Proxy to data-persistence-service) ---
|
||||||
|
|
||||||
use common_contracts::config_models::{LlmProvidersConfig, AnalysisModulesConfig};
|
use common_contracts::config_models::{LlmProvidersConfig, AnalysisModulesConfig, DataSourcesConfig};
|
||||||
|
|
||||||
/// [GET /v1/configs/llm_providers]
|
/// [GET /v1/configs/llm_providers]
|
||||||
async fn get_llm_providers_config(
|
async fn get_llm_providers_config(
|
||||||
@ -195,16 +197,35 @@ async fn update_analysis_modules_config(
|
|||||||
Ok(Json(updated_config))
|
Ok(Json(updated_config))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// [GET /v1/configs/data_sources]
|
||||||
|
async fn get_data_sources_config(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
) -> Result<impl IntoResponse> {
|
||||||
|
let config = state.persistence_client.get_data_sources_config().await?;
|
||||||
|
Ok(Json(config))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [PUT /v1/configs/data_sources]
|
||||||
|
async fn update_data_sources_config(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
Json(payload): Json<DataSourcesConfig>,
|
||||||
|
) -> Result<impl IntoResponse> {
|
||||||
|
let updated_config = state.persistence_client.update_data_sources_config(&payload).await?;
|
||||||
|
Ok(Json(updated_config))
|
||||||
|
}
|
||||||
|
|
||||||
/// [GET /v1/discover-models/:provider_id]
|
/// [GET /v1/discover-models/:provider_id]
|
||||||
async fn discover_models(
|
async fn discover_models(
|
||||||
State(state): State<AppState>,
|
State(state): State<AppState>,
|
||||||
Path(provider_id): Path<String>,
|
Path(provider_id): Path<String>,
|
||||||
) -> Result<impl IntoResponse> {
|
) -> Result<impl IntoResponse> {
|
||||||
|
info!("discover_models: provider_id={}", provider_id);
|
||||||
let providers = state.persistence_client.get_llm_providers_config().await?;
|
let providers = state.persistence_client.get_llm_providers_config().await?;
|
||||||
|
|
||||||
if let Some(provider) = providers.get(&provider_id) {
|
if let Some(provider) = providers.get(&provider_id) {
|
||||||
let client = reqwest::Client::new();
|
let client = reqwest::Client::new();
|
||||||
let url = format!("{}/models", provider.api_base_url.trim_end_matches('/'));
|
let url = format!("{}/models", provider.api_base_url.trim_end_matches('/'));
|
||||||
|
info!("discover_models: target_url={} (provider_id={})", url, provider_id);
|
||||||
|
|
||||||
let response = client
|
let response = client
|
||||||
.get(&url)
|
.get(&url)
|
||||||
@ -215,10 +236,7 @@ async fn discover_models(
|
|||||||
if !response.status().is_success() {
|
if !response.status().is_success() {
|
||||||
let status = response.status();
|
let status = response.status();
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
warn!(
|
warn!("discover_models failed: provider_id={} status={} body={}", provider_id, status, error_text);
|
||||||
"Failed to discover models for provider '{}'. Status: {}, Body: {}",
|
|
||||||
provider_id, status, error_text
|
|
||||||
);
|
|
||||||
// Return a structured error to the frontend
|
// Return a structured error to the frontend
|
||||||
return Ok((
|
return Ok((
|
||||||
StatusCode::BAD_GATEWAY,
|
StatusCode::BAD_GATEWAY,
|
||||||
@ -232,9 +250,52 @@ async fn discover_models(
|
|||||||
let models_json: serde_json::Value = response.json().await?;
|
let models_json: serde_json::Value = response.json().await?;
|
||||||
Ok((StatusCode::OK, Json(models_json)).into_response())
|
Ok((StatusCode::OK, Json(models_json)).into_response())
|
||||||
} else {
|
} else {
|
||||||
|
warn!("discover_models: provider not found: {}", provider_id);
|
||||||
Ok((
|
Ok((
|
||||||
StatusCode::NOT_FOUND,
|
StatusCode::NOT_FOUND,
|
||||||
Json(serde_json::json!({ "error": "Provider not found" })),
|
Json(serde_json::json!({ "error": "Provider not found" })),
|
||||||
).into_response())
|
).into_response())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct DiscoverPreviewRequest {
|
||||||
|
api_base_url: String,
|
||||||
|
api_key: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [POST /v1/discover-models]
|
||||||
|
/// Preview discovery without persisting provider configuration.
|
||||||
|
async fn discover_models_preview(
|
||||||
|
Json(payload): Json<DiscoverPreviewRequest>,
|
||||||
|
) -> Result<impl IntoResponse> {
|
||||||
|
let redacted_key = if payload.api_key.is_empty() { "<empty>" } else { "<redacted>" };
|
||||||
|
info!(
|
||||||
|
"discover_models_preview: target_url={}/models api_key={}",
|
||||||
|
payload.api_base_url.trim_end_matches('/'),
|
||||||
|
redacted_key
|
||||||
|
);
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
let url = format!("{}/models", payload.api_base_url.trim_end_matches('/'));
|
||||||
|
let response = client
|
||||||
|
.get(&url)
|
||||||
|
.bearer_auth(&payload.api_key)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let error_text = response.text().await?;
|
||||||
|
warn!("discover_models_preview failed: status={} body={}", status, error_text);
|
||||||
|
return Ok((
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
Json(serde_json::json!({
|
||||||
|
"error": "Failed to fetch models from provider",
|
||||||
|
"provider_error": error_text,
|
||||||
|
})),
|
||||||
|
).into_response());
|
||||||
|
}
|
||||||
|
|
||||||
|
let models_json: serde_json::Value = response.json().await?;
|
||||||
|
Ok((StatusCode::OK, Json(models_json)).into_response())
|
||||||
|
}
|
||||||
|
|||||||
@ -7,11 +7,36 @@ mod persistence;
|
|||||||
use crate::config::AppConfig;
|
use crate::config::AppConfig;
|
||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
use crate::state::AppState;
|
use crate::state::AppState;
|
||||||
use tracing::info;
|
use tracing::{error, info};
|
||||||
|
use tracing_subscriber::EnvFilter;
|
||||||
use std::process;
|
use std::process;
|
||||||
|
use std::io::{self, Write};
|
||||||
|
use tower_http::trace::TraceLayer;
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
async fn main() {
|
async fn main() {
|
||||||
|
// Ensure panics are clearly printed with backtraces (independent of env var)
|
||||||
|
std::panic::set_hook(Box::new(|panic_info| {
|
||||||
|
eprintln!("panic in api-gateway: {}", panic_info);
|
||||||
|
eprintln!("backtrace:\n{:?}", std::backtrace::Backtrace::force_capture());
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Emit earliest visibility logs to stderr for containers that restart quickly
|
||||||
|
let ts = std::time::SystemTime::now()
|
||||||
|
.duration_since(std::time::UNIX_EPOCH)
|
||||||
|
.map(|d| d.as_secs())
|
||||||
|
.unwrap_or(0);
|
||||||
|
eprintln!("api-gateway launching: pid={}, ts_unix={}", process::id(), ts);
|
||||||
|
// Print critical environment variables relevant to configuration (no secrets)
|
||||||
|
eprintln!(
|
||||||
|
"env: SERVER_PORT={:?}, NATS_ADDR={:?}, DATA_PERSISTENCE_SERVICE_URL={:?}, PROVIDER_SERVICES.len={}",
|
||||||
|
std::env::var("SERVER_PORT").ok(),
|
||||||
|
std::env::var("NATS_ADDR").ok(),
|
||||||
|
std::env::var("DATA_PERSISTENCE_SERVICE_URL").ok(),
|
||||||
|
std::env::var("PROVIDER_SERVICES").ok().map(|s| s.len()).unwrap_or(0),
|
||||||
|
);
|
||||||
|
let _ = io::stderr().flush();
|
||||||
|
|
||||||
if let Err(e) = run().await {
|
if let Err(e) = run().await {
|
||||||
eprintln!("api-gateway failed to start: {}", e);
|
eprintln!("api-gateway failed to start: {}", e);
|
||||||
process::exit(1);
|
process::exit(1);
|
||||||
@ -19,9 +44,16 @@ async fn main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async fn run() -> Result<()> {
|
async fn run() -> Result<()> {
|
||||||
// Initialize logging
|
// Initialize deterministic logging, default to info if not provided
|
||||||
|
let env_filter = EnvFilter::try_from_default_env()
|
||||||
|
.unwrap_or_else(|_| EnvFilter::new("info,axum=info,hyper=info"));
|
||||||
tracing_subscriber::fmt()
|
tracing_subscriber::fmt()
|
||||||
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
.with_env_filter(env_filter)
|
||||||
|
.with_target(true)
|
||||||
|
.with_thread_ids(true)
|
||||||
|
.with_thread_names(true)
|
||||||
|
.with_ansi(false)
|
||||||
|
.compact()
|
||||||
.init();
|
.init();
|
||||||
|
|
||||||
info!("Starting api-gateway service...");
|
info!("Starting api-gateway service...");
|
||||||
@ -29,20 +61,36 @@ async fn run() -> Result<()> {
|
|||||||
// Load configuration
|
// Load configuration
|
||||||
let config = AppConfig::load().map_err(|e| error::AppError::Configuration(e.to_string()))?;
|
let config = AppConfig::load().map_err(|e| error::AppError::Configuration(e.to_string()))?;
|
||||||
let port = config.server_port;
|
let port = config.server_port;
|
||||||
|
info!(
|
||||||
|
server_port = port,
|
||||||
|
nats_addr = %config.nats_addr,
|
||||||
|
persistence_url = %config.data_persistence_service_url,
|
||||||
|
"Loaded configuration"
|
||||||
|
);
|
||||||
info!("Configured provider services: {:?}", config.provider_services);
|
info!("Configured provider services: {:?}", config.provider_services);
|
||||||
|
|
||||||
// Initialize application state
|
// Initialize application state
|
||||||
let app_state = AppState::new(config).await?;
|
let app_state = AppState::new(config).await?;
|
||||||
|
|
||||||
// Create the Axum router
|
// Create the Axum router
|
||||||
let app = api::create_router(app_state);
|
let app = api::create_router(app_state)
|
||||||
|
// Request-level tracing for better observability in Tilt/Compose logs
|
||||||
|
.layer(TraceLayer::new_for_http());
|
||||||
|
|
||||||
// Start the HTTP server
|
// Start the HTTP server
|
||||||
let listener = tokio::net::TcpListener::bind(format!("0.0.0.0:{}", port))
|
let addr = format!("0.0.0.0:{}", port);
|
||||||
.await
|
let listener = match tokio::net::TcpListener::bind(&addr).await {
|
||||||
.unwrap();
|
Ok(l) => l,
|
||||||
|
Err(e) => {
|
||||||
|
error!(%addr, err = %e, "Failed to bind TCP listener");
|
||||||
|
return Err(error::AppError::Anyhow(anyhow::anyhow!(e)));
|
||||||
|
}
|
||||||
|
};
|
||||||
info!("HTTP server listening on port {}", port);
|
info!("HTTP server listening on port {}", port);
|
||||||
axum::serve(listener, app).await.unwrap();
|
if let Err(e) = axum::serve(listener, app).await {
|
||||||
|
error!(err = %e, "HTTP server terminated with error");
|
||||||
|
return Err(error::AppError::Anyhow(anyhow::anyhow!(e)));
|
||||||
|
}
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
use common_contracts::dtos::CompanyProfileDto;
|
use common_contracts::dtos::CompanyProfileDto;
|
||||||
use common_contracts::config_models::{LlmProvidersConfig, AnalysisModulesConfig};
|
use common_contracts::config_models::{LlmProvidersConfig, AnalysisModulesConfig, DataSourcesConfig};
|
||||||
|
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct PersistenceClient {
|
pub struct PersistenceClient {
|
||||||
@ -88,4 +88,31 @@ impl PersistenceClient {
|
|||||||
.await?;
|
.await?;
|
||||||
Ok(updated_config)
|
Ok(updated_config)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub async fn get_data_sources_config(&self) -> Result<DataSourcesConfig> {
|
||||||
|
let url = format!("{}/configs/data_sources", self.base_url);
|
||||||
|
let config = self
|
||||||
|
.client
|
||||||
|
.get(&url)
|
||||||
|
.send()
|
||||||
|
.await?
|
||||||
|
.error_for_status()?
|
||||||
|
.json::<DataSourcesConfig>()
|
||||||
|
.await?;
|
||||||
|
Ok(config)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn update_data_sources_config(&self, payload: &DataSourcesConfig) -> Result<DataSourcesConfig> {
|
||||||
|
let url = format!("{}/configs/data_sources", self.base_url);
|
||||||
|
let updated_config = self
|
||||||
|
.client
|
||||||
|
.put(&url)
|
||||||
|
.json(payload)
|
||||||
|
.send()
|
||||||
|
.await?
|
||||||
|
.error_for_status()?
|
||||||
|
.json::<DataSourcesConfig>()
|
||||||
|
.await?;
|
||||||
|
Ok(updated_config)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -34,3 +34,29 @@ pub struct AnalysisModuleConfig {
|
|||||||
|
|
||||||
// 整个分析模块配置集合的数据结构
|
// 整个分析模块配置集合的数据结构
|
||||||
pub type AnalysisModulesConfig = HashMap<String, AnalysisModuleConfig>; // Key: module_id, e.g., "bull_case"
|
pub type AnalysisModulesConfig = HashMap<String, AnalysisModuleConfig>; // Key: module_id, e.g., "bull_case"
|
||||||
|
|
||||||
|
#[derive(Serialize, Deserialize, Debug, Clone, Default)]
|
||||||
|
pub struct SystemConfig {
|
||||||
|
pub llm_providers: LlmProvidersConfig,
|
||||||
|
pub analysis_modules: AnalysisModulesConfig,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, ToSchema)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum DataSourceProvider {
|
||||||
|
Tushare,
|
||||||
|
Finnhub,
|
||||||
|
Alphavantage,
|
||||||
|
Yfinance,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, ToSchema)]
|
||||||
|
pub struct DataSourceConfig {
|
||||||
|
pub provider: DataSourceProvider,
|
||||||
|
pub api_key: Option<String>,
|
||||||
|
pub api_url: Option<String>,
|
||||||
|
pub enabled: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
// 数据源配置集合(集中、强类型、单一来源)
|
||||||
|
pub type DataSourcesConfig = HashMap<String, DataSourceConfig>;
|
||||||
|
|||||||
@ -1,6 +1,9 @@
|
|||||||
use axum::{extract::State, Json};
|
use axum::{extract::State, Json};
|
||||||
use common_contracts::config_models::{LlmProvidersConfig, AnalysisModulesConfig};
|
use common_contracts::config_models::{
|
||||||
|
AnalysisModulesConfig, DataSourceConfig, LlmProvidersConfig,
|
||||||
|
};
|
||||||
use service_kit::api;
|
use service_kit::api;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
use crate::{db::system_config, AppState, ServerError};
|
use crate::{db::system_config, AppState, ServerError};
|
||||||
|
|
||||||
@ -41,3 +44,32 @@ pub async fn update_analysis_modules_config(
|
|||||||
let updated_config = system_config::update_config(pool, "analysis_modules", &payload).await?;
|
let updated_config = system_config::update_config(pool, "analysis_modules", &payload).await?;
|
||||||
Ok(Json(updated_config))
|
Ok(Json(updated_config))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub type DataSourcesConfig = HashMap<String, DataSourceConfig>;
|
||||||
|
|
||||||
|
#[api(
|
||||||
|
GET,
|
||||||
|
"/api/v1/configs/data_sources",
|
||||||
|
output(detail = "DataSourcesConfig")
|
||||||
|
)]
|
||||||
|
pub async fn get_data_sources_config(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
) -> Result<Json<DataSourcesConfig>, ServerError> {
|
||||||
|
let pool = state.pool();
|
||||||
|
let config = system_config::get_config::<DataSourcesConfig>(pool, "data_sources").await?;
|
||||||
|
Ok(Json(config))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[api(
|
||||||
|
PUT,
|
||||||
|
"/api/v1/configs/data_sources",
|
||||||
|
output(detail = "DataSourcesConfig")
|
||||||
|
)]
|
||||||
|
pub async fn update_data_sources_config(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
Json(payload): Json<DataSourcesConfig>,
|
||||||
|
) -> Result<Json<DataSourcesConfig>, ServerError> {
|
||||||
|
let pool = state.pool();
|
||||||
|
let updated_config = system_config::update_config(pool, "data_sources", &payload).await?;
|
||||||
|
Ok(Json(updated_config))
|
||||||
|
}
|
||||||
|
|||||||
@ -1 +1,12 @@
|
|||||||
|
use common_contracts::config_models::{AnalysisModulesConfig, LlmProvidersConfig, DataSourceConfig};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
pub use common_contracts::models::*;
|
pub use common_contracts::models::*;
|
||||||
|
|
||||||
|
#[derive(Serialize, Deserialize, Debug, Clone, Default)]
|
||||||
|
pub struct SystemConfig {
|
||||||
|
pub llm_providers: LlmProvidersConfig,
|
||||||
|
pub analysis_modules: AnalysisModulesConfig,
|
||||||
|
pub data_sources: HashMap<String, DataSourceConfig>,
|
||||||
|
}
|
||||||
|
|||||||
@ -6,7 +6,7 @@ use axum::{
|
|||||||
Router,
|
Router,
|
||||||
};
|
};
|
||||||
use common_contracts::observability::{HealthStatus, ServiceStatus, TaskProgress};
|
use common_contracts::observability::{HealthStatus, ServiceStatus, TaskProgress};
|
||||||
use crate::state::AppState;
|
use crate::state::{AppState, ServiceOperationalStatus};
|
||||||
|
|
||||||
pub fn create_router(app_state: AppState) -> Router {
|
pub fn create_router(app_state: AppState) -> Router {
|
||||||
Router::new()
|
Router::new()
|
||||||
@ -17,14 +17,22 @@ pub fn create_router(app_state: AppState) -> Router {
|
|||||||
|
|
||||||
/// [GET /health]
|
/// [GET /health]
|
||||||
/// Provides the current health status of the module.
|
/// Provides the current health status of the module.
|
||||||
async fn health_check(State(_state): State<AppState>) -> Json<HealthStatus> {
|
async fn health_check(State(state): State<AppState>) -> Json<HealthStatus> {
|
||||||
let mut details = HashMap::new();
|
let mut details = HashMap::new();
|
||||||
// In a real scenario, we would check connections to the message bus, etc.
|
let operational_status = state.status.read().await;
|
||||||
details.insert("message_bus_connection".to_string(), "ok".to_string());
|
|
||||||
|
let (service_status, reason) = match &*operational_status {
|
||||||
|
ServiceOperationalStatus::Active => (ServiceStatus::Ok, "ok".to_string()),
|
||||||
|
ServiceOperationalStatus::Degraded { reason } => {
|
||||||
|
(ServiceStatus::Degraded, reason.clone())
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
details.insert("operational_status".to_string(), reason);
|
||||||
|
|
||||||
let status = HealthStatus {
|
let status = HealthStatus {
|
||||||
module_id: "finnhub-provider-service".to_string(),
|
module_id: "finnhub-provider-service".to_string(),
|
||||||
status: ServiceStatus::Ok,
|
status: service_status,
|
||||||
version: env!("CARGO_PKG_VERSION").to_string(),
|
version: env!("CARGO_PKG_VERSION").to_string(),
|
||||||
details,
|
details,
|
||||||
};
|
};
|
||||||
|
|||||||
@ -7,7 +7,7 @@ pub struct AppConfig {
|
|||||||
pub nats_addr: String,
|
pub nats_addr: String,
|
||||||
pub data_persistence_service_url: String,
|
pub data_persistence_service_url: String,
|
||||||
pub finnhub_api_url: String,
|
pub finnhub_api_url: String,
|
||||||
pub finnhub_api_key: SecretString,
|
pub finnhub_api_key: Option<SecretString>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppConfig {
|
impl AppConfig {
|
||||||
@ -39,11 +39,6 @@ impl AppConfig {
|
|||||||
"FINNHUB_API_URL must not be empty".to_string(),
|
"FINNHUB_API_URL must not be empty".to_string(),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
if cfg.finnhub_api_key.expose_secret().trim().is_empty() {
|
|
||||||
return Err(config::ConfigError::Message(
|
|
||||||
"FINNHUB_API_KEY must not be empty".to_string(),
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(cfg)
|
Ok(cfg)
|
||||||
}
|
}
|
||||||
|
|||||||
56
services/finnhub-provider-service/src/config_poller.rs
Normal file
56
services/finnhub-provider-service/src/config_poller.rs
Normal file
@ -0,0 +1,56 @@
|
|||||||
|
use crate::error::Result;
|
||||||
|
use crate::state::AppState;
|
||||||
|
use common_contracts::config_models::{DataSourceConfig, DataSourceProvider};
|
||||||
|
use secrecy::SecretString;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::time::Duration;
|
||||||
|
use tracing::{error, info, instrument};
|
||||||
|
|
||||||
|
const POLLING_INTERVAL_SECONDS: u64 = 60;
|
||||||
|
|
||||||
|
#[instrument(skip(state))]
|
||||||
|
pub async fn run_config_poller(state: AppState) {
|
||||||
|
info!("Starting configuration poller...");
|
||||||
|
let mut interval = tokio::time::interval(Duration::from_secs(POLLING_INTERVAL_SECONDS));
|
||||||
|
interval.tick().await; // Initial tick happens immediately
|
||||||
|
|
||||||
|
loop {
|
||||||
|
if let Err(e) = poll_and_update_config(&state).await {
|
||||||
|
error!("Failed to poll and update config: {:?}", e);
|
||||||
|
}
|
||||||
|
interval.tick().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn poll_and_update_config(state: &AppState) -> Result<()> {
|
||||||
|
info!("Polling for data source configurations...");
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
let url = format!(
|
||||||
|
"{}/configs/data_sources",
|
||||||
|
state.config.data_persistence_service_url
|
||||||
|
);
|
||||||
|
|
||||||
|
let response = client.get(&url).send().await?;
|
||||||
|
response.error_for_status_ref()?;
|
||||||
|
|
||||||
|
let configs: HashMap<String, DataSourceConfig> = response.json().await?;
|
||||||
|
|
||||||
|
let finnhub_config = configs.values().find(|cfg| {
|
||||||
|
matches!(cfg.provider, DataSourceProvider::Finnhub) && cfg.enabled
|
||||||
|
});
|
||||||
|
|
||||||
|
if let Some(config) = finnhub_config {
|
||||||
|
if let Some(api_key) = &config.api_key {
|
||||||
|
state.update_provider(Some(SecretString::from(api_key.clone()))).await;
|
||||||
|
info!("Successfully updated Finnhub provider with new configuration.");
|
||||||
|
} else {
|
||||||
|
state.update_provider(None).await;
|
||||||
|
info!("Finnhub provider is enabled but API key is missing. Service is degraded.");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
state.update_provider(None).await;
|
||||||
|
info!("No enabled Finnhub configuration found. Service is degraded.");
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@ -1,40 +1,38 @@
|
|||||||
|
use anyhow::anyhow;
|
||||||
|
use reqwest::Error as ReqwestError;
|
||||||
use thiserror::Error;
|
use thiserror::Error;
|
||||||
|
|
||||||
pub type Result<T> = std::result::Result<T, AppError>;
|
|
||||||
|
|
||||||
#[derive(Error, Debug)]
|
#[derive(Error, Debug)]
|
||||||
pub enum AppError {
|
pub enum AppError {
|
||||||
#[error("Configuration error: {0}")]
|
#[error("Configuration error: {0}")]
|
||||||
Configuration(String),
|
Configuration(String),
|
||||||
|
|
||||||
#[error("Message bus error: {0}")]
|
|
||||||
MessageBus(#[from] async_nats::Error),
|
|
||||||
|
|
||||||
#[error("Message bus publish error: {0}")]
|
|
||||||
MessageBusPublish(#[from] async_nats::PublishError),
|
|
||||||
|
|
||||||
#[error("Message bus subscribe error: {0}")]
|
|
||||||
MessageBusSubscribe(String),
|
|
||||||
|
|
||||||
#[error("Message bus connect error: {0}")]
|
|
||||||
MessageBusConnect(String),
|
|
||||||
|
|
||||||
#[error("HTTP request to another service failed: {0}")]
|
|
||||||
ServiceRequest(#[from] reqwest::Error),
|
|
||||||
|
|
||||||
#[error("Data parsing error: {0}")]
|
#[error("Data parsing error: {0}")]
|
||||||
DataParsing(#[from] anyhow::Error),
|
DataParsing(#[from] anyhow::Error),
|
||||||
|
|
||||||
|
#[error("Provider not available: {0}")]
|
||||||
|
ProviderNotAvailable(String),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Reqwest(#[from] ReqwestError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Nats(#[from] async_nats::Error),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsSubscribe(#[from] async_nats::client::SubscribeError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsUnsubscribe(#[from] async_nats::UnsubscribeError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsPublish(#[from] async_nats::error::Error<async_nats::client::PublishErrorKind>),
|
||||||
}
|
}
|
||||||
|
|
||||||
// 手动实现针对 async-nats 泛型错误类型的 From 转换
|
impl From<config::ConfigError> for AppError {
|
||||||
impl From<async_nats::error::Error<async_nats::ConnectErrorKind>> for AppError {
|
fn from(e: config::ConfigError) -> Self {
|
||||||
fn from(err: async_nats::error::Error<async_nats::ConnectErrorKind>) -> Self {
|
AppError::Configuration(e.to_string())
|
||||||
AppError::MessageBusConnect(err.to_string())
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl From<async_nats::SubscribeError> for AppError {
|
pub type Result<T, E = AppError> = std::result::Result<T, E>;
|
||||||
fn from(err: async_nats::SubscribeError) -> Self {
|
|
||||||
AppError::MessageBusSubscribe(err.to_string())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@ -55,6 +55,7 @@ pub struct ReportItem {
|
|||||||
pub label: String,
|
pub label: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
pub struct FinnhubDataProvider {
|
pub struct FinnhubDataProvider {
|
||||||
client: FinnhubClient,
|
client: FinnhubClient,
|
||||||
}
|
}
|
||||||
|
|||||||
@ -8,6 +8,7 @@ mod message_consumer;
|
|||||||
mod persistence;
|
mod persistence;
|
||||||
mod state;
|
mod state;
|
||||||
mod worker;
|
mod worker;
|
||||||
|
mod config_poller;
|
||||||
|
|
||||||
use crate::config::AppConfig;
|
use crate::config::AppConfig;
|
||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
@ -30,6 +31,9 @@ async fn main() -> Result<()> {
|
|||||||
// Initialize application state
|
// Initialize application state
|
||||||
let app_state = AppState::new(config);
|
let app_state = AppState::new(config);
|
||||||
|
|
||||||
|
// --- Start the config poller ---
|
||||||
|
tokio::spawn(config_poller::run_config_poller(app_state.clone()));
|
||||||
|
|
||||||
// Create the Axum router
|
// Create the Axum router
|
||||||
let app = api::create_router(app_state.clone());
|
let app = api::create_router(app_state.clone());
|
||||||
|
|
||||||
|
|||||||
@ -1,27 +1,58 @@
|
|||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
use crate::state::AppState;
|
use crate::state::{AppState, ServiceOperationalStatus};
|
||||||
use common_contracts::messages::FetchCompanyDataCommand;
|
use common_contracts::messages::FetchCompanyDataCommand;
|
||||||
use futures_util::StreamExt;
|
use futures_util::StreamExt;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use tracing::{error, info};
|
use std::time::Duration;
|
||||||
|
use tracing::{error, info, warn};
|
||||||
|
|
||||||
const SUBJECT_NAME: &str = "data_fetch_commands";
|
const SUBJECT_NAME: &str = "data_fetch_commands";
|
||||||
|
|
||||||
pub async fn run(state: AppState) -> Result<()> {
|
pub async fn run(state: AppState) -> Result<()> {
|
||||||
info!("Starting NATS message consumer...");
|
info!("Starting NATS message consumer...");
|
||||||
|
|
||||||
let client = async_nats::connect(&state.config.nats_addr).await?;
|
loop {
|
||||||
info!("Connected to NATS.");
|
let status = state.status.read().await.clone();
|
||||||
|
if let ServiceOperationalStatus::Degraded { reason } = status {
|
||||||
|
warn!(
|
||||||
|
"Service is in degraded state (reason: {}). Pausing message consumption for 30s.",
|
||||||
|
reason
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_secs(30)).await;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
// This is a simple subscriber. For production, consider JetStream for durability.
|
info!("Service is Active. Connecting to NATS...");
|
||||||
|
match async_nats::connect(&state.config.nats_addr).await {
|
||||||
|
Ok(client) => {
|
||||||
|
info!("Successfully connected to NATS.");
|
||||||
|
if let Err(e) = subscribe_and_process(state.clone(), client).await {
|
||||||
|
error!("NATS subscription error: {}. Reconnecting in 10s...", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
error!("Failed to connect to NATS: {}. Retrying in 10s...", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn subscribe_and_process(state: AppState, client: async_nats::Client) -> Result<()> {
|
||||||
let mut subscriber = client.subscribe(SUBJECT_NAME.to_string()).await?;
|
let mut subscriber = client.subscribe(SUBJECT_NAME.to_string()).await?;
|
||||||
|
|
||||||
info!(
|
info!(
|
||||||
"Consumer started, waiting for messages on subject '{}'",
|
"Consumer started, waiting for messages on subject '{}'",
|
||||||
SUBJECT_NAME
|
SUBJECT_NAME
|
||||||
);
|
);
|
||||||
|
|
||||||
while let Some(message) = subscriber.next().await {
|
while let Some(message) = subscriber.next().await {
|
||||||
|
let current_status = state.status.read().await.clone();
|
||||||
|
if matches!(current_status, ServiceOperationalStatus::Degraded {..}) {
|
||||||
|
warn!("Service became degraded. Disconnecting from NATS and pausing consumption.");
|
||||||
|
subscriber.unsubscribe().await?;
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
info!("Received NATS message.");
|
info!("Received NATS message.");
|
||||||
let state_clone = state.clone();
|
let state_clone = state.clone();
|
||||||
let publisher_clone = client.clone();
|
let publisher_clone = client.clone();
|
||||||
@ -30,6 +61,16 @@ pub async fn run(state: AppState) -> Result<()> {
|
|||||||
match serde_json::from_slice::<FetchCompanyDataCommand>(&message.payload) {
|
match serde_json::from_slice::<FetchCompanyDataCommand>(&message.payload) {
|
||||||
Ok(command) => {
|
Ok(command) => {
|
||||||
info!("Deserialized command for symbol: {}", command.symbol);
|
info!("Deserialized command for symbol: {}", command.symbol);
|
||||||
|
|
||||||
|
// Skip processing if market is 'CN'
|
||||||
|
if command.market.to_uppercase() == "CN" {
|
||||||
|
info!(
|
||||||
|
"Skipping command for symbol '{}' as its market ('{}') is 'CN'.",
|
||||||
|
command.symbol, command.market
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
if let Err(e) =
|
if let Err(e) =
|
||||||
crate::worker::handle_fetch_command(state_clone, command, publisher_clone)
|
crate::worker::handle_fetch_command(state_clone, command, publisher_clone)
|
||||||
.await
|
.await
|
||||||
|
|||||||
@ -1,32 +1,73 @@
|
|||||||
use std::sync::Arc;
|
|
||||||
|
|
||||||
use dashmap::DashMap;
|
|
||||||
use uuid::Uuid;
|
|
||||||
|
|
||||||
use common_contracts::observability::TaskProgress;
|
use common_contracts::observability::TaskProgress;
|
||||||
use secrecy::ExposeSecret;
|
use secrecy::{ExposeSecret, SecretString};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::RwLock;
|
||||||
|
|
||||||
use crate::config::AppConfig;
|
use crate::config::AppConfig;
|
||||||
use crate::finnhub::FinnhubDataProvider;
|
use crate::finnhub::FinnhubDataProvider;
|
||||||
|
use dashmap::DashMap;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
|
pub enum ServiceOperationalStatus {
|
||||||
|
Active,
|
||||||
|
Degraded { reason: String },
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct AppState {
|
pub struct AppState {
|
||||||
pub tasks: Arc<DashMap<Uuid, TaskProgress>>,
|
pub tasks: Arc<DashMap<Uuid, TaskProgress>>,
|
||||||
pub config: Arc<AppConfig>,
|
pub config: Arc<AppConfig>,
|
||||||
pub finnhub_provider: Arc<FinnhubDataProvider>,
|
pub status: Arc<RwLock<ServiceOperationalStatus>>,
|
||||||
|
finnhub_provider: Arc<RwLock<Option<FinnhubDataProvider>>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppState {
|
impl AppState {
|
||||||
pub fn new(config: AppConfig) -> Self {
|
pub fn new(config: AppConfig) -> Self {
|
||||||
let provider = Arc::new(FinnhubDataProvider::new(
|
let (initial_provider, initial_status) =
|
||||||
config.finnhub_api_url.clone(),
|
if let Some(api_key) = config.finnhub_api_key.as_ref() {
|
||||||
config.finnhub_api_key.expose_secret().to_string(),
|
let provider = FinnhubDataProvider::new(
|
||||||
));
|
config.finnhub_api_url.clone(),
|
||||||
|
api_key.expose_secret().to_string(),
|
||||||
|
);
|
||||||
|
(Some(provider), ServiceOperationalStatus::Active)
|
||||||
|
} else {
|
||||||
|
(
|
||||||
|
None,
|
||||||
|
ServiceOperationalStatus::Degraded {
|
||||||
|
reason: "Finnhub API Key is not configured.".to_string(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
};
|
||||||
|
|
||||||
Self {
|
Self {
|
||||||
tasks: Arc::new(DashMap::new()),
|
tasks: Arc::new(DashMap::new()),
|
||||||
config: Arc::new(config),
|
config: Arc::new(config),
|
||||||
finnhub_provider: provider,
|
status: Arc::new(RwLock::new(initial_status)),
|
||||||
|
finnhub_provider: Arc::new(RwLock::new(initial_provider)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn get_provider(&self) -> Option<FinnhubDataProvider> {
|
||||||
|
self.finnhub_provider.read().await.clone()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn update_provider(&self, api_key: Option<SecretString>) {
|
||||||
|
let mut provider_guard = self.finnhub_provider.write().await;
|
||||||
|
let mut status_guard = self.status.write().await;
|
||||||
|
|
||||||
|
if let Some(key) = api_key {
|
||||||
|
let new_provider = FinnhubDataProvider::new(
|
||||||
|
self.config.finnhub_api_url.clone(),
|
||||||
|
key.expose_secret().to_string(),
|
||||||
|
);
|
||||||
|
*provider_guard = Some(new_provider);
|
||||||
|
*status_guard = ServiceOperationalStatus::Active;
|
||||||
|
} else {
|
||||||
|
*provider_guard = None;
|
||||||
|
*status_guard = ServiceOperationalStatus::Degraded {
|
||||||
|
reason: "Finnhub API Key is not configured.".to_string(),
|
||||||
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,11 +1,11 @@
|
|||||||
use crate::error::Result;
|
use crate::error::{AppError, Result};
|
||||||
use crate::persistence::PersistenceClient;
|
use crate::persistence::PersistenceClient;
|
||||||
use crate::state::AppState;
|
use crate::state::AppState;
|
||||||
use chrono::Datelike;
|
use chrono::Datelike;
|
||||||
use common_contracts::dtos::{CompanyProfileDto, TimeSeriesFinancialDto};
|
use common_contracts::dtos::{CompanyProfileDto, TimeSeriesFinancialDto};
|
||||||
use common_contracts::messages::{CompanyProfilePersistedEvent, FetchCompanyDataCommand, FinancialsPersistedEvent};
|
use common_contracts::messages::{CompanyProfilePersistedEvent, FetchCompanyDataCommand, FinancialsPersistedEvent};
|
||||||
use common_contracts::observability::TaskProgress;
|
use common_contracts::observability::TaskProgress;
|
||||||
use tracing::info;
|
use tracing::{error, info};
|
||||||
|
|
||||||
pub async fn handle_fetch_command(
|
pub async fn handle_fetch_command(
|
||||||
state: AppState,
|
state: AppState,
|
||||||
@ -26,11 +26,22 @@ pub async fn handle_fetch_command(
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
|
let provider = match state.get_provider().await {
|
||||||
|
Some(p) => p,
|
||||||
|
None => {
|
||||||
|
let reason = "Execution failed: Finnhub provider is not available (misconfigured).".to_string();
|
||||||
|
error!("{}", reason);
|
||||||
|
if let Some(mut task) = state.tasks.get_mut(&command.request_id) {
|
||||||
|
task.status = "Failed".to_string();
|
||||||
|
task.details = reason.clone();
|
||||||
|
}
|
||||||
|
return Err(AppError::ProviderNotAvailable(reason));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// 1. Fetch data via provider
|
// 1. Fetch data via provider
|
||||||
let (profile, financials): (CompanyProfileDto, Vec<TimeSeriesFinancialDto>) = state
|
let (profile, financials): (CompanyProfileDto, Vec<TimeSeriesFinancialDto>) =
|
||||||
.finnhub_provider
|
provider.fetch_all_data(&command.symbol).await?;
|
||||||
.fetch_all_data(&command.symbol)
|
|
||||||
.await?;
|
|
||||||
|
|
||||||
// 2. Persist
|
// 2. Persist
|
||||||
{
|
{
|
||||||
|
|||||||
11
services/tushare-provider-service/Cargo.lock
generated
11
services/tushare-provider-service/Cargo.lock
generated
@ -2296,6 +2296,16 @@ version = "4.1.0"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b"
|
checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "secrecy"
|
||||||
|
version = "0.8.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "9bd1c54ea06cfd2f6b63219704de0b9b4f72dcc2b8fdef820be6cd799780e91e"
|
||||||
|
dependencies = [
|
||||||
|
"serde",
|
||||||
|
"zeroize",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "security-framework"
|
name = "security-framework"
|
||||||
version = "2.11.1"
|
version = "2.11.1"
|
||||||
@ -3312,6 +3322,7 @@ dependencies = [
|
|||||||
"rmcp",
|
"rmcp",
|
||||||
"rust_decimal",
|
"rust_decimal",
|
||||||
"rust_decimal_macros",
|
"rust_decimal_macros",
|
||||||
|
"secrecy",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"thiserror 2.0.17",
|
"thiserror 2.0.17",
|
||||||
|
|||||||
@ -31,3 +31,4 @@ chrono = "0.4.38"
|
|||||||
rust_decimal = "1.35.0"
|
rust_decimal = "1.35.0"
|
||||||
rust_decimal_macros = "1.35.0"
|
rust_decimal_macros = "1.35.0"
|
||||||
itertools = "0.14.0"
|
itertools = "0.14.0"
|
||||||
|
secrecy = { version = "0.8", features = ["serde"] }
|
||||||
|
|||||||
@ -1,7 +1,7 @@
|
|||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use axum::{routing::get, Router, extract::State, response::Json};
|
use axum::{routing::get, Router, extract::State, response::Json};
|
||||||
|
|
||||||
use crate::state::AppState;
|
use crate::state::{AppState, ServiceOperationalStatus};
|
||||||
use common_contracts::observability::{HealthStatus, ServiceStatus};
|
use common_contracts::observability::{HealthStatus, ServiceStatus};
|
||||||
|
|
||||||
pub fn create_router(app_state: AppState) -> Router {
|
pub fn create_router(app_state: AppState) -> Router {
|
||||||
@ -11,12 +11,22 @@ pub fn create_router(app_state: AppState) -> Router {
|
|||||||
.with_state(app_state)
|
.with_state(app_state)
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn health_check(State(_state): State<AppState>) -> Json<HealthStatus> {
|
async fn health_check(State(state): State<AppState>) -> Json<HealthStatus> {
|
||||||
let mut details = HashMap::new();
|
let mut details = HashMap::new();
|
||||||
details.insert("message_bus_connection".to_string(), "ok".to_string());
|
let operational_status = state.status.read().await;
|
||||||
|
|
||||||
|
let (service_status, reason) = match &*operational_status {
|
||||||
|
ServiceOperationalStatus::Active => (ServiceStatus::Ok, "ok".to_string()),
|
||||||
|
ServiceOperationalStatus::Degraded { reason } => {
|
||||||
|
(ServiceStatus::Degraded, reason.clone())
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
details.insert("operational_status".to_string(), reason);
|
||||||
|
|
||||||
let status = HealthStatus {
|
let status = HealthStatus {
|
||||||
module_id: "tushare-provider-service".to_string(),
|
module_id: "tushare-provider-service".to_string(),
|
||||||
status: ServiceStatus::Ok,
|
status: service_status,
|
||||||
version: env!("CARGO_PKG_VERSION").to_string(),
|
version: env!("CARGO_PKG_VERSION").to_string(),
|
||||||
details,
|
details,
|
||||||
};
|
};
|
||||||
|
|||||||
@ -1,4 +1,5 @@
|
|||||||
use serde::Deserialize;
|
use serde::Deserialize;
|
||||||
|
use secrecy::SecretString;
|
||||||
|
|
||||||
#[derive(Debug, Deserialize, Clone)]
|
#[derive(Debug, Deserialize, Clone)]
|
||||||
pub struct AppConfig {
|
pub struct AppConfig {
|
||||||
@ -6,7 +7,7 @@ pub struct AppConfig {
|
|||||||
pub nats_addr: String,
|
pub nats_addr: String,
|
||||||
pub data_persistence_service_url: String,
|
pub data_persistence_service_url: String,
|
||||||
pub tushare_api_url: String,
|
pub tushare_api_url: String,
|
||||||
pub tushare_api_token: String,
|
pub tushare_api_token: Option<SecretString>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppConfig {
|
impl AppConfig {
|
||||||
@ -36,11 +37,6 @@ impl AppConfig {
|
|||||||
"TUSHARE_API_URL must not be empty".to_string(),
|
"TUSHARE_API_URL must not be empty".to_string(),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
if cfg.tushare_api_token.trim().is_empty() || cfg.tushare_api_token.trim() == "YOUR_TUSHARE_API_TOKEN" {
|
|
||||||
return Err(config::ConfigError::Message(
|
|
||||||
"TUSHARE_API_TOKEN must be provided (non-empty, non-placeholder)".to_string(),
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(cfg)
|
Ok(cfg)
|
||||||
}
|
}
|
||||||
|
|||||||
56
services/tushare-provider-service/src/config_poller.rs
Normal file
56
services/tushare-provider-service/src/config_poller.rs
Normal file
@ -0,0 +1,56 @@
|
|||||||
|
use crate::error::Result;
|
||||||
|
use crate::state::AppState;
|
||||||
|
use common_contracts::config_models::{DataSourceConfig, DataSourceProvider};
|
||||||
|
use secrecy::SecretString;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::time::Duration;
|
||||||
|
use tracing::{error, info, instrument};
|
||||||
|
|
||||||
|
const POLLING_INTERVAL_SECONDS: u64 = 60;
|
||||||
|
|
||||||
|
#[instrument(skip(state))]
|
||||||
|
pub async fn run_config_poller(state: AppState) {
|
||||||
|
info!("Starting configuration poller...");
|
||||||
|
let mut interval = tokio::time::interval(Duration::from_secs(POLLING_INTERVAL_SECONDS));
|
||||||
|
interval.tick().await; // Initial tick happens immediately
|
||||||
|
|
||||||
|
loop {
|
||||||
|
if let Err(e) = poll_and_update_config(&state).await {
|
||||||
|
error!("Failed to poll and update config: {:?}", e);
|
||||||
|
}
|
||||||
|
interval.tick().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn poll_and_update_config(state: &AppState) -> Result<()> {
|
||||||
|
info!("Polling for data source configurations...");
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
let url = format!(
|
||||||
|
"{}/configs/data_sources",
|
||||||
|
state.config.data_persistence_service_url
|
||||||
|
);
|
||||||
|
|
||||||
|
let response = client.get(&url).send().await?;
|
||||||
|
response.error_for_status_ref()?;
|
||||||
|
|
||||||
|
let configs: HashMap<String, DataSourceConfig> = response.json().await?;
|
||||||
|
|
||||||
|
let tushare_config = configs.values().find(|cfg| {
|
||||||
|
matches!(cfg.provider, DataSourceProvider::Tushare) && cfg.enabled
|
||||||
|
});
|
||||||
|
|
||||||
|
if let Some(config) = tushare_config {
|
||||||
|
if let Some(api_key) = &config.api_key {
|
||||||
|
state.update_provider(Some(SecretString::from(api_key.clone()))).await;
|
||||||
|
info!("Successfully updated Tushare provider with new configuration.");
|
||||||
|
} else {
|
||||||
|
state.update_provider(None).await;
|
||||||
|
info!("Tushare provider is enabled but API key is missing. Service is degraded.");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
state.update_provider(None).await;
|
||||||
|
info!("No enabled Tushare configuration found. Service is degraded.");
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@ -1,27 +1,47 @@
|
|||||||
|
use anyhow::anyhow;
|
||||||
|
use reqwest::Error as ReqwestError;
|
||||||
use thiserror::Error;
|
use thiserror::Error;
|
||||||
|
|
||||||
#[derive(Error, Debug)]
|
#[derive(Error, Debug)]
|
||||||
pub enum ProviderError {
|
pub enum AppError {
|
||||||
#[error("API request failed: {0}")]
|
|
||||||
ApiRequest(#[from] reqwest::Error),
|
|
||||||
|
|
||||||
#[error("Failed to parse JSON response: {0}")]
|
|
||||||
JsonParsing(#[from] serde_json::Error),
|
|
||||||
|
|
||||||
#[error("Tushare API returned an error: code={code}, message='{msg}'")]
|
|
||||||
TushareApi { code: i64, msg: String },
|
|
||||||
|
|
||||||
#[error("Configuration error: {0}")]
|
#[error("Configuration error: {0}")]
|
||||||
Configuration(String),
|
Configuration(String),
|
||||||
|
|
||||||
#[error("Data mapping error: {0}")]
|
#[error("Data parsing error: {0}")]
|
||||||
|
DataParsing(#[from] anyhow::Error),
|
||||||
|
|
||||||
|
#[error("Mapping error: {0}")]
|
||||||
Mapping(String),
|
Mapping(String),
|
||||||
|
|
||||||
#[error("Persistence client error: {0}")]
|
|
||||||
Persistence(String),
|
|
||||||
|
|
||||||
#[error("Internal error: {0}")]
|
#[error("Internal error: {0}")]
|
||||||
Internal(#[from] anyhow::Error),
|
Internal(String),
|
||||||
|
|
||||||
|
#[error("Provider not available: {0}")]
|
||||||
|
ProviderNotAvailable(String),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Reqwest(#[from] ReqwestError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Nats(#[from] async_nats::Error),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsSubscribe(#[from] async_nats::client::SubscribeError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsUnsubscribe(#[from] async_nats::UnsubscribeError),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
NatsPublish(#[from] async_nats::error::Error<async_nats::client::PublishErrorKind>),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
SerdeJson(#[from] serde_json::Error),
|
||||||
}
|
}
|
||||||
|
|
||||||
pub type Result<T> = std::result::Result<T, ProviderError>;
|
impl From<config::ConfigError> for AppError {
|
||||||
|
fn from(e: config::ConfigError) -> Self {
|
||||||
|
AppError::Configuration(e.to_string())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub type Result<T, E = AppError> = std::result::Result<T, E>;
|
||||||
|
|||||||
@ -8,9 +8,10 @@ mod state;
|
|||||||
mod ts_client;
|
mod ts_client;
|
||||||
mod tushare;
|
mod tushare;
|
||||||
mod worker;
|
mod worker;
|
||||||
|
mod config_poller;
|
||||||
|
|
||||||
use crate::config::AppConfig;
|
use crate::config::AppConfig;
|
||||||
use crate::error::{Result, ProviderError};
|
use crate::error::{Result, AppError};
|
||||||
use crate::state::AppState;
|
use crate::state::AppState;
|
||||||
use tracing::info;
|
use tracing::info;
|
||||||
|
|
||||||
@ -24,12 +25,15 @@ async fn main() -> Result<()> {
|
|||||||
info!("Starting tushare-provider-service...");
|
info!("Starting tushare-provider-service...");
|
||||||
|
|
||||||
// Load configuration
|
// Load configuration
|
||||||
let config = AppConfig::load().map_err(|e| ProviderError::Configuration(e.to_string()))?;
|
let config = AppConfig::load().map_err(|e| AppError::Configuration(e.to_string()))?;
|
||||||
let port = config.server_port;
|
let port = config.server_port;
|
||||||
|
|
||||||
// Initialize application state
|
// Initialize application state
|
||||||
let app_state = AppState::new(config);
|
let app_state = AppState::new(config);
|
||||||
|
|
||||||
|
// --- Start the config poller ---
|
||||||
|
tokio::spawn(config_poller::run_config_poller(app_state.clone()));
|
||||||
|
|
||||||
// Create the Axum router
|
// Create the Axum router
|
||||||
let app = api::create_router(app_state.clone());
|
let app = api::create_router(app_state.clone());
|
||||||
|
|
||||||
|
|||||||
@ -8,7 +8,7 @@ use rust_decimal::prelude::*;
|
|||||||
use rust_decimal_macros::dec;
|
use rust_decimal_macros::dec;
|
||||||
|
|
||||||
use crate::{
|
use crate::{
|
||||||
error::ProviderError,
|
error::AppError,
|
||||||
tushare::{
|
tushare::{
|
||||||
BalanceSheet, Cashflow, Dividend, FinaIndicator, Income, Repurchase, StkHolderNumber,
|
BalanceSheet, Cashflow, Dividend, FinaIndicator, Income, Repurchase, StkHolderNumber,
|
||||||
},
|
},
|
||||||
@ -28,7 +28,7 @@ pub struct TushareFinancials {
|
|||||||
pub fn map_financial_statements(
|
pub fn map_financial_statements(
|
||||||
symbol: &str,
|
symbol: &str,
|
||||||
raw_data: TushareFinancials,
|
raw_data: TushareFinancials,
|
||||||
) -> Result<Vec<TimeSeriesFinancialDto>, ProviderError> {
|
) -> Result<Vec<TimeSeriesFinancialDto>, AppError> {
|
||||||
// 1. Merge all financial data by end_date
|
// 1. Merge all financial data by end_date
|
||||||
let mut by_date = merge_financial_data(&raw_data);
|
let mut by_date = merge_financial_data(&raw_data);
|
||||||
|
|
||||||
@ -251,12 +251,12 @@ fn calculate_derived_metrics(series: &mut SeriesMap) {
|
|||||||
series.extend(new_series);
|
series.extend(new_series);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn flatten_series_to_dtos(symbol: &str, series: SeriesMap) -> Result<Vec<TimeSeriesFinancialDto>, ProviderError> {
|
fn flatten_series_to_dtos(symbol: &str, series: SeriesMap) -> Result<Vec<TimeSeriesFinancialDto>, AppError> {
|
||||||
let mut dtos: Vec<TimeSeriesFinancialDto> = Vec::new();
|
let mut dtos: Vec<TimeSeriesFinancialDto> = Vec::new();
|
||||||
for (metric_name, data_points) in series {
|
for (metric_name, data_points) in series {
|
||||||
for point in data_points {
|
for point in data_points {
|
||||||
let period_date = NaiveDate::parse_from_str(&point.period, "%Y%m%d")
|
let period_date = NaiveDate::parse_from_str(&point.period, "%Y%m%d")
|
||||||
.map_err(|e| ProviderError::Mapping(format!("Invalid period '{}': {}", point.period, e)))?;
|
.map_err(|e| AppError::Mapping(format!("Invalid period '{}': {}", point.period, e)))?;
|
||||||
dtos.push(TimeSeriesFinancialDto {
|
dtos.push(TimeSeriesFinancialDto {
|
||||||
symbol: symbol.to_string(),
|
symbol: symbol.to_string(),
|
||||||
metric_name: metric_name.clone(),
|
metric_name: metric_name.clone(),
|
||||||
|
|||||||
@ -1,5 +1,5 @@
|
|||||||
use crate::error::{Result, ProviderError};
|
use crate::error::Result;
|
||||||
use crate::state::AppState;
|
use crate::state::{AppState, ServiceOperationalStatus};
|
||||||
use common_contracts::messages::FetchCompanyDataCommand;
|
use common_contracts::messages::FetchCompanyDataCommand;
|
||||||
use futures_util::StreamExt;
|
use futures_util::StreamExt;
|
||||||
use tracing::{error, info, warn};
|
use tracing::{error, info, warn};
|
||||||
@ -7,34 +7,60 @@ use tracing::{error, info, warn};
|
|||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use tokio::sync::mpsc;
|
use tokio::sync::mpsc;
|
||||||
use chrono::Utc;
|
use chrono::Utc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
const SUBJECT_NAME: &str = "data_fetch_commands";
|
const SUBJECT_NAME: &str = "data_fetch_commands";
|
||||||
|
|
||||||
pub async fn run(state: AppState) -> Result<()> {
|
pub async fn run(state: AppState) -> Result<()> {
|
||||||
info!("Starting NATS message consumer...");
|
info!("Starting NATS message consumer...");
|
||||||
|
|
||||||
let client = async_nats::connect(&state.config.nats_addr)
|
loop {
|
||||||
.await
|
let status = state.status.read().await.clone();
|
||||||
.map_err(|e| ProviderError::Internal(anyhow::anyhow!("NATS connect failed: {}", e)))?;
|
if let ServiceOperationalStatus::Degraded { reason } = status {
|
||||||
info!("Connected to NATS.");
|
warn!(
|
||||||
subscribe_to_data_commands(Arc::new(state), client).await
|
"Service is in degraded state (reason: {}). Pausing message consumption for 30s.",
|
||||||
|
reason
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_secs(30)).await;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Service is Active. Connecting to NATS...");
|
||||||
|
match async_nats::connect(&state.config.nats_addr).await {
|
||||||
|
Ok(client) => {
|
||||||
|
info!("Successfully connected to NATS.");
|
||||||
|
if let Err(e) = subscribe_and_process(state.clone(), client).await {
|
||||||
|
error!("NATS subscription error: {}. Reconnecting in 10s...", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
error!("Failed to connect to NATS: {}. Retrying in 10s...", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn subscribe_to_data_commands(app_state: Arc<AppState>, nats_client: async_nats::Client) -> Result<()> {
|
async fn subscribe_and_process(
|
||||||
// This is a simple subscriber. For production, consider JetStream for durability.
|
state: AppState,
|
||||||
let mut subscriber = nats_client
|
client: async_nats::Client,
|
||||||
.subscribe(SUBJECT_NAME.to_string())
|
) -> Result<()> {
|
||||||
.await
|
let mut subscriber = client.subscribe(SUBJECT_NAME.to_string()).await?;
|
||||||
.map_err(|e| ProviderError::Internal(anyhow::anyhow!("NATS subscribe failed: {}", e)))?;
|
|
||||||
|
|
||||||
info!(
|
info!(
|
||||||
"Consumer started, waiting for messages on subject '{}'",
|
"Consumer started, waiting for messages on subject '{}'",
|
||||||
SUBJECT_NAME
|
SUBJECT_NAME
|
||||||
);
|
);
|
||||||
|
|
||||||
while let Some(message) = subscriber.next().await {
|
while let Some(message) = subscriber.next().await {
|
||||||
|
let current_status = state.status.read().await.clone();
|
||||||
|
if matches!(current_status, ServiceOperationalStatus::Degraded {..}) {
|
||||||
|
warn!("Service became degraded. Disconnecting from NATS and pausing consumption.");
|
||||||
|
subscriber.unsubscribe().await?;
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
info!("Received NATS message.");
|
info!("Received NATS message.");
|
||||||
let state_for_closure = app_state.clone();
|
let state_for_closure = Arc::new(state.clone());
|
||||||
|
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
if let Err(e) = serde_json::from_slice::<FetchCompanyDataCommand>(&message.payload) {
|
if let Err(e) = serde_json::from_slice::<FetchCompanyDataCommand>(&message.payload) {
|
||||||
|
|||||||
@ -1,31 +1,72 @@
|
|||||||
use std::sync::Arc;
|
|
||||||
|
|
||||||
use dashmap::DashMap;
|
|
||||||
use uuid::Uuid;
|
|
||||||
|
|
||||||
use common_contracts::observability::TaskProgress;
|
|
||||||
|
|
||||||
use crate::config::AppConfig;
|
use crate::config::AppConfig;
|
||||||
use crate::tushare::TushareDataProvider;
|
use crate::tushare::TushareDataProvider;
|
||||||
|
use common_contracts::observability::TaskProgress;
|
||||||
|
use dashmap::DashMap;
|
||||||
|
use secrecy::{ExposeSecret, SecretString};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::RwLock;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
|
pub enum ServiceOperationalStatus {
|
||||||
|
Active,
|
||||||
|
Degraded { reason: String },
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct AppState {
|
pub struct AppState {
|
||||||
pub tasks: Arc<DashMap<Uuid, TaskProgress>>,
|
pub tasks: Arc<DashMap<Uuid, TaskProgress>>,
|
||||||
pub config: Arc<AppConfig>,
|
pub config: Arc<AppConfig>,
|
||||||
pub tushare_provider: Arc<TushareDataProvider>,
|
pub status: Arc<RwLock<ServiceOperationalStatus>>,
|
||||||
|
tushare_provider: Arc<RwLock<Option<TushareDataProvider>>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppState {
|
impl AppState {
|
||||||
pub fn new(config: AppConfig) -> Self {
|
pub fn new(config: AppConfig) -> Self {
|
||||||
let provider = Arc::new(TushareDataProvider::new(
|
let (initial_provider, initial_status) =
|
||||||
config.tushare_api_url.clone(),
|
if let Some(api_key) = config.tushare_api_token.as_ref() {
|
||||||
config.tushare_api_token.clone(),
|
let provider = TushareDataProvider::new(
|
||||||
));
|
config.tushare_api_url.clone(),
|
||||||
|
api_key.expose_secret().clone(),
|
||||||
|
);
|
||||||
|
(Some(provider), ServiceOperationalStatus::Active)
|
||||||
|
} else {
|
||||||
|
(
|
||||||
|
None,
|
||||||
|
ServiceOperationalStatus::Degraded {
|
||||||
|
reason: "Tushare API Key is not configured.".to_string(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
};
|
||||||
|
|
||||||
Self {
|
Self {
|
||||||
tasks: Arc::new(DashMap::new()),
|
tasks: Arc::new(DashMap::new()),
|
||||||
config: Arc::new(config),
|
config: Arc::new(config),
|
||||||
tushare_provider: provider,
|
status: Arc::new(RwLock::new(initial_status)),
|
||||||
|
tushare_provider: Arc::new(RwLock::new(initial_provider)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn get_provider(&self) -> Option<TushareDataProvider> {
|
||||||
|
self.tushare_provider.read().await.clone()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn update_provider(&self, api_key: Option<SecretString>) {
|
||||||
|
let mut provider_guard = self.tushare_provider.write().await;
|
||||||
|
let mut status_guard = self.status.write().await;
|
||||||
|
|
||||||
|
if let Some(key) = api_key {
|
||||||
|
let new_provider = TushareDataProvider::new(
|
||||||
|
self.config.tushare_api_url.clone(),
|
||||||
|
key.expose_secret().clone(),
|
||||||
|
);
|
||||||
|
*provider_guard = Some(new_provider);
|
||||||
|
*status_guard = ServiceOperationalStatus::Active;
|
||||||
|
} else {
|
||||||
|
*provider_guard = None;
|
||||||
|
*status_guard = ServiceOperationalStatus::Degraded {
|
||||||
|
reason: "Tushare API Key is not configured.".to_string(),
|
||||||
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,4 +1,4 @@
|
|||||||
use crate::error::ProviderError;
|
use crate::error::AppError;
|
||||||
use serde::{de::DeserializeOwned, Deserialize, Serialize};
|
use serde::{de::DeserializeOwned, Deserialize, Serialize};
|
||||||
use tracing::info;
|
use tracing::info;
|
||||||
|
|
||||||
@ -46,7 +46,7 @@ impl TushareClient {
|
|||||||
api_name: &str,
|
api_name: &str,
|
||||||
params: serde_json::Value,
|
params: serde_json::Value,
|
||||||
fields: &str,
|
fields: &str,
|
||||||
) -> Result<Vec<T>, ProviderError> {
|
) -> Result<Vec<T>, AppError> {
|
||||||
let request_payload = TushareRequest {
|
let request_payload = TushareRequest {
|
||||||
api_name,
|
api_name,
|
||||||
token: &self.api_token,
|
token: &self.api_token,
|
||||||
@ -67,15 +67,16 @@ impl TushareClient {
|
|||||||
let response: TushareResponse<T> = serde_json::from_str(&text)?;
|
let response: TushareResponse<T> = serde_json::from_str(&text)?;
|
||||||
|
|
||||||
if response.code != 0 {
|
if response.code != 0 {
|
||||||
return Err(ProviderError::TushareApi {
|
return Err(AppError::DataParsing(anyhow::anyhow!(format!(
|
||||||
code: response.code,
|
"Tushare API error code {}: {}",
|
||||||
msg: response.msg,
|
response.code, response.msg
|
||||||
});
|
))));
|
||||||
}
|
}
|
||||||
|
|
||||||
let data = response.data.ok_or_else(|| ProviderError::TushareApi {
|
let data = response.data.ok_or_else(|| {
|
||||||
code: -1,
|
AppError::DataParsing(anyhow::anyhow!(
|
||||||
msg: "No data field in response".to_string(),
|
"Tushare response missing data field"
|
||||||
|
))
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
let items = data
|
let items = data
|
||||||
|
|||||||
@ -4,7 +4,7 @@ use serde_json::json;
|
|||||||
use tokio;
|
use tokio;
|
||||||
|
|
||||||
use crate::{
|
use crate::{
|
||||||
error::ProviderError,
|
error::AppError,
|
||||||
mapping::{map_financial_statements, TushareFinancials},
|
mapping::{map_financial_statements, TushareFinancials},
|
||||||
ts_client::TushareClient,
|
ts_client::TushareClient,
|
||||||
};
|
};
|
||||||
@ -25,7 +25,7 @@ impl TushareDataProvider {
|
|||||||
pub async fn fetch_all_data(
|
pub async fn fetch_all_data(
|
||||||
&self,
|
&self,
|
||||||
symbol: &str,
|
symbol: &str,
|
||||||
) -> Result<(CompanyProfileDto, Vec<TimeSeriesFinancialDto>), ProviderError> {
|
) -> Result<(CompanyProfileDto, Vec<TimeSeriesFinancialDto>), AppError> {
|
||||||
let (
|
let (
|
||||||
stock_basic,
|
stock_basic,
|
||||||
stock_company,
|
stock_company,
|
||||||
@ -42,18 +42,18 @@ impl TushareDataProvider {
|
|||||||
let ts_code = stock_basic
|
let ts_code = stock_basic
|
||||||
.get(0)
|
.get(0)
|
||||||
.map(|r| r.ts_code.clone())
|
.map(|r| r.ts_code.clone())
|
||||||
.ok_or_else(|| ProviderError::Mapping("stock_basic missing first row".to_string()))?;
|
.ok_or_else(|| AppError::Mapping("stock_basic missing first row".to_string()))?;
|
||||||
let name = stock_basic
|
let name = stock_basic
|
||||||
.get(0)
|
.get(0)
|
||||||
.and_then(|r| r.name.clone())
|
.and_then(|r| r.name.clone())
|
||||||
.ok_or_else(|| ProviderError::Mapping("stock_basic.name missing".to_string()))?;
|
.ok_or_else(|| AppError::Mapping("stock_basic.name missing".to_string()))?;
|
||||||
let industry = stock_basic.get(0).and_then(|r| r.industry.clone());
|
let industry = stock_basic.get(0).and_then(|r| r.industry.clone());
|
||||||
let list_date = stock_basic
|
let list_date = stock_basic
|
||||||
.get(0)
|
.get(0)
|
||||||
.and_then(|r| r.list_date.clone())
|
.and_then(|r| r.list_date.clone())
|
||||||
.map(|d| NaiveDate::parse_from_str(&d, "%Y%m%d"))
|
.map(|d| NaiveDate::parse_from_str(&d, "%Y%m%d"))
|
||||||
.transpose()
|
.transpose()
|
||||||
.map_err(|e| ProviderError::Mapping(format!("Invalid list_date: {}", e)))?;
|
.map_err(|e| AppError::Mapping(format!("Invalid list_date: {}", e)))?;
|
||||||
|
|
||||||
let profile = CompanyProfileDto {
|
let profile = CompanyProfileDto {
|
||||||
symbol: ts_code,
|
symbol: ts_code,
|
||||||
@ -94,7 +94,7 @@ impl TushareDataProvider {
|
|||||||
Vec<Dividend>,
|
Vec<Dividend>,
|
||||||
Vec<StkHolderNumber>,
|
Vec<StkHolderNumber>,
|
||||||
),
|
),
|
||||||
ProviderError,
|
AppError,
|
||||||
> {
|
> {
|
||||||
let params = json!({ "ts_code": symbol });
|
let params = json!({ "ts_code": symbol });
|
||||||
|
|
||||||
|
|||||||
@ -5,42 +5,52 @@ use common_contracts::{
|
|||||||
messages::{CompanyProfilePersistedEvent, FetchCompanyDataCommand, FinancialsPersistedEvent},
|
messages::{CompanyProfilePersistedEvent, FetchCompanyDataCommand, FinancialsPersistedEvent},
|
||||||
};
|
};
|
||||||
use tokio::sync::mpsc;
|
use tokio::sync::mpsc;
|
||||||
use tracing::info;
|
use tracing::{info, error};
|
||||||
use chrono::Datelike;
|
use chrono::Datelike;
|
||||||
|
|
||||||
use crate::{error::ProviderError, persistence::PersistenceClient, state::AppState};
|
use crate::{error::AppError, persistence::PersistenceClient, state::AppState};
|
||||||
|
|
||||||
pub async fn run_tushare_workflow(
|
pub async fn run_tushare_workflow(
|
||||||
state: Arc<AppState>,
|
state: Arc<AppState>,
|
||||||
command: FetchCompanyDataCommand,
|
command: FetchCompanyDataCommand,
|
||||||
completion_tx: mpsc::Sender<()>,
|
completion_tx: mpsc::Sender<()>,
|
||||||
) -> Result<(), ProviderError> {
|
) -> Result<(), AppError> {
|
||||||
let task_id = command.request_id;
|
let task_id = command.request_id;
|
||||||
let symbol = command.symbol.clone();
|
let symbol = command.symbol.clone();
|
||||||
|
|
||||||
|
let provider = match state.get_provider().await {
|
||||||
|
Some(p) => p,
|
||||||
|
None => {
|
||||||
|
let reason = "Execution failed: Tushare provider is not available (misconfigured).".to_string();
|
||||||
|
error!("{}", reason);
|
||||||
|
if let Some(mut task) = state.tasks.get_mut(&task_id) {
|
||||||
|
task.status = "Failed".to_string();
|
||||||
|
task.details = reason.clone();
|
||||||
|
}
|
||||||
|
return Err(AppError::ProviderNotAvailable(reason));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// 1. Update task progress: Fetching data
|
// 1. Update task progress: Fetching data
|
||||||
{
|
{
|
||||||
let mut entry = state
|
let mut entry = state
|
||||||
.tasks
|
.tasks
|
||||||
.get_mut(&task_id)
|
.get_mut(&task_id)
|
||||||
.ok_or_else(|| ProviderError::Internal(anyhow::anyhow!("Task not found")))?;
|
.ok_or_else(|| AppError::Internal("Task not found".to_string()))?;
|
||||||
entry.status = "FetchingData".to_string();
|
entry.status = "FetchingData".to_string();
|
||||||
entry.progress_percent = 10;
|
entry.progress_percent = 10;
|
||||||
entry.details = "Starting data fetch from Tushare".to_string();
|
entry.details = "Starting data fetch from Tushare".to_string();
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Fetch data using the provider
|
// 2. Fetch data using the provider
|
||||||
let (profile, financials) = state
|
let (profile, financials) = provider.fetch_all_data(&symbol).await?;
|
||||||
.tushare_provider
|
|
||||||
.fetch_all_data(&symbol)
|
|
||||||
.await?;
|
|
||||||
|
|
||||||
// 3. Update task progress: Persisting data
|
// 3. Update task progress: Persisting data
|
||||||
{
|
{
|
||||||
let mut entry = state
|
let mut entry = state
|
||||||
.tasks
|
.tasks
|
||||||
.get_mut(&task_id)
|
.get_mut(&task_id)
|
||||||
.ok_or_else(|| ProviderError::Internal(anyhow::anyhow!("Task not found")))?;
|
.ok_or_else(|| AppError::Internal("Task not found".to_string()))?;
|
||||||
entry.status = "PersistingData".to_string();
|
entry.status = "PersistingData".to_string();
|
||||||
entry.progress_percent = 60;
|
entry.progress_percent = 60;
|
||||||
entry.details = "Data fetched, persisting to database".to_string();
|
entry.details = "Data fetched, persisting to database".to_string();
|
||||||
@ -60,7 +70,7 @@ pub async fn run_tushare_workflow(
|
|||||||
// 5. Publish events
|
// 5. Publish events
|
||||||
let nats_client = async_nats::connect(&state.config.nats_addr)
|
let nats_client = async_nats::connect(&state.config.nats_addr)
|
||||||
.await
|
.await
|
||||||
.map_err(|e| ProviderError::Internal(anyhow::anyhow!("NATS connection failed: {}", e)))?;
|
.map_err(|e| AppError::Internal(format!("NATS connection failed: {}", e)))?;
|
||||||
|
|
||||||
publish_events(&nats_client, &command, &financials).await?;
|
publish_events(&nats_client, &command, &financials).await?;
|
||||||
|
|
||||||
@ -69,7 +79,7 @@ pub async fn run_tushare_workflow(
|
|||||||
let mut entry = state
|
let mut entry = state
|
||||||
.tasks
|
.tasks
|
||||||
.get_mut(&task_id)
|
.get_mut(&task_id)
|
||||||
.ok_or_else(|| ProviderError::Internal(anyhow::anyhow!("Task not found")))?;
|
.ok_or_else(|| AppError::Internal("Task not found".to_string()))?;
|
||||||
entry.status = "Completed".to_string();
|
entry.status = "Completed".to_string();
|
||||||
entry.progress_percent = 100;
|
entry.progress_percent = 100;
|
||||||
entry.details = "Workflow finished successfully".to_string();
|
entry.details = "Workflow finished successfully".to_string();
|
||||||
@ -91,7 +101,7 @@ async fn persist_data(
|
|||||||
financials: &[TimeSeriesFinancialDto],
|
financials: &[TimeSeriesFinancialDto],
|
||||||
state: &Arc<AppState>,
|
state: &Arc<AppState>,
|
||||||
task_id: uuid::Uuid,
|
task_id: uuid::Uuid,
|
||||||
) -> Result<(), ProviderError> {
|
) -> Result<(), AppError> {
|
||||||
// In a real implementation, we'd use tokio::try_join! to run these in parallel.
|
// In a real implementation, we'd use tokio::try_join! to run these in parallel.
|
||||||
if let Err(e) = client.upsert_company_profile(profile.clone()).await {
|
if let Err(e) = client.upsert_company_profile(profile.clone()).await {
|
||||||
state
|
state
|
||||||
@ -128,7 +138,7 @@ async fn publish_events(
|
|||||||
nats_client: &async_nats::Client,
|
nats_client: &async_nats::Client,
|
||||||
command: &FetchCompanyDataCommand,
|
command: &FetchCompanyDataCommand,
|
||||||
financials: &[TimeSeriesFinancialDto],
|
financials: &[TimeSeriesFinancialDto],
|
||||||
) -> Result<(), ProviderError> {
|
) -> Result<(), AppError> {
|
||||||
let profile_event = CompanyProfilePersistedEvent {
|
let profile_event = CompanyProfilePersistedEvent {
|
||||||
request_id: command.request_id,
|
request_id: command.request_id,
|
||||||
symbol: command.symbol.clone(),
|
symbol: command.symbol.clone(),
|
||||||
@ -138,8 +148,7 @@ async fn publish_events(
|
|||||||
"events.data.company_profile_persisted",
|
"events.data.company_profile_persisted",
|
||||||
serde_json::to_vec(&profile_event).unwrap().into(),
|
serde_json::to_vec(&profile_event).unwrap().into(),
|
||||||
)
|
)
|
||||||
.await
|
.await?;
|
||||||
.map_err(|e| ProviderError::Internal(anyhow::anyhow!("Event publishing failed: {}", e)))?;
|
|
||||||
|
|
||||||
let years: std::collections::BTreeSet<u16> = financials
|
let years: std::collections::BTreeSet<u16> = financials
|
||||||
.iter()
|
.iter()
|
||||||
@ -155,8 +164,7 @@ async fn publish_events(
|
|||||||
"events.data.financials_persisted",
|
"events.data.financials_persisted",
|
||||||
serde_json::to_vec(&financials_event).unwrap().into(),
|
serde_json::to_vec(&financials_event).unwrap().into(),
|
||||||
)
|
)
|
||||||
.await
|
.await?;
|
||||||
.map_err(|e| ProviderError::Internal(anyhow::anyhow!("Event publishing failed: {}", e)))?;
|
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user