Commit Graph

21 Commits

Author SHA1 Message Date
Lv, Qi
fbb1703b00 feat: fix provider test endpoints and update deployment scripts
- Finnhub: Add missing /test endpoint
- AlphaVantage: Fix test endpoint deserialization (handle null api_url)
- Mock Provider: Add /test endpoint and fix Zodios validation error by adding Mock enum
- Deployment: Remove Mock Provider from production deployment script
- Infrastructure: Add production Dockerfiles and compose configs
2025-12-01 02:00:21 +08:00
Lv, Qi
7933c706d1 refactor: strict typing for workflow events using WorkflowEventType enum
- refactor(frontend): replace string literals with WorkflowEventType enum for event handling
- feat(backend): export WorkflowEventType in common-contracts and openapi
- fix(tests): update end-to-end tests to match new ContextSelectorConfig and LlmConfig types
- chore: regenerate openapi.json and frontend client schemas
2025-11-30 19:28:57 +08:00
Lv, Qi
e9e4d0c1b3 chore: massive update covering recent refactoring and bug fixes
- fix: infinite message loop in workflow orchestrator
- feat: restore realtime LLM streaming from report generator to frontend
- refactor: major update to provider services (generic workers, workflow adapters)
- refactor: common contracts and message definitions updated
- feat: enhanced logging and observability in orchestrator
- docs: update project management tasks and status
- chore: dependency updates and config adjustments
2025-11-30 19:17:02 +08:00
Lv, Qi
70b30b39d8 Refactor: Complete transition from analysis_results to workflow_history
- Removed dependencies on analysis_results table
- Implemented workflow_history storage in Data Persistence Service
- Updated Workflow Orchestrator to save workflow snapshots to history
- Refactored Frontend to consume workflow_history and fetch reports via VGCS
- Fixed Data Providers (Tushare, YFinance) to report output paths in metadata
- Updated documentation and task status
2025-11-29 17:55:54 +08:00
Lv, Qi
eee1eb8b3f fix: resolve compilation errors and dependency conflicts
- Fix 'Path' macro parsing issue in service-kit-macros
- Resolve 'reedline'/'sqlite' dependency conflict in service-kit
- Consolidate workspace configuration and lockfile
- Fix 'data-persistence-service' compilation errors
- Update docker-compose and dev configurations
2025-11-29 16:29:56 +08:00
Lv, Qi
0c975bb8f1 Refactor: Remove legacy analysis results and implement workflow history
- **Common Contracts**: Updated DTOs and models to support workflow history; removed legacy analysis result DTOs.
- **Data Persistence Service**:
    - Removed `analysis_results` table logic and API endpoints.
    - Implemented `workflow_history` API and DB access (`history.rs`).
    - Fixed compilation errors and updated tests.
    - Exposed Postgres port in `docker-compose.yml` for easier debugging/offline checks.
- **API Gateway**:
    - Implemented `history` endpoints (get history list, get by ID).
    - Removed legacy `analysis-results` endpoints.
    - Fixed routing and handler logic in `api.rs`.
- **Report Generator Service**:
    - Removed dependency on legacy `analysis-results` persistence calls.
    - Fixed compilation errors.
- **Workflow Orchestrator**: Fixed warnings and minor logic issues.
- **Providers**: Updated provider services (alphavantage, tushare, finnhub, yfinance, mock) to align with contract changes.
- **Frontend**:
    - Updated `ReportPage` and stores to use new workflow history.
    - Added `RecentReportsDropdown` component.
    - Cleaned up `RealtimeLogs` component.
- **Documentation**: Moved completed design tasks to `completed/` and added refactoring context docs.

Confirmed all services pass `cargo check`.
2025-11-29 14:46:44 +08:00
Lv, Qi
03b53aed71 feat: Refactor Analysis Context Mechanism and Generic Worker
- Implemented Unified Context Mechanism (Task 20251127):
  - Decoupled intent (Module) from resolution (Orchestrator).
  - Added ContextResolver for resolving input bindings (Manual Glob/Auto LLM).
  - Added IOBinder for managing physical paths.
  - Updated GenerateReportCommand to support explicit input bindings and output paths.

- Refactored Report Worker to Generic Execution (Task 20251128):
  - Removed hardcoded financial DTOs and specific formatting logic.
  - Implemented Generic YAML-based context assembly for better LLM readability.
  - Added detailed execution tracing (Sidecar logs).
  - Fixed input data collision bug by using full paths as context keys.

- Updated Tushare Provider to support dynamic output paths.
- Updated Common Contracts with new configuration models.
2025-11-28 20:11:17 +08:00
Lv, Qi
91a6dfc4c1 feat: Implement Context Inspector and fix workflow merge base
- **Observability**: Added `ContextExplorer` component for viewing task input/output file trees and diffs.
- **Workflow Engine**:
  - Implemented Initial Commit logic in `StartWorkflow` to ensure all tasks share a common merge base.
  - Added `input_commit` tracking to `DagScheduler` and events.
  - Exposed `vgcs` (Git) operations via Orchestrator API (tree, blob, diff).
- **API Gateway**: Added proxy routes for Context Inspector (`/api/context/...`).
- **UI**:
  - Refactored `ReportPage` to remove legacy "Fundamental Data" tab.
  - Added "Context Inspector" button in Task Detail view with Dialog support.
  - Visualized Added/Modified/Deleted files with color coding in Context Explorer.
- **Cleanup**: Removed unused `FinancialTable` component.
2025-11-28 02:53:25 +08:00
Lv, Qi
b90388b76e feat: add task display names and improve report page UX
- Backend: Add display_name to TaskNode and DagNode for human-readable names
- Frontend: Update visualizer and tabs to use display_name
- UX: Implement global scrolling layout for ReportPage
- UX: Add sticky behavior with visual feedback for Workflow Sidebar
- UX: Fix RealtimeLogs scrolling and layout issues
2025-11-28 01:01:17 +08:00
Lv, Qi
b8eab4dfd5 feat: fix error propagation in report-generator and workflow-orchestrator 2025-11-27 23:07:42 +08:00
Lv, Qi
a59b994a92 WIP: Commit all pending changes across services, frontend, and docs
- Sync updates for provider services (AlphaVantage, Finnhub, YFinance, Tushare)
- Update Frontend components and pages for recent config changes
- Update API Gateway and Registry
- Include design docs and tasks status
2025-11-27 02:45:56 +08:00
Lv, Qi
ca1eddd244 Refactor Orchestrator to DAG-based scheduler with Git Context merging
- Implemented `DagScheduler` in `workflow-orchestrator` to manage task dependencies and commit history.
- Added `vgcs.merge_commits` in `workflow-context` for smart 3-way merge of parallel task branches.
- Introduced generic `WorkflowTaskCommand` and `WorkflowTaskEvent` in `common-contracts`.
- Adapted `tushare-provider-service` with `generic_worker` to support Git-based context read/write.
- Updated NATS subjects to support wildcard routing for generic workflow commands.
2025-11-27 02:42:25 +08:00
Lv, Qi
70b1a27978 feat: implement API DTO export and verify E2E tests
- contracts: Add #[api_dto] macros to core structs for OpenAPI support
- api-gateway: Integrate utoipa for Swagger UI and OpenAPI spec generation
- config: Implement dynamic configuration APIs for DataSources and LLM Providers
- tests: Refactor E2E tests to support dynamic provider configuration and fix timeouts
- docs: Update backend status in backend_todos.md
2025-11-22 22:14:01 +08:00
Lv, Qi
0cb31e363e Refactor E2E tests and improve error handling in Orchestrator
- Fix `simple_test_analysis` template in E2E test setup to align with Orchestrator's data fetch logic.
- Implement and verify additional E2E scenarios:
    - Scenario C: Partial Provider Failure (verified error propagation fix in Orchestrator).
    - Scenario D: Invalid Symbol input.
    - Scenario E: Analysis Module failure.
- Update `WorkflowStateMachine::handle_report_failed` to correctly scope error broadcasting to the specific task instead of failing effectively silently or broadly.
- Update testing strategy documentation to reflect completed Phase 4 testing.
- Skip Scenario B (Orchestrator Restart) as persistence is not yet implemented (decision made to defer persistence).
2025-11-21 20:44:32 +08:00
Lv, Qi
d28f3c5266 feat: update analysis workflow and fix LLM client connection issues
- Enhance LlmClient to handle malformed URLs and HTML error responses
- Improve logging in report-generator-service worker
- Update frontend API routes and hooks for analysis
- Update various service configurations and persistence logic
2025-11-19 17:30:52 +08:00
Lv, Qi
e855a16c69 feat(analysis): integrate analysis templates into report workflow
- Backend: Add template_id to DataRequest and AnalysisResult metadata
- Frontend: Add TemplateSelector to Report page
- Frontend: Refactor ReportPage to use AnalysisTemplateSets for dynamic tabs
- Frontend: Strict mode for report rendering based on template_id
- Cleanup: Remove legacy analysis config hooks
2025-11-19 06:59:36 +08:00
Lv, Qi
427776b863 feat(analysis): Implement Configurable Analysis Template Engine
This commit introduces a comprehensive, template-based analysis orchestration system, refactoring the entire analysis generation workflow from the ground up.

Key Changes:

1.  **Backend Architecture (`report-generator-service`):**
    *   Replaced the naive analysis workflow with a robust orchestrator based on a Directed Acyclic Graph (DAG) of module dependencies.
    *   Implemented a full topological sort (`petgraph`) to determine the correct execution order and detect circular dependencies.

2.  **Data Models (`common-contracts`, `data-persistence-service`):**
    *   Introduced the concept of `AnalysisTemplateSets` to allow for multiple, independent, and configurable analysis workflows.
    *   Created a new `analysis_results` table to persist the output of each module for every analysis run, ensuring traceability.
    *   Implemented a file-free data seeding mechanism to populate default analysis templates on service startup.

3.  **API Layer (`api-gateway`):**
    *   Added a new asynchronous endpoint (`POST /analysis-requests/{symbol}`) to trigger analysis workflows via NATS messages.
    *   Updated all configuration endpoints to support the new `AnalysisTemplateSets` model.

4.  **Frontend UI (`/config`, `/query`):**
    *   Completely refactored the "Analysis Config" page into a two-level management UI for "Template Sets" and the "Modules" within them, supporting full CRUD operations.
    *   Updated the "Query" page to allow users to select which analysis template to use when generating a report.

This new architecture provides a powerful, flexible, and robust foundation for all future development of our intelligent analysis capabilities.
2025-11-18 07:47:08 +08:00
Lv, Qi
60e6c8f61b feat(config): 将服务配置全面迁移至数据库
本次提交完成了一项重要的架构重构,将所有外部服务的API凭证管理从环境变量迁移到了中心化的数据库配置中。

主要变更:

1.  **统一配置源**:
    -   `data-persistence-service` 现已提供 `/api/v1/configs/data_sources` 端点,用于统一管理数据源配置。
    -   所有配置(LLM 和数据源)现在都通过数据库的 `system_config` 表进行管理,实现了“单一事实源”。

2.  **增强服务韧性**:
    -   重构了 `finnhub-`, `tushare-`, `alphavantage-provider-service`。
    -   这些服务在启动时不再强制要求 API Key。
    -   引入了动态配置轮询器 (`config_poller`),服务现在可以定期从数据库获取最新配置。
    -   实现了“降级模式”:当配置缺失时,服务会进入 `Degraded` 状态并暂停处理消息,而不是直接崩溃。配置恢复后,服务会自动回到 `Active` 状态。
    -   `/health` 端点现在能准确反映服务的真实运行状态。

3.  **前端易用性提升**:
    -   您在 `/config` 页面上增加了“数据源配置”面板,允许用户通过 UI 动态更新所有 API Token。

4.  **部署简化**:
    -   从 `docker-compose.yml` 中移除了所有已废弃的 `_API_KEY` 环境变量,消除了启动时的警告。

这项重构显著提升了系统的可维护性、健壮性和用户体验,为未来的功能扩展奠定了坚实的基础。
2025-11-18 05:58:18 +08:00
Lv, Qi
a1e4b265ba feat(config): Implement database-centric LLM provider architecture
本次提交完成了一次全面的架构重构,实现了以数据库为中心的、支持多供应商的LLM配置体系。

**核心变更**:
1.  **数据库驱动配置**: 废弃了所有基于本地文件的配置方案 (`analysis-config.json`),将LLM Provider和分析模块的配置作为结构化数据存入数据库的`system_config`表中,由`data-persistence-service`统一管理。
2.  **Schema-in-Code**: 在`common-contracts`中定义了所有配置的Rust Structs,作为整个系统的“单一事实源”,确保了端到端的类型安全。
3.  **服务职责重构**:
    *   `data-persistence-service`吸收了配置管理功能,成为配置的“守门人”。
    *   `config-service-rs`服务已被彻底移除。
    *   `report-generator-service`重构为可以为每个任务动态创建使用不同Provider配置的LLM客户端。
4.  **前端功能增强**:
    *   新增了独立的`/llm-config`页面,用于对LLM Providers及其可用模型进行完整的CRUD管理,并支持模型自动发现。
    *   重构了旧的`/config`页面,为分析模块提供了级联选择器来精确指定所需的Provider和Model。

此次重构极大地提升了系统的灵活性和可扩展性,完全对齐了“配置即数据”的现代化设计原则。
2025-11-17 04:41:36 +08:00
Lv, Qi
ceffa0d95c chore(build): unify Dockerfiles and root .dockerignore; fix services after deps upgrade
- secrecy: migrate Secret<String> -> SecretString across services; adjust usages
- warnings: clean unused imports/structs (no behavior change)
- docker-compose: remove unnecessary host port mappings; rely on app-network; keep frontend only
- build: set build.context to repo root; reference per-service Dockerfiles
- add: root .dockerignore to exclude target/node_modules/ref/archive/docs and logs
- data-persistence-service: refine cargo-chef with path deps; slim context; copy correct targets
- Dockerfiles: normalize multi-stage builds; fix WORKDIR and release binary paths
- nats: resolve 4222 conflict by not binding to host
- verified: all rust services build successfully with new flow
2025-11-16 23:34:28 +08:00
Lv, Qi
5327e76aaa chore: 提交本轮 Rust 架构迁移相关改动
- docker-compose: 下线 Python backend/config-service,切换至 config-service-rs
- archive: 归档 legacy Python 目录至 archive/python/*
- services: 新增/更新 common-contracts、api-gateway、各 provider、report-generator-service、config-service-rs
- data-persistence-service: API/system 模块与模型/DTO 调整
- frontend: 更新 useApi 与 API 路由
- docs: 更新路线图并勾选光荣退役
- cleanup: 移除 data-distance-service 占位测试
2025-11-16 20:55:46 +08:00