targetONCO

Turning weeks of cancer analysis into minutes

We built TargetONCO, the world's FIRST and ONLY fully automated agentic AI system for end-to-end precision oncology radiation & pathological analysis: from X-ray radiology to complex spatial proteomics tissue analysis and treatment insights — all orchestrated autonomously by a single AI agent system, powered by VLMs, Machine Learning & Deep Learning, pathological & proteomics tools, clinical big-data, and more.

Created by Mohammad Zoraiz (lead), YuCheng (Tom) Yuan, Sheldon Lewis, and Suhas Kurapati.

---

Table of Contents

1. Overview
2. System Architecture
3. Backend Components
4. Frontend Architecture
5. Installation & Setup
6. Environment Variables
7. Running the System
8. Testing
9. Technical Implementation Details
10. Credits & Acknowledgments
11. License
12. Citation
13. Additional Resources

---

Overview

targetONCO is a comprehensive biomedical AI platform that combines three specialized systems:

1. OncoPathology: Framework for spatial proteomics analysis (segmentation → quantification → analysis) extended from base biomni
2. Vector Search: Elasticsearch-based similarity search for finding similar tissues and cells across patient cohorts
3. OncoRAX: Multi-turn agentic pipeline for radiology report analysis with cancer detection based off of MedRAX

The system provides both a web-based frontend (React + FastAPI) and command-line interface for interacting with these capabilities.

---

System Architecture

Simplified Architecture Overview

The following diagram provides a high-level overview of the targetONCO system architecture, showing the main data flow from user input through the frontend to specialized processing pipelines:

This diagram illustrates:

• Frontend Interface: User interaction point for uploading X-Ray and OME-Tiff images
• X-Ray Processing: OncoRAX pipeline for radiology analysis and report generation
• OME-Tiff Processing: Two pathways - OncoPathology pipeline and direct segmentation/quantification
• Output Generation: Reports for doctors and patients, embeddings for vector search

Detailed System Architecture

For a comprehensive view of all components, data flows, integrations, and technical details, see the detailed architecture diagram:

This detailed diagram illustrates:

• Frontend Components: Chat Panel, Terminal, Dashboard/Report Viewer with WebSocket/FastAPI communication
• Agent Core Engine: Claude Agent SDK with session management, permission levels, memory states, and dynamic skills library
• Execution Backends: Modal Cloud (isolated REPL, batch processing), Local Docker/Mac, and HPC (Apptainer/Singularity)
• Multi-Turn Agent Framework: Hierarchical orchestration with process nodes (tool calling) and execute nodes (result processing)
• OncoRAX Pipeline: X-ray analysis with multi-turn information pipelining (Inference → Critique → Retrieval → Differential → Report Generation)
• OncoPathology Pipeline: Complete tissue analysis workflow (Stitching → TMA Dearray → Probability Mapping → Segmentation → Quantification → Clustering/Annotation → 3D Visualization)
• Vector Search System: Elasticsearch integration with dimension reduction, tissue-level and cell-level vector similarity, and clinical data retrieval
• Data Sources: Orion-CRC embeddings, pipeline outputs, and 25+ biomedical API schemas

High-Level Architecture (Text)

┌─────────────────────────────────────────────────────────────────┐
│                         targetONCO System                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────────┐         ┌──────────────────┐              │
│  │   Frontend       │         │   Backend API    │              │
│  │   (React/Vite)   │◄───────►│   (FastAPI)      │              │
│  │   Port: 5173     │  HTTP   │   Port: 8000     │              │
│  └──────────────────┘         └──────────────────┘              │
│         │                              │                        │
│         │ WebSocket/SSE                 │                       │
│         │                              │                        │
│         └──────────────┬───────────────┘                        │
│                        │                                        │
│         ┌─────────────▼─────────────┐                           │
│         │   Backend Components      │                           │
│         ├───────────────────────────┤                           │
│         │ 1. OncoPathology          │                           │
│         │    (biomni/)               │                          │
│         │ 2. Vector Search           │                          │
│         │    (vector_search/)        │                          │
│         │ 3. OncoRAX                 │                          │
│         │    (xray/OncoRAX/)         │                          │
│         └───────────────────────────┘                           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Communication Flow

User (Browser/CLI)
    ↓
Frontend (React) ←→ FastAPI Server (WebSocket/SSE)
    ↓
ClaudeSDKAgent (biomni/agent/claude_sdk_agent.py)
    ↓
Tool Execution
    ├─→ OncoPathology Pipeline (Docker containers)
    ├─→ Vector Search (Elasticsearch)
    ├─→ Database Queries (DepMap, GTEx, etc.)
    └─→ Analysis Tools (100+ tools)
    ↓
Results → Frontend/CLI

---

Backend Components

1. OncoPathology (biomni/)

Location: backend/biomni/

Extended Biomni implementation for spatial proteomics and oncology research.

Core Architecture

• Agent Framework: Two agent implementations

  • ClaudeSDKAgent (agent/claude_sdk_agent.py): Uses official Claude Agent SDK with Claude Code CLI
  • A1 (agent/a1.py): LangGraph-based agent with Anthropic API

• Tool Registry (tool/tool_registry.py): Automatic tool discovery and description generation

• Skills Library (skills/): Domain-specific workflows and best practices

• Configuration (config.py): Centralized settings via default_config

OncoPathology Pipeline

Complete spatial proteomics workflow:

Raw OME-TIFF Image
    ↓
[1] Probability Map Generation (UnMicst)
    → segmentation_unmicst.py
    → Docker: labsyspharm/unmicst:arm64-local
    → Output: NucleiPM_*.tif, ContoursPM_*.tif
    ↓
[2] Cell Segmentation (S3segmenter)
    → segmentation_s3segmenter.py
    → Docker: labsyspharm/s3segmenter:arm64-local
    → Output: cell.ome.tif, nuclei.ome.tif
    ↓
[3] Single-Cell Quantification (mcquant)
    → quantification.py
    → Docker: labsyspharm/mcquant:arm64-local
    → Output: {image_name}_cell.csv (histoCAT-compatible)

Key Implementation Details:

• Container Runtime Detection: Auto-detects Docker/Singularity/Apptainer
• Long-Running Task Management: Proper timeout handling (2-15+ minutes for multi-GB images)
• Error Recovery: Comprehensive error messages and retry logic
• Format Support: OME-TIFF, TIF, TIFF, H5, HDF5

Extended Tool Ecosystem (100+ Tools)

Segmentation Tools:

• segmentation_unmicst.py: UnMicst probability map generation
• segmentation_s3segmenter.py: S3segmenter watershed segmentation
• deep_imcyto.py: Nextflow-based segmentation pipeline
• typex.py: Cell type classification
• unetcoreograph.py: TMA dearraying

Spatial Analysis Tools:

• spatial_phlex.py: Advanced spatial analysis (DBSCAN, barrier scoring, GPU-accelerated)
• clustering.py: Cell clustering algorithms

Pathology & Imaging Tools:

• pathology.py: Specialized pathology functions (aortic analysis, ATP assays, thrombus histology, calcium imaging, corneal nerve quantification, bone morphometry)
• bioimaging.py: Medical image processing (nnUNet, SimpleITK registration)
• imaging.py: General imaging utilities
• background_subtraction.py: Background correction
• basic_illumination.py: Illumination correction

Database Tools:

• database.py: Database query utilities
• genetics.py, genomics.py: Genetic/genomic data access
• molecular_biology.py: Molecular biology tools

Visualization Tools:

• visualization.py: Plotting utilities
• volume_render.py: 3D volume rendering

Documentation: See backend/biomni/README.md

2. Vector Search (vector_search/)

Location: backend/vector_search/

Elasticsearch-based similarity search for tissues and cells.

Architecture

Quantification CSV (Orion-CRC or OncoPathology)
    ↓
Preprocessing (preprocess.py)
    ├─→ Dataset Detection (auto-detects Orion-CRC vs OncoPathology)
    ├─→ Marker Extraction (10 common markers)
    ├─→ Normalization (arcsinh cofactor=5.0 + z-score)
    ├─→ Tissue Embedding (70-dim: 7 stats × 10 markers)
    └─→ Cell Embeddings (10-dim: normalized intensities, subsampled 5000/patient)
    ↓
JSONL Files
    ├─→ tissue_embeddings.jsonl
    └─→ cell_embeddings.jsonl
    ↓
Elasticsearch Ingestion (elastic.py)
    ├─→ biomni-tissue-embeddings index (70-dim dense_vector, cosine similarity)
    └─→ biomni-cell-embeddings index (10-dim dense_vector, cosine similarity)
    ↓
kNN Search
    ├─→ Tissue-level: Find similar patient profiles
    └─→ Cell-level: Find similar individual cells

Embedding Strategy

Tissue Embeddings (70 dimensions):

• For each of 10 markers: [mean, std, q10, q25, median, q75, q90]
• Captures overall tissue phenotype distribution

Cell Embeddings (10 dimensions):

• Direct normalized marker intensities (arcsinh + z-score)
• One vector per cell (subsampled: 5000 cells per patient)

Clinical Metadata

Each document includes:

• Patient ID, diagnosis, location, stage (AJCC), TNM
• MMR status (pMMR/dMMR), PFS (days), recurrence, treatment
• Spatial coordinates (X_centroid, Y_centroid)
• Morphology features (Area, MajorAxisLength, Eccentricity, etc.)

Documentation: See backend/vector_search/README.md

3. OncoRAX (xray/OncoRAX/)

Location: backend/xray/OncoRAX/

Multi-turn agentic pipeline for radiology report analysis.

Pipeline Architecture

Chest X-Ray Image
    ↓
[1] OncoRAX Inference
    → Connects to OncoRAX agent
    → Initial findings extraction
    ↓
[2] Quality Critique
    → critique.py
    → Identifies missing checks, contradictions, uncertainties
    ↓
[3] Case Retrieval
    → retrieval.py
    → Elasticsearch Cloud (BM25 + vector search, 1024-dim embeddings)
    → Jina Embeddings v3 API
    → 310 indexed cases (300 cancer + 10 normal)
    ↓
[4] Impression Revision
    → Refines with cautious medical language
    ↓
[5] Differential Diagnosis
    → differential.py
    → Bayesian probabilistic ranking (8+ conditions)
    → Dataset-derived priors (NIH Chest X-ray Dataset)
    → 100% cancer detection rate (Mass/Nodule in top 3)
    ↓
[6] Report Composition
    → report.py
    → Structured doctor reports + patient summaries
    → JSON + Markdown output
    ↓
Complete Clinical Report

Key Features

• Bayesian Differential Diagnosis: Normalized probability distributions, evidence-based priors, likelihood mappings
• Quality Assurance: Automated critique catches errors before finalization
• Case Retrieval: Hybrid search (BM25 + vector) with 100% retrieval success rate
• Structured Reports: EMR-ready formats with complete audit trail
• Cancer Detection: Trained on NIH dataset (112,120 X-rays), 100% detection rate

Documentation: See backend/xray/OncoRAX/README.md

---

Frontend Architecture

Location: frontend/

Tech Stack

• Framework: React 19 + Vite 6
• Language: TypeScript (strict mode)
• Styling: TailwindCSS 3.4
• Backend Bridge: FastAPI (Python) with WebSocket and SSE
• Terminal Emulation: xterm.js for log display
• Markdown Rendering: react-markdown with syntax highlighting

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Frontend (React)                      │
│  Port: 5173 (Vite Dev Server)                            │
├─────────────────────────────────────────────────────────┤
│                                                           │
│  ┌──────────────┐    ┌──────────────┐                   │
│  │   App.tsx   │    │  Components  │                   │
│  │  (Main)     │───►│  - Chat      │                   │
│  └──────────────┘    │  - Terminal  │                   │
│         │            │  - Reports  │                   │
│         │            │  - Metrics  │                   │
│         ▼            └──────────────┘                   │
│  ┌──────────────┐                                       │
│  │ useAgentSocket│                                      │
│  │  (Hook)      │                                       │
│  └──────────────┘                                       │
│         │                                               │
│         │ WebSocket (/api/ws)                           │
│         │ SSE (/api/chat)                               │
│         ▼                                               │
│  ┌──────────────┐                                       │
│  │ FastAPI      │                                       │
│  │ Server       │                                       │
│  │ Port: 8000   │                                       │
│  └──────────────┘                                       │
│         │                                               │
│         │ Imports biomni package                         │
│         ▼                                               │
│  ┌──────────────┐                                       │
│  │ ClaudeSDKAgent│                                      │
│  │ (Backend)    │                                       │
│  └──────────────┘                                       │
└─────────────────────────────────────────────────────────┘

Key Components

Core Hooks:

• useAgentSocket.ts: Manages WebSocket connection, session state, streaming
• useTheme.ts: Dark/light theme management

Components:

• App.tsx: Main application orchestrator
• ChatPanel.tsx: Interactive chat interface
• TerminalPanel.tsx: Real-time log display (xterm.js)
• ReportDrawer.tsx: Structured report viewer
• InsightsPanel.tsx: Analysis insights display
• PipelineSelector.tsx: Pipeline selection UI

Server (frontend/server/):

• main.py: FastAPI server with WebSocket and SSE endpoints
• chat_agent.py: OpenAI-powered Q&A with web-search tool calling

Communication Protocols

WebSocket (/api/ws):

• Real-time bidirectional communication
• Events: chat, reset, status, assistant_text, tool_exec, tool_result, artifact
• Used for pipeline execution via ClaudeSDKAgent

Server-Sent Events (/api/chat):

• Streaming chat responses
• OpenAI-powered Q&A with analysis context
• View modes: doctor (clinical) vs patient (simplified)

REST Endpoints:

• POST /api/upload: File upload for pipeline inputs
• POST /api/reset: Reset agent session
• GET /api/health: Health check

---

Installation & Setup

Prerequisites

• Python ≥ 3.11
• Node.js ≥ 18 (for Claude Code CLI and frontend)
• Docker Desktop (for containerized bioimaging tools)
• Elasticsearch (optional, for vector search)
• Conda (recommended for environment management)

Step-by-Step Setup

1. Clone Repository

git clone git@github.com:tomtommyyuan/targetONCO.git
cd targetONCO

2. Create Conda Environment

conda create -n treehacks26 python=3.11 -y
conda activate treehacks26

Important: Always use python -m pip (not bare pip) when installing packages inside conda environments.

3. Install Node.js & Claude Code CLI

# macOS
brew install node

# Or download from https://nodejs.org/

# Install Claude Code CLI globally
npm install -g @anthropic-ai/claude-code

# Authenticate (one-time interactive step)
claude login

This stores your Claude Pro/Max/Team credentials for the SDK.

4. Install Backend Package

cd backend
python -m pip install -e .

This installs all Python dependencies from pyproject.toml:

• claude-agent-sdk: Claude Agent SDK
• langchain, langgraph: Agent framework (A1 agent)
• anthropic: Anthropic API client
• pandas, numpy, tifffile: Data handling
• fastapi, uvicorn, websockets: Frontend server
• elasticsearch: Vector search (optional)

5. Install Frontend Dependencies

cd ../frontend
npm install

Installs React, Vite, TypeScript, TailwindCSS, and other frontend dependencies.

6. Install Docker Images (Optional but Recommended)

# Core OncoPathology pipeline tools (ARM64 for Mac)
docker pull labsyspharm/mcquant:arm64-local
docker pull labsyspharm/unmicst:arm64-local
docker pull labsyspharm/scimap:arm64-local
docker pull labsyspharm/s3segmenter:arm64-local

# Additional imaging tools
docker pull labsyspharm/basic-illumination:latest
docker pull labsyspharm/ashlar:latest

# TMA dearraying
docker pull labsyspharm/unetcoreograph:latest

# Verify
docker images | grep labsyspharm

Docker images are pulled automatically when tools run, so this step is optional.

7. Setup Elasticsearch (Optional, for Vector Search)

Option A: Local Elasticsearch

# macOS
brew install elasticsearch
brew services start elasticsearch

# Or download from https://www.elastic.co/downloads/elasticsearch

Option B: Elastic Cloud

Sign up at https://cloud.elastic.co and create a cluster.

8. Verify Installation

# Check SDK import
python -c "from claude_agent_sdk import query; print('claude-agent-sdk OK')"

# Check biomni import
python -c "from biomni.agent import ClaudeSDKAgent; print('ClaudeSDKAgent OK')"

# Check vector search
python -c "from vector_search.elastic import VectorSearchClient; print('VectorSearchClient OK')"

# Check version
biomni --version

---

Environment Variables

Required

# Anthropic API key (for Claude models)
export ANTHROPIC_API_KEY="sk-ant-..."

Optional: Backend Configuration

# Working directory (default: data/ relative to project root)
export BIOMNI_PATH="data"

# Timeout in seconds (default: 1800 = 30 min)
export BIOMNI_TIMEOUT_SECONDS="1800"

# Model (default: claude-sonnet-4-20250514)
export BIOMNI_LLM="claude-sonnet-4-20250514"

# Temperature (default: 0.7, only for A1 agent)
export BIOMNI_TEMPERATURE="0.7"

# Commercial mode — exclude non-commercial datasets (default: false)
export BIOMNI_COMMERCIAL_MODE="false"

Optional: Vector Search (Elasticsearch)

# For local Elasticsearch
export ES_URL="http://localhost:9200"
export ES_API_KEY=""  # Leave empty for local

# For Elastic Cloud
export ES_URL="https://your-cluster.es.us-east-1.aws.cloud.es.io:9243"
export ES_API_KEY="your-api-key-here"

Optional: Frontend Chat Agent (OpenAI)

# For /api/chat endpoint (OpenAI-powered Q&A)
export OPENAI_API_KEY="sk-..."
export OPENAI_MODEL="gpt-4o"  # Default

Optional: OncoRAX

# OncoRAX model directory
export MODEL_DIR="~/OncoRAX-models"

# OncoRAX Elasticsearch Cloud (for case retrieval)
export ELASTIC_CLOUD_ID="your-cloud-id"
export ELASTIC_API_KEY="your-api-key"

# Jina API (for embeddings generation)
export JINA_API_KEY="your-jina-api-key"

Environment File

Create .env in project root:

# .env
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
ES_URL=http://localhost:9200
BIOMNI_PATH=data
BIOMNI_TIMEOUT_SECONDS=1800

---

Running the System

Option 1: Frontend (Recommended for Web UI)

# From project root
./frontend/dev.sh

This script:

1. Installs frontend dependencies if needed
2. Starts FastAPI backend on port 8000
3. Starts Vite dev server on port 5173
4. Opens http://localhost:5173 in browser

Features:

• Real-time WebSocket communication
• Interactive chat interface
• Terminal log display
• Report viewer
• Pipeline selection (OncoPathology, OncoRAX)

Option 2: CLI (Command-Line Interface)

# Interactive session
biomni

# Non-interactive (single query)
biomni -p "Run the full OncoPathology pipeline on image.ome.tif"

# With options
biomni -p "Your prompt" \
    --model claude-sonnet-4-5 \
    --commercial \
    --path ./data

CLI Options:

biomni [options] [prompt]

Options:
  -p, --print                     Print response and exit (non-interactive)
  -v, --verbose                   Show detailed progress
  --commercial                    Use commercial mode
  --path <path>                   Path to working directory
  --model <model>                 Model to use
  --dangerously-skip-permissions  Skip permission prompts
  --version                       Output version number
  -h, --help                      Display help

Option 3: Python API

from biomni.agent import ClaudeSDKAgent

# Initialize agent
agent = ClaudeSDKAgent(
    path='./data',
    model='claude-sonnet-4-20250514',
    permission_mode='acceptEdits'  # or 'default', 'bypassPermissions'
)

# Single query
result = agent.go("Analyze this tissue image")
print(result)

# Multi-turn (session maintained automatically)
result = agent.go("What about the CD8+ cells?")
print(result)

# Reset conversation
agent.reset_session()

# Streaming (async)
async for msg in agent.go_stream("Your task"):
    print(msg)

Option 4: Runner Script

# Basic usage
python backend/run_agent.py "Your prompt here" --api-key sk-ant-...

# With prompt file
python backend/run_agent.py --prompt-file prompt.txt --output-dir ./logs

# With proxy API
python backend/run_agent.py "Your prompt" \
    --base-url http://proxy:3888/ \
    --auth-token sk-...

Features:

• Automatic log file generation with timestamps
• API configuration (native or proxy)
• PYTHONPATH management
• Output streaming to console and log file

---

Testing

Backend Testing

Test OncoPathology Pipeline

# Test probability map generation
python -c "
from biomni.tool.segmentation_unmicst import generate_probability_maps
result = generate_probability_maps(
    input_image='test_image.ome.tif',
    output_dir='./test_output/',
    channel=0
)
print(result)
"

# Test cell segmentation
python -c "
from biomni.tool.segmentation_s3segmenter import segment_cells
result = segment_cells(
    input_image='test_image.ome.tif',
    probability_maps_dir='./test_output/',
    output_dir='./test_output/segmentation/',
    segment_cytoplasm=True,
    cytoplasm_channels=[2]
)
print(result)
"

# Test quantification
python -c "
from biomni.tool.quantification import quantify_cells
result = quantify_cells(
    image_path='test_image.ome.tif',
    mask_paths=['./test_output/segmentation/test_image/cell.ome.tif'],
    channel_names='markers.csv',
    output_dir='./test_output/'
)
print(result)
"

Test Vector Search

# Setup Elasticsearch indices
python -m vector_search.elastic setup

# Preprocess sample data
python -m vector_search.preprocess \
    --csv sample_cell.csv \
    --patient-id test_sample \
    --output-dir ./test_embeddings

# Ingest into Elasticsearch
python -m vector_search.elastic ingest \
    --tissue ./test_embeddings/tissue_embeddings.jsonl \
    --cells ./test_embeddings/cell_embeddings.jsonl

# Test search
python -m vector_search.example_search

# Check status
python -m vector_search.elastic status

Test OncoRAX

cd backend/xray/OncoRAX

# Run end-to-end pipeline
python scripts/run_case.py \
    --image demo/chest/normal1.jpg \
    --output report.json

# Run tests
pytest tests/

Frontend Testing

cd frontend

# Start dev server
npm run dev

# Run in browser
# Open http://localhost:5173
# Test WebSocket connection
# Test chat interface
# Test file uploads

Integration Testing

# Test full stack
./frontend/dev.sh

# In browser:
# 1. Upload test image
# 2. Select OncoPathology pipeline
# 3. Run pipeline
# 4. Verify results in terminal/logs
# 5. Test chat interface
# 6. Test report viewing

---

Technical Implementation Details

Backend Implementation

Agent Architecture

ClaudeSDKAgent (backend/biomni/agent/claude_sdk_agent.py):

• Uses official claude-agent-sdk Python package
• Leverages Claude Code CLI (installed via npm)
• Session management via session_id from SDK
• Streaming support via go_stream() async generator
• Permission modes: default, acceptEdits, bypassPermissions
• Skill injection: Automatically loads OncoPathology pipeline skills

A1 Agent (backend/biomni/agent/a1.py):

• LangGraph-based agent framework
• Uses Anthropic API directly
• Configurable temperature
• Tool discovery via tool_registry.py
• Environment description via env_desc.py or env_desc_cm.py

Tool Execution

Container Runtime Detection:

• Auto-detects Docker, Singularity, or Apptainer
• Prefers Apptainer/Singularity over Docker (HPC environments)
• Falls back gracefully if no container runtime available

Long-Running Task Management:

• Configurable timeouts per tool
• Polling strategy for background tasks (60-second intervals)
• Comprehensive error messages and recovery
• Progress tracking via research logs

Format Support:

• Images: OME-TIFF, TIF, TIFF, H5, HDF5
• Quantification: CSV (histoCAT-compatible)
• Markers: CSV files with marker names
• Metadata: JSON, CSV for clinical data

Vector Search Implementation

Preprocessing (backend/vector_search/preprocess.py):

• Automatic dataset detection (Orion-CRC vs OncoPathology)
• Column name mapping via config.py
• Arcsinh transformation (cofactor = 5.0, standard for CyCIF)
• Z-score normalization per marker
• Subsampling (5000 cells per patient by default)

Elasticsearch Integration (backend/vector_search/elastic.py):

• Dense vector indices with cosine similarity
• kNN search with configurable num_candidates
• Filtered search with Elasticsearch query DSL
• Bulk ingestion with error handling
• Index management (create, delete, status)

Frontend Implementation

WebSocket Communication

Protocol (frontend/server/main.py):

• WebSocket endpoint: /api/ws
• Message types: chat, reset
• Event types: status, assistant_text, tool_exec, tool_result, artifact
• Error handling with traceback reporting

Client-Side (frontend/src/hooks/useAgentSocket.ts):

• React hook for WebSocket management
• Session state management
• Real-time message streaming
• Terminal log accumulation
• Artifact extraction (base64 images)

Server-Sent Events (SSE)

Chat Endpoint (frontend/server/main.py):

• POST /api/chat with streaming SSE
• OpenAI-powered Q&A with analysis context
• View mode support (doctor vs patient)
• Pipeline-aware context loading
• Web-search tool calling for supplementary information

Implementation (frontend/server/chat_agent.py):

• Loads analysis context from report_demo/data/
• Builds system prompts based on view mode
• Streams responses as JSON events
• Handles tool calls (web search)

Component Architecture

State Management:

• React Context for agent state (contexts/AgentContext.tsx)
• React Context for report state (contexts/ReportContext.tsx)
• Local state for UI components

Real-Time Updates:

• WebSocket for pipeline execution
• SSE for chat responses
• Terminal emulation with xterm.js
• Markdown rendering with syntax highlighting

---

Credits & Acknowledgments

Base Framework

Biomni:

• Original framework: snap-stanford/biomni
• Paper: Huang, Kexin et al. "Biomni: A General-Purpose Biomedical AI Agent" (bioRxiv 2025)
• License: Apache 2.0

Tool Developers

OncoPathology Pipeline Tools:

• UnMicst: Universal Models for Identifying Cells and Segmenting Tissue
• S3segmenter: Watershed-based segmentation tool
• mcquant: Single-cell quantification tool
• Spatial-PHLEX: Advanced spatial analysis pipeline (TRACERx-PHLEX suite)
• Deep-IMCYTO: Nextflow-based segmentation pipeline
• TYPEx: Cell type classification pipeline

Container Images: Provided by LabSysPharm (labsyspharm/*)

OncoRAX

Original OncoRAX:

• Repository: bowang-lab/OncoRAX
• Paper: Fallahpour, Adibvafa et al. "OncoRAX: Medical Reasoning Agent for Chest X-ray" (ICML 2025, arXiv:2502.02673)
• Authors: Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, Bo Wang

Training Dataset:

• NIH Chest X-ray Dataset: 112,120 X-ray images from 30,805 unique patients
• Reference: Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. "ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases." IEEE CVPR 2017

Vector Search

Embedding Strategy:

• Based on Orion-CRC dataset (Lin et al. 2023, Nature Cancer)
• Clinical metadata from colorectal cancer cohort

Frontend

Technologies:

• React 19, Vite 6, TypeScript, TailwindCSS
• FastAPI, uvicorn, websockets
• xterm.js for terminal emulation
• react-markdown for content rendering

Contributors

This implementation extends Biomni with:

• OncoPathology spatial proteomics pipeline
• Vector search system for tissue/cell similarity
• Enhanced OncoRAX agentic workflow
• Full-stack web interface
• Comprehensive tool ecosystem

---

License

Apache 2.0 — see LICENSE at the repository root, mirrored at backend/LICENSE.

Individual tools may have their own licenses. See tool directories for specific license information.

---

Citation

If you use targetONCO, please cite:

Base Biomni:

@article{huang2025biomni,
  title={Biomni: A General-Purpose Biomedical AI Agent},
  author={Huang, Kexin and Zhang, Serena and Wang, Hanchen and others},
  journal={bioRxiv},
  year={2025}
}

OncoPathology & OncoPath Extensions:

@software{targetonco2025,
  title={targetONCO: Extended Biomni for Spatial Proteomics and Oncology},
  author={Mohammad Zoraiz, YuCheng (Tom) Yuan, Sheldon Lewis, Suhas Kurapati},
  year={2025},
  url={https://github.com/zoraizmohammad/targetONCO}
}

MedRAX:

@misc{fallahpour2025OncoRAX,
  title={MedRAX: Medical Reasoning Agent for Chest X-ray},
  author={Fallahpour, Adibvafa and Ma, Jun and Munim, Alif and Lyu, Hongwei and Wang, Bo},
  year={2025},
  eprint={2502.02673},
  archivePrefix={arXiv}
}

---

Additional Resources

• Backend Documentation: backend/README.md
• OncoPathology Details: backend/biomni/README.md
• Vector Search Guide: backend/vector_search/README.md
• OncoRAX Documentation: backend/xray/OncoRAX/README.md
• Frontend README: frontend/README.md