Corelib Monorepo

A high-performance, resilient, and multi-runtime monorepo for TypeScript and Rust. This workspace provides foundational utilities, cloud extensions, and financial market tooling designed for Node.js, Bun, and Deno.

🚀 Overview

This repository is structured as a pnpm monorepo, integrating TypeScript's flexibility with Rust's performance through FFI (Foreign Function Interface). It is built on principles of resilience, performance, and cross-runtime compatibility.

Key Features

• Isomorphic Core: All foundational utilities in ts-core support Node.js, Bun, and Deno.
• Resilient HTTP: Standardized fetch wrapper (RequestUnlimited) with automatic retries and consistent error serialization. RequestProxied extends this with self-healing, rotating proxy pools that automatically prune dead endpoints.
• Strict Logging: Structured logging API with (msg: string, extras?: object) signature and telemetry support.
• Native Rust Performance: Performance-critical paths (streaming engines, WS clients) implemented in Rust via N-API/FFI — zero JS event-loop blocking via napi worker threads.
• Real-Time Streaming: Live market data from Alpaca, Finnhub, and Yahoo Finance; the Rust engine owns the WebSocket, delivering data to JS as events via napi ThreadsafeFunction.
• Unified Configuration: Centralized ConfigManager supporting overrides via files, environment variables, and CLI.

---

Architecture

graph TD
    A[Your App] -->|npm install| B["@ckirg/corelib (ts-core)"]
    A -->|npm install| C["@ckirg/corelib-markets (ts-markets)"]
    A -->|npm install| D["@ckirg/corelib-cloud (ts-cloud)"]
    C --> B
    D --> B
    D --> C
    B <-->|N-API FFI| E["corelib-rust (.node binary)"]
    C -->|"streaming via FFI\n(napi ThreadsafeFunction)"| E
    E -->|"WS: Alpaca / Finnhub / Yahoo"| F[Live Market Feeds]

Loading

Full detail: ARCHITECTURE.md — package map, runtime matrix, FFI surface, proxy semantics, CI/CD stages.

---

📂 Project Structure & Packages

Package	Description	Documentation
@ckirg/corelib	Core logic, FFI bridge, resilient HTTP, and database abstractions.	README
@ckirg/corelib-markets	Market data tooling (Nasdaq, Yahoo) and financial indicators.	README
@ckirg/corelib-cloud	Cloud-specific extensions for AWS, GCP, and Cloudflare.	README
corelib-rust	Native Rust core exposed via N-API (FFI).	README

---

📦 Installation for External Projects

These packages are published to the public npm registry under the @ckirg scope.

1. Install

npm install @ckirg/corelib @ckirg/corelib-markets
# or: pnpm add @ckirg/corelib @ckirg/corelib-markets
# or: bun add  @ckirg/corelib @ckirg/corelib-markets

2. Required .npmrc (when using @ckirg/corelib-markets)

@ckirg/corelib-markets depends on @gadicc/yahoo-finance2, distributed via JSR. Add the JSR registry mapping to your project's .npmrc, or installs will 404 on the transitive @jsr/... package:

@jsr:registry=https://npm.jsr.io

🦀 The Native Rust Binary

@ckirg/corelib ships a postinstall that downloads the correct prebuilt Rust binary (corelib-rust-*.node) for your OS/arch from the matching GitHub Release.

• Supported targets: linux-x64, linux-arm64, win32-x64, darwin-x64, darwin-arm64. The Linux modules are built against glibc 2.34, so they also load on AWS Lambda and older distros (linux-arm64, for AWS Graviton, Ampere or a Raspberry Pi, ships from v0.3.1; v0.3.0's linux-x64 needs glibc 2.38).
• Bun consumers: Bun blocks dependency postinstall scripts by default. Add corelib to trustedDependencies in your package.json, or the binary is never fetched and the FFI fails at runtime:

  "trustedDependencies": ["@ckirg/corelib"]

• MODE=development caveat: the postinstall skips the download when MODE=development (or under CI's GITHUB_ACTIONS). Don't set MODE=development in a consuming app's .env, or you'll hit a "module not found" for the addon.
• Manual trigger if the download was skipped or blocked:

  node node_modules/@ckirg/corelib/scripts/postinstall.js

---

🛠️ Getting Started

Prerequisites

• Node.js (v24+ recommended)
• pnpm (v11.5+ required)
• Rust (for native module builds)

Common Commands

All workspace-wide commands are managed through pnpm:

Command	Description
pnpm install	Install all dependencies and link packages
pnpm build-all	Build all packages in correct order
pnpm test-all	Run all unit and integration tests
pnpm lint-all	Run Biome linting and formatting checks
pnpm format-all	Apply Biome formatting automatically
pnpm docs-all	Regenerate all API documentation
pnpm clean-all	Wipe all node_modules, dist, and target folders

---

📖 Usage Examples

1. Core Utilities (@ckirg/corelib)

import { logger, endPoint, RequestProxied, ConfigManager } from '@ckirg/corelib';

// 1. Resilient Fetch (ky-powered with retries)
const result = await endPoint('https://api.nasdaq.com/api/market-info');

// 2. Proxied Fetch (Automatic rotation & fallback)
const client = new RequestProxied(["https://proxy1.com", "https://proxy2.com"]);
const proxiedResult = await client.endPoint("https://target.com");

// 3. Configuration Management
const config = ConfigManager.getInstance();
await config.initialize();              // single-flight; safe under concurrency
if (!config.isInitialized) await config.whenReady();
const port = config.get("database.port"); // read at the use site, don't cache slices

2. Market Data (@ckirg/corelib-markets)

import { MarketMonitor, MarketSymbols, Historical, type MarketPhase } from '@ckirg/corelib-markets';

// 1. Resilient Status Poller
const monitor = new MarketMonitor();
monitor.on("status-change", (phase: MarketPhase) => {
  console.log(`Market phase changed to ${phase}`);
});
monitor.start();

// 2. Persistent Symbol Database
const symbols = new MarketSymbols();
const aapl = await symbols.get("AAPL"); // Auto-refreshes if needed

// 3. Resilient Historical Data (Yahoo Finance v3)
const history = await Historical.getData("AAPL", { period1: "2023-01-01" });
if (history.status === 'success') {
  console.log(`Retrieved ${history.value.length} quotes`);
}

2b. Real-Time Streaming (flagship — Rust WS engine, zero event-loop blocking)

import { AlpacaStreaming } from '@ckirg/corelib-markets';

const stream = new AlpacaStreaming();

// Credentials via init() or APCA_API_KEY_ID / APCA_API_SECRET_KEY env vars
await stream.init({ keyId: 'YOUR_KEY', secretKey: 'YOUR_SECRET' });
await stream.start();

stream.subscribe(['AAPL', 'TSLA', 'NVDA']);

stream.on('pricing', (data) => {
  console.log(`${data.symbol}: $${data.price}`);
});
stream.on('connected',    () => console.log('connected'));
stream.on('disconnected', () => console.log('disconnected'));
stream.on('error',        (e) => console.error('error', e));

// FinnhubStreaming and YahooStreaming follow the same EventEmitter pattern
// (Finnhub uses token auth; Yahoo is unauthenticated — see ts-markets/README.md)

3. Edge Proxy Services (@ckirg/corelib-cloud)

A portable TypeScript service exposing corelib logic on Cloudflare Workers, AWS Lambda, and Cloud Run.

# Example: Fetching market data via the Edge Proxy
curl -X POST https://your-edge-service/api/v1/markets/nasdaq \
     -H "Content-Type: application/json" \
     -d '{
       "url": "https://api.nasdaq.com/api/quote/AAPL/info?assetclass=stocks"
     }'

---

📖 Documentation

The project publishes a documentation site at ckir.github.io/corelib:

• Landing site
• API Documentation Index
  • Core Utilities Reference (@ckirg/corelib)
  • Market Data Reference (@ckirg/corelib-markets)
  • Cloud Extensions Reference (@ckirg/corelib-cloud)
  • Rust Native Core Reference (corelib-rust; redirects to the crate page)
• User Guide (PDF)

The site is authored in site/; the generated API reference and the user guide are merged into _site/ by pnpm site:build. pnpm site:preview serves it locally at http://127.0.0.1:4173.

---

📜 License

Refer to the LICENSE file for details.