Confidential Model as a Service on Alibaba Cloud Model Studio

中文版：README_zh.md

Project Overview

CMaaS (Confidential Model as a Service) is an attested inference solution built on Trusted Execution Environment (TEE, e.g., Intel TDX) technology. It addresses data privacy challenges in AI inference through TEE isolation, end-to-end encryption, and remote attestation.

The system builds trust through three key dimensions: Secure Channel for end-to-end encrypted communication, Attested Node for verifiable execution environment, and Verified Code for auditable software provenance. Together, these dimensions provide comprehensive security guarantees:

End-to-End Encryption An application-layer secure channel is established on top of HTTPS between the client and the confidential inference node. Each TEE node generates a random root private key for session key negotiation, with an ephemeral key derived for each session. All keys are held in memory on their respective side: client memory or the node's TEE-protected memory. Neither the SDK nor the inference service persists them to disk.
Security Hardening & Integrity Measurement Leveraging a dual-layered defense of chip-level hardware isolation and software hardening, the system performs full-stack integrity measurement of the boot chain and runtime environment. At the block storage level, the root partition's integrity is protected by a Merkle Tree, while the data partition is encrypted with one-time keys to ensure confidentiality.
Software & Hardware Transparency Clients can directly verify the authenticity of the hardware-generated remote attestation report with the TEE vendor. Furthermore, they can inspect a public transparency log to confirm the integrity of the server-side software artifacts.

For the complete system security architecture design, see the architecture document.

How to Use

Overview

The CMaaS Client SDK abstracts the complexity of various security hardening into simple, easy-to-use APIs, enabling developers to integrate confidential inference capabilities into their applications with minimal code changes.

SDK Library (sdk/): Core client implementation with attestation, verification and encryption
CLI Tool (cmd/cmaas-client/): Command-line tool demonstrating SDK usage
OpenAI-Compatible Proxy (cmd/openai-proxy/): HTTP server that exposes /v1/chat/completions backed by the SDK, so existing OpenAI clients (Python/JS SDKs, vLLM benchmark tools, etc.) can access the confidential inference service without code changes
Anthropic-Compatible Proxy (cmd/anthropic-proxy/): HTTP server that exposes /v1/messages for the Anthropic Messages API, so Anthropic SDKs and Claude Code can target the confidential inference service by only changing base_url
Audit Tool (cmd/cmaas-audit/): Offline verification of local transparency reports

The SDK manages the following areas:

Transport Management: Manages same-node routing, concurrent requests, and automatic session recovery over the Model Studio HTTP/SSE path
TEE Attestation: Verifies TEE Quote signatures and EventLog integrity, with public key caching to reduce overhead in load-balanced deployments
Transparency Log Verification: Verifies release manifests against transparency logs service, ensuring the inference node runs a known, audited software version
Encrypted Requests: Implements RA-based key negotiation and end-to-end encryption for all inference data

The design ensures that plaintext prompts and responses never leave the client's local environment or the TEE's isolated memory, providing end-to-end confidentiality throughout the inference lifecycle.

Quick Start (CLI)

The CMaaS CLI serves as a reference implementation and testing utility for confidential inference.

Build and Run

# From the repository root
make client

export DASHSCOPE_API_KEY=your_api_key
bin/cmaas-client \
  --endpoint https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services \
  --model    <code> \
  --prompt   "What is confidential computing?" \
  --stream

CLI Options

Flag	Required	Default	Description
`--endpoint`	Yes	(none)	Model Studio gateway endpoint
`--model`	Yes	(none)	Model name for gateway routing
`--prompt`	Yes	(none)	Inference prompt to send
`--stream`	No	`false`	Enable streaming mode
`--transparency-report-dir`	No	OS-specific	Directory for local transparency reports
`--no-prompt-in-transparency-report`	No	`false`	Exclude prompts from local transparency reports

Note: The DASHSCOPE_API_KEY environment variable must be set for authentication.

Quick Start (SDK)

The SDK provides Client and SessionManager APIs, both with attestation, request encryption, and response decryption.

The ChatCompletionRequest sampling parameters (e.g. Temperature, TopP, MaxTokens) follow the naming conventions of the OpenAI Chat Completion API for familiarity. The SDK also supports function/tool calling (Tools, ToolChoice, ParallelToolCalls), reasoning models (ReasoningContent/Reasoning fields and ThinkingTokenBudget), and text, reasoning, logprobs, and tool-call fragments through the unified Chunks streaming channel.

Client

Client selects a model at initialization and sends inference requests through Chat.

client, err := sdk.NewClient(&sdk.Options{
    Endpoint: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services", // Model Studio gateway endpoint
    APIKey:   os.Getenv("DASHSCOPE_API_KEY"),                                  // Model Studio API key
    Model:    "<code>",                                                        // Model code on Model Studio
})
if err != nil {
    return err
}
defer client.Close()

completion, err := client.Chat(context.Background(), &sdk.ChatCompletionRequest{
    Messages: []sdk.Message{
        {Role: "user", Content: "What is confidential computing?"},
    },
    MaxTokens:   sdk.Int(2048),   // maximum response tokens (adjust as-needed)
    Temperature: sdk.Float(0.7),  // temperature (adjust as needed)
})
if err != nil {
    return err
}

// Decrypted inference result
fmt.Println("Response:", completion.Content)

// Inspect attestation result
fmt.Printf("Verified At: %v\n", completion.Attestation.VerifiedAt)
fmt.Printf("TEE Type: %s\n", completion.Attestation.Bundle.TEEType)

SessionManager

SessionManager accepts a model on each Chat call and manages encrypted sessions by model and request affinity. Requests with the same model and affinity reuse a session whenever possible, reducing repeated key negotiation and providing affinity routing.

manager, err := sdk.NewSessionManager(&sdk.SessionManagerOptions{
    Options: sdk.Options{
        Endpoint: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services",
        APIKey:   os.Getenv("DASHSCOPE_API_KEY"),
    },
})
if err != nil {
    return err
}
defer manager.Close()

completion, err := manager.Chat(context.Background(), "<code>", nil, &sdk.ChatCompletionRequest{
    Messages: []sdk.Message{{Role: "user", Content: "What is confidential computing?"}},
    MaxTokens: sdk.Int(2048),
})
if err != nil {
    return err
}
fmt.Println("Response:", completion.Content)

Streaming Inference

Both APIs enable streaming with Stream: true and return incremental results through Chunks. The example below uses Client; for SessionManager, replace the call with manager.Chat(ctx, model, nil, req). Result handling is the same.

completion, err := client.Chat(ctx, &sdk.ChatCompletionRequest{
    Messages: []sdk.Message{{Role: "user", Content: "Write a story"}},
    Stream:   true,
    MaxTokens:   sdk.Int(2048),
    Temperature: sdk.Float(0.7),
})
if err != nil {
    return err
}

// Read and print text from the unified streaming channel
for chunk := range completion.Chunks {
    fmt.Print(chunk.Content)
}
fmt.Println() // Newline after streaming completes
if completion.Error != nil {
    return completion.Error
}

// Inspect attestation result
fmt.Printf("Verified At: %v\n", completion.Attestation.VerifiedAt)
fmt.Printf("TEE Type: %s\n", completion.Attestation.Bundle.TEEType)

Consume Chunks until the channel closes, then check completion.Error. Cancel the request context if you stop reading early. Both APIs cancel unfinished streams on Close() and wait for their related transparency audit reports to finish writing before returning.

See the standalone examples for reasoning output and tool calls.

SDK Options

Common Options fields are listed below. Client uses Options directly; SessionManager uses SessionManagerOptions.Options for the same gateway, security verification, and audit settings. Model is set at initialization only for Client; SessionManager takes the model in each Chat call.

type Options struct {
    Endpoint string     // Model Studio gateway endpoint (required)
    APIKey   string     // Model Studio API key (required)
    Model    string     // Model code on Model Studio (required for Client)

    PubKeyCacheTTL           time.Duration // Public key cache TTL (default: 12h, -1 to disable)
    GatewayExtraHeaders      http.Header   // Additional Model Studio gateway headers

    TransparencyLogs          []sdk.TransparencyLogConfig // Trusted transparency log instance configurations
    TransparencyReportDir     string                      // Audit report directory (default: OS-specific)
    DisableTransparencyReport bool                        // Disable local audit report persistence
    NoPromptInTransparencyReport bool                     // Exclude prompts from local transparency reports

    AllowAttestationFallback bool // Allow automatic fallback on attestation failure (default: false)
}

Options Details:

Endpoint: Model Studio gateway endpoint.
PubKeyCacheTTL: Remote attestation verification requires fetching collateral from the TEE vendor and querying transparency log inclusion proofs, which may introduce significant latency. To reduce this overhead, the SDK caches verified server public keys. In load-balanced deployments, the client may connect to multiple backend nodes, and the cache significantly accelerates subsequent connections to previously verified nodes. Set this value to -1 to disable the cache and force full verification every time.
GatewayExtraHeaders: Headers added to Model Studio gateway requests.
TransparencyLogs: Configures trusted transparency log instances. When empty, the SDK uses its built-in Rekor configuration.
TransparencyReportDir: Specifies the directory where transparency reports which are auditable are stored. When empty, OS-specific defaults are used:
- macOS: ~/Library/Application Support/dashscope-confidential-maas/transparency-reports/
- Linux: $XDG_DATA_HOME/dashscope-confidential-maas/transparency-reports/
- Windows: %APPDATA%\dashscope-confidential-maas\transparency-reports\
NoPromptInTransparencyReport: Set to true to exclude prompts from local transparency reports while preserving all other report fields and attestation materials.
DisableTransparencyReport: Set to true to disable local persistence of transparency reports. (Not recommended.)
AllowAttestationFallback: Disabled by default. When enabled, failures in remote attestation reports, transparency logs, or measurement verification may fall back instead of aborting. Ephemeral-key signature verification and key derivation remain mandatory, but the returned result has a zero VerifiedAt.

SessionManager session settings:

Option	Default	Description
`SessionsPerModel`	`8192`	Maximum cached sessions per model, with LRU eviction
`SessionTTL`	`480s`	Idle session TTL

Environment variables:

The following environment variables apply to applications built on the SDK.

CMAAS_GATEWAY_EXTRA_HEADERS: Sets headers added to Model Studio gateway requests as comma-separated name=value pairs. Its priority is lower than GatewayExtraHeaders.
CMAAS_DEBUG: Set to true to enable SDK diagnostic logs. The two proxies also log complete plaintext HTTP request and response bodies.

Session Affinity

SessionManager automatically derives session affinity identifiers from commonly used agents' session identifiers and request characteristics such as content prefixes, helping maintain affinity between related requests where possible.

To specify session affinity explicitly, set X-DashScope-CMaaS-Proxy-Affinity: pass it through the headers argument to Chat when using the SDK directly, or as an HTTP request header when using a proxy.

Compatible HTTP Proxies

For integrations that already speak OpenAI or Anthropic protocols, the repository ships two local HTTP proxies built on the SDK. Starting with v0.3.2, both proxies are built on the SessionManager API in place of the previous Client API.

Note: Both proxies handle plaintext API requests and must run within the user's trusted environment.

Both proxies listen on 127.0.0.1:8080 by default. To run both proxies on the same host, set a different --listen port for one of them. To accept connections from other hosts, explicitly set --listen to a non-loopback address (for example, 0.0.0.0:8080) and enforce appropriate network access controls; do not expose an unprotected proxy directly to an untrusted network.

OpenAI-Compatible Proxy (`cmd/openai-proxy/`)

Implements the relevant subset of the OpenAI HTTP API, with support for streaming, function/tool calling, and reasoning model fields (reasoning_content, thinking_budget, enable_thinking).

# From the repository root
make openai-proxy

export DASHSCOPE_API_KEY=your_api_key
bin/openai-proxy \
  --endpoint https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services \
  --listen   127.0.0.1:8080

Endpoints:

POST /v1/chat/completions — OpenAI Chat Completions (streaming and non-streaming)

Flags:

Flag	Required	Default	Description
`--endpoint`	Yes	(none)	Model Studio gateway endpoint
`--model`	No	(none)	Comma-separated model list
`--served-model-name`	No	(none)	Model alias; requires a single-model configuration
`--listen`	No	`127.0.0.1:8080`	HTTP listen address; explicitly use a non-loopback address for remote access
`--sessions-per-model`	No	`8192`	Maximum cached sessions per model
`--session-ttl`	No	`480s`	Idle session TTL
`--workers`	No	`0`	No longer has any effect
`--max-concurrency`	No	`0`	Maximum concurrent requests (`0` disables the limit)
`--cache-ttl`	No	`12h`	Public key cache TTL (`0` disables the cache)
`--transparency-report-dir`	No	OS-specific	Directory for local transparency reports
`--no-prompt-in-transparency-report`	No	`false`	Exclude prompts from local transparency reports

Parameter and endpoint removal notice: Starting with v0.3.2, the --model, --served-model-name, and --workers flags are deprecated. v0.4.0 will remove these three flags and the /v1/models and /models endpoints.

Starting with v0.3.2, each request selects its inference model through model, independently of the --model list.

DASHSCOPE_API_KEY must be set in the environment.

Example using curl against the chat endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<code>",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user",   "content": "What is confidential computing?"}
    ],
    "temperature": 0.7,
    "max_tokens": 128,
    "stream": false
  }'

Example with the OpenAI Python SDK (chat completions):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="ignored")
resp = client.chat.completions.create(
    model="<code>",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user",   "content": "What is confidential computing?"},
    ],
    temperature=0.7,
    max_tokens=128,
)
print(resp.choices[0].message.content)

Anthropic-Compatible Proxy (`cmd/anthropic-proxy/`)

Exposes the Anthropic Messages API so the official Anthropic SDKs and the Claude Code CLI can be pointed at a CMaaS backend. The proxy translates the Anthropic protocol into the SDK's OpenAI-shaped request, and translates the response back into Anthropic-style streaming events (message_start / content_block_* / message_delta / message_stop).

# From the repository root
make anthropic-proxy

export DASHSCOPE_API_KEY=your_api_key
bin/anthropic-proxy \
  --endpoint https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services \
  --listen   127.0.0.1:8080

Endpoints:

POST /v1/messages — Anthropic Messages API (streaming and non-streaming)

Flags are generally the same as the OpenAI proxy. DASHSCOPE_API_KEY must also be set in the environment.

Example using curl against the messages endpoint:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<code>",
    "max_tokens": 128,
    "messages": [
      {"role": "user", "content": "What is confidential computing?"}
    ],
    "stream": false
  }'

Example with the Anthropic Python SDK:

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8080", api_key="ignored")
resp = client.messages.create(
    model="<code>",
    max_tokens=128,
    messages=[{"role": "user", "content": "What is confidential computing?"}],
)
print(resp.content[0].text)

Both proxies log the locally generated local_request_id and, when available, the gateway-returned request_id to support troubleshooting.

Environment variables:

CMAAS_ALLOW_ATTESTATION_FALLBACK: Set to true to enable attestation fallback. It is disabled by default, and either proxy emits a security warning when it is enabled.

Attestation and Auditing

Attestation Result

Each inference will bind an AttestationResult that provides transparency into the TEE server's identity and integrity:

VerifiedAt: Timestamp when the attestation was performed
Bundle: Contains the full attestation bundle including TEE type, server public key, TEE evidence, and transparency data

Call result.TEEDetailsJSON() to extract TEE-specific key fields as JSON. For Intel TDX environments (currently supported), this includes:

MRTD: Initial TD measurement (VM configuration and initial code)
RTMR[0-3]: Runtime measurement registers for firmware, kernel, and application layers
TD Attributes: Trust domain configuration attributes
Debug Enabled: Whether the TEE is running in debug mode

TDX Collateral Service and Cache

TDX Quote verification requires collateral from Intel PCS or a PCCS caching service. The SDK uses Alibaba Cloud PCCS by default and also supports specifying another service through an environment variable or the QCNL configuration file.

Related settings

PCCS_URL: Specifies the PCS/PCCS service URL. It takes precedence over PCCS_URL in the QCNL configuration file /etc/sgx_default_qcnl.conf and the default Alibaba Cloud PCCS. For example, in a cn-beijing VPC environment, set PCCS_URL=https://sgx-dcap-server-vpc.cn-beijing.aliyuncs.com, or set PCCS_URL=https://sgx-dcap-server-vpc.cn-beijing.aliyuncs.com/sgx/certification/v4/ in the QCNL configuration file.

PCCS_EXTRA_HEADERS: Sets headers added to PCS/PCCS requests as comma-separated name=value pairs.

Transparency Log Verification

Transparency log verification strengthens the trust chain by ensuring the inference node runs a known, audited software version—not just any code that happens to produce valid TEE measurements.

During key negotiation, the SDK automatically performs the following transparency verification steps:

Verify Log Entry Integrity: Validates each proof entry's DSSE envelope signature, RFC 6962 Merkle inclusion proof, and signed checkpoint against the transparency log's public keys
Verify Payload Hash: Confirms that the release manifest's SHA-256 hash (JCS-canonicalized) matches the payload hash recorded in the log entry
Validate TEE Measurements: Validates the TEE measurements from the TEE evidence (for TDX: measurements from Quote, UKI hash and container digests from EventLog) against the expected values in the verified release manifest

If any step fails, key negotiation is aborted; only explicitly enabling AllowAttestationFallback permits fallback and continuation.

Transparency Reports

After each completed inference call, the SDK appends a record to a daily, process-specific reports-YYYY-MM-DD-PID.jsonl file. Identical attestation materials are stored separately under attestation-materials/ and referenced by content hash. Records include:

The request prompt
Request status and error information
Whether attestation fallback was used and the corresponding verification failure reason
Session binding fields and verification timestamps
A reference to the TEE evidence and transparency data when negotiation completed

These reports provide a locally persisted and verifiable trail for offline auditing and security review. For offline verification, see the CMaaS Audit Tool.

Name		Name	Last commit message	Last commit date
assets		assets
cmd		cmd
docs		docs
examples		examples
internal/json		internal/json
pkg		pkg
sdk		sdk
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
README_zh.md		README_zh.md
go.mod		go.mod
go.sum		go.sum

Navigation Menu

Repository navigation

About

Repository files navigation

Confidential Model as a Service on Alibaba Cloud Model Studio

Project Overview

How to Use

Overview

Quick Start (CLI)

Build and Run

CLI Options

Quick Start (SDK)

Client

SessionManager

Streaming Inference

SDK Options

Session Affinity

Compatible HTTP Proxies

OpenAI-Compatible Proxy (`cmd/openai-proxy/`)

Anthropic-Compatible Proxy (`cmd/anthropic-proxy/`)

Attestation and Auditing

Attestation Result

Transparency Log Verification

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Confidential Model as a Service on Alibaba Cloud Model Studio

Project Overview

How to Use

Overview

Quick Start (CLI)

Build and Run

CLI Options

Quick Start (SDK)

Client

SessionManager

Streaming Inference

SDK Options

Session Affinity

Compatible HTTP Proxies

OpenAI-Compatible Proxy (cmd/openai-proxy/)

Anthropic-Compatible Proxy (cmd/anthropic-proxy/)

Attestation and Auditing

Attestation Result

Transparency Log Verification

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

OpenAI-Compatible Proxy (`cmd/openai-proxy/`)

Anthropic-Compatible Proxy (`cmd/anthropic-proxy/`)