HUATUO 华佗

Kernel-wide Insight, Instant Observability, AutoTracing, Continuous Profiling

中文文档 · Quick Start · Documentation ·

---

What is HUATUO

HUATUO, named after Hua Tuo, a legendary physician in ancient China, is a cloud-native OS observability project open-sourced by DiDi and incubated under the CCF. It provides kernel-level observability for general-purpose cloud-native computing, AI computing, and bare-metal infrastructure services. By integrating Linux kernel dynamic tracing technologies such as kprobes, tracepoints, ftrace, and eBPF, HUATUO provides kernel-wide visibility through finer-grained metrics, automatic runtime context capture in the kernel, and intelligent tracing. Deployed at scale in DiDi's production environment, HUATUO helps troubleshoot system failures and improves the high availability and performance of cloud-native operating systems.

HUATUO is now listed in the CNCF Landscape.

Key Features

• Kernel-Wide Insight: Leverages eBPF to keep performance overhead below 1%, delivering full-stack, low-level observability insights into Linux kernel subsystems such as memory management, CPU scheduling, networking, and block I/O.
• Instant Observability: An event-driven runtime context capture mechanism that instruments kernel slow paths. It automatically triggers on events such as page faults, scheduling delays, generating detailed data for immediate analysis.
• AutoTracing: Employs automated snapshot retention to resolve performance jitter typical in cloud‑native and AI infrastructure environments, addressing issues such as CPU idle drops, CPU system-time spikes, I/O surges, and load average spikes.
• Continuous Profiling: A comprehensive and continuous performance profiling of the operating system and applications, covering CPU, Memory, I/O, and Locks. This drives business innovation and plays a key role in Chaos, HA and Stability Engineering.
• Heterogeneous Computing Hardware: Encompassing core system components such as CPUs, memory, PCIe interconnects, network adapters, and storage, alongside specialized AI accelerators including GPUs and NPUs.
• Ecosystem Integration: Integrates with mainstream open-source observability stacks such as Prometheus, Grafana, Pyroscope, and Elasticsearch, and automatically associates Kubernetes container labels/annotations. This is achieved through zero-instrumentation, kernel-level eBPF programming, ensuring broad compatibility across hardware platforms and Linux distributions.

Big Picture

Ecosystem

Getting Started

• Quick Run

  To launch the HUATUO service with Docker:

    $ docker run --privileged --pid=host --cgroupns=host --network=host -v /sys:/sys -v /run:/run huatuo/huatuo-bamai:latest
  

  To pull metrics from another terminal:

    $ curl -s localhost:19704/metrics
  

• Quick Setup

  To launch the full stack (Elasticsearch, Prometheus, Grafana, and huatuo) using Docker Compose:

    $ docker compose --project-directory ./build/docker up
  

  Once running, access the monitoring dashboard at http://localhost:3000.

  

  

• NOTE

  Do not deploy images with the latest tag to production environments, as this is a development and testing image. Use a formal release image or binary.

Go Client

The public Go client is in client/. Node client files use the node_ prefix.

package main

import (
	"context"
	"encoding/json"

	nodeapi "github.com/ccfos/huatuo/apis/v1/node"
	"github.com/ccfos/huatuo/client"
)

func updateNodeConfig(ctx context.Context, address client.NodeAddress) error {
	nodeClient, err := client.NewNode(&client.NodeConfig{
		BearerToken: "node-token",
	})
	if err != nil {
		return err
	}
	return nodeClient.UpdateConfig(ctx, address, &nodeapi.UpdateConfigRequest{
		Config: map[string]json.RawMessage{
			"Runtime.CPULimitCores": json.RawMessage("1.5"),
		},
	})
}

Kernel Versions

The project supports kernel version 4.18 and later. The following kernel and OS distribution are primarily tested.

HUATUO	Kernel	OS
1.0.0	4.18.x	CentOS 8.x
2.4.0	4.18.x	Rocky Linux 8.10
1.0.0	5.4.x	OpenCloudOS V8
1.0.0	5.4.x	Ubuntu 20.04.6 LTS
1.0.0	5.10.x	Anolis OS 8.10
1.0.0	5.10.x	openEuler 22.03 LTS-SP4
2.4.0	5.14.x	Rocky Linux 9.8
1.0.0	5.15.x	Ubuntu 22.04.5 LTS
2.4.0	6.1.x	Debian 12 (bookworm)
1.0.0	6.6.x	Anolis OS 23.3
1.0.0	6.6.x	OpenCloudOS V9
1.0.0	6.6.x	openEuler 24.03 LTS-SP4
1.0.0	6.8.x	Ubuntu 24.04.4 LTS
2.4.0	6.12.x	Debian 13 (trixie)
1.0.0	6.14.x	Fedora 42
2.4.0	6.19.x	Fedora Linux 44
2.3.0	7.0.x	Ubuntu 26.04 LTS

Documentation

For more information, visit https://docs.huatuo.tech

Co-Building

• ❇️ We welcome all users, developers, companies, and organizations to use Huatuo, report bugs, submit feature requests, share best practices, and help us build a professional and vibrant open-source community.
• ❤️ Contributors

Contact Us

• WeChat Group and Official Account:

License

This project is open source under the Apache License 2.0. The BPF code is licensed under the GPL license.