lpipe — Service-Oriented Shared-Memory IPC

Zero-dependency, high-performance local IPC library for C++17 over POSIX shared memory.

• Service-Oriented Architecture — services and attributes are defined via JSON; one Provider (writer) to many Consumers (readers) per service
• JSON-driven code generation — define attributes in a JSON config, get type-safe typed Provider/Consumer C++ headers automatically
• Lock-free SPSC per attribute — zero mutex contention, minimal memory footprint, deterministic latencies suitable for real-time loops
• Automatic shared memory lifecycle — initialization (create/open/mmap) through de-initialization (unmap/unlink) managed by the library via RAII
• Per-attribute crash recovery — Provider restart detected via session_id; consumers auto-reset counter tracking without data loss
• Zero external dependencies — C++17 + POSIX only; no Boost, no protobuf, no Google Test

Table of Contents

• How It Works
• JSON Configuration Reference
• Workflow
• Build
• API Reference
  • Public API (generated)
  • Error Codes
• Losing Ring Buffer Behavior
• Project Structure
• License

How It Works

lpipe organizes IPC around services. Each service is defined by a JSON config file and corresponds to exactly one POSIX shared memory segment.

stateDiagram
    direction LR

    cfg: Service-A.json

    p: [Process] Service A/Provider
    c0: [Process] Service A/Consumer 0
    c1: [Process] Service A/Consumer 1
    shm: Shared Memory (/lpipe_<name>)

    state shm {
        direction TB
        a0: Attribute 0 Buffer
        a1: Attribute 1 Buffer
        a2: ...
    }

    p --> a0
    p --> a1
    p --> a2
    a0 --> c0
    a1 --> c1

Loading

Key properties:

• One Provider per service — Only one process can write to a service's shared memory segment at a time.
• Multiple Consumers — Multiple processes can read from a service simultaneously. Each attribute supports exactly one reader (SPSC). Two processes must not read the same attribute concurrently. Each Consumer can read from different attributes.
• Multi-service Consumers — A single process can consume from multiple services at once, using multiple Consumer instances.
• Provider never blocks — The ring buffer operates in losing/drop-oldest mode; writes always succeed.

JSON Configuration Reference

{
  "version": "1",          // Required, must be "1"
  "name": "service_name",  // Required, valid C identifier, 1-63 chars
  "attributes": [           // Required, non-empty array (max 64)
    {
      "name": "attr_name",  // Required, valid C identifier, 1-63 chars, UNIQUE
      "max_records": 64,    // Required, integer 1-65536
      "fields": [           // Required, non-empty array
        {
          "name": "field",  // Required, valid C identifier, unique within attribute
          "type": "uint8",  // Required: uint8 | uint32 | int32 (signed) | float32 | bytes
          "size": 256       // Required for 'bytes' type, integer 1-65536
        }
      ]
    }
  ]
}

• Attribute Data Types

JSON type	C++ Type	sizeof
uint8	uint8_t	1
uint32	uint32_t	4
int32	int32_t	4
float32	float	4
bytes	uint8_t[N]	N (1–65536)

The sizeof column shows the user struct size. Total record size in shared memory includes a 16-byte RecordHeader and alignment padding. See architecture.md §4 for layout computation.

Workflow

1. Define attributes in a JSON config file
2. Generate headers via CMake custom targets using scripts/lpipe_gen.py
3. Write application code using the generated Provider/Consumer wrappers

For full integration instructions — CMake setup and API usage patterns — see INTEGRATION.md

Builtin examples: examples/README.md

Build

mkdir -p build && cd build && cmake .. && make -j$(nproc)
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake .. -DLPIPE_BUILD_BENCHMARKS=ON   # needs Google Benchmark
cmake .. -DLPIPE_BUILD_TESTS=OFF       # skip test suite
cmake .. -DLPIPE_BUILD_EXAMPLES=OFF    # skip examples

Build Options

Option	Default	Description
CMAKE_BUILD_TYPE	(empty)	Set to Release for -O3 -march=native
LPIPE_BUILD_EXAMPLES	ON	Build example applications (bcm, esp, ivi)
LPIPE_BUILD_TESTS	ON	Build and enable test suite
LPIPE_BUILD_BENCHMARKS	OFF	Build benchmarks (needs Google Benchmark)

• Codegen (scripts/lpipe_gen.py) runs automatically via CMake custom targets created by the lpipe_generate_headers() function (defined in top-level CMakeLists.txt)
• Generated headers land at build/generated/{hvac,ego}/ (gitignored)
• librt linked conditionally on older glibc (check_library_exists(rt shm_open))
• fcntl advisory locks on /dev/shm (tmpfs) may be silently ignored on some kernels → compile with -DLPIPE_NO_FCNTL_LOCK if duplicate Providers can start

API Reference

Public API (generated)

// Generated per config: lpipe_gen_<name>_typed.hpp
namespace lpipe_gen_<name> {

class Provider {
public:
    // Default constructor — uses kServiceConfig from the generated layout header
    Provider();

    // Non-owning reference (for tests/advanced use)
    explicit Provider(lpipe::internal::Provider& provider);

    lpipe::Error write_ATTRNAME(const ATTRNAME& data);  // one per attribute
    lpipe::Error write_ATTRNAME(ATTRNAME&& data);
    lpipe::Error read_ATTRNAME(ATTRNAME& data, ...);
    std::unique_ptr<ATTRNAME> read_ATTRNAME_ptr(
        uint64_t* out_ts = nullptr, uint8_t* out_counter = nullptr);
    std::unique_ptr<ATTRNAME> read_ATTRNAME_ptr(
        lpipe::Error& out_err,                           // discriminated error
        uint64_t* out_ts = nullptr, uint8_t* out_counter = nullptr);
    // ... repeated for each attribute
};

class Consumer {
public:
    // Default constructor — uses kServiceConfig from the generated layout header
    Consumer();

    // Non-owning reference (for tests/advanced use)
    explicit Consumer(lpipe::internal::Consumer& consumer);

    lpipe::Error read_ATTRNAME(ATTRNAME& data, ...);
    std::unique_ptr<ATTRNAME> read_ATTRNAME_ptr(
        uint64_t* out_ts = nullptr, uint8_t* out_counter = nullptr);
    std::unique_ptr<ATTRNAME> read_ATTRNAME_ptr(
        lpipe::Error& out_err,                           // discriminated error
        uint64_t* out_ts = nullptr, uint8_t* out_counter = nullptr);
    // ... repeated for each attribute
};

} // namespace lpipe_gen_<name>

Error Codes

Code	Value	Description
Ok	0	Success
BufferFull	-1	Reserved; never returned in losing mode
BufferEmpty	-2	No data available for reading
CounterMismatch	-4	Counter mismatch (auto-recovered by catch-up; only returned for genuine corruption)
ShmCreateFailed	-10	Failed to create shared memory
ShmOpenFailed	-11	Failed to open shared memory
ShmTruncFailed	-12	Failed to set shared memory size
ShmMapFailed	-13	Failed to mmap shared memory
ShmLockFailed	-14	fcntl lock failed (another Provider exists)
InvalidArg	-20	Invalid argument (e.g., unknown attribute name)
InvalidLayout	-21	Invalid shared memory layout
LayoutMismatch	-22	AttributeDesc mismatch between endpoints
AccessDenied	-30	Consumer attempted to write

Losing Ring Buffer Behavior

In losing/drop-oldest (overwrite-on-full) mode:

• Provider always writes — no BufferFull return. try_push() skips the free-space check entirely. If the buffer is logically full (write_index - read_index >= max_records), the oldest unread record slot is overwritten.
• Consumer detects when it fell behind (write_index − read_index > max_records) inside try_pop(). It catches up by jumping read_index to write_index - max_records and re-synchronizing expected_counter from the catch-up slot's record_counter. After catch-up, CounterMismatch is NOT returned — the record is consumed normally with Error::Ok. CounterMismatch is only returned for genuine corruption (not during catch-up).

The catch-up logic is fully contained within lpipe::internal::TypedCircularBuffer<T>::try_pop() — no extra flags in AttributeState.

Project Structure

lpipe/
├── CMakeLists.txt
├── README.md                    # This file
├── architecture.md              # CMake build system architecture
├── architecture-app.md          # Application-level architecture design
├── implementation-plan.md       # CMake refactoring plan
├── iteration-1.md               # Iteration details
├── iteration-2.md
├── iteration-3.md
│
├── examples/
│   ├── config/
│   │   ├── hvac.json            # HVAC service config
│   │   └── ego.json             # EGO service config
│   ├── bcm_example.cpp          # Provider of hvac config
│   ├── esp_example.cpp          # Provider of ego config
│   └── ivi_example.cpp          # Consumer of both hvac and ego
│
├── include/lpipe/
│   ├── common.hpp               # AttributeDef, ServiceConfig, lock-free assertions
│   ├── error.hpp                # Error enum
│   └── internal.hpp             # Single re-export header for all lpipe::internal symbols
│
├── src/
│   ├── internal_types.hpp       # ShmHeader, AttributeDesc, RecordHeader, AttributeMeta, LPIPE_VERSION
│   ├── circular_buffer.hpp      # lpipe::internal::TypedCircularBuffer<T>
│   ├── shm.hpp                  # lpipe::internal::ShmRegion RAII class
│   ├── endpoint.hpp             # lpipe::internal::Endpoint class (header-only templates)
│   ├── provider.hpp             # lpipe::internal::Provider
│   ├── consumer.hpp             # lpipe::internal::Consumer
│   ├── shm.cpp
│   └── endpoint.cpp             # Endpoint constructor, layout, name-to-index map
│
├── scripts/
│   ├── schema.py                # JSON validation
│   ├── lpipe_gen.py             # Code generator
│   └── test_validation.py       # Validation tests
│
├── templates/
│   ├── attribute_hpp.tmpl       # Per-attribute struct template
│   ├── layout_hpp.tmpl          # Layout/offsets template
│   ├── all_hpp.tmpl             # Umbrella include template
│   └── typed_hpp.tmpl           # Generated Provider/Consumer wrapper template
│
├── tests/
│   ├── test_shm.cpp             # ShmRegion tests (9 tests)
│   ├── test_circular_buffer.cpp # Ring buffer tests (18 tests)
│   ├── test_generated.cpp       # Generated code + wrapper tests (28 tests)
│   └── test_integration.cpp     # End-to-end integration tests (30 tests)
│
├── benchmarks/
│   ├── bench_latency.cpp
│   └── bench_throughput.cpp
│
└── tools/
    └── lpipe_cleanup.cpp        # Stale shm cleanup utility

License

MIT — see LICENSE