elab-sync-hub

elab-sync-hub is an open-source synchronization hub for distributed eLabFTW instances.
It is designed for collaborative research environments in which samples, experiments, and measurements are carried out across multiple universities, laboratories, and instrument platforms.

In our use case, different partner sites provide different instruments and measurement capabilities. A complete characterization workflow for one sample may therefore require synthesis, treatment, and measurements at several institutions. This creates two practical needs:

1. Sample tracking: researchers must quickly see what has already been done for a given sample.
2. ELN synchronization: relevant eLabFTW records must remain connected and accessible across different local instances.

This repository addresses the second point directly and is intended to work in combination with a Sample-ID / QR-code based tracking workflow.

---

Motivation

In distributed collaborative projects, each participating site may operate its own local eLabFTW instance for daily laboratory work. While this is practical locally, it creates a challenge for cross-site collaboration:

• the same logical sample may appear in multiple ELN systems
• experiments performed at different sites should stay connected
• uploaded files and embedded figures should remain usable after transfer
• links between samples and experiments should not be lost
• manual copy-paste of records is error-prone and inefficient

elab-sync-hub provides a practical interoperability layer between local and shared eLabFTW instances.

---

Main features

The synchronization workflow is split into three stages:

1. Entity synchronization

Synchronizes the core ELN objects:

• experiments
• database items
• metadata-based global identity
• basic record content (title, body, metadata, etc.)

This stage establishes and maintains a mapping between the same logical research object on different eLabFTW instances.

2. Link synchronization

Synchronizes forward links between ELN objects:

• item → item
• item → experiment
• experiment → item
• experiment → experiment

This helps preserve the graph structure of connected research objects across instances.

3. Upload and embedded image synchronization

Synchronizes:

• uploads attached to records
• embedded image references inside the experiment/item body
• rewritten app/download.php?f=... references on the target instance

This is especially important for rich-text ELN records containing figures or screenshots.

---

Architecture

The current synchronization logic is organized as:

• entities: synchronize experiments/items and establish global identity
• links: synchronize forward relations between records
• uploads: synchronize uploads and embedded image references

For uploads and embedded images, the current logic follows a:

collect → decide winner → apply winner to loser

workflow.

This avoids naïve bidirectional overwriting and is more robust in environments with heterogeneous permissions.

---

Repository structure

elab-sync-hub/
├── README.md
├── requirements.txt
├── config.example.yaml
├── .env.example
├── data/
│   └── synchub.sqlite
├── reports/
├── scripts/
│   ├── sync_full.py
│   └── sync_incremental.py
└── sync/
    ├── __init__.py
    ├── client.py
    ├── db.py
    ├── pipeline.py
    └── stages/
        ├── __init__.py
        ├── entities.py
        ├── links.py
        └── uploads.py

---

Related sample tracking workflow

This synchronization project is closely connected to a Sample-ID / QR-code based sample tracking workflow.

A dedicated web-based sample tracking interface was developed to query eLabFTW records by Sample-ID and provide a structured overview of the experimental history of a sample:

https://flair.mw.tu-darmstadt.de/

This web interface allows users to:

• search for a sample by Sample-ID
• view linked experiment/database records
• quickly see which experimental steps have already been completed
• directly open the corresponding eLabFTW entries
• use QR codes attached to physical samples for fast lookup

The synchronization hub and the sample tracking tool solve complementary parts of the same problem:

• sample tracking answers:
  “What has already been done with this sample, and where?”

• elab-sync-hub ensures that the underlying ELN records from different sites can be synchronized and kept connected.

---

Installation

1. Clone the repository

git clone git@github.com:wanghao-github/elab-sync-hub.git
cd elab-sync-hub

2. Create a Python environment

python -m venv .venv
source .venv/bin/activate

or with your preferred environment manager.

3. Install dependencies

pip install -r requirements.txt

---

Configuration

Environment variables

API keys are read from environment variables.

Example:

export ELAB_API_KEY_DARMSTADT="..."
export ELAB_API_KEY_DEMO="..."

Config file

Create a config file, for example config.yaml, based on config.example.yaml.

Example structure:

database:
  path: data/synchub.sqlite

nodes:
  darmstadt:
    url: "https://eln01-t.ca.hrz.tu-darmstadt.de/api/v2"
    api_key_env: "ELAB_API_KEY_DARMSTADT"
    verify_ssl: false
    can_write: true

  demo:
    url: "https://wanghao93.xyz/api/v2"
    api_key_env: "ELAB_API_KEY_DEMO"
    verify_ssl: false
    can_write: true

item_category_id_map:
  darmstadt:
    Project: 154
    Instrument: 156
    Chemicals: 60

  demo:
    Project: 2
    Instrument: 3
    Chemicals: 1

Notes

• can_write controls whether the hub is allowed to modify a node.
• item_category_id_map is required for creating items on target instances.
• verify_ssl: false may be useful for testing or self-hosted instances, but should be used carefully.

---

Usage

Full synchronization

Run the full pipeline:

python scripts/sync_full.py \
  --config config.yaml \
  --db data/synchub.sqlite \
  --node-a darmstadt \
  --node-b demo

Incremental synchronization

Run the incremental pipeline:

python scripts/sync_incremental.py \
  --config config.yaml \
  --db data/synchub.sqlite \
  --node-a darmstadt \
  --node-b demo

Restrict to a specific owner

python scripts/sync_full.py \
  --config config.yaml \
  --db data/synchub.sqlite \
  --node-a darmstadt \
  --node-b demo \
  --owner-fullname "Hao Wang"

Restrict to items only

python scripts/sync_full.py \
  --config config.yaml \
  --db data/synchub.sqlite \
  --node-a darmstadt \
  --node-b demo \
  --include-items

Restrict to experiments only

python scripts/sync_full.py \
  --config config.yaml \
  --db data/synchub.sqlite \
  --node-a darmstadt \
  --node-b demo \
  --include-experiments

Dry run

python scripts/sync_full.py \
  --config config.yaml \
  --db data/synchub.sqlite \
  --node-a darmstadt \
  --node-b demo \
  --dry-run

---

Reports

All stages generate JSON reports in the reports/ directory.

Typical reports include:

• entities_sync_report.json
• links_sync_report.json
• uploads_sync_report.json
• pipeline_full_report.json
• pipeline_incremental_report.json

These reports are useful for:

• debugging synchronization issues
• identifying permission problems
• checking whether uploads were reused or transferred
• understanding why a particular node was selected as the winner

---

Current status

The current implementation is already able to run the main synchronization workflow:

• entity synchronization works
• link synchronization works
• upload / embedded image synchronization works on the main path

The project is still under active development, and some edge cases are being improved, especially for:

• heterogeneous permissions across nodes
• partially incomplete source records
• incremental optimization and cursor-based narrowing
• further cleanup and packaging

---

Intended deployment model

elab-sync-hub is intended to be configurable and deployable in different institutional environments.

A typical setup is:

• each university keeps its own local eLabFTW workflow
• a shared hub instance maintains cross-site mapping and synchronization state
• synchronization runs periodically
• sample tracking provides the user-facing overview layer

This makes the software suitable for collaborative projects in which data should remain:

• transparent
• accessible
• structured
• interoperable
• manageable across several sites

---

Why this matters

A full experimental workflow for one sample often requires instruments that are only available at different universities. In such a setting, both of the following are essential:

1. the ability to track the current status of a sample
2. the ability to synchronize the corresponding ELN records across sites

Without both, collaboration becomes inefficient, records fragment into local silos, and the practical value of structured ELN documentation is reduced.

elab-sync-hub was developed precisely to address this problem.

---

License

---

Acknowledgement

This software was developed in the context of cross-site experimental collaboration and FAIR-oriented research data management workflows using eLabFTW.

It is intended as a practical open-source tool for connecting distributed ELN environments and improving cross-institutional sample- and experiment-level interoperability.