L1ES: Elasticsearch Plugin for Log10x

Search and query compact Log10x events directly within Elasticsearch and OpenSearch. This open-source Log10x plugin expands compact events at query time and rewrites standard match, match_phrase and multi_match queries, so searches of those types, from Kibana or any client, match the original log text while the index stores the smaller compact form.

L1ES is built for Elasticsearch 8.17.0 and OpenSearch 2.19.0. Elasticsearch 9.3 and later store templates natively with the pattern_text field type, and no 9.x build of L1ES exists.

Blog: Cutting Elasticsearch log storage roughly in half without changing queries. How Elasticsearch stores fewer bytes and still returns the original log lines.

Version	1.0.1
Elasticsearch	8.17.0
OpenSearch	2.19.0
Java	17+
License	MIT

How It Works

Log10x compacts log events into two parts:

1. Templates: the recurring structure, with $ placeholders for variable values
2. Encoded events: a compact representation in the form ~<template-hash>,<value1>,<value2>,...

L1ES stores the templates in an internal index and, at query time:

1. Matches your search terms against the template patterns
2. Finds compact events that use matching templates
3. Decodes each result back to the original log line
4. Returns the expanded content in a field you specify

Standard Elasticsearch queries see only the compact text and cannot match original content. L1ES bridges this gap in two ways:

1. Custom query types (l1es_match, l1es_match_phrase, l1es_multi_match): explicit queries for compact fields
2. Transparent query rewriting: automatically converts standard match, match_phrase, and multi_match queries to their L1ES equivalents, so Kibana dashboards, saved searches, and KQL queries built on those query types work without changes

Quick Start

1. Download or Build the Plugin

The 1.0.1 release carries the two zips: l1es-plugin-1.0.1.es.8.17.0.zip for Elasticsearch and l1es-plugin-1.0.1.os.2.19.0.zip for OpenSearch. To build from source:

./gradlew build

This produces the plugin zip at build/distributions/l1es-plugin-1.0.1.es.8.17.0.zip. See Building from Source for prerequisites and alternative build methods.

2. Install the Plugin

bin/elasticsearch-plugin install file:///path/to/l1es-plugin-1.0.1.es.8.17.0.zip

Or with Docker:

FROM docker.elastic.co/elasticsearch/elasticsearch:8.17.0
COPY l1es-plugin-1.0.1.es.8.17.0.zip /tmp/l1es-plugin.zip
RUN elasticsearch-plugin install --batch file:///tmp/l1es-plugin.zip

3. Initialize the Plugin

After Elasticsearch starts:

curl -X POST 'http://localhost:9200/_l1es/setup'

This creates the internal indices (l1es_dml for template storage, l1es_dml_indices for field mappings). DML (Data Matching Library) is the internal template lookup engine that maps compact events back to their original structure.

4. Load Templates

Bulk-load your template file (one JSON object per line) into the l1es_dml index:

# Each line of templates.json: {"templateHash":"<hash>","template":"<pattern>"}
# Convert to bulk format and load:
while IFS= read -r line; do
  hash=$(echo "$line" | python3 -c "import sys,json; print(json.load(sys.stdin)['templateHash'])")
  pattern=$(echo "$line" | python3 -c "import sys,json; print(json.dumps(json.load(sys.stdin)['template']))")
  printf '{"index":{"_index":"l1es_dml","_id":"%s"}}\n{"pattern":%s}\n' "$hash" "$pattern"
done < templates.json > /tmp/bulk_templates.ndjson

curl -X POST 'http://localhost:9200/_bulk' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @/tmp/bulk_templates.ndjson

5. Register the Encoded Field

Tell L1ES which index and field contain compact data:

curl -X POST 'http://localhost:9200/_l1es/add-dml-index' \
  -H 'Content-Type: application/json' \
  -d '{
    "index_name": "my-logs",
    "source": "message",
    "dest": "decoded_message"
  }'

• index_name: your data index
• source: the field containing compact events
• dest: the field name for expanded output (defaults to source if omitted)

6. Load Encoded Data

Index your compact log events into the data index as usual:

curl -X POST 'http://localhost:9200/my-logs/_bulk' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @encoded_events.ndjson

7. Search

Use L1ES query types to search the expanded content:

# Phrase search
curl -X POST 'http://localhost:9200/my-logs/_search' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "l1es_match_phrase": {
        "message": {
          "query": "Accounting service started"
        }
      }
    },
    "fields": ["message"]
  }'

The fields parameter is required to trigger the fetch sub-phase that decodes results. The expanded content appears in the decoded_message field (or whatever you specified as dest).

Transparent Kibana Support

L1ES can rewrite standard match, match_phrase and multi_match queries so that Kibana dashboards, saved searches, and KQL queries built on those query types work against compact data without changes.

When query_rewrite_enabled is true (the default), L1ES intercepts incoming search requests and converts standard match, match_phrase, and multi_match queries to their L1ES equivalents. If the target field is not compact, the query falls back to standard Elasticsearch behavior automatically.

When source_decoding_enabled is true (the default), L1ES decodes compact fields in _source for all search responses on registered indices. Kibana document views, Discover, and dashboards display the original log text instead of the encoded ~hash,val1,val2,... format.

Together, these two features mean:

• Kibana Discover: searches match original log content; documents display expanded text
• Kibana dashboards: visualizations, filters, and saved searches built on those query types work unchanged
• KQL queries: message: "error" AND message: "database" matches against expanded content
• Alerts: Kibana alerting rules built on those query types fire on the original log text

For those query types, no changes are needed to index mappings, index patterns, Kibana saved objects, or client applications.

Disabling Transparent Rewriting

To use only the explicit l1es_* query types (e.g., for programmatic access where you control the query DSL), set in config/l1es.yml:

flags:
  query_rewrite_enabled: false
  source_decoding_enabled: false

OpenSearch

On OpenSearch, transparent query rewriting runs as the l1es_query_rewrite search request processor in place of an action filter. Create a search pipeline with that processor and set it as the default search pipeline of each registered index:

curl -X PUT 'http://localhost:9200/_search/pipeline/l1es_query_rewrite' \
  -H 'Content-Type: application/json' \
  -d '{"request_processors": [{"l1es_query_rewrite": {}}]}'

curl -X PUT 'http://localhost:9200/my-logs/_settings' \
  -H 'Content-Type: application/json' \
  -d '{"index.search.default_pipeline": "l1es_query_rewrite"}'

The behavior is otherwise identical.

Query Types

L1ES provides three query types that mirror standard Elasticsearch queries:

l1es_match

Token-based search. Supports AND/OR operators.

{
  "l1es_match": {
    "message": {
      "query": "error database",
      "operator": "AND"
    }
  }
}

Parameters: query (required), operator, analyzer, fuzziness, prefix_length, max_expansions, minimum_should_match, fuzzy_rewrite, fuzzy_transpositions, lenient, zero_terms_query, auto_generate_synonyms_phrase_query.

l1es_match_phrase

Exact phrase search. Tokens must appear in order.

{
  "l1es_match_phrase": {
    "message": {
      "query": "service started successfully",
      "slop": 0
    }
  }
}

Parameters: query (required), analyzer, slop, zero_terms_query.

l1es_multi_match

Search across multiple fields.

{
  "l1es_multi_match": {
    "query": "connection timeout",
    "fields": ["message", "error_log"],
    "operator": "AND"
  }
}

Parameters: query (required), fields, type (best_fields, most_fields, phrase, phrase_prefix), operator, analyzer, slop, fuzziness, prefix_length, max_expansions, minimum_should_match, fuzzy_rewrite, tie_breaker, lenient, zero_terms_query, auto_generate_synonyms_phrase_query, fuzzy_transpositions.

REST API

Endpoint	Method	Description
_l1es	GET	Plugin info (version, description)
_l1es/setup	POST	Create internal indices
_l1es/cleanup	POST	Remove internal indices
_l1es/add-dml-index	POST	Register compact field mapping
_l1es/remove-dml-index	POST	Unregister compact field mapping

Configuration

The plugin configuration is in config/l1es.yml inside the plugin directory:

flags:
  enabled: true                      # Master switch
  decoding_enabled: true             # Enable fetch sub-phase decoding
  match_query_enabled: true          # Enable l1es_match
  match_pharse_query_enabled: true   # Enable l1es_match_phrase
  multi_match_query_enabled: true    # Enable l1es_multi_match
  query_rewrite_enabled: true        # Transparent rewriting of standard queries for Kibana
  source_decoding_enabled: true      # Decode compact fields in _source responses

encoder:
  hasEncodedLinePrefix: true         # Encoded lines start with a prefix character
  encodedLinePrefix: "~"             # The prefix character
  valueSeperator: ","                # Separator between values in compact lines

dmldb:
  indicesIndexNumberOfShards: 1      # Shards for l1es_dml_indices
  indicesIndexNumberOfReplicas: 1    # Replicas for l1es_dml_indices
  dmlIndexNumberOfShards: 1          # Shards for l1es_dml
  dmlIndexNumberOfReplicas: 1        # Replicas for l1es_dml
  dmlSizeToSearch: 10000             # Max templates to scan per query

The encoder section must match the format produced by your Log10x encoder. The defaults above match the standard Log10x output format (~<hash>,<val1>,<val2>,...).

Building from Source

Prerequisites

• Java 17 (JDK)
• Gradle 8.5+

All dependencies (including log10x-decoder-core) are resolved automatically from Maven Central.

Build

./gradlew build

The plugin zip is produced at build/distributions/l1es-plugin-1.0.1.es.8.17.0.zip.

Build with Docker (if your local Java version is incompatible)

docker run --rm \
  -v "$(pwd)":/project \
  -v "$HOME/.m2":/root/.m2 \
  -w /project \
  eclipse-temurin:17-jdk \
  ./gradlew build

Docker Image

docker build -t l1es:8.17.0 .
docker run -d --name l1es -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  l1es:8.17.0

Project Structure

├── build.gradle                     # Build configuration
├── gradle.properties                # Version numbers
├── Dockerfile                       # Docker image for ES + plugin
├── src/main/
│   ├── config/l1es.yml              # Plugin configuration
│   ├── resources/
│   │   └── plugin-descriptor.properties
│   └── java/
│       ├── com/log10x/l1es/
│       │   ├── analysis/            # Tokenization utilities
│       │   ├── dml/                 # Template storage and lookup
│       │   ├── fetch/               # Fetch sub-phase (expand results + _source expansion)
│       │   ├── filter/              # ActionFilter for transparent query rewriting
│       │   ├── job/                 # Internal ES operations
│       │   ├── query/               # Custom query types
│       │   │   ├── match/           # Query builders and query implementations
│       │   │   ├── predicate/       # Cross-check predicates (AND, phrase)
│       │   │   └── wrap/            # TwoPhaseIterator wrappers
│       │   └── util/                # ES and JSON utilities
│       └── org/elasticsearch/plugin/log10x/
│           ├── L1esPlugin.java      # Plugin entry point
│           ├── config/              # Configuration classes
│           └── handler/             # REST endpoint handlers

OpenSearch Support

L1ES also supports OpenSearch 2.19.0. The OpenSearch variant is built as a separate Gradle subproject and is functionally identical. See opensearch/README.md for OpenSearch-specific instructions.

Build the OpenSearch variant:

./gradlew :opensearch:build -x test

The plugin zip is produced at opensearch/build/distributions/l1es-plugin-1.0.1.os.2.19.0.zip.

Compatibility

L1ES Version	Elasticsearch	OpenSearch	Java	Lucene
1.0.1	8.17.0	2.19.0	17+	9.12.0

Documentation

• Log10x Documentation
• Receiver Documentation
• Compact & Expand

License

This repository is licensed under the MIT License.

The Receiver that writes compact events

This repository holds the Elasticsearch/OpenSearch plugin that expands Log10x compact events at search time. The Log10x Receiver, which writes them, is not in this repository; it runs on the built-in Free tier on up to 10 nodes, with no licence token, and a commercial licence lifts the node count.

Component	License
This repository (Elasticsearch/OpenSearch plugin)	MIT (open source)
Log10x Receiver	Free tier on up to 10 nodes; commercial licence above that

Get Started:

• Log10x Pricing
• Documentation
• Contact Sales

Contributing

Contributions are welcome. Open a pull request against this repository.

Support

For issues and feature requests:

• Open an issue on GitHub
• Contact the Log10x team at support@log10x.com