diff --git a/.gitignore b/.gitignore index e68c6be..023d013 100644 --- a/.gitignore +++ b/.gitignore @@ -220,6 +220,16 @@ __marimo__/ # benchmark datasets and results (large, generated) benchmarks/*/datasets/ benchmarks/*/results/ +!benchmarks/realtime-data-indexing/results/ +benchmarks/realtime-data-indexing/results/.rerun.lock +# Keep raw stdout/stderr logs local; commit only benchmark measurements/reports. +benchmarks/realtime-data-indexing/results/**/*.txt.gz +benchmarks/realtime-data-indexing/results/previous/ +benchmarks/realtime-data-indexing/results/diagnostics/ +# Local outputs regenerated from the committed measurements. +benchmarks/realtime-data-indexing/results/*-memory.png +benchmarks/realtime-data-indexing/results/table.md +benchmarks/realtime-data-indexing/results/consistency-check.json benchmarks/*/compat/*.so # Runtime working directory of `serviette demo` (regenerable: corpus copy, diff --git a/README.md b/README.md index 8118d87..5b41cae 100644 --- a/README.md +++ b/README.md @@ -76,11 +76,11 @@ organisations such as [La Poste](https://pathway.com/success-stories/la-poste), [Transdev](https://pathway.com/success-stories/transdev) and [CMA CGM](https://pathway.com/success-stories/cma-cgm). -serviette's own numbers are measured, not estimated: indexing -up to 50 GB of text (26 million pages, 17 million files) in constant memory, -retrieval accuracy on a public benchmark (FRAMES), eight vector databases -under integration tests against real instances — all reproducible from this -repository. +serviette's benchmarks cover indexing up to 50 GB of text (26 million pages, +17 million files), with indexer memory measured throughout the run, and +retrieval accuracy on a public benchmark (FRAMES). Eight vector databases +are under integration tests against real instances. Scripts, recorded results +and measurement limitations are available in this repository. **Where it stands today.** It answers one question at a time, from the current state of the documents. Conversation memory, user accounts and a @@ -207,62 +207,67 @@ every byte below is extracted text (a PDF collection with the same text content would weigh several times more) — see [benchmarks/realtime-data-indexing](https://github.com/pathwaycom/serviette/tree/main/benchmarks/realtime-data-indexing): -| corpus | ≈ pages | files | chunks | indexing time | peak memory (PSS) | in Qdrant | +| corpus target | ≈ pages | files | chunks (approx.) | indexing time | peak memory (PSS, GiB) | in Qdrant (GiB) | |---|---|---|---|---|---|---| -| 100 MB | 52 000 | 12 969 | 66 136 | 39 s | 6.6 GB | 0.6 GB | -| 1 GB | 524 000 | 240 516 | 836 595 | 4.8 min | 6.9 GB | 2.2 GB | -| 3 GB | 1 573 000 | 841 890 | 2 703 850 | 15 min | 7.3 GB | 6.0 GB | -| 10 GB | 5 243 000 | 3 423 359 | 10 093 514 | 58 min | 7.7 GB | 20.8 GB | -| 30 GB | 15 729 000 | 9 202 620 | 29 817 294 | 2.9 h | 10.1 GB | 61.3 GB | -| 50 GB | 26 214 000 | 17 083 603 | 53 913 774 | 5.5 h | 13.1 GB | 107.9 GB | - -Documents flow through the pipeline rather than accumulating in it, so -what stays in memory is short and worth spelling out. - -**Grows with the corpus — one thing.** The file-watch index: to detect live -edits and deletions, the indexer keeps a record (path, mtime, size, owner) -per watched file. Measured cost: **~318 bytes per file** (paths of typical -length; ±20% with the hash-table's load factor), verified from 13 thousand -to 17 million files (right-hand plot: six corpus sizes against one fitted -line). It scales with the *number of files*, not bytes: the same corpus -packed into fewer, larger files costs proportionally less. - -**Constant, regardless of corpus size.** The embedding stack (PyTorch -runtime + model, per worker), the engine baseline (~200 MB per process), -connector machinery (~0.4 GB), and working buffers that reach a plateau in -the first minutes of a run and stay there — identical on 3 GB and 10 GB. - -**On disk, not in memory.** Parsed-text cache, persistence snapshots, and -the embeddings themselves (in the vector database). That is why the curves -plateau: a **500× larger corpus costs 2.5× the memory** — and the growth -that remains is the file-watch index above, i.e. the corpus in fewer files -would cost less. Indexing time scales linearly with bytes throughout. +| 100 MB | 52 431 | 12 969 | 53 651 | 44 s | 6.86 | 0.61 | +| 1 GB | 524 288 | 240 516 | 604 299 | 4.9 min | 6.82 | 1.85 | +| 3 GB | 1 572 864 | 841 890 | 1 888 846 | 15.4 min | 7.18 | 4.89 | +| 10 GB | 5 242 884 | 3 423 359 | 6 761 214 | 1.0 h | 8.16 | 16.07 | +| 30 GB | 15 728 646 | 9 202 620 | 20 933 962 | 3.1 h | 10.40 | 48.57 | +| 50 GB | 26 214 382 | 17 083 603 | 37 472 786 | 6.0 h | 14.65 | 84.45 | + +The table and both figures come from the same six runs. Raw summaries, +memory samples, per-process snapshots and resolved configurations are in +[results](benchmarks/realtime-data-indexing/results/README.md). +One final run per size is published; run-to-run variability was not measured. +Memory and disk sizes use GiB; input labels are nominal targets +(1 GB = 1,000 MiB of extracted text). A page is approximately 2,000 bytes. +Chunks are Qdrant's approximate count at completion; the per-size curves +retain the maximum observed count used by the completion detector. + +Documents flow through the pipeline rather than accumulating in it. +The file-watch index retains metadata per watched file to detect edits and +deletions, so its cost depends on file count. The right panel estimates this +cost from the connector worker's extra private memory: the fitted slope is +**~361 bytes per file**. This proxy also includes unequal worker +buffers; it does not isolate individual allocations. + +The embedding runtime, engine and working buffers account for the worker +baseline. The curves show how total memory evolves during indexing, +including any steps as file-watch metadata and buffers grow. +Parsed-text cache, persistence snapshots and stored embeddings live on disk. +Across these runs a **500× larger input target costs 2.1× the peak PSS**. +Both file count and text volume change between sizes; this comparison does +not establish constant memory usage.

- Left: indexer PSS over time for corpora from 100 MB to 50 GB, from the run before the allocator tuning; every curve reaches its plateau in the first minutes and holds it for hours. Right: connector-worker extra memory across six corpus sizes follows ~318 bytes per watched file + Measured indexer PSS over time for all six corpora; legends use the same peaks as the table. Right: estimated file-watch memory versus file count, fitted at approximately 361 bytes per file.

-

Curves from the run before the allocator tuning (peaks 6.6–16.4 GB); the table above has the final numbers from the tuned rerun (6.6–13.1 GB). The shape is the point: each curve reaches its plateau in the first minutes and holds it for the rest of the run — 5.5 hours for 50 GB.

- -The peak itself is dominated by the embedding stack, not the engine — a -Pathway worker process is ~200 MB; the rest is the price of running -embeddings locally (8 × PyTorch runtime + model), i.e. of paying no -per-token API fees. Fewer workers or an API embedder shrink it accordingly. +

All curves and table values come from the same runs with allocator background purging enabled. File-watch memory is estimated from rank-0 private-memory excess over the median of the other workers.

- Breakdown of the 10 GB corpus peak as measured in the run before the allocator tuning (8.8 GB): three quarters is the local PyTorch embedding stack across 8 workers; file-watch metadata is about 1.1 GB; supervisors and shared code make up the rest + Breakdown at the 10 GB run's 8.16 GiB PSS peak: estimated local-embedder private and shared memory, estimated file-watch memory, remaining private memory and other shared pages.

-

Same earlier run (8.8 GB peak on the 10 GB corpus; 7.7 GB after tuning). The proportions are what matters here: the embedding stack dominates, the engine and the file-watch index are small.

- -Memory is measured as PSS (proportional set size) summed over the container: -shared pages — e.g. the PyTorch libraries mapped by every worker — are -counted once, not once per process. Setup: 96-core CPU host, streaming mode, -8 worker processes, local `static-retrieval-mrl-en-v1` embeddings (no API -calls; Matryoshka-truncated to 256 dims), 512-token chunks, Qdrant, and -jemalloc's `background_thread` purging enabled in the indexer containers -(measured free; it keeps idle workers from retaining freed pages). Numbers -were measured on a nightly Pathway build whose engine matches the released -wheel (pathway ≥ 0.32.1 — what the benchmark's docker image and the -Development section install), so they are reproducible as-is. +

Attribution at the sampled 10 GB peak. Local-embedder memory is estimated from a separate eight-process initialization diagnostic and library mappings; libraries already needed without the embedder remain in the residual. The 3.27 GiB estimate without the local embedder is calculated by subtraction, not measured in an external-embedder indexing run. It excludes the embedding service and does not measure client overhead or changes in working buffers. Shared pages are accounted for proportionally through PSS.

+ +Setup: 96-logical-CPU host (see the saved CPU model), +streaming mode, 8 worker processes with 12 OpenMP threads each, +local `static-retrieval-mrl-en-v1` embeddings, Matryoshka-truncated to 256 dims, +512-token chunks, Qdrant, and jemalloc `background_thread:true`. +No embedding API calls. Installed versions: serviette 0.1.7, +Pathway 0.33.0, PyTorch 2.14.1, +sentence-transformers 6.1.0. +Peak memory is the maximum sampled PSS summed over the indexer container. +Qdrant and API-server memory, unmapped filesystem cache and kernel memory +are outside that measurement; it is not a RAM requirement for the whole stack. +Brief peaks between samples may be missed. + +Indexing time is an estimate ending at the last increase in the running +maximum of Qdrant's approximate chunk count. That moment can precede the +final write; output-completeness checks do not revise the timestamp or +measure when the backend's search index is ready. The recorded environment, +cache conditions and limits on reproducing it are described in +[the measurement notes](benchmarks/realtime-data-indexing/results/README.md). ### Retrieval accuracy (FRAMES) diff --git a/benchmarks/realtime-data-indexing/README.md b/benchmarks/realtime-data-indexing/README.md index 70f212a..8f56b8d 100644 --- a/benchmarks/realtime-data-indexing/README.md +++ b/benchmarks/realtime-data-indexing/README.md @@ -1,13 +1,13 @@ # Real-time data indexing benchmark -Measures the two claims that matter for the serviette indexer: +Measures two aspects of the serviette indexer: -1. **Indexing speed** — wall-clock time to index a document collection end to - end (parse → chunk → embed → write to the vector DB). -2. **Flat memory** — the indexer's memory consumption does **not** grow with - the size of the input collection (`only_metadata` sources keep paths, not - bytes, in the pipeline). The output is a memory-over-time plot and the - physical peak memory (PSS: shared pages counted once, not per process). +1. **Indexing speed** — estimated wall-clock time for the indexing pipeline + (parse → chunk → embed → write to the vector DB), using the approximate + output-counter timing described below. +2. **Memory scaling** — document bytes flow through the pipeline, while + file-watch metadata grows with the number of files. The output is a + memory-over-time plot and sampled peak PSS (shared pages counted once). Everything runs locally and costs nothing: embeddings are computed by a local sentence-transformers model (`static-retrieval-mrl-en-v1`, @@ -21,16 +21,45 @@ Three containers via docker-compose: incrementally on upsert (fast indexing), it is designed for concurrent search-while-writing, and the Pathway sink keys points internally (no user primary key), so writes parallelize across Pathway workers. -- **indexer** — `serviette indexer` over the dataset (static mode: the run ends - when the collection is fully indexed, so wall-clock = indexing time). +- **indexer** — `serviette indexer` in streaming mode. Completion uses the + chunk-count, parsing-progress and CPU-idleness checks in `run_bench.py`. + Published indexing time ends at the last increase in the running maximum + of Qdrant's approximate chunk count. - **server** — `serviette server`; used after indexing to replay the accuracy questions through `/api/v1/retrieve`. -Memory is sampled from `/proc/1/status` inside the indexer container every -few seconds: `VmRSS` and `Pss` for the time series (PSS divides pages shared -between workers — e.g. the PyTorch libraries — proportionally, so its sum is -the honest physical footprint), `VmHWM` (kernel-tracked high-water -mark) for the exact peak — no sampling artifacts. +Memory sampling starts after server readiness, with a three-second sleep +between iterations; collecting memory and querying progress adds to that +interval. Samples cover the indexer container's processes: RSS from +`/proc//status`, PSS and private-page counts from +`/proc//smaps_rollup`. Peak PSS is the maximum sampled sum. Shared pages +are counted once through PSS. Summed `VmHWM` is retained in CSV as an upper +bound on RSS, since different processes can peak at different times. +Per-process samples include the Pathway worker rank, excluding the sampler. +Qdrant and API-server memory, unmapped filesystem cache and kernel memory +are outside this PSS measurement. It is not a RAM requirement for the whole +stack, and sampling can miss brief peaks. + +The current completion detector also requires successful parsing of every input +file. A chunk-rich subset can exceed the total file count and remain quiet +while the source is still being enumerated; this must not end sampling. +For a frozen corpus, `--expected-chunks N` additionally requires an exact +Qdrant count of N before accepting completion. This validates complete +output; the published approximate counter, 180-second quiet window, +30-core activity threshold and last-counter-increase time formula remain +the same. The saved results distinguish the checks available for each run. + +The timer starts before launching the indexer and server; dataset preparation, +model downloads and Qdrant startup are outside it. Approximate counts can +overshoot and later fall during segment optimization, so the last new maximum +need not coincide with the final write. An exact-count check validates output +completeness when performed, but does not change the published time estimate +or establish when the backend finished building its search index. + +`count_expected_chunks.py` independently counts the frozen `.txt` inputs +using UTF-8 decoding and the product's configured 512-token splitter. +Run it in the same immutable benchmark image, outside measured indexing, +and save its `-expected-output.json` files alongside the raw results. ## Dataset @@ -38,6 +67,15 @@ mark) for the exact peak — no sampling artifacts. (public, fast CDN) and writes one `.txt` file per article until the target size is reached. It also samples distinctive articles into `questions.json`; each question must retrieve its own article (top-k accuracy sanity check). +Sources continue in this order: English, German, French, Spanish, Italian, +Portuguese, Dutch. The last two are needed to reach the published 50 GB +target. Preparation records actual bytes and file count in `preparation.json` +and fails if sources run out before the target; publication also checks the +measured corpus size. +The current product splitter merges trailing fragments shorter than 100 +characters into the preceding chunk. Older published measurements used the +splitter without this merge, so old and new chunk counts, storage sizes and +timings do not isolate the effect of allocator tuning. ## Running @@ -48,13 +86,68 @@ themselves never touch the network. ```bash python prepare_dataset.py --size-mb 100 # also: 1000, 3000, 10000 -python run_bench.py --size 100mb +BENCH_JEMALLOC_CONF=background_thread:true python run_bench.py --size 100mb ``` Outputs land in `results/`: `-memory.png` (memory over time), `-memory.csv`, `-summary.json` (duration, MB/min, chunk count, -peak RSS/PSS, retrieval accuracy). +peak RSS/PSS, retrieval accuracy). Each run also saves per-process samples, +the process snapshot at peak PSS, a redacted resolved compose configuration, +and compressed indexer logs. CSV measurements, JSON results and the two +published figures in `docs/assets/` are committed. Raw `*.txt.gz` logs, +per-size `-memory.png` plots, the generated `table.md` and the +`consistency-check.json` audit report stay local and are ignored by Git. + +To regenerate the entire publication from a single series of runs: + +```bash +export BENCH_DATA_ROOT=/path/on/a/large/disk/serviette-bench-datasets +export BENCH_JEMALLOC_CONF=background_thread:true +export BENCH_OMP_THREADS=12 +export COMPOSE_PROJECT_NAME=serviette-consistency-bench +# PATHWAY_LICENSE_KEY must already be set. Build once for the full series. +docker compose build +python rerun_all.py +``` + +This runs 100mb → 1gb → 3gb → 10gb → 30gb → 50gb sequentially, records +an eight-process local-embedder initialization diagnostic, and then +calls `plot_results.py --update-readme`. Run it under nohup or tmux for the +multi-hour sizes. Datasets remain on the chosen disk after completion; +remove only this run's datasets when they are no longer needed. +Completed datasets (with `questions.json`) are reused, preserving historical +corpus boundaries. Missing datasets are prepared to the full nominal target; +nonempty incomplete datasets are rejected. Before each measured run, the driver +independently counts chunks in that corpus using the recorded image and passes +`--expected-chunks` to require an exact Qdrant count at completion. The audit +may download tokenizer data; it does not embed text or call an embedding API. +Reading the corpus for this audit can warm the filesystem cache. Containers +start with fresh database and persistence state, but the workflow does not +control the operating system's cache state or measure cold-cache I/O. + +`plot_results.py` reads all six CSVs and summaries, validates their agreement, +and generates both README images and the table. It also saves `derived.json` +with the fitted file-watch estimate and peak-memory ratio. The pie separates +estimated local-embedder private memory, estimated shared embedding-library +PSS, estimated file-watch memory, remaining private memory and other shared +pages. It estimates what remains in the indexer with an external embedder, +excluding the embedding service itself. Anonymous-memory attribution uses +the separate diagnostic and remains an estimate; shared library pages come +from the main run's mappings. +See `results/README.md` for the recorded setup and attribution formula. + +The publication contains one final run per corpus size, with no estimate of +run-to-run variability. Its memory scaling describes this configuration and +these corpora; file count and text volume both change between sizes. The +saved image IDs and package versions identify the measured environment, but +the build recipe uses floating image tags and dependency versions, and model +and dataset downloads have no pinned revision. A future rebuild can therefore +use a different environment or inputs. -To make the flat-memory point, run the same benchmark at increasing sizes -(100mb → 1gb → 3gb → 10gb) and compare `peak_rss_mb` across summaries: the -corpus grows 100×, the peak stays in the same band. +Run `python verify_results.py` directly from a checkout to independently check +every aggregate sample against its per-process snapshot, recompute peaks +and indexing times, compare all published tables, and verify the pie sum +and complete 50 GB output. No benchmark rerun or plot regeneration is needed. +The command prints its report and writes a local `results/consistency-check.json` +with the checks and hashes of both published figures. `plot_results.py` can +regenerate the per-size plots and standalone table from the saved measurements. diff --git a/benchmarks/realtime-data-indexing/count_expected_chunks.py b/benchmarks/realtime-data-indexing/count_expected_chunks.py new file mode 100644 index 0000000..6420332 --- /dev/null +++ b/benchmarks/realtime-data-indexing/count_expected_chunks.py @@ -0,0 +1,55 @@ +"""Audit frozen input using the benchmark image's actual product splitter. + +Run outside measured indexing. UTF8Parser decodes .txt bytes without alteration. +Each directory is a task; no embeddings or backend writes are involved. +""" +import argparse +import json +import multiprocessing as mp +import time +from pathlib import Path + + +def initialize(): + global splitter + from serviette.config import SplitterConfig + from serviette.indexer.graph import build_xpack_splitter + splitter = build_xpack_splitter(SplitterConfig(type="token_count", chunk_size=512)) + + +def count_directory(directory): + documents = chunks = corpus_bytes = 0 + for path in Path(directory).glob("*.txt"): + payload = path.read_bytes() + text = payload.decode("utf-8") + documents += 1 + corpus_bytes += len(payload) + chunks += len(splitter.chunk(text)) + return documents, chunks, corpus_bytes + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--data-root", required=True) + parser.add_argument("--results", required=True) + parser.add_argument("--sizes", nargs="+", default=["3gb", "10gb", "30gb"]) + parser.add_argument("--workers", type=int, default=12) + args = parser.parse_args() + with mp.Pool(args.workers, initializer=initialize) as pool: + for size in args.sizes: + started = time.monotonic() + directories = sorted((Path(args.data_root) / size / "docs").iterdir()) + totals = [0, 0, 0] + for completed, counts in enumerate(pool.imap_unordered(count_directory, directories), 1): + totals = [a + b for a, b in zip(totals, counts)] + if completed % 20 == 0 or completed == len(directories): + print(f"{size}: directories {completed}/{len(directories)}, documents {totals[0]}, chunks {totals[1]}, elapsed {time.monotonic()-started:.0f}s", flush=True) + result = dict(zip(["documents", "exact_chunks", "corpus_bytes"], totals)) + result.update(dataset=size, elapsed_seconds=round(time.monotonic()-started, 1), + source="Frozen corpus files; UTF-8 decode and actual build_xpack_splitter, token_count=512; executed in immutable serviette-bench image") + (Path(args.results) / f"{size}-expected-output.json").write_text(json.dumps(result, indent=2) + "\n") + print(json.dumps(result), flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/realtime-data-indexing/measure_embedder.py b/benchmarks/realtime-data-indexing/measure_embedder.py new file mode 100755 index 0000000..648da1c --- /dev/null +++ b/benchmarks/realtime-data-indexing/measure_embedder.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""Diagnostic: eight workers, before/after local embedder initialization. + +Run between indexing sizes, in the same image and environment. This does +not replace the indexing benchmark. Anonymous local-embedding memory is +estimated by the private-page increment; shared code comes from main-run +mapping samples. Keep workers alive at each barrier to count shared pages +proportionally across all eight rather than multiplying a single RSS value. +""" +from __future__ import annotations + +import argparse +import gc +import json +import os +import subprocess +import sys +import time +from pathlib import Path + + +def snapshot(): + result = {} + for line in Path('/proc/self/smaps_rollup').read_text().splitlines(): + if line.startswith(('Pss:', 'Private_Clean:', 'Private_Dirty:')): + key, value = line.split(':', 1) + result[key] = int(value.split()[0]) + embedding_pss = embedding_private = 0 + embedding = False + for line in Path('/proc/self/smaps').read_text().splitlines(): + parts = line.split() + if parts and '-' in parts[0] and ':' not in parts[0]: + path = parts[-1] if len(parts) >= 6 else '' + embedding = any(part in path for part in + ('/torch/', '/nvidia/', '/triton/', '/sentence_transformers/', + '/tokenizers/', '/safetensors/', + '/models--sentence-transformers--static-retrieval-mrl-en-v1/')) + elif embedding and line.startswith('Pss:'): + embedding_pss += int(parts[1]) + elif embedding and line.startswith(('Private_Clean:', 'Private_Dirty:')): + embedding_private += int(parts[1]) + result['embedding_shared_kb'] = max(0, embedding_pss - embedding_private) + return result + + +def worker(rank, directory, config): + # Match modules used by the actual indexer before constructing the model. + import yaml + from pathway.xpacks.llm import embedders # noqa: F401 + + from serviette.config.schema import EmbedderConfig, SplitterConfig + from serviette.indexer.graph import ParserRegistry, build_xpack_embedder, build_xpack_splitter + settings = yaml.safe_load(config.read_text()) + cfg = EmbedderConfig(**settings['embedder']) + # These remain necessary with a remote embedder. Include their lazy + # imports and tokenizer allocations in the pre-embedder baseline. + parser = ParserRegistry() + splitter = build_xpack_splitter(SplitterConfig(**settings['splitter'])) + text = 'Wikipedia articles describe history, science and geography. ' * 70 + parsed = parser.parse(text.encode(), '.txt') + assert splitter.chunk(parsed) + stages = ['baseline', 'runtime', 'model_warm', 'model_released'] + for stage in stages: + while not (directory / f'{stage}.go').exists(): + time.sleep(0.05) + if stage == 'runtime': + import sentence_transformers # noqa: F401 + elif stage == 'model_warm': + embedder = build_xpack_embedder(cfg) + for _ in range(3): + vectors = embedder.__wrapped__([text] * cfg.batch_size) + assert len(vectors[0]) == 256 + del vectors + elif stage == 'model_released': + del embedder + gc.collect() + # Let background purging settle, without changing allocator settings. + time.sleep(2) + (directory / f'{stage}.{rank}.ready').write_text('ready') + while not (directory / f'{stage}.sample').exists(): + time.sleep(0.05) + (directory / f'{stage}.{rank}.json').write_text(json.dumps(snapshot())) + while not (directory / 'finish').exists(): + time.sleep(0.05) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--worker', type=int) + parser.add_argument('--directory', type=Path, default=Path('/tmp/embedder-calibration')) + parser.add_argument('--config', type=Path, default=Path('/bench/config.yaml')) + args = parser.parse_args() + if args.worker is not None: + worker(args.worker, args.directory, args.config) + return + args.directory.mkdir() + children = [subprocess.Popen([sys.executable, __file__, '--worker', str(rank), + '--directory', str(args.directory), '--config', str(args.config)], + env={**os.environ, 'PATHWAY_PROCESS_ID': str(rank)}, + stdout=sys.stderr) for rank in range(8)] + result = {'description': __doc__, 'workers': 8, + 'baseline_includes': 'graph imports, UTF-8 parser, initialized and warmed configured splitter', + 'jemalloc_conf': os.environ.get('_RJEM_MALLOC_CONF'), + 'omp_threads': os.environ.get('OMP_NUM_THREADS'), 'stages': {}} + try: + for stage in ['baseline', 'runtime', 'model_warm', 'model_released']: + (args.directory / f'{stage}.go').touch() + for suffix in ['ready', 'json']: + deadline = time.monotonic() + 180 + while not all((args.directory / f'{stage}.{rank}.{suffix}').exists() for rank in range(8)): + if any(child.poll() is not None for child in children): + raise RuntimeError('Diagnostic worker exited') + if time.monotonic() > deadline: + raise TimeoutError(stage) + time.sleep(0.1) + if suffix == 'ready': + (args.directory / f'{stage}.sample').touch() + result['stages'][stage] = [json.loads((args.directory / f'{stage}.{rank}.json').read_text()) for rank in range(8)] + print(json.dumps(result, indent=2)) + finally: + (args.directory / 'finish').touch() + for child in children: + if child.poll() is None: + child.terminate() + for child in children: + child.wait() + + +if __name__ == '__main__': + main() diff --git a/benchmarks/realtime-data-indexing/plot_results.py b/benchmarks/realtime-data-indexing/plot_results.py new file mode 100755 index 0000000..375278d --- /dev/null +++ b/benchmarks/realtime-data-indexing/plot_results.py @@ -0,0 +1,455 @@ +#!/usr/bin/env python3 +"""Render the published table and figures exclusively from saved measurements.""" +from __future__ import annotations + +import argparse +import csv +import gzip +import json +import statistics +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +ROOT = Path(__file__).resolve().parent +REPO = ROOT.parent.parent +SIZES = [("100mb", "100 MB"), ("1gb", "1 GB"), ("3gb", "3 GB"), + ("10gb", "10 GB"), ("30gb", "30 GB"), ("50gb", "50 GB")] + + +def private_mb(process): + return (process.get("Private_Clean", 0) + process.get("Private_Dirty", 0)) / 1024 + + +def components(processes): + """Measured private/shared pages; rank-0 excess is an attribution estimate. + + Rank 0 owns the filesystem connector. Its excess over the median private + footprint of the other workers is a proxy, not an allocation profiler. + Do not label the entire worker baseline as 'PyTorch': it also includes + the engine and working buffers, which smaps cannot distinguish. + """ + workers = [p for p in processes if p["worker_rank"] is not None] + rank0 = [p for p in workers if p["worker_rank"] == 0] + others = [p for p in workers if p["worker_rank"] != 0] + if len(workers) != 8 or len(rank0) != 1 or len(others) != 7: + raise ValueError("Expected exactly eight Pathway workers including rank 0") + extra = max(0, private_mb(rank0[0]) - statistics.median(private_mb(p) for p in others)) + worker_private = sum(private_mb(p) for p in workers) + supervisor_private = sum(private_mb(p) for p in processes if p["worker_rank"] is None) + total = sum(p.get("Pss", 0) for p in processes) / 1024 + shared = total - worker_private - supervisor_private + if shared < -0.01: + raise ValueError("Private pages exceed total PSS") + return {"workers_private_mb": worker_private - extra, + "connector_extra_mb_estimate": extra, + "supervisors_private_mb": supervisor_private, + "shared_pss_mb": max(0, shared)} + + +def duration(seconds): + if seconds < 60: + return f"{seconds:.0f} s" + if seconds < 3600: + return f"{seconds / 60:.1f} min" + return f"{seconds / 3600:.1f} h" + + +def embedder_breakdown(run, calibration): + def private_total(stage): + return sum(p['Private_Clean'] + p['Private_Dirty'] for p in calibration['stages'][stage]) / 1024 + baseline = private_total('baseline') + worker_embedder_private = private_total('model_warm') - baseline + processes = run['summary']['peak_processes'] + if not all('mapping_groups' in p for p in processes): + raise ValueError('10 GB peak must include embedding-library mapping measurements') + c = run['peak_components'] + # Initialization and live batches have different buffer lifetimes. Never + # attribute more private memory than the measured worker footprint minus + # its independently measured pre-embedder baseline. Supervisor private + # pages remain in the residual; some may also be removable with a remote + # embedder, so this estimate does not subtract them speculatively. + embedding_private = min(worker_embedder_private, c['workers_private_mb'] - baseline) + embedding_shared = sum(max(0, p['mapping_groups']['embedding']['Pss'] - + p['mapping_groups']['embedding']['private_kb']) for p in processes) / 1024 + shared_increment = (sum(p['embedding_shared_kb'] for p in calibration['stages']['model_warm']) - + sum(p['embedding_shared_kb'] for p in calibration['stages']['baseline'])) / 1024 + embedding_shared = min(embedding_shared, max(0, shared_increment)) + metadata = c['connector_extra_mb_estimate'] + remaining_private = c['workers_private_mb'] + c['supervisors_private_mb'] - embedding_private + other_shared = c['shared_pss_mb'] - embedding_shared + if min(embedding_private, remaining_private, other_shared) < -1: + raise ValueError('Embedder attribution is incompatible with measured peak pages') + return {'local_embedder_private_mb_estimate': embedding_private, + 'local_embedder_shared_libraries_mb': embedding_shared, + 'file_watch_mb_estimate': metadata, + 'remaining_indexer_private_mb_estimate': remaining_private, + 'other_shared_mb': max(0, other_shared)} + + +def load(results, sizes=SIZES): + runs = [] + for size, label in sizes: + summary = json.loads((results / f"{size}-summary.json").read_text()) + target_mb = int(size[:-2]) * 1000 if size.endswith('gb') else int(size[:-2]) + if summary['corpus_mb'] < target_mb - 0.1: + raise ValueError(f"{size}: prepared corpus is smaller than its published target") + expected_path = results / f"{size}-expected-output.json" + if expected_path.exists(): + expected = json.loads(expected_path.read_text()) + if expected['documents'] != summary['documents']: + raise ValueError(f"{size}: frozen-corpus file count differs") + if ('corpus_bytes' in expected + and abs(expected['corpus_bytes'] / 1024**2 - summary['corpus_mb']) > 0.11): + raise ValueError(f"{size}: frozen-corpus byte count differs") + if size == '50gb': + if (summary.get('parsed_documents') != expected['documents'] + or summary.get('verified_exact_chunks') != expected['exact_chunks']): + raise ValueError('50gb: full input/output completion was not verified') + elif expected['exact_chunks'] != summary.get('verified_exact_chunks', summary['chunks']): + raise ValueError(f"{size}: output differs from independent splitter count") + else: + raise ValueError(f"{size}: missing independent output-completeness proof") + with (results / f"{size}-memory.csv").open() as fh: + samples = [{k: float(v) for k, v in row.items()} for row in csv.DictReader(fh)] + peak = max(s["pss_mb"] for s in samples) + if abs(peak - summary["peak_pss_mb"]) > 0.11: + raise ValueError(f"{size}: CSV peak does not match summary") + if int(samples[-1]["chunks"]) != summary["max_observed_chunks"]: + raise ValueError(f"{size}: CSV high-water chunk count does not match summary") + if summary["jemalloc_conf"] != "background_thread:true" or summary["omp_threads"] != 12: + raise ValueError(f"{size}: different measured configuration") + with gzip.open(results / f"{size}-processes.jsonl.gz", "rt") as fh: + last = None + for line in fh: + last = json.loads(line) + if abs(last["elapsed_s"] - samples[-1]["elapsed_s"]) > 0.11: + raise ValueError(f"{size}: process and aggregate series differ") + if abs(sum(p.get("Pss", 0) for p in summary["peak_processes"]) / 1024 - peak) > 0.11: + raise ValueError(f"{size}: peak process snapshot does not match total PSS") + runs.append({"size": size, "label": label, "summary": summary, + "samples": samples, "final_components": components(last["processes"]), + "peak_components": components(summary["peak_processes"])}) + return runs + + +def table(runs): + lines = ["| corpus target | ≈ pages | files | chunks (approx.) | indexing time | peak memory (PSS, GiB) | in Qdrant (GiB) |", + "|---|---|---|---|---|---|---|"] + for run in runs: + s = run["summary"] + ints = [f"{s[k]:,}".replace(",", " ") for k in ("approx_text_pages", "documents", "chunks")] + lines.append(f"| {run['label']} | {' | '.join(ints)} | {duration(s['indexing_seconds'])} | {s['peak_pss_mb'] / 1024:.2f} | {s['vector_db_disk_mb'] / 1024:.2f} |") + return "\n".join(lines) + + +def load_calibration(results, environment): + calibration = json.loads((results / 'embedder-calibration.json').read_text()) + if (calibration['image'] != environment['image'] or calibration['workers'] != 8 + or calibration['jemalloc_conf'] != 'background_thread:true' + or calibration['omp_threads'] != '12' or 'baseline_includes' not in calibration): + raise ValueError('Embedder diagnostic and indexing runs have different setups') + return calibration + + +def draw_run(run, output_path): + """Regenerate each size's plot from the same validated CSV and summary.""" + fig, ax = plt.subplots(figsize=(9, 4.5)) + minutes = [s['elapsed_s'] / 60 for s in run['samples']] + peak_gib = run['summary']['peak_pss_mb'] / 1024 + ax.plot(minutes, [s['pss_mb'] / 1024 for s in run['samples']], + label='indexer PSS', linewidth=1.8) + ax.axhline(peak_gib, linestyle='--', linewidth=1, alpha=0.6, + label=f'peak {peak_gib:.2f} GiB') + ax.set(xlabel='minutes since start', ylabel='memory (PSS), GiB', ylim=(0, None), + title=f"serviette indexer — {run['size']} corpus ({run['summary']['documents']} docs, streaming)") + chunks = ax.twinx() + chunks.plot(minutes, [s['chunks'] for s in run['samples']], color='tab:green', + alpha=0.6, linewidth=1.4, label='maximum observed chunks') + chunks.set(ylabel='maximum observed chunks', ylim=(0, None)) + lines, labels = ax.get_legend_handles_labels() + other_lines, other_labels = chunks.get_legend_handles_labels() + ax.legend(lines + other_lines, labels + other_labels, loc='center right') + ax.grid(alpha=0.3) + fig.tight_layout() + fig.savefig(output_path, dpi=130) + plt.close(fig) + + +def draw_breakdown(ten, c, remaining_mb, output_path): + values = list(c.values()) + labels = ["Local embedder\nPrivate memory (estimate)", + "Local embedder\nShared libraries (estimate)", + "File-watch memory\n(rank-0 excess estimate)", + "Other private memory\nEngine, buffers, supervisors (estimate)", + "Other shared pages\n(PSS residual)"] + labels = [f"{label}\n{value / 1024:.2f} GiB" for label, value in zip(labels, values)] + fig, ax = plt.subplots(figsize=(11.5, 6.5)) + wedges, _, _ = ax.pie(values, autopct=lambda p: f'{p:.0f}%' if p >= 6 else '', + startangle=90, colors=["#5145ef", "#a5b4fc", "#10a34a", "#64748b", "#cbd5e1"], + wedgeprops={"edgecolor": "white"}) + # Space callouts independently of tiny wedge angles, so shared-page and + # metadata labels remain readable without overlapping. + anchors = [(np.cos(np.deg2rad((w.theta1 + w.theta2) / 2)), + np.sin(np.deg2rad((w.theta1 + w.theta2) / 2))) for w in wedges] + for side in [-1, 1]: + indexes = sorted([i for i, (x, y) in enumerate(anchors) if (1 if x >= 0 else -1) == side], + key=lambda i: anchors[i][1]) + positions = np.linspace(-1.0, 1.0, len(indexes)) if len(indexes) > 1 else [0] + for index, y in zip(indexes, positions): + ax.annotate(labels[index], xy=anchors[index], xytext=(side * 1.4, y), + ha='left' if side > 0 else 'right', va='center', fontsize=12, + arrowprops={'arrowstyle': '-', 'color': '#64748b', + 'connectionstyle': 'angle3,angleA=0,angleB=90'}) + ax.set_xlim(-1.85, 1.85) + ax.set_title(f"10 GB corpus: measured peak {ten['summary']['peak_pss_mb'] / 1024:.2f} GiB PSS\n" + f"Estimated indexer without local embedder: {remaining_mb / 1024:.2f} GiB") + fig.tight_layout() + fig.savefig(output_path, dpi=130, bbox_inches='tight') + plt.close(fig) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--results", type=Path, default=ROOT / "results") + parser.add_argument("--assets", type=Path, default=REPO / "docs/assets") + parser.add_argument("--update-readme", action="store_true") + args = parser.parse_args() + runs = load(args.results) # Fail before touching published files if any run is missing. + environment = json.loads((args.results / "run-environment.json").read_text()) + ten = next(r for r in runs if r["size"] == "10gb") + calibration = load_calibration(args.results, environment) + for run in runs: + draw_run(run, args.results / f"{run['size']}-memory.png") + c = embedder_breakdown(ten, calibration) + files = np.array([r["summary"]["documents"] for r in runs], dtype=float) + extra = np.array([r["final_components"]["connector_extra_mb_estimate"] for r in runs]) + # Allow fixed connector overhead rather than attributing it to file count. + slope, intercept = np.polyfit(files, extra * 1024**2, 1) + slope, intercept = float(slope), float(intercept) + ratio = runs[-1]["summary"]["peak_pss_mb"] / runs[0]["summary"]["peak_pss_mb"] + derived = {"connector_bytes_per_file_estimate": slope, + "connector_fixed_overhead_bytes_estimate": intercept, "peak_memory_ratio": ratio, + "runs": [{"size": r["size"], "peak_components": r["peak_components"], + "final_components": r["final_components"]} for r in runs]} + (args.results / "derived.json").write_text(json.dumps(derived, indent=2) + "\n") + args.assets.mkdir(parents=True, exist_ok=True) + fig, (ax, right) = plt.subplots(1, 2, figsize=(13.5, 4.8)) + for run in runs: + s = run["summary"] + ax.plot([row["elapsed_s"] / 3600 for row in run["samples"]], + [row["pss_mb"] / 1024 for row in run["samples"]], + label=f"{run['label']} (peak {s['peak_pss_mb'] / 1024:.2f} GiB)") + ax.set(title="Indexer PSS over time", xlabel="hours", ylabel="PSS, GiB", ylim=(0, None)) + ax.legend(fontsize=9) + right.scatter(files / 1000, extra, color="#5145ef", label="rank-0 private-memory excess (estimate)") + x = np.linspace(0, files.max() * 1.05, 100) + right.plot(x / 1000, (intercept + slope * x) / 1024**2, "--", color="#10a34a", label=f"fit: fixed overhead + ~{slope:.0f} bytes × files") + for run, xx, yy in zip(runs, files / 1000, extra): + dy = {'100mb': -16, '1gb': 0, '3gb': 12}.get(run['size'], 0) + right.annotate(run["label"], (xx, yy), xytext=(5, dy), textcoords="offset points", fontsize=9) + right.set(title="File-watch memory proxy", xlabel="watched files, thousands", ylabel="extra private memory, MiB") + right.legend(fontsize=8) + for axes in (ax, right): + axes.grid(alpha=0.25) + fig.tight_layout() + fig.savefig(args.assets / "bench-memory.png", dpi=130) + plt.close(fig) + + derived['10gb_embedder_breakdown'] = c + derived['10gb_indexer_without_local_embedder_mb_estimate'] = ( + ten['summary']['peak_pss_mb'] - c['local_embedder_private_mb_estimate'] - c['local_embedder_shared_libraries_mb']) + (args.results / "derived.json").write_text(json.dumps(derived, indent=2) + "\n") + draw_breakdown(ten, c, derived['10gb_indexer_without_local_embedder_mb_estimate'], + args.assets / 'bench-memory-breakdown.png') + + published_table = table(runs) + (args.results / "table.md").write_text(published_table + "\n") + cutoff_note = '' + preparation_path = args.results / '50gb-preparation.json' + if preparation_path.exists(): + preparation = json.loads(preparation_path.read_text()) + if preparation.get('trimmed_trailing_documents'): + cutoff_note = """The 50 GB input restores the historical Portuguese/Dutch continuation and +matches the archived file count and approximate page count. Strictly filling +the nominal byte target adds 20 trailing Dutch articles (35,697 bytes); +these are preserved outside the input to reproduce the published cutoff. +`50gb-preparation.json` records the final boundary and the trimmed bytes +and document count. +""" + fifty_expected = json.loads((args.results / '50gb-expected-output.json').read_text()) + notes = f"""# Indexing benchmark results + +Measured from {environment['started_utc']}, on {environment['logical_cpus']} logical CPUs. +Exact CPU description, RAM, kernel, image ID and installed package versions: +`run-environment.json`. Resolved compose configurations redact the license key. +The workload, 180-second quiet window, 30-core activity threshold and +approximate-counter timing formula are retained. The current completion +detector also requires all input files to be parsed and, when an expected +count is supplied, verifies the exact output count. Independent splitter +counts match the saved final counts for all six sizes. The saved 50 GB run +additionally records complete parsing progress and an exact Qdrant count; +the five smaller runs record approximate backend counts. +The separate 50 GB audit scanned {fifty_expected['documents']:,} frozen input files ({fifty_expected['corpus_bytes']:,} +bytes) in the original immutable benchmark image and produced exactly +{fifty_expected['exact_chunks']:,} chunks, matching the saved run and exact Qdrant count. +It ran outside the measured indexing workload; memory and timing results +are unchanged. `50gb-expected-output.json` records this independent audit. + +{published_table} + +Input labels retain the published nominal sizes: 1 GB means a target of +1,000 MiB of text; 100 MB means 100 MiB. Memory and disk columns use GiB; +CSV columns ending in `_mb` contain MiB. Pages use 2,000 bytes per page. +{cutoff_note} +Peaks are the maximum sampled total PSS, not kernel high-water marks. +Sampling continues through the existing completion window. +Only indexer processes are included: Qdrant and API-server memory, unmapped +filesystem cache and kernel memory are outside this measurement. It is not +a RAM requirement for the whole stack. Sampling starts after server readiness; +each iteration includes a three-second sleep plus collection and polling +overhead, so brief peaks may be missed. +The chunk-count methodology is unchanged: the summary uses Qdrant's +approximate count at completion, while CSV `chunks` retains the maximum +observed count used by the completion detector. Segment optimization can +reduce approximate counts, so these two fields need not be identical; +`max_observed_chunks` records the CSV's value in the summary explicitly. + +Published indexing time is an estimate: elapsed time from before launching +the indexer and server to the last increase in the running maximum of the +approximate chunk count. Preparation, model downloads and Qdrant startup are +outside this interval. A temporary overcount can make the last new maximum +precede the final write. Checking the exact output count at completion does +not revise that timestamp or measure when the backend's search index is ready. +The saved timing values retain this definition. + +`*-memory.csv` and `*-processes.jsonl.gz` record aggregate and per-process +measurements at the same times; `*-summary.json` includes the process snapshot +at peak PSS. `*-indexer.txt.gz` contains the local container log and is +excluded from Git. Measurements, derived values, this report and the two +published figures in `docs/assets/` are committed. Per-size `*-memory.png` +plots, the standalone `table.md` and `consistency-check.json` stay local and +are ignored by Git; they can be regenerated from the saved measurements. + +One final run per corpus size is published, without an estimate of run-to-run +variability. File count and text volume both change across sizes, so these +measurements do not establish constant memory at a fixed file count. They +describe this local static-embedding model, plain-text workload and configuration. +The product splitter merges trailing fragments shorter than 100 characters +into the preceding chunk; older published measurements predate that behavior. +Differences in chunk counts, disk usage and timing therefore cannot be +attributed to allocator tuning alone. + +The saved runs do not establish a controlled cold-cache result. Fresh database +and persistence state do not imply a cold operating-system cache. The current +rerun driver reads the entire corpus to count expected chunks before indexing, +which can warm that cache. Saved image IDs and key package versions identify +the measured environment; the build recipe still uses floating image tags +and dependency versions, and model and dataset downloads have no pinned +revision. A future rebuild can use different software or inputs. + +The right panel estimates file-watch memory as rank 0's private-page excess +over the median of the seven other workers at the final sample. It includes +any other unequal worker allocations; it is not an allocation-level measurement. +The line is a least-squares fit with an intercept (~{slope:.0f} bytes per file; +fixed overhead estimate {intercept / 1024**2:.1f} MiB). +The pie attributes local-embedding private memory using an eight-process +controlled diagnostic in the same image: graph imports before the embedder, +UTF-8 parser and configured splitter initialized and warmed, embedding +runtime imported, model loaded and warmed on three 2,048-text batches, then +model released. `embedder-calibration.json` records all stages. The private +increment after warming estimates the local stack's private cost across eight +workers. This is capped at the observed non-file-watch worker private memory +minus the calibrated pre-embedder baseline, since live and diagnostic buffers +have different lifetimes. Supervisor private pages are left in the residual; +any embedding allocations there are not subtracted speculatively. +Shared embedding-library PSS starts from mapping samples at the 10 GB peak +and is capped by the diagnostic's shared-library increment, so libraries +already needed by the parser are retained in the residual. Other shared +pages are separated. Anonymous attribution +and the indexer-without-local-embedder figure are estimates, not an actual +external-service benchmark; client overhead and changes in working buffers +are not measured. The embedding service's own memory is outside that figure. +Derived values are saved in `derived.json`. + +Regenerate with `python plot_results.py --update-readme`. +The script checks CSV peaks, chunk counts and process snapshots against summaries +and requires all six sizes before updating the publication. +Run `python verify_results.py` directly from a checkout to audit every process +sample, both published tables and the pie sum. It does not require a benchmark +rerun or regenerated per-size plots. It prints the result and writes a local +`consistency-check.json` with hashes of both published figures. +""" + (args.results / "README.md").write_text(notes) + if args.update_readme: + readme = REPO / "README.md" + text = readme.read_text() + start = text.index("| corpus", text.index("### Indexing resources")) + end = text.index("### Retrieval accuracy", start) + versions = environment["versions"] + replacement = f"""{published_table} + +The table and both figures come from the same six runs. Raw summaries, +memory samples, per-process snapshots and resolved configurations are in +[results](benchmarks/realtime-data-indexing/results/README.md). +One final run per size is published; run-to-run variability was not measured. +Memory and disk sizes use GiB; input labels are nominal targets +(1 GB = 1,000 MiB of extracted text). A page is approximately 2,000 bytes. +Chunks are Qdrant's approximate count at completion; the per-size curves +retain the maximum observed count used by the completion detector. + +Documents flow through the pipeline rather than accumulating in it. +The file-watch index retains metadata per watched file to detect edits and +deletions, so its cost depends on file count. The right panel estimates this +cost from the connector worker's extra private memory: the fitted slope is +**~{slope:.0f} bytes per file**. This proxy also includes unequal worker +buffers; it does not isolate individual allocations. + +The embedding runtime, engine and working buffers account for the worker +baseline. The curves show how total memory evolves during indexing, +including any steps as file-watch metadata and buffers grow. +Parsed-text cache, persistence snapshots and stored embeddings live on disk. +Across these runs a **500× larger input target costs {ratio:.1f}× the peak PSS**. +Both file count and text volume change between sizes; this comparison does +not establish constant memory usage. + +

+ Measured indexer PSS over time for all six corpora; legends use the same peaks as the table. Right: estimated file-watch memory versus file count, fitted at approximately {slope:.0f} bytes per file. +

+

All curves and table values come from the same runs with allocator background purging enabled. File-watch memory is estimated from rank-0 private-memory excess over the median of the other workers.

+ +

+ Breakdown at the 10 GB run's {ten['summary']['peak_pss_mb'] / 1024:.2f} GiB PSS peak: estimated local-embedder private and shared memory, estimated file-watch memory, remaining private memory and other shared pages. +

+

Attribution at the sampled 10 GB peak. Local-embedder memory is estimated from a separate eight-process initialization diagnostic and library mappings; libraries already needed without the embedder remain in the residual. The {derived['10gb_indexer_without_local_embedder_mb_estimate'] / 1024:.2f} GiB estimate without the local embedder is calculated by subtraction, not measured in an external-embedder indexing run. It excludes the embedding service and does not measure client overhead or changes in working buffers. Shared pages are accounted for proportionally through PSS.

+ +Setup: {environment['logical_cpus']}-logical-CPU host (see the saved CPU model), +streaming mode, 8 worker processes with 12 OpenMP threads each, +local `static-retrieval-mrl-en-v1` embeddings, Matryoshka-truncated to 256 dims, +512-token chunks, Qdrant, and jemalloc `background_thread:true`. +No embedding API calls. Installed versions: serviette {versions['serviette']}, +Pathway {versions['pathway']}, PyTorch {versions['torch']}, +sentence-transformers {versions['sentence-transformers']}. +Peak memory is the maximum sampled PSS summed over the indexer container. +Qdrant and API-server memory, unmapped filesystem cache and kernel memory +are outside that measurement; it is not a RAM requirement for the whole stack. +Brief peaks between samples may be missed. + +Indexing time is an estimate ending at the last increase in the running +maximum of Qdrant's approximate chunk count. That moment can precede the +final write; output-completeness checks do not revise the timestamp or +measure when the backend's search index is ready. The recorded environment, +cache conditions and limits on reproducing it are described in +[the measurement notes](benchmarks/realtime-data-indexing/results/README.md). + +""" + readme.write_text(text[:start] + replacement + text[end:]) + print(published_table) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/realtime-data-indexing/prepare_dataset.py b/benchmarks/realtime-data-indexing/prepare_dataset.py old mode 100644 new mode 100755 index e85eb3e..ca7755e --- a/benchmarks/realtime-data-indexing/prepare_dataset.py +++ b/benchmarks/realtime-data-indexing/prepare_dataset.py @@ -3,7 +3,7 @@ Downloads Wikipedia parquet shards from Hugging Face (public, no auth, fast CDN) and explodes articles into one ``.txt`` file each until the requested -size is reached — so the same script scales from 100 MB to 10 GB by changing +size is reached — so the same script scales from 100 MB to 50 GB by changing ``--size-mb``. Also emits ``questions.json``: distinctive articles sampled across the corpus with queries that must retrieve exactly that file (retrieval-accuracy check for the benchmark). @@ -24,7 +24,10 @@ # Language cascade: English alone tops out around 20 GB of plain text, so # larger corpora continue into further languages. Accuracy questions are # sampled from the head of the corpus, i.e. always from English. -HF_CONFIGS = ["20231101.en", "20231101.de", "20231101.fr", "20231101.es", "20231101.it"] +# The published 50 GB corpus also continues into Portuguese and Dutch. +# The first five languages alone stop at about 45,324 MiB. +HF_CONFIGS = ["20231101.en", "20231101.de", "20231101.fr", "20231101.es", "20231101.it", + "20231101.pt", "20231101.nl"] # Shard the docs tree: N files per subdirectory, so multi-million-file # corpora don't create a single pathological directory. FILES_PER_DIR = 10_000 @@ -103,6 +106,12 @@ def main() -> None: ) article_index += 1 + (out / "preparation.json").write_text(json.dumps({ + "target_bytes": target_bytes, "written_bytes": written, + "documents": article_index, "language_configs": HF_CONFIGS, + }, indent=2) + "\n") + if written < target_bytes: + raise RuntimeError(f"Corpus sources exhausted: {written} bytes < {target_bytes} target") (out / "questions.json").write_text(json.dumps(questions, indent=2)) print( f"done: {article_index} files, {written / 1024 / 1024:.1f} MB, " diff --git a/benchmarks/realtime-data-indexing/rerun_all.py b/benchmarks/realtime-data-indexing/rerun_all.py new file mode 100755 index 0000000..9b0a2ed --- /dev/null +++ b/benchmarks/realtime-data-indexing/rerun_all.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Run the six published sizes sequentially, then render one consistent report.""" +from __future__ import annotations + +import datetime +import fcntl +import gzip +import json +import os +import platform +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +SIZES = [("100mb", 100), ("1gb", 1000), ("3gb", 3000), + ("10gb", 10000), ("30gb", 30000), ("50gb", 50000)] + + +def main(): + if not os.environ.get("PATHWAY_LICENSE_KEY"): + raise SystemExit("PATHWAY_LICENSE_KEY must be set") + if os.environ.get("BENCH_JEMALLOC_CONF") != "background_thread:true": + raise SystemExit("Set BENCH_JEMALLOC_CONF=background_thread:true") + results = ROOT / "results" + results.mkdir(exist_ok=True) + data_root = Path(os.environ["BENCH_DATA_ROOT"]) + data_root.mkdir(parents=True, exist_ok=True) + with (results / ".rerun.lock").open("w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + _run(results, data_root) + + +def _run(results, data_root): + def output(cmd): + return subprocess.check_output(cmd, text=True).strip() + + versions = output(["docker", "run", "--rm", "--entrypoint", "python", + "serviette-bench", "-c", + "import importlib.metadata as m,json; print(json.dumps({k:m.version(k) for k in ['serviette','pathway','torch','sentence-transformers','qdrant-client']}))"]) + metadata = { + "started_utc": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "git_commit": output(["git", "rev-parse", "HEAD"]), + "cpu": output(["lscpu"]), "logical_cpus": os.cpu_count(), + "memory": Path("/proc/meminfo").read_text(), "kernel": platform.release(), + "docker": output(["docker", "--version"]), + "versions": json.loads(versions), + "image": output(["docker", "image", "inspect", "serviette-bench", "--format", "{{.Id}}"]), + "environment": {key: os.environ.get(key) for key in + ["BENCH_DATA_ROOT", "BENCH_JEMALLOC_CONF", "BENCH_OMP_THREADS", "COMPOSE_PROJECT_NAME"]}, + } + (results / "run-environment.json").write_text(json.dumps(metadata, indent=2) + "\n") + for size, mb in SIZES: + print(f"Preparing {size}", flush=True) + dataset = data_root / size + if not (dataset / "questions.json").exists(): + if (dataset / "docs").exists() and any((dataset / "docs").iterdir()): + raise RuntimeError(f"Incomplete dataset at {dataset}; use a fresh BENCH_DATA_ROOT") + subprocess.run([sys.executable, str(ROOT / "prepare_dataset.py"), + "--size-mb", str(mb), "--out", str(dataset)], check=True) + if (dataset / 'preparation.json').exists(): + shutil.copy2(dataset / 'preparation.json', results / f'{size}-preparation.json') + print(f"Indexing {size}", flush=True) + # Count the actual frozen corpus, including its chosen byte boundary, + # outside the measured run and using the same immutable product image. + subprocess.run([ + "docker", "run", "--rm", + "-e", "HF_HUB_OFFLINE=1", "-e", "OMP_NUM_THREADS=1", + "-v", f"{data_root.resolve()}:/datasets:ro", + "-v", f"{ROOT / 'count_expected_chunks.py'}:/bench/count_expected_chunks.py:ro", + "-v", f"{results}:/results", metadata["image"], + "python", "/bench/count_expected_chunks.py", "--data-root", "/datasets", + "--results", "/results", "--sizes", size, + ], check=True) + expected = json.loads((results / f"{size}-expected-output.json").read_text()) + expected["image"] = metadata["image"] + (results / f"{size}-expected-output.json").write_text(json.dumps(expected, indent=2) + "\n") + try: + subprocess.run([sys.executable, str(ROOT / "run_bench.py"), "--size", size, + "--expected-chunks", str(expected["exact_chunks"])], check=True) + except BaseException: + logs = subprocess.run(["docker", "compose", "logs", "--no-color"], + cwd=ROOT, capture_output=True, text=True, check=False) + with gzip.open(results / f"{size}-failure.txt.gz", "wt") as fh: + fh.write(logs.stdout + logs.stderr) + raise + summary = json.loads((results / f"{size}-summary.json").read_text()) + if summary["chunks"] < summary["documents"]: + raise RuntimeError(f"Incomplete indexing: {size}") + print(f"Finished {size}: {summary['chunks']} chunks, {summary['peak_pss_mb']} MiB PSS", flush=True) + print("Measuring local embedder initialization separately from indexing", flush=True) + diagnostic = subprocess.run( + ["docker", "compose", "run", "--rm", "--no-deps", "-T", "-v", + f"{ROOT / 'measure_embedder.py'}:/bench/measure_embedder.py:ro", + "indexer", "python", "/bench/measure_embedder.py"], + cwd=ROOT, capture_output=True, text=True, check=True) + calibration = json.loads(diagnostic.stdout) + calibration["image"] = metadata["image"] + (results / "embedder-calibration.json").write_text(json.dumps(calibration, indent=2) + "\n") + with gzip.open(results / "embedder-calibration.txt.gz", "wt") as fh: + fh.write(diagnostic.stderr) + subprocess.run([sys.executable, str(ROOT / "plot_results.py"), "--update-readme"], check=True) + print("All six sizes measured; README and figures regenerated.", flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/realtime-data-indexing/results/100mb-compose.json b/benchmarks/realtime-data-indexing/results/100mb-compose.json new file mode 100644 index 0000000..efcf5f5 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/100mb-compose.json @@ -0,0 +1,170 @@ +{ + "name": "serviette-consistency-bench", + "networks": { + "default": { + "name": "serviette-consistency-bench_default", + "ipam": {} + } + }, + "services": { + "indexer": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "indexer", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OMP_NUM_THREADS": "12", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/100mb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + }, + "qdrant": { + "command": null, + "entrypoint": null, + "image": "qdrant/qdrant:latest", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 6333, + "published": "8301", + "protocol": "tcp" + } + ] + }, + "server": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "server", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 8000, + "published": "8300", + "protocol": "tcp" + } + ], + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/100mb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + } + }, + "x-serviette": { + "build": { + "context": "../..", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "PV3L-4L4N-TF7R-JWM7-LKVK-PX3E-FHTX-KMNC", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "volumes": [ + "~/.cache/huggingface:/hf-cache:ro", + "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/100mb/docs:/data/docs:ro", + "./config.yaml:/bench/config.yaml:ro" + ] + } +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/100mb-expected-output.json b/benchmarks/realtime-data-indexing/results/100mb-expected-output.json new file mode 100644 index 0000000..6a5d89f --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/100mb-expected-output.json @@ -0,0 +1,5 @@ +{ + "documents": 12969, + "exact_chunks": 53651, + "source": "Independent complete frozen-input splitter count; matches the final indexing run" +} diff --git a/benchmarks/realtime-data-indexing/results/100mb-memory.csv b/benchmarks/realtime-data-indexing/results/100mb-memory.csv new file mode 100644 index 0000000..ca44ac7 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/100mb-memory.csv @@ -0,0 +1,61 @@ +elapsed_s,rss_mb,hwm_mb,pss_mb,chunks +11.1,1102.0,1102.0,858.5,0 +14.4,2943.1,2951.4,1997.0,0 +18.0,7277.3,7277.3,4609.2,0 +21.6,9281.2,9281.3,5830.7,0 +25.5,10957.6,10957.6,6886.7,0 +29.3,11146.3,11146.6,7020.2,23909 +33.0,11182.3,11188.0,7028.8,36391 +36.6,11214.0,11219.1,7017.7,44463 +40.4,11258.4,11275.5,7011.3,51349 +44.0,11254.7,11276.0,7007.0,53651 +47.7,11241.9,11276.0,6994.2,53651 +51.3,11225.8,11276.0,6977.7,53651 +55.0,11220.4,11276.0,6972.0,53651 +58.7,11216.6,11276.0,6968.1,53651 +62.4,11216.9,11276.0,6967.8,53651 +66.0,11217.0,11276.0,6967.8,53651 +69.7,11216.2,11276.0,6967.0,53651 +73.4,11216.2,11276.0,6967.0,53651 +77.1,11216.2,11276.0,6966.9,53651 +80.8,11215.6,11276.0,6966.6,53651 +84.3,11215.8,11276.0,6966.6,53651 +87.9,11215.7,11276.0,6966.5,53651 +91.4,11215.4,11276.0,6966.4,53651 +95.1,11215.5,11276.0,6966.4,53651 +98.8,11215.5,11276.2,6966.3,53651 +102.5,11215.4,11276.2,6966.2,53651 +106.0,11215.6,11276.2,6966.2,53651 +109.7,11215.5,11276.2,6966.1,53651 +113.4,11215.3,11276.2,6966.0,53651 +117.1,11215.3,11276.2,6966.0,53651 +120.7,11215.2,11276.2,6965.9,53651 +124.3,11215.2,11276.2,6965.9,53651 +128.0,11215.1,11276.2,6965.9,53651 +131.7,11215.1,11276.2,6965.9,53651 +135.2,11215.2,11276.2,6965.8,53651 +138.8,11215.2,11276.2,6965.8,53651 +142.5,11215.0,11276.2,6965.6,53651 +146.2,11215.0,11276.2,6965.7,53651 +150.0,11215.0,11276.2,6965.6,53651 +153.5,11214.7,11276.2,6965.6,53651 +157.0,11214.9,11276.2,6965.5,53651 +160.8,11214.8,11276.2,6965.4,53651 +164.4,11214.6,11276.2,6965.3,53651 +168.1,11214.7,11276.2,6965.3,53651 +171.7,11214.5,11276.2,6965.1,53651 +175.4,11214.2,11276.2,6964.8,53651 +179.1,11214.2,11276.2,6964.9,53651 +182.6,11214.2,11276.2,6964.8,53651 +186.3,11214.0,11276.2,6964.7,53651 +189.9,11214.1,11276.2,6964.7,53651 +193.7,11213.7,11276.2,6964.6,53651 +197.4,11213.7,11276.2,6964.6,53651 +201.2,11213.7,11276.2,6964.5,53651 +204.8,11213.5,11276.2,6956.1,53651 +208.6,11213.7,11276.2,6956.0,53651 +212.3,11213.5,11276.2,6955.9,53651 +216.0,11213.2,11276.2,6955.7,53651 +219.8,11213.2,11276.2,6955.6,53651 +223.3,11213.1,11276.2,6955.5,53651 +226.8,11212.9,11276.2,6955.3,53651 diff --git a/benchmarks/realtime-data-indexing/results/100mb-processes.jsonl.gz b/benchmarks/realtime-data-indexing/results/100mb-processes.jsonl.gz new file mode 100644 index 0000000..83c61f4 Binary files /dev/null and b/benchmarks/realtime-data-indexing/results/100mb-processes.jsonl.gz differ diff --git a/benchmarks/realtime-data-indexing/results/100mb-summary.json b/benchmarks/realtime-data-indexing/results/100mb-summary.json new file mode 100644 index 0000000..57c702f --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/100mb-summary.json @@ -0,0 +1,115 @@ +{ + "dataset": "100mb", + "corpus_mb": 100.0, + "documents": 12969, + "chunks": 53651, + "indexing_seconds": 44.0, + "throughput_mb_per_min": 136.24, + "peak_rss_mb": 11258.4, + "peak_pss_mb": 7028.8, + "peak_processes": [ + { + "pid": 1, + "worker_rank": null, + "VmHWM": 990416, + "VmRSS": 990416, + "Pss": 626795, + "Private_Clean": 308, + "Private_Dirty": 583568 + }, + { + "pid": 131, + "worker_rank": null, + "VmHWM": 182760, + "VmRSS": 182760, + "Pss": 106876, + "Private_Clean": 32, + "Private_Dirty": 98524 + }, + { + "pid": 159, + "worker_rank": 0, + "VmHWM": 1383516, + "VmRSS": 1383516, + "Pss": 897528, + "Private_Clean": 764, + "Private_Dirty": 834220 + }, + { + "pid": 160, + "worker_rank": 1, + "VmHWM": 1268900, + "VmRSS": 1268628, + "Pss": 796966, + "Private_Clean": 484, + "Private_Dirty": 733792 + }, + { + "pid": 161, + "worker_rank": 2, + "VmHWM": 1286812, + "VmRSS": 1286812, + "Pss": 797217, + "Private_Clean": 300, + "Private_Dirty": 734384 + }, + { + "pid": 162, + "worker_rank": 3, + "VmHWM": 1268028, + "VmRSS": 1267004, + "Pss": 798308, + "Private_Clean": 916, + "Private_Dirty": 734284 + }, + { + "pid": 163, + "worker_rank": 4, + "VmHWM": 1266920, + "VmRSS": 1264620, + "Pss": 797776, + "Private_Clean": 436, + "Private_Dirty": 734024 + }, + { + "pid": 164, + "worker_rank": 5, + "VmHWM": 1253988, + "VmRSS": 1253888, + "Pss": 781688, + "Private_Clean": 24, + "Private_Dirty": 719472 + }, + { + "pid": 165, + "worker_rank": 6, + "VmHWM": 1285488, + "VmRSS": 1284748, + "Pss": 797871, + "Private_Clean": 3348, + "Private_Dirty": 731096 + }, + { + "pid": 166, + "worker_rank": 7, + "VmHWM": 1269672, + "VmRSS": 1268264, + "Pss": 796461, + "Private_Clean": 2032, + "Private_Dirty": 731308 + } + ], + "peak_elapsed_s": 32.992267817258835, + "jemalloc_conf": "background_thread:true", + "omp_threads": 12, + "indexer_cpu_seconds": 370.4, + "avg_parallelism": 8.4, + "vector_db_disk_mb": 625.9, + "vector_db_mb_per_corpus_mb": 6.26, + "retrieval_accuracy": "5/5", + "embedder": "sentence-transformers/static-retrieval-mrl-en-v1 (local, 256 dims)", + "approx_text_pages": 52431, + "accuracy_misses": [], + "max_observed_chunks": 53651, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count" +} diff --git a/benchmarks/realtime-data-indexing/results/10gb-compose.json b/benchmarks/realtime-data-indexing/results/10gb-compose.json new file mode 100644 index 0000000..9262467 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/10gb-compose.json @@ -0,0 +1,170 @@ +{ + "name": "serviette-consistency-bench", + "networks": { + "default": { + "name": "serviette-consistency-bench_default", + "ipam": {} + } + }, + "services": { + "indexer": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "indexer", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OMP_NUM_THREADS": "12", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/10gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + }, + "qdrant": { + "command": null, + "entrypoint": null, + "image": "qdrant/qdrant:latest", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 6333, + "published": "8301", + "protocol": "tcp" + } + ] + }, + "server": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "server", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 8000, + "published": "8300", + "protocol": "tcp" + } + ], + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/10gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + } + }, + "x-serviette": { + "build": { + "context": "../..", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "PV3L-4L4N-TF7R-JWM7-LKVK-PX3E-FHTX-KMNC", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "volumes": [ + "~/.cache/huggingface:/hf-cache:ro", + "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/10gb/docs:/data/docs:ro", + "./config.yaml:/bench/config.yaml:ro" + ] + } +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/10gb-expected-output.json b/benchmarks/realtime-data-indexing/results/10gb-expected-output.json new file mode 100644 index 0000000..9768592 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/10gb-expected-output.json @@ -0,0 +1,8 @@ +{ + "documents": 3423359, + "exact_chunks": 6761214, + "corpus_bytes": 10485769151, + "dataset": "10gb", + "elapsed_seconds": 271.9, + "source": "Frozen corpus files; UTF-8 decode and actual build_xpack_splitter, token_count=512; executed in immutable serviette-bench image" +} diff --git a/benchmarks/realtime-data-indexing/results/10gb-memory.csv b/benchmarks/realtime-data-indexing/results/10gb-memory.csv new file mode 100644 index 0000000..b9488d2 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/10gb-memory.csv @@ -0,0 +1,814 @@ +elapsed_s,rss_mb,hwm_mb,pss_mb,chunks +11.3,1123.8,1123.8,875.2,0 +14.9,2858.6,2858.6,1924.1,0 +19.3,7743.8,7743.8,4880.5,0 +23.5,8907.5,8907.5,5540.6,0 +27.8,9026.4,9026.4,5657.6,0 +32.1,9159.2,9159.3,5789.0,0 +36.4,9458.7,9491.6,6088.8,0 +40.9,9784.5,9815.6,6414.6,0 +45.7,11309.0,11626.4,7338.0,10898 +50.4,11500.1,11793.5,7473.7,30992 +55.0,11531.9,11847.4,7437.3,40166 +59.7,11536.7,11850.1,7422.4,51444 +64.3,11556.4,11859.4,7430.0,61863 +68.8,11609.8,11886.3,7443.6,72087 +73.5,11643.2,11919.0,7453.2,82887 +78.3,11662.1,11931.2,7462.7,92901 +82.9,11697.2,11957.2,7482.0,103265 +87.5,11720.8,11975.5,7487.4,115205 +92.3,11737.6,11993.0,7494.9,124252 +96.9,11750.7,12004.1,7506.3,135307 +101.6,11773.4,12020.8,7525.6,147134 +106.1,11776.5,12027.2,7524.8,157892 +110.7,11788.0,12031.2,7535.6,167446 +115.5,11796.3,12036.7,7539.4,176294 +120.2,11797.9,12039.0,7540.0,185944 +125.0,11801.5,12042.9,7543.0,196981 +129.6,11803.3,12045.6,7543.9,206934 +134.2,11812.4,12050.0,7551.2,216718 +138.9,11818.1,12052.8,7557.8,226637 +143.4,11820.1,12059.1,7560.7,239204 +148.0,11828.0,12062.8,7568.5,250199 +152.6,11836.5,12070.6,7577.8,256827 +157.2,11835.5,12072.8,7577.2,268971 +161.7,11847.4,12077.4,7589.1,278861 +166.3,11846.1,12081.3,7587.5,290248 +171.0,11853.2,12082.3,7593.7,301931 +175.5,11863.3,12089.0,7603.5,309446 +180.1,11865.0,12090.8,7603.8,320964 +184.8,11866.6,12094.4,7606.3,330683 +189.4,11876.9,12100.7,7616.4,343154 +194.2,11875.6,12105.6,7613.6,353829 +198.8,11881.0,12107.8,7618.7,364015 +203.3,11883.9,12110.0,7621.6,376508 +207.9,11886.9,12112.0,7623.9,384669 +212.5,11890.2,12116.1,7627.8,394917 +217.1,11888.4,12116.6,7626.3,405738 +221.8,11889.9,12116.8,7628.4,415977 +226.4,11890.9,12118.1,7629.4,426151 +231.0,11892.9,12120.8,7632.2,435949 +235.6,11893.3,12121.1,7632.5,446890 +240.3,11899.9,12121.4,7638.6,457229 +244.8,11897.6,12122.4,7636.1,468221 +249.4,11898.8,12124.2,7637.8,477059 +254.1,11901.4,12125.5,7640.3,490857 +258.7,11903.9,12126.8,7642.9,500198 +263.4,11905.6,12129.0,7643.7,512904 +268.0,11909.3,12132.3,7647.7,524275 +272.7,11912.6,12135.6,7649.7,532197 +277.5,11917.0,12138.4,7654.9,545423 +282.2,11918.7,12140.2,7656.1,552707 +286.8,11918.7,12141.1,7655.9,563584 +291.6,11920.1,12141.5,7657.6,572080 +296.2,11938.0,12147.2,7675.3,582419 +300.8,11939.3,12148.1,7676.8,594591 +305.5,11943.0,12154.2,7680.9,603712 +310.1,11940.7,12154.6,7679.1,609951 +314.7,11943.1,12157.0,7681.6,620259 +319.2,11948.3,12159.2,7686.2,629652 +323.9,11947.1,12160.7,7685.1,640172 +328.6,11948.1,12160.9,7686.3,650540 +333.1,11949.4,12162.6,7687.9,660359 +337.8,11949.5,12163.3,7688.1,671603 +342.5,11953.3,12163.7,7691.8,683511 +347.2,11952.5,12164.8,7690.4,695610 +351.9,11960.3,12166.3,7698.7,704574 +356.5,11957.6,12168.4,7695.8,718161 +361.1,11961.6,12168.6,7700.2,724247 +365.7,11959.0,12169.5,7697.0,732508 +370.4,11960.4,12169.6,7698.9,742714 +375.0,11961.5,12170.2,7698.9,756337 +379.7,11959.8,12170.2,7697.1,765987 +384.4,11958.8,12170.5,7696.0,773290 +389.0,11970.2,12172.4,7707.1,782400 +393.6,12026.2,12275.1,7763.6,794545 +398.3,12030.1,12275.1,7767.2,801864 +402.9,12033.7,12275.1,7771.7,811512 +407.6,12038.8,12275.1,7775.7,823204 +412.2,12038.1,12275.1,7774.8,831265 +416.9,12037.1,12275.1,7773.7,851185 +421.5,12040.1,12275.1,7777.9,859997 +426.0,12042.9,12275.1,7780.8,868876 +430.7,12042.5,12275.1,7781.0,878974 +435.2,12045.0,12275.1,7783.0,887450 +440.1,12044.3,12275.1,7782.2,899453 +444.8,12046.1,12275.1,7784.7,907481 +449.3,12048.0,12275.1,7786.1,913627 +454.1,12050.0,12275.1,7787.7,924842 +458.6,12050.4,12275.1,7788.5,935530 +463.3,12058.4,12275.1,7796.1,941448 +467.9,12055.4,12275.1,7793.0,950386 +472.4,12057.6,12275.1,7794.9,969578 +477.1,12057.7,12275.1,7794.8,976636 +481.7,12060.5,12275.1,7797.2,983204 +486.3,12059.4,12275.1,7796.2,991314 +491.0,12062.4,12275.1,7799.4,1009992 +495.6,12061.1,12275.1,7798.0,1018470 +500.3,12064.6,12275.1,7801.3,1022753 +504.8,12070.4,12275.1,7807.8,1030962 +509.5,12070.9,12275.1,7807.9,1038826 +514.1,12069.2,12275.1,7807.0,1047636 +518.6,12076.5,12276.8,7814.3,1067544 +523.3,12075.5,12277.0,7813.5,1076151 +527.9,12074.9,12277.0,7813.2,1084072 +532.6,12096.4,12277.0,7835.2,1091142 +537.2,12094.3,12277.0,7834.0,1108746 +541.8,12098.1,12277.0,7836.9,1110785 +546.4,12096.4,12277.0,7835.3,1119469 +551.0,12096.2,12277.0,7835.1,1125868 +555.6,12096.6,12277.0,7835.6,1146684 +560.2,12100.7,12277.0,7838.9,1155119 +564.8,12103.2,12278.2,7841.7,1159675 +569.4,12103.3,12278.4,7840.8,1163773 +574.2,12110.4,12281.6,7848.4,1187199 +578.7,12110.5,12283.3,7848.1,1195426 +583.3,12112.3,12284.5,7849.7,1198875 +588.0,12111.3,12285.4,7847.1,1206755 +592.6,12116.6,12288.0,7854.7,1227135 +597.3,12117.1,12290.2,7854.8,1232616 +602.0,12114.6,12290.2,7852.3,1250996 +606.7,12112.9,12291.2,7851.2,1256970 +611.2,12114.5,12291.6,7852.2,1256970 +615.9,12116.1,12292.0,7853.4,1278340 +620.5,12118.8,12292.6,7856.1,1281954 +625.2,12120.4,12293.6,7858.0,1284860 +629.8,12116.2,12293.6,7853.9,1303976 +634.3,12121.7,12294.1,7859.1,1311296 +638.9,12124.7,12294.9,7862.6,1314068 +643.6,12121.9,12295.5,7859.8,1336334 +648.2,12147.0,12309.4,7884.3,1347236 +652.9,12143.8,12310.6,7881.0,1350255 +657.5,12134.7,12311.5,7871.9,1372507 +662.0,12133.8,12311.5,7870.7,1372507 +666.7,12129.1,12311.5,7865.9,1386434 +671.4,12131.7,12311.5,7869.5,1388280 +676.1,12134.7,12311.5,7872.7,1403892 +680.9,12134.7,12311.5,7871.9,1411833 +685.6,12139.9,12312.4,7877.4,1413960 +690.3,12140.3,12312.4,7877.4,1430330 +695.0,12136.1,12312.4,7874.0,1430330 +699.7,12141.6,12312.4,7879.1,1447655 +704.3,12140.2,12312.7,7878.1,1451809 +709.1,12142.7,12313.2,7880.7,1462429 +713.8,12144.0,12313.8,7881.4,1463919 +718.7,12145.0,12315.3,7882.6,1482045 +723.4,12149.9,12316.2,7887.7,1490016 +728.1,12146.6,12317.4,7884.1,1492260 +732.8,12149.6,12317.9,7887.7,1511936 +737.5,12150.3,12320.3,7889.0,1511936 +742.3,12153.6,12321.3,7891.3,1527236 +747.0,12151.8,12321.5,7890.0,1527236 +751.6,12153.6,12321.8,7892.1,1544755 +756.3,12156.8,12324.7,7894.9,1546895 +760.9,12156.4,12325.5,7894.5,1568889 +765.6,12157.2,12326.6,7895.1,1570012 +770.1,12158.0,12327.4,7895.8,1587958 +774.9,12156.3,12327.6,7894.8,1587958 +779.6,12169.3,12331.4,7907.4,1601103 +784.5,12167.3,12334.6,7905.3,1619477 +789.1,12165.6,12336.1,7903.4,1623892 +793.8,12166.6,12336.7,7904.0,1625506 +798.5,12163.6,12336.7,7901.5,1647526 +803.0,12163.8,12336.7,7901.5,1647526 +807.7,12166.1,12337.0,7904.1,1664484 +812.4,12168.3,12337.7,7906.4,1664484 +817.1,12166.0,12337.7,7904.4,1679022 +821.9,12171.9,12341.5,7909.7,1697098 +826.5,12171.1,12341.9,7908.9,1700114 +831.2,12173.9,12342.1,7911.4,1712948 +835.9,12178.5,12345.0,7919.5,1719731 +840.5,12178.7,12346.1,7919.6,1719731 +845.3,12178.7,12346.4,7919.2,1724727 +850.2,12194.1,12355.4,7934.6,1744445 +854.9,12195.7,12361.5,7935.4,1749102 +859.6,12194.3,12362.4,7934.5,1749102 +864.2,12185.8,12362.4,7926.2,1767364 +868.7,12184.8,12362.4,7924.7,1768694 +873.3,12184.1,12362.4,7923.3,1789974 +878.0,12185.4,12362.4,7924.0,1789974 +882.7,12187.9,12362.4,7926.8,1800063 +887.5,12192.7,12363.8,7931.4,1804585 +892.2,12198.2,12363.9,7936.5,1806539 +897.0,12204.6,12369.0,7942.7,1813407 +901.7,12208.2,12371.1,7946.2,1833795 +906.4,12196.1,12371.4,7934.1,1833795 +911.3,12201.3,12373.1,7939.2,1848106 +916.0,12190.9,12373.1,7929.0,1848106 +920.8,12189.6,12373.1,7927.8,1855991 +925.4,12193.1,12373.1,7931.3,1862061 +930.1,12205.2,12373.1,7943.6,1862061 +935.0,12214.7,12378.4,7953.0,1862061 +939.7,12228.5,12388.6,7966.4,1867293 +944.5,12235.2,12395.9,7972.9,1867293 +949.1,12250.5,12406.1,7987.6,1867293 +953.9,12257.6,12413.3,7994.8,1870345 +958.8,12275.2,12428.9,8012.5,1876039 +963.6,12290.6,12442.2,8027.5,1876039 +968.3,12292.5,12447.8,8030.2,1936067 +972.9,12240.5,12448.1,7977.4,1936067 +977.6,12214.6,12448.1,7952.4,1936067 +982.2,12216.5,12448.1,7954.2,1952512 +986.8,12210.3,12448.1,7948.1,1952512 +991.5,12212.7,12448.1,7950.4,1963270 +996.2,12203.6,12448.1,7941.5,1963270 +1000.8,12204.2,12448.1,7942.6,1972075 +1005.3,12209.2,12448.1,7947.8,1990155 +1010.0,12213.9,12448.1,7953.5,1992275 +1014.6,12217.1,12448.1,7955.4,2012131 +1019.4,12214.2,12448.1,7952.7,2012131 +1023.9,12213.7,12448.1,7951.8,2024300 +1028.6,12213.8,12448.1,7952.5,2024300 +1033.2,12259.2,12448.1,7997.3,2042355 +1037.8,12256.3,12448.1,7994.0,2042355 +1042.5,12260.5,12448.1,7998.3,2051384 +1047.1,12258.8,12448.1,7996.6,2070594 +1051.9,12261.6,12448.1,7998.9,2071732 +1056.5,12265.1,12448.1,8002.6,2082324 +1061.3,12272.9,12448.1,8010.2,2082324 +1065.8,12271.1,12448.1,8008.6,2101126 +1070.5,12270.1,12448.1,8007.4,2118244 +1075.3,12267.7,12448.1,8004.7,2118244 +1080.1,12266.1,12448.1,8003.1,2130596 +1084.7,12270.4,12448.1,8007.6,2130596 +1089.3,12276.3,12448.1,8013.3,2148895 +1094.1,12275.8,12448.1,8012.8,2148895 +1098.8,12275.8,12448.1,8013.6,2162962 +1103.3,12274.6,12448.1,8012.5,2181420 +1108.0,12272.9,12448.1,8010.7,2181420 +1112.6,12271.8,12448.1,8009.7,2194850 +1117.2,12274.4,12448.1,8012.8,2194850 +1121.8,12275.2,12448.1,8014.3,2210293 +1126.5,12276.2,12448.1,8014.8,2223573 +1131.2,12281.4,12448.1,8019.8,2239113 +1135.9,12288.0,12448.1,8026.3,2239113 +1140.3,12283.4,12448.1,8021.7,2253464 +1145.0,12286.2,12448.1,8023.7,2266890 +1149.7,12281.0,12448.1,8017.7,2266890 +1154.4,12280.3,12448.1,8018.1,2281472 +1159.1,12280.6,12448.1,8018.4,2281472 +1163.6,12279.5,12448.1,8017.1,2285825 +1168.3,12280.6,12448.1,8018.0,2299519 +1173.0,12280.8,12448.1,8018.3,2301472 +1177.8,12286.6,12448.1,8023.8,2321892 +1182.3,12289.4,12448.1,8026.4,2321892 +1187.0,12288.3,12448.1,8025.4,2331314 +1191.6,12291.2,12448.1,8027.7,2349934 +1196.2,12289.6,12448.1,8026.4,2349934 +1200.8,12290.1,12448.1,8027.6,2364190 +1205.5,12286.6,12448.1,8023.9,2364190 +1210.0,12291.2,12448.1,8028.2,2379881 +1214.8,12290.7,12448.1,8027.6,2394477 +1219.4,12287.9,12448.1,8024.9,2394477 +1224.1,12293.6,12448.1,8030.0,2408596 +1228.9,12289.8,12448.1,8026.6,2408596 +1233.6,12290.2,12448.1,8026.4,2422315 +1238.4,12289.7,12448.1,8025.3,2432519 +1243.0,12290.1,12448.1,8025.6,2432519 +1247.5,12291.2,12448.1,8026.2,2447393 +1252.2,12293.6,12448.1,8028.6,2462509 +1256.8,12291.5,12448.1,8025.9,2462509 +1261.5,12293.8,12448.1,8028.3,2474886 +1266.0,12293.8,12448.1,8027.4,2489564 +1270.8,12292.9,12448.1,8026.5,2489564 +1275.3,12293.8,12448.1,8027.2,2504887 +1279.8,12293.5,12448.1,8026.7,2504887 +1284.4,12296.1,12448.1,8029.0,2513434 +1289.3,12294.3,12448.1,8027.5,2530988 +1294.0,12294.6,12448.1,8027.6,2530988 +1298.5,12295.0,12448.1,8028.0,2547091 +1303.2,12292.3,12448.1,8025.7,2565579 +1307.9,12289.5,12448.1,8022.9,2565579 +1312.6,12295.2,12448.1,8028.2,2566180 +1317.3,12294.0,12448.1,8027.2,2582810 +1321.9,12293.9,12448.1,8027.2,2582810 +1326.5,12293.4,12448.1,8026.6,2598004 +1331.0,12295.3,12448.1,8028.8,2614844 +1335.7,12295.5,12448.1,8028.8,2614844 +1340.3,12293.6,12448.1,8026.9,2622576 +1345.1,12294.6,12448.1,8027.9,2622576 +1349.7,12298.8,12448.1,8031.9,2635013 +1354.3,12298.1,12448.1,8031.3,2654489 +1359.0,12295.2,12448.1,8028.2,2654489 +1363.7,12293.7,12448.1,8026.4,2667512 +1368.3,12291.1,12448.1,8024.0,2684038 +1373.0,12292.1,12448.1,8025.3,2684038 +1377.7,12293.0,12448.1,8025.9,2694231 +1382.5,12293.6,12448.1,8026.7,2694231 +1387.1,12294.8,12448.1,8027.7,2706994 +1391.6,12295.0,12448.1,8028.1,2727444 +1396.3,12292.8,12448.1,8026.0,2742130 +1400.9,12291.3,12448.1,8024.9,2742130 +1405.6,12294.5,12448.1,8028.1,2750405 +1410.3,12293.2,12448.1,8026.4,2769171 +1414.9,12292.1,12448.1,8025.4,2769171 +1419.6,12290.9,12448.1,8023.7,2774532 +1424.3,12296.2,12448.1,8029.3,2792946 +1429.1,12296.0,12448.1,8029.2,2792946 +1433.8,12292.0,12448.1,8025.0,2797522 +1438.4,12292.3,12448.1,8025.5,2810460 +1443.1,12294.2,12448.1,8027.5,2810460 +1447.8,12301.6,12448.1,8035.2,2826475 +1452.5,12302.3,12448.1,8035.6,2842813 +1457.1,12299.8,12491.5,8033.0,2842813 +1461.8,12302.5,12524.2,8035.5,2854322 +1466.5,12295.3,12524.2,8028.5,2870216 +1471.2,12291.8,12524.2,8024.4,2870216 +1476.0,12297.9,12524.2,8031.1,2875982 +1480.6,12292.9,12524.2,8025.7,2893146 +1485.4,12298.1,12524.2,8031.0,2893146 +1490.1,12296.5,12524.2,8029.2,2901667 +1494.7,12298.8,12524.2,8032.4,2918733 +1499.4,12308.3,12524.2,8041.7,2918733 +1504.1,12303.8,12524.2,8036.9,2927528 +1508.8,12294.0,12524.2,8027.1,2944394 +1513.5,12293.4,12524.2,8026.6,2944394 +1518.2,12294.3,12524.2,8027.1,2955159 +1522.8,12297.4,12524.2,8030.3,2971411 +1527.5,12295.4,12524.2,8028.4,2971411 +1532.2,12298.8,12524.2,8032.3,2986071 +1536.9,12299.5,12524.2,8032.8,3006147 +1541.6,12297.3,12524.2,8030.8,3006147 +1546.2,12297.6,12524.2,8030.3,3008951 +1550.8,12296.9,12524.2,8029.9,3026621 +1555.5,12300.8,12524.2,8033.9,3026621 +1560.1,12298.0,12524.2,8031.2,3038097 +1564.7,12295.7,12524.2,8028.7,3054493 +1569.4,12295.9,12524.2,8028.8,3054493 +1574.0,12297.2,12524.2,8030.3,3059854 +1578.7,12295.6,12524.2,8028.7,3074222 +1583.6,12293.7,12524.2,8027.2,3090918 +1588.3,12294.4,12524.2,8027.8,3090918 +1593.0,12294.3,12524.2,8027.8,3100585 +1597.7,12296.4,12524.2,8029.7,3117125 +1602.4,12295.0,12524.2,8027.9,3117125 +1607.3,12296.3,12524.2,8029.5,3127959 +1611.9,12292.9,12524.2,8025.7,3142187 +1616.5,12295.2,12524.2,8028.3,3142187 +1621.3,12298.1,12524.2,8031.7,3142187 +1626.0,12306.1,12524.2,8039.3,3160758 +1630.8,12302.5,12524.2,8035.1,3178004 +1635.6,12300.1,12524.2,8033.4,3178004 +1640.3,12297.4,12524.2,8030.5,3185337 +1645.0,12296.3,12524.2,8029.3,3203175 +1649.7,12293.8,12524.2,8026.6,3203175 +1654.3,12297.4,12524.2,8030.6,3206132 +1659.0,12295.9,12524.2,8029.1,3225276 +1663.7,12296.1,12524.2,8029.1,3225276 +1668.4,12299.5,12524.2,8032.7,3230133 +1673.0,12301.7,12524.2,8034.6,3248543 +1677.8,12301.7,12524.2,8034.8,3264951 +1682.6,12298.6,12524.2,8031.7,3264951 +1687.3,12297.0,12524.2,8030.3,3278138 +1692.0,12295.5,12524.2,8028.9,3298214 +1696.6,12296.6,12524.2,8030.0,3298214 +1701.1,12296.5,12524.2,8029.3,3302196 +1705.8,12308.2,12524.2,8041.2,3319360 +1710.5,12308.1,12524.2,8041.2,3319360 +1715.1,12303.3,12524.2,8036.4,3328227 +1719.7,12300.9,12524.2,8033.8,3343307 +1724.5,12299.7,12524.2,8032.8,3358289 +1729.1,12297.5,12524.2,8030.4,3358289 +1733.8,12296.9,12524.2,8030.1,3364907 +1738.4,12297.2,12524.2,8029.8,3377091 +1743.1,12297.3,12524.2,8030.6,3377091 +1747.8,12296.6,12524.2,8029.9,3381676 +1752.4,12297.7,12524.2,8030.6,3398916 +1756.9,12298.3,12524.2,8031.5,3413290 +1761.6,12294.8,12524.2,8028.1,3413290 +1766.2,12296.9,12524.2,8030.0,3418019 +1770.8,12297.6,12524.2,8030.8,3432015 +1775.4,12297.5,12524.2,8030.4,3432015 +1780.1,12299.9,12524.2,8033.3,3438878 +1784.7,12301.8,12524.2,8034.6,3457120 +1789.5,12298.8,12524.2,8031.6,3474600 +1794.2,12298.5,12524.2,8031.4,3474600 +1798.9,12293.0,12524.2,8025.8,3474600 +1803.6,12297.6,12524.2,8030.4,3491964 +1808.3,12297.6,12524.2,8030.6,3506564 +1812.8,12301.0,12524.2,8034.1,3506564 +1817.5,12300.0,12524.2,8033.2,3510939 +1822.2,12298.4,12524.2,8031.5,3529555 +1826.7,12296.1,12524.2,8029.3,3529555 +1831.4,12295.6,12524.2,8028.7,3534143 +1836.1,12298.1,12524.2,8031.3,3550685 +1840.7,12296.6,12524.2,8029.5,3569573 +1845.5,12297.5,12524.2,8030.2,3569573 +1850.0,12297.8,12524.2,8031.0,3572357 +1854.8,12297.8,12524.2,8030.2,3589315 +1859.5,12295.4,12524.2,8028.7,3605585 +1864.3,12297.9,12524.2,8031.3,3605585 +1868.9,12301.3,12524.2,8034.5,3612229 +1873.5,12303.2,12524.2,8036.4,3624971 +1878.2,12303.8,12524.2,8036.7,3624971 +1883.0,12301.3,12524.2,8034.1,3631503 +1887.7,12300.8,12524.2,8033.5,3651011 +1892.5,12304.2,12524.2,8037.2,3665147 +1897.2,12300.6,12524.2,8033.7,3679499 +1902.0,12296.0,12524.2,8029.1,3679499 +1906.7,12300.7,12524.2,8034.0,3686236 +1911.4,12300.9,12524.2,8034.2,3701522 +1916.1,12301.5,12524.2,8034.7,3701522 +1920.9,12299.0,12524.2,8032.2,3703724 +1925.5,12296.0,12524.2,8029.1,3722912 +1930.4,12296.8,12524.2,8029.6,3728394 +1935.2,12301.1,12524.2,8033.9,3732738 +1939.9,12389.8,12564.9,8122.8,3753710 +1944.5,12389.9,12564.9,8122.6,3753710 +1949.2,12382.5,12564.9,8115.0,3753710 +1953.8,12379.9,12564.9,8112.5,3769557 +1958.4,12380.1,12564.9,8112.8,3769557 +1963.0,12380.6,12564.9,8113.9,3773868 +1967.8,12383.3,12564.9,8116.1,3790174 +1972.7,12378.5,12564.9,8112.1,3790908 +1977.7,12390.0,12564.9,8122.4,3793998 +1982.6,12407.1,12564.9,8139.4,3816826 +1987.4,12403.5,12564.9,8135.6,3816826 +1992.0,12398.3,12564.9,8130.7,3816826 +1996.7,12385.6,12564.9,8118.0,3836120 +2001.4,12380.9,12564.9,8113.0,3848474 +2006.2,12380.6,12564.9,8112.7,3864336 +2010.7,12382.4,12564.9,8114.6,3864336 +2015.3,12385.5,12564.9,8117.9,3864532 +2019.9,12385.3,12564.9,8118.1,3879600 +2024.6,12383.3,12564.9,8115.5,3897832 +2029.2,12382.3,12564.9,8114.7,3897832 +2033.9,12380.1,12564.9,8112.2,3902707 +2038.5,12382.8,12564.9,8115.3,3917511 +2043.1,12383.0,12564.9,8115.7,3933515 +2047.8,12382.8,12564.9,8115.4,3933515 +2052.5,12381.5,12564.9,8114.0,3936704 +2057.5,12376.8,12564.9,8109.7,3951516 +2062.1,12378.8,12564.9,8111.1,3951516 +2066.8,12380.7,12564.9,8112.9,3952107 +2071.6,12381.2,12564.9,8113.4,3965947 +2076.3,12382.1,12564.9,8114.2,3986641 +2081.0,12385.1,12564.9,8117.7,4003541 +2085.7,12386.8,12564.9,8119.1,4003541 +2090.4,12385.6,12564.9,8118.0,4009813 +2095.1,12383.0,12564.9,8115.6,4026343 +2099.9,12380.9,12564.9,8113.6,4041561 +2104.5,12379.4,12564.9,8112.1,4041561 +2109.2,12379.3,12564.9,8111.8,4042470 +2113.7,12378.0,12564.9,8110.1,4057350 +2118.4,12378.0,12564.9,8110.0,4073198 +2123.1,12380.4,12564.9,8112.4,4073198 +2127.8,12381.1,12564.9,8113.7,4076436 +2132.5,12383.5,12564.9,8115.8,4091880 +2137.2,12380.3,12564.9,8112.5,4107006 +2141.8,12378.9,12564.9,8111.6,4107006 +2146.5,12382.2,12564.9,8115.3,4107006 +2151.2,12382.5,12564.9,8115.3,4122711 +2155.9,12383.2,12564.9,8115.6,4143161 +2160.6,12381.0,12564.9,8113.4,4161815 +2165.2,12381.6,12564.9,8114.1,4161815 +2169.8,12380.0,12564.9,8112.5,4161815 +2174.4,12377.6,12564.9,8110.0,4169467 +2179.0,12378.9,12564.9,8111.4,4187613 +2183.7,12378.7,12564.9,8111.9,4187613 +2188.5,12379.8,12564.9,8112.0,4187613 +2193.2,12378.3,12564.9,8110.9,4191596 +2198.0,12376.8,12564.9,8109.3,4205192 +2202.9,12380.0,12564.9,8112.0,4222904 +2207.7,12380.6,12564.9,8112.7,4222904 +2212.4,12383.7,12564.9,8116.2,4226457 +2217.1,12383.6,12564.9,8115.9,4246031 +2221.8,12382.6,12564.9,8114.7,4260007 +2226.5,12381.8,12564.9,8114.1,4260007 +2231.2,12379.7,12564.9,8111.8,4260007 +2235.9,12380.8,12564.9,8113.1,4275295 +2240.6,12379.1,12564.9,8111.4,4290131 +2245.2,12378.4,12564.9,8110.4,4290131 +2249.9,12377.5,12564.9,8110.3,4290131 +2254.6,12378.0,12564.9,8110.7,4307817 +2259.2,12380.0,12564.9,8112.9,4327477 +2264.0,12378.8,12564.9,8111.2,4344221 +2268.8,12380.8,12564.9,8112.9,4344221 +2273.5,12379.1,12564.9,8111.6,4344221 +2278.2,12380.0,12564.9,8112.3,4357887 +2282.9,12379.6,12564.9,8111.7,4371353 +2287.6,12380.9,12564.9,8113.4,4371353 +2292.1,12379.5,12564.9,8112.0,4371353 +2296.8,12384.1,12564.9,8116.6,4387341 +2301.5,12381.7,12564.9,8113.8,4403443 +2306.1,12379.1,12564.9,8111.5,4415667 +2310.7,12378.3,12564.9,8110.8,4415667 +2315.5,12382.6,12564.9,8115.3,4419238 +2320.2,12381.2,12564.9,8113.8,4433746 +2324.9,12377.8,12564.9,8110.1,4452430 +2329.5,12378.2,12564.9,8110.2,4452430 +2334.3,12379.0,12564.9,8111.2,4452430 +2338.9,12379.2,12564.9,8111.7,4459157 +2343.6,12380.2,12564.9,8112.3,4478285 +2348.1,12384.7,12564.9,8117.2,4493031 +2352.8,12383.7,12564.9,8115.9,4493031 +2357.5,12380.4,12564.9,8112.7,4493031 +2362.2,12380.0,12564.9,8112.2,4500743 +2366.9,12378.4,12564.9,8110.9,4520167 +2371.5,12378.6,12564.9,8111.2,4534157 +2376.2,12378.6,12564.9,8111.3,4534157 +2380.8,12376.0,12564.9,8108.5,4534157 +2385.6,12376.5,12564.9,8108.9,4549112 +2390.2,12375.8,12564.9,8108.2,4565284 +2394.8,12375.3,12564.9,8107.5,4565284 +2399.4,12378.2,12564.9,8111.1,4565284 +2403.9,12377.7,12564.9,8110.1,4576467 +2408.7,12381.0,12564.9,8113.0,4597659 +2413.4,12380.8,12564.9,8112.8,4611885 +2418.1,12378.0,12564.9,8110.1,4611885 +2422.8,12378.9,12564.9,8111.4,4611885 +2427.6,12379.8,12564.9,8112.6,4622628 +2432.1,12378.3,12564.9,8110.6,4637318 +2437.1,12379.6,12564.9,8112.5,4652948 +2441.8,12380.5,12564.9,8113.4,4652948 +2446.7,12378.4,12564.9,8111.0,4652948 +2451.4,12381.5,12564.9,8113.8,4665061 +2456.0,12384.3,12564.9,8116.6,4680049 +2460.8,12380.4,12564.9,8112.7,4680049 +2465.4,12377.9,12564.9,8110.7,4680049 +2470.1,12381.6,12564.9,8115.8,4689518 +2474.8,12396.9,12564.9,8133.0,4714674 +2479.6,12393.6,12566.8,8128.2,4728352 +2484.3,12384.6,12566.8,8119.2,4743244 +2488.9,12380.8,12566.8,8115.7,4743244 +2493.6,12373.7,12566.8,8108.6,4743244 +2498.2,12374.1,12566.8,8108.4,4750157 +2502.9,12379.4,12566.8,8112.5,4769169 +2507.6,12378.8,12566.8,8111.6,4784597 +2512.3,12386.8,12566.8,8120.5,4784597 +2517.1,12436.6,12566.8,8169.7,4786996 +2521.9,12466.9,12571.2,8200.1,4822996 +2526.6,12466.5,12573.1,8199.3,4853108 +2531.4,12460.8,12573.1,8194.0,4881396 +2536.0,12456.8,12573.7,8189.5,4881396 +2540.7,12461.8,12573.7,8194.8,4881396 +2545.5,12462.4,12574.1,8195.4,4900303 +2550.3,12458.9,12574.1,8191.7,4927399 +2555.0,12457.3,12574.1,8189.9,4961581 +2559.8,12463.1,12574.1,8195.9,4985361 +2564.4,12450.9,12574.1,8183.4,4985361 +2569.1,12455.7,12574.1,8188.8,4985361 +2573.8,12451.2,12574.1,8183.8,4985361 +2578.6,12446.4,12574.1,8179.1,4996166 +2583.2,12449.2,12574.1,8181.7,5024152 +2588.0,12435.5,12574.1,8168.3,5043090 +2592.6,12424.2,12574.1,8156.2,5058956 +2597.4,12436.6,12574.1,8169.4,5058956 +2602.1,12443.0,12574.1,8175.3,5058956 +2606.8,12434.1,12574.1,8165.7,5058956 +2611.5,12422.7,12574.1,8154.7,5079456 +2616.3,12424.2,12574.1,8156.6,5097838 +2621.1,12451.2,12574.1,8183.4,5120992 +2626.6,12464.9,12574.6,8196.9,5120992 +2631.3,12454.8,12576.7,8186.7,5120992 +2636.1,12433.4,12576.7,8165.1,5126283 +2640.7,12417.6,12576.7,8149.2,5137319 +2645.3,12412.4,12576.7,8143.8,5153503 +2650.1,12409.5,12576.7,8141.3,5153503 +2654.8,12408.3,12576.7,8140.5,5153503 +2659.5,12406.7,12576.7,8138.4,5153503 +2664.2,12405.7,12576.7,8137.7,5168315 +2669.0,12405.8,12576.7,8137.7,5184367 +2673.6,12407.4,12576.7,8139.6,5184367 +2678.2,12407.2,12576.7,8139.3,5184367 +2682.7,12406.4,12576.7,8138.6,5192784 +2687.2,12403.7,12576.7,8135.7,5209998 +2692.1,12408.2,12576.7,8140.1,5227930 +2696.7,12403.8,12576.7,8135.5,5227930 +2701.4,12405.0,12576.7,8137.3,5227930 +2706.0,12407.4,12576.7,8139.3,5234280 +2710.8,12406.6,12576.7,8138.8,5250102 +2715.5,12404.9,12576.7,8136.9,5266496 +2720.1,12404.9,12576.7,8136.5,5266496 +2724.7,12408.4,12576.7,8140.7,5266496 +2729.3,12410.3,12576.7,8142.6,5276923 +2733.9,12405.8,12576.7,8138.1,5284147 +2738.6,12406.1,12576.7,8138.1,5299521 +2743.2,12405.9,12576.7,8137.6,5316453 +2747.7,12405.9,12576.7,8138.0,5316453 +2752.3,12406.5,12576.7,8138.3,5316453 +2757.0,12415.5,12576.7,8147.7,5324477 +2761.7,12409.9,12576.7,8141.7,5342571 +2766.2,12412.0,12576.7,8143.7,5354305 +2771.0,12409.8,12576.7,8142.2,5354305 +2775.7,12407.4,12576.7,8139.3,5354305 +2780.3,12405.1,12576.7,8137.5,5362755 +2784.9,12404.4,12576.7,8136.8,5378419 +2789.7,12403.6,12576.7,8136.0,5393019 +2794.1,12407.4,12576.7,8139.3,5393019 +2798.8,12409.4,12576.7,8140.9,5393019 +2803.5,12406.9,12576.7,8138.7,5399853 +2808.2,12401.1,12576.7,8132.9,5407047 +2812.9,12401.1,12576.7,8132.8,5423403 +2817.5,12399.0,12576.7,8131.4,5437265 +2822.1,12404.4,12576.7,8136.7,5437265 +2826.7,12404.6,12576.7,8137.0,5437265 +2831.3,12407.0,12576.7,8139.3,5448998 +2836.1,12405.5,12576.7,8137.7,5467712 +2840.9,12405.9,12576.7,8138.4,5481910 +2845.6,12404.1,12576.7,8135.4,5481910 +2850.3,12404.8,12576.7,8136.8,5481910 +2855.0,12404.1,12576.7,8136.1,5486011 +2859.7,12406.2,12576.7,8138.0,5504597 +2864.4,12406.8,12576.7,8139.1,5519295 +2869.0,12405.5,12576.7,8137.5,5519295 +2873.7,12406.1,12576.7,8137.8,5519295 +2878.4,12403.3,12576.7,8135.3,5524809 +2883.0,12402.6,12576.7,8134.9,5538771 +2887.7,12404.1,12576.7,8136.3,5554857 +2892.4,12402.2,12576.7,8134.6,5568849 +2897.1,12404.0,12576.7,8136.3,5568849 +2901.9,12405.6,12576.7,8137.6,5568849 +2906.6,12404.7,12576.7,8136.9,5572436 +2911.3,12404.4,12576.7,8137.0,5590580 +2916.0,12404.1,12576.7,8136.3,5605062 +2920.7,12405.7,12576.7,8137.6,5620264 +2925.3,12403.8,12576.7,8135.7,5620264 +2929.9,12402.5,12576.7,8134.6,5620264 +2934.4,12405.1,12576.7,8137.7,5620264 +2939.1,12408.2,12576.7,8140.5,5638061 +2943.8,12407.7,12576.7,8139.8,5654017 +2948.5,12403.8,12576.7,8135.7,5667967 +2953.1,12407.2,12576.7,8139.0,5667967 +2957.8,12402.7,12576.7,8134.7,5667967 +2962.5,12403.5,12576.7,8135.8,5670179 +2967.1,12400.8,12576.7,8132.7,5687819 +2971.9,12402.6,12576.7,8134.5,5703317 +2976.5,12402.1,12576.7,8134.4,5703317 +2981.1,12403.0,12576.7,8135.3,5703317 +2985.7,12405.8,12576.7,8137.7,5704676 +2990.2,12404.9,12576.7,8137.6,5722178 +2995.0,12407.4,12576.7,8139.5,5736894 +2999.8,12406.5,12576.7,8138.2,5752056 +3004.6,12404.7,12576.7,8136.5,5752056 +3009.3,12401.9,12576.7,8133.8,5752056 +3013.9,12401.9,12576.7,8133.6,5752056 +3018.6,12400.9,12576.7,8132.9,5765673 +3023.4,12403.7,12576.7,8135.6,5781887 +3028.0,12401.3,12576.7,8133.6,5794389 +3032.6,12406.8,12576.7,8139.4,5794389 +3037.2,12403.5,12576.7,8135.8,5794389 +3041.8,12403.7,12576.7,8135.3,5802017 +3046.6,12401.8,12576.7,8133.6,5820685 +3051.3,12401.4,12576.7,8133.5,5831589 +3055.9,12400.9,12576.7,8132.8,5843973 +3060.6,12399.9,12576.7,8132.3,5843973 +3065.2,12401.5,12576.7,8133.7,5843973 +3069.9,12396.7,12576.7,8129.1,5843973 +3074.5,12401.8,12576.7,8133.6,5859973 +3079.1,12401.2,12576.7,8133.2,5876951 +3083.8,12402.5,12576.7,8134.8,5890695 +3088.5,12402.8,12576.7,8134.9,5890695 +3093.4,12403.5,12576.7,8135.4,5890695 +3098.1,12417.4,12576.7,8149.4,5890695 +3102.8,12420.0,12576.7,8151.5,5900508 +3107.4,12411.3,12576.7,8143.3,5914468 +3112.0,12408.5,12576.7,8140.3,5928196 +3116.7,12401.8,12576.7,8133.8,5928196 +3121.4,12401.7,12576.7,8134.1,5928196 +3126.1,12405.4,12576.7,8137.2,5936790 +3130.7,12407.2,12576.7,8138.9,5954312 +3135.3,12403.3,12576.7,8134.7,5970784 +3140.1,12399.7,12576.7,8131.6,5984114 +3144.7,12400.1,12576.7,8132.3,5984114 +3149.6,12402.3,12576.7,8134.6,5984114 +3154.4,12401.4,12576.7,8132.6,5984114 +3159.0,12403.2,12576.7,8134.8,5996887 +3163.6,12401.2,12576.7,8133.3,6014233 +3168.3,12399.8,12576.7,8131.5,6014233 +3172.9,12405.1,12576.7,8137.2,6014233 +3177.5,12401.0,12576.7,8133.2,6017260 +3182.2,12404.0,12576.7,8135.9,6031210 +3186.8,12399.7,12576.7,8131.2,6048552 +3191.4,12400.8,12576.7,8132.4,6065202 +3196.0,12413.4,12576.7,8146.0,6078354 +3200.6,12409.8,12576.7,8142.1,6078354 +3205.3,12406.6,12576.7,8138.6,6078354 +3210.0,12407.1,12576.7,8139.1,6082891 +3214.6,12402.2,12576.7,8133.9,6096601 +3219.2,12404.0,12576.7,8135.7,6113535 +3223.9,12403.6,12576.7,8135.5,6127965 +3228.6,12403.2,12576.7,8135.3,6127965 +3233.3,12402.9,12576.7,8135.4,6127965 +3237.9,12402.9,12576.7,8135.1,6127965 +3242.4,12406.8,12576.7,8138.9,6139085 +3247.1,12405.9,12576.7,8138.5,6156889 +3251.7,12405.3,12576.7,8137.8,6171215 +3256.4,12404.2,12576.7,8136.2,6171215 +3261.1,12404.0,12576.7,8136.2,6171215 +3265.8,12404.6,12576.7,8136.5,6171215 +3270.4,12404.2,12576.7,8136.3,6178664 +3275.1,12399.8,12576.7,8132.1,6194568 +3279.7,12402.0,12576.7,8134.1,6208396 +3284.5,12401.1,12576.7,8133.3,6223142 +3289.4,12402.6,12576.7,8134.9,6223142 +3294.5,12401.7,12576.7,8133.5,6223142 +3299.2,12402.5,12576.7,8134.9,6223142 +3303.8,12404.7,12576.7,8136.6,6239169 +3308.4,12403.2,12576.7,8135.3,6252035 +3313.1,12401.7,12576.7,8133.8,6265777 +3317.8,12401.6,12576.7,8133.7,6265777 +3322.5,12406.4,12576.7,8138.7,6265777 +3327.2,12404.1,12576.7,8136.3,6269893 +3331.8,12405.5,12576.7,8137.6,6286095 +3336.4,12407.4,12576.7,8139.2,6303305 +3341.1,12410.3,12576.7,8142.4,6318133 +3345.7,12414.9,12576.7,8147.1,6318133 +3350.4,12426.9,12576.7,8159.1,6318133 +3355.1,12433.2,12576.7,8165.2,6323153 +3359.8,12421.7,12576.7,8153.6,6341639 +3364.4,12417.9,12576.7,8149.6,6356581 +3369.0,12407.0,12576.7,8138.6,6374705 +3373.7,12411.5,12576.7,8143.4,6389905 +3378.3,12409.9,12576.7,8142.0,6389905 +3383.0,12409.2,12576.7,8141.4,6389905 +3387.7,12410.4,12576.7,8142.4,6389905 +3392.4,12407.6,12576.7,8139.4,6396510 +3397.0,12407.8,12576.7,8139.2,6412262 +3401.6,12405.5,12576.7,8136.9,6428174 +3406.2,12406.3,12576.7,8137.9,6428174 +3410.9,12409.0,12576.7,8141.1,6428174 +3415.4,12408.8,12576.7,8141.2,6428174 +3420.1,12407.5,12576.7,8139.0,6440689 +3424.7,12405.4,12576.7,8137.4,6457599 +3429.2,12404.6,12576.7,8136.8,6471515 +3433.8,12403.6,12576.7,8135.9,6471515 +3438.5,12406.5,12576.7,8139.3,6471515 +3443.2,12405.7,12576.7,8137.7,6471515 +3447.7,12403.4,12576.7,8135.1,6483051 +3452.4,12404.9,12576.7,8136.6,6499293 +3457.0,12403.1,12576.7,8134.5,6513827 +3461.8,12401.4,12576.7,8132.9,6528751 +3466.5,12405.2,12576.7,8137.4,6528751 +3471.3,12405.1,12576.7,8137.2,6528751 +3476.1,12402.5,12576.7,8134.3,6528751 +3480.9,12405.7,12576.7,8137.7,6541868 +3485.6,12408.5,12576.7,8140.5,6560840 +3490.5,12414.9,12576.7,8146.7,6578672 +3495.2,12425.4,12576.7,8157.4,6578672 +3499.8,12414.7,12576.7,8146.8,6578672 +3504.4,12412.1,12576.7,8144.1,6578672 +3509.0,12408.2,12576.7,8140.0,6594008 +3513.6,12411.3,12576.7,8143.2,6606770 +3518.2,12408.9,12576.7,8140.8,6623214 +3523.0,12406.4,12576.7,8138.0,6636330 +3527.6,12403.6,12576.7,8135.8,6636330 +3532.1,12403.8,12576.7,8135.3,6636330 +3536.9,12403.4,12576.7,8135.5,6636330 +3541.4,12406.1,12576.7,8138.4,6645840 +3545.9,12408.4,12576.7,8140.6,6664016 +3550.7,12408.5,12576.7,8139.8,6678214 +3555.4,12405.5,12576.7,8137.1,6690742 +3560.4,12403.0,12576.7,8134.9,6690742 +3565.0,12406.1,12576.7,8138.3,6690742 +3569.6,12403.2,12576.7,8135.8,6690742 +3574.3,12411.9,12576.7,8144.0,6692736 +3578.9,12406.4,12576.7,8137.8,6709374 +3583.6,12407.1,12576.7,8139.2,6722856 +3588.2,12439.4,12577.2,8171.2,6733056 +3592.8,12460.8,12579.0,8192.7,6733056 +3597.5,12499.5,12583.9,8231.4,6733056 +3602.3,12498.3,12585.3,8230.2,6733192 +3606.9,12498.8,12585.8,8230.8,6744880 +3611.6,12496.2,12585.9,8228.0,6760584 +3616.2,12499.2,12586.7,8231.0,6776450 +3620.7,12501.1,12586.7,8233.2,6788200 +3625.4,12497.5,12586.7,8229.5,6788200 +3630.2,12499.0,12586.7,8230.6,6788200 +3634.8,12497.5,12586.8,8229.4,6788200 +3639.4,12490.7,12586.8,8222.4,6788200 +3644.0,12488.8,12586.8,8220.6,6788200 +3648.6,12500.8,12586.8,8232.8,6788200 +3653.2,12547.8,12602.6,8281.6,6788200 +3657.9,12526.8,12656.1,8258.6,6788200 +3662.6,12502.3,12656.1,8234.1,6788200 +3667.2,12480.8,12656.1,8212.6,6788200 +3671.9,12496.8,12656.1,8229.2,6788200 +3676.4,12538.4,12656.1,8271.0,6788200 +3681.2,12526.1,12656.1,8257.9,6788200 +3685.8,12509.4,12656.1,8241.0,6788200 +3690.2,12478.7,12656.1,8210.4,6788200 +3695.0,12491.5,12656.1,8223.3,6788200 +3699.7,12527.9,12656.1,8260.0,6788200 +3704.3,12627.0,12656.1,8358.6,6788200 +3709.0,12521.8,12656.1,8253.4,6788200 +3713.6,12480.5,12656.1,8212.1,6788200 +3718.2,12486.9,12656.1,8218.7,6788200 +3722.7,12520.9,12656.1,8252.9,6788200 +3727.4,12623.4,12656.1,8355.8,6788200 +3732.0,12528.5,12656.1,8260.1,6788200 +3736.7,12478.2,12656.1,8209.8,6788200 +3741.3,12482.5,12656.1,8213.9,6788200 +3745.9,12515.4,12656.1,8247.2,6788200 +3750.4,12605.9,12656.1,8337.9,6788200 +3755.0,12536.2,12656.1,8267.6,6788200 +3759.7,12477.9,12656.1,8209.4,6788200 +3764.5,12477.9,12656.1,8209.4,6788200 +3769.1,12510.3,12656.1,8242.0,6788200 +3773.8,12588.9,12656.1,8320.8,6788200 +3778.4,12538.6,12656.1,8270.0,6788200 +3783.0,12477.8,12656.1,8209.1,6788200 +3787.7,12477.4,12656.1,8208.9,6788200 +3792.3,12502.9,12656.1,8234.8,6788200 +3796.9,12558.0,12656.1,8290.3,6788200 +3801.5,12546.0,12656.1,8277.4,6788200 diff --git a/benchmarks/realtime-data-indexing/results/10gb-processes.jsonl.gz b/benchmarks/realtime-data-indexing/results/10gb-processes.jsonl.gz new file mode 100644 index 0000000..ac30769 Binary files /dev/null and b/benchmarks/realtime-data-indexing/results/10gb-processes.jsonl.gz differ diff --git a/benchmarks/realtime-data-indexing/results/10gb-summary.json b/benchmarks/realtime-data-indexing/results/10gb-summary.json new file mode 100644 index 0000000..d38d31d --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/10gb-summary.json @@ -0,0 +1,250 @@ +{ + "dataset": "10gb", + "corpus_mb": 10000.0, + "documents": 3423359, + "chunks": 6761214, + "max_observed_chunks": 6788200, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count", + "indexing_seconds": 3620.7, + "throughput_mb_per_min": 165.71, + "peak_rss_mb": 12627.0, + "peak_pss_mb": 8358.6, + "peak_processes": [ + { + "pid": 1, + "worker_rank": null, + "VmHWM": 991480, + "VmRSS": 991480, + "Pss": 627866, + "Private_Clean": 300, + "Private_Dirty": 584176, + "mapping_groups": { + "embedding": { + "Pss": 50606, + "private_kb": 19868 + }, + "other": { + "Pss": 576747, + "private_kb": 564608 + } + } + }, + { + "pid": 131, + "worker_rank": null, + "VmHWM": 183284, + "VmRSS": 183284, + "Pss": 107115, + "Private_Clean": 24, + "Private_Dirty": 98496, + "mapping_groups": { + "embedding": { + "Pss": 0, + "private_kb": 0 + }, + "other": { + "Pss": 106905, + "private_kb": 98520 + } + } + }, + { + "pid": 159, + "worker_rank": 0, + "VmHWM": 2537796, + "VmRSS": 2535608, + "Pss": 2038745, + "Private_Clean": 844, + "Private_Dirty": 1974920, + "mapping_groups": { + "embedding": { + "Pss": 64792, + "private_kb": 19900 + }, + "other": { + "Pss": 1973441, + "private_kb": 1955864 + } + } + }, + { + "pid": 160, + "worker_rank": 1, + "VmHWM": 1313820, + "VmRSS": 1311024, + "Pss": 825942, + "Private_Clean": 992, + "Private_Dirty": 763860, + "mapping_groups": { + "embedding": { + "Pss": 64335, + "private_kb": 20536 + }, + "other": { + "Pss": 761095, + "private_kb": 744316 + } + } + }, + { + "pid": 161, + "worker_rank": 2, + "VmHWM": 1320756, + "VmRSS": 1317052, + "Pss": 834678, + "Private_Clean": 7104, + "Private_Dirty": 766412, + "mapping_groups": { + "embedding": { + "Pss": 69382, + "private_kb": 26580 + }, + "other": { + "Pss": 764782, + "private_kb": 746936 + } + } + }, + { + "pid": 162, + "worker_rank": 3, + "VmHWM": 1331092, + "VmRSS": 1328404, + "Pss": 828667, + "Private_Clean": 1108, + "Private_Dirty": 764368, + "mapping_groups": { + "embedding": { + "Pss": 65118, + "private_kb": 20144 + }, + "other": { + "Pss": 763033, + "private_kb": 745332 + } + } + }, + { + "pid": 163, + "worker_rank": 4, + "VmHWM": 1329896, + "VmRSS": 1323816, + "Pss": 826374, + "Private_Clean": 744, + "Private_Dirty": 761748, + "mapping_groups": { + "embedding": { + "Pss": 65322, + "private_kb": 19896 + }, + "other": { + "Pss": 760540, + "private_kb": 742596 + } + } + }, + { + "pid": 164, + "worker_rank": 5, + "VmHWM": 1326200, + "VmRSS": 1322140, + "Pss": 824952, + "Private_Clean": 496, + "Private_Dirty": 761512, + "mapping_groups": { + "embedding": { + "Pss": 65575, + "private_kb": 20044 + }, + "other": { + "Pss": 758868, + "private_kb": 741964 + } + } + }, + { + "pid": 165, + "worker_rank": 6, + "VmHWM": 1314068, + "VmRSS": 1311012, + "Pss": 822568, + "Private_Clean": 52, + "Private_Dirty": 761316, + "mapping_groups": { + "embedding": { + "Pss": 63213, + "private_kb": 19844 + }, + "other": { + "Pss": 758846, + "private_kb": 741524 + } + } + }, + { + "pid": 166, + "worker_rank": 7, + "VmHWM": 1311492, + "VmRSS": 1306240, + "Pss": 822334, + "Private_Clean": 512, + "Private_Dirty": 759796, + "mapping_groups": { + "embedding": { + "Pss": 63997, + "private_kb": 19920 + }, + "other": { + "Pss": 757831, + "private_kb": 740388 + } + } + } + ], + "peak_elapsed_s": 3704.339952558279, + "jemalloc_conf": "background_thread:true", + "omp_threads": 12, + "indexer_cpu_seconds": 33262.7, + "avg_parallelism": 9.2, + "vector_db_disk_mb": 16455.5, + "qdrant_image_id": "sha256:0ad2e23181e5646b286253c04cea23f07040ca313cc377d9fa7b21db184b6406", + "vector_db_mb_per_corpus_mb": 1.65, + "retrieval_accuracy": "17/20", + "embedder": "sentence-transformers/static-retrieval-mrl-en-v1 (local, 256 dims)", + "approx_text_pages": 5242884, + "accuracy_misses": [ + { + "expected": "0000/6000-Tom-Tango.txt", + "query": "Tom Tango: Tom Tango and \"TangoTiger\" are aliases used online by a baseball sabermetrics and ice hockey statistics analy", + "got": [ + "0240/2077-Tango-disambiguation.txt", + "0240/2077-Tango-disambiguation.txt", + "0295/8960-Histoire-du-Tango.txt", + "0150/4615-In-tango.txt", + "0001/0504-List-of-public-domain-tangos.txt" + ] + }, + { + "expected": "0003/6000-Black-capped-fruit-bat.txt", + "query": "Black-capped fruit bat: The black-capped fruit bat (Chironax melanocephalus) is a species of megabat in the monotypic ge", + "got": [ + "0033/2843-Black-winged-little-yellow-bat.txt", + "0083/8702-Wrinkle-faced-bat.txt", + "0046/2629-Antillean-fruit-eating-bat.txt", + "0100/5548-List-of-bats.txt", + "0083/8676-Honduran-fruit-eating-bat.txt" + ] + }, + { + "expected": "0005/6000-Jamie-Mueller.txt", + "query": "Jamie Mueller: Jamie F.", + "got": [ + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0062/3994-Jamie-Marsh.txt" + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/1gb-compose.json b/benchmarks/realtime-data-indexing/results/1gb-compose.json new file mode 100644 index 0000000..8f6bb6f --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/1gb-compose.json @@ -0,0 +1,170 @@ +{ + "name": "serviette-consistency-bench", + "networks": { + "default": { + "name": "serviette-consistency-bench_default", + "ipam": {} + } + }, + "services": { + "indexer": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "indexer", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OMP_NUM_THREADS": "12", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/1gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + }, + "qdrant": { + "command": null, + "entrypoint": null, + "image": "qdrant/qdrant:latest", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 6333, + "published": "8301", + "protocol": "tcp" + } + ] + }, + "server": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "server", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 8000, + "published": "8300", + "protocol": "tcp" + } + ], + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/1gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + } + }, + "x-serviette": { + "build": { + "context": "../..", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "PV3L-4L4N-TF7R-JWM7-LKVK-PX3E-FHTX-KMNC", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "volumes": [ + "~/.cache/huggingface:/hf-cache:ro", + "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/1gb/docs:/data/docs:ro", + "./config.yaml:/bench/config.yaml:ro" + ] + } +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/1gb-expected-output.json b/benchmarks/realtime-data-indexing/results/1gb-expected-output.json new file mode 100644 index 0000000..a6fadf7 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/1gb-expected-output.json @@ -0,0 +1,5 @@ +{ + "documents": 240516, + "exact_chunks": 604299, + "source": "Independent complete frozen-input splitter count; matches the final indexing run" +} diff --git a/benchmarks/realtime-data-indexing/results/1gb-memory.csv b/benchmarks/realtime-data-indexing/results/1gb-memory.csv new file mode 100644 index 0000000..b799636 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/1gb-memory.csv @@ -0,0 +1,129 @@ +elapsed_s,rss_mb,hwm_mb,pss_mb,chunks +11.1,936.3,936.3,756.9,0 +14.4,2809.1,2809.1,1876.6,0 +18.0,6614.2,6614.2,4086.7,0 +21.5,8759.8,8759.8,5449.3,0 +25.0,9110.6,9130.0,5656.5,0 +28.7,10700.9,10702.3,6675.8,11211 +32.3,10781.1,10788.1,6736.0,31009 +36.0,10838.5,10848.9,6747.8,39046 +39.7,10928.1,10940.3,6737.6,46691 +43.3,10934.0,10957.0,6722.4,54695 +47.2,10960.7,10971.9,6731.8,63350 +50.9,10979.7,10986.3,6736.9,71766 +54.5,10985.3,10992.7,6739.6,79250 +58.2,10993.1,11000.6,6750.7,87038 +61.8,11006.9,11012.5,6754.3,94647 +65.5,11019.9,11026.4,6763.9,103401 +69.2,11035.5,11040.2,6773.6,114637 +72.9,11050.3,11053.8,6780.8,120659 +76.6,11070.8,11071.8,6788.7,130324 +80.3,11092.7,11093.8,6800.7,138015 +83.9,11106.1,11112.9,6810.5,146329 +87.5,11112.8,11118.6,6817.1,154898 +91.2,11121.3,11129.2,6825.0,164456 +94.9,11129.6,11133.1,6831.3,173243 +98.5,11137.2,11140.8,6836.4,180483 +102.2,11140.3,11147.0,6837.7,190361 +105.8,11150.0,11152.7,6847.4,198379 +109.5,11153.5,11157.3,6851.0,211457 +113.1,11163.2,11166.5,6853.5,215801 +116.8,11168.5,11171.8,6856.0,222701 +120.5,11170.3,11175.5,6858.2,229904 +124.2,11174.2,11177.1,6861.4,244393 +127.8,11181.2,11183.2,6866.6,251119 +131.5,11184.1,11187.6,6870.4,257622 +135.1,11189.4,11190.4,6874.8,265679 +138.9,11188.9,11192.9,6873.8,274613 +142.5,11189.2,11193.8,6874.4,282968 +146.2,11191.3,11196.8,6877.1,291413 +149.9,11197.3,11200.2,6881.7,303496 +153.5,11210.3,11212.3,6894.4,309678 +157.2,11215.4,11217.9,6896.5,315621 +160.8,11212.3,11220.6,6893.3,322846 +164.4,11213.4,11221.3,6894.5,338520 +168.1,11223.6,11229.0,6904.6,344210 +171.8,11224.5,11232.6,6904.7,349265 +175.5,11228.4,11234.5,6908.3,356450 +179.1,11228.9,11235.7,6908.7,365271 +182.9,11231.5,11236.6,6911.0,376615 +186.6,11235.6,11241.8,6912.0,386953 +190.2,11239.6,11246.3,6916.1,394682 +193.9,11243.0,11247.8,6919.5,399406 +197.5,11246.1,11250.3,6922.2,407253 +201.2,11239.0,11251.0,6915.3,419384 +204.9,11243.9,11251.0,6919.7,426836 +208.6,11245.1,11251.7,6920.7,432591 +212.2,11245.0,11253.7,6920.7,439119 +215.9,11248.9,11254.8,6925.1,448139 +219.6,11250.5,11255.9,6926.7,460629 +223.4,11251.6,11256.7,6928.4,469834 +227.0,11261.5,11263.5,6938.2,474504 +230.7,11259.8,11264.5,6936.8,480360 +234.3,11259.9,11265.5,6936.5,487925 +238.0,11265.2,11266.5,6942.1,501217 +241.7,11259.3,11267.4,6935.7,502886 +245.4,11261.5,11267.6,6937.8,510547 +249.1,11265.3,11268.4,6941.2,529309 +252.9,11263.4,11269.8,6939.2,533727 +257.0,11263.9,11269.9,6939.3,538209 +260.6,11267.5,11270.6,6942.6,542261 +264.3,11271.4,11273.1,6946.5,546211 +268.0,11274.2,11275.1,6948.0,552947 +271.6,11268.6,11276.3,6942.4,567341 +275.3,11272.0,11276.5,6946.4,572794 +278.9,11285.5,11297.6,6960.6,579667 +282.6,11288.7,11300.8,6963.9,585940 +286.2,11290.9,11301.7,6965.3,591544 +289.9,11296.4,11307.9,6970.9,604389 +293.6,11304.9,11313.5,6979.3,604598 +297.3,11306.9,11313.5,6980.8,604598 +301.0,11296.6,11313.5,6970.5,604598 +304.6,11282.5,11313.5,6956.5,604598 +308.2,11281.0,11313.5,6955.1,604598 +311.9,11277.7,11313.5,6951.7,604598 +315.5,11277.5,11313.5,6951.5,604598 +319.2,11277.5,11313.5,6951.5,604598 +322.9,11277.4,11313.5,6951.5,604598 +326.5,11277.6,11313.5,6951.6,604598 +330.2,11274.2,11313.5,6948.2,604598 +333.8,11275.1,11313.5,6949.1,604598 +337.5,11273.8,11313.5,6948.0,604598 +341.2,11273.6,11313.5,6947.7,604598 +344.8,11272.9,11313.5,6946.9,604598 +348.5,11272.9,11313.5,6946.9,604598 +352.1,11274.5,11313.5,6948.6,604598 +355.7,11266.6,11313.5,6940.9,604598 +359.4,11269.0,11313.5,6943.1,604598 +363.0,11274.8,11313.6,6948.9,604598 +366.7,11272.9,11313.6,6947.2,604598 +370.4,11275.1,11313.6,6949.2,604598 +374.0,11277.2,11313.6,6951.3,604598 +377.7,11273.6,11313.6,6947.6,604598 +381.4,11278.6,11313.6,6952.9,604598 +385.1,11275.9,11313.6,6950.1,604598 +388.8,11270.5,11313.6,6944.8,604598 +392.4,11268.5,11313.6,6942.5,604598 +396.2,11276.5,11313.6,6950.6,604598 +399.8,11265.7,11313.6,6939.9,604598 +403.4,11267.0,11313.6,6941.2,604598 +407.1,11278.0,11313.6,6952.1,604598 +410.8,11273.1,11313.6,6947.4,604598 +414.4,11275.2,11313.6,6949.1,604598 +418.1,11276.6,11313.6,6950.7,604598 +421.8,11277.4,11313.6,6951.7,604598 +425.4,11278.5,11313.6,6952.4,604598 +429.0,11274.5,11313.6,6948.5,604598 +432.7,11274.3,11313.6,6948.4,604598 +436.4,11277.1,11313.6,6951.1,604598 +440.0,11276.6,11313.6,6950.7,604598 +443.7,11277.3,11313.6,6951.5,604598 +447.4,11277.0,11313.6,6951.4,604598 +451.1,11277.5,11313.6,6951.5,604598 +454.8,11277.4,11313.6,6951.5,604598 +458.4,11277.5,11313.6,6951.4,604598 +462.0,11277.3,11313.6,6951.4,604598 +465.7,11277.4,11313.6,6951.3,604598 +469.3,11271.8,11313.6,6945.9,604598 +472.9,11271.3,11313.6,6945.4,604598 +476.7,11271.1,11313.6,6945.3,604598 diff --git a/benchmarks/realtime-data-indexing/results/1gb-processes.jsonl.gz b/benchmarks/realtime-data-indexing/results/1gb-processes.jsonl.gz new file mode 100644 index 0000000..b14b80b Binary files /dev/null and b/benchmarks/realtime-data-indexing/results/1gb-processes.jsonl.gz differ diff --git a/benchmarks/realtime-data-indexing/results/1gb-summary.json b/benchmarks/realtime-data-indexing/results/1gb-summary.json new file mode 100644 index 0000000..7778c69 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/1gb-summary.json @@ -0,0 +1,138 @@ +{ + "dataset": "1gb", + "corpus_mb": 1000.0, + "documents": 240516, + "chunks": 604299, + "indexing_seconds": 293.6, + "throughput_mb_per_min": 204.35, + "peak_rss_mb": 11306.9, + "peak_pss_mb": 6980.8, + "peak_processes": [ + { + "pid": 1, + "worker_rank": null, + "VmHWM": 989848, + "VmRSS": 989848, + "Pss": 627136, + "Private_Clean": 212, + "Private_Dirty": 583568 + }, + { + "pid": 144, + "worker_rank": null, + "VmHWM": 183220, + "VmRSS": 183220, + "Pss": 107278, + "Private_Clean": 84, + "Private_Dirty": 98528 + }, + { + "pid": 159, + "worker_rank": 0, + "VmHWM": 1479060, + "VmRSS": 1477968, + "Pss": 979990, + "Private_Clean": 1456, + "Private_Dirty": 915160 + }, + { + "pid": 160, + "worker_rank": 1, + "VmHWM": 1279256, + "VmRSS": 1278216, + "Pss": 777325, + "Private_Clean": 108, + "Private_Dirty": 714564 + }, + { + "pid": 161, + "worker_rank": 2, + "VmHWM": 1278908, + "VmRSS": 1277308, + "Pss": 778555, + "Private_Clean": 1452, + "Private_Dirty": 714220 + }, + { + "pid": 162, + "worker_rank": 3, + "VmHWM": 1272260, + "VmRSS": 1271864, + "Pss": 772901, + "Private_Clean": 1088, + "Private_Dirty": 708544 + }, + { + "pid": 163, + "worker_rank": 4, + "VmHWM": 1279164, + "VmRSS": 1278276, + "Pss": 778984, + "Private_Clean": 540, + "Private_Dirty": 715968 + }, + { + "pid": 164, + "worker_rank": 5, + "VmHWM": 1273708, + "VmRSS": 1273036, + "Pss": 773988, + "Private_Clean": 316, + "Private_Dirty": 710180 + }, + { + "pid": 165, + "worker_rank": 6, + "VmHWM": 1271656, + "VmRSS": 1271300, + "Pss": 772349, + "Private_Clean": 1056, + "Private_Dirty": 708356 + }, + { + "pid": 166, + "worker_rank": 7, + "VmHWM": 1277992, + "VmRSS": 1277204, + "Pss": 779792, + "Private_Clean": 740, + "Private_Dirty": 716088 + } + ], + "peak_elapsed_s": 297.30629689991474, + "jemalloc_conf": "background_thread:true", + "omp_threads": 12, + "indexer_cpu_seconds": 3078.4, + "avg_parallelism": 10.5, + "vector_db_disk_mb": 1890.3, + "vector_db_mb_per_corpus_mb": 1.89, + "retrieval_accuracy": "18/20", + "embedder": "sentence-transformers/static-retrieval-mrl-en-v1 (local, 256 dims)", + "approx_text_pages": 524288, + "accuracy_misses": [ + { + "expected": "0004/4000-Fat-tailed-distribution.txt", + "query": "Fat-tailed distribution: A fat-tailed distribution is a probability distribution that exhibits a large skewness or kurto", + "got": [ + "0002/1834-Overdispersion.txt", + "0000/2592-Cumulative-distribution-function.txt", + "0019/3908-Von-Mises-Fisher-distribution.txt", + "0000/1640-Binomial-distribution.txt", + "0000/2592-Cumulative-distribution-function.txt" + ] + }, + { + "expected": "0005/6000-Jamie-Mueller.txt", + "query": "Jamie Mueller: Jamie F.", + "got": [ + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt" + ] + } + ], + "max_observed_chunks": 604598, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count" +} diff --git a/benchmarks/realtime-data-indexing/results/30gb-compose.json b/benchmarks/realtime-data-indexing/results/30gb-compose.json new file mode 100644 index 0000000..22353f0 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/30gb-compose.json @@ -0,0 +1,170 @@ +{ + "name": "serviette-consistency-bench", + "networks": { + "default": { + "name": "serviette-consistency-bench_default", + "ipam": {} + } + }, + "services": { + "indexer": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "indexer", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OMP_NUM_THREADS": "12", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/30gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + }, + "qdrant": { + "command": null, + "entrypoint": null, + "image": "qdrant/qdrant:latest", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 6333, + "published": "8301", + "protocol": "tcp" + } + ] + }, + "server": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "server", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 8000, + "published": "8300", + "protocol": "tcp" + } + ], + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/30gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + } + }, + "x-serviette": { + "build": { + "context": "../..", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "PV3L-4L4N-TF7R-JWM7-LKVK-PX3E-FHTX-KMNC", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "volumes": [ + "~/.cache/huggingface:/hf-cache:ro", + "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/30gb/docs:/data/docs:ro", + "./config.yaml:/bench/config.yaml:ro" + ] + } +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/30gb-expected-output.json b/benchmarks/realtime-data-indexing/results/30gb-expected-output.json new file mode 100644 index 0000000..582c294 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/30gb-expected-output.json @@ -0,0 +1,8 @@ +{ + "documents": 9202620, + "exact_chunks": 20933962, + "corpus_bytes": 31457292331, + "dataset": "30gb", + "elapsed_seconds": 862.2, + "source": "Frozen corpus files; UTF-8 decode and actual build_xpack_splitter, token_count=512; executed in immutable serviette-bench image" +} diff --git a/benchmarks/realtime-data-indexing/results/30gb-memory.csv b/benchmarks/realtime-data-indexing/results/30gb-memory.csv new file mode 100644 index 0000000..563419a --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/30gb-memory.csv @@ -0,0 +1,2309 @@ +elapsed_s,rss_mb,hwm_mb,pss_mb,chunks +12.4,1947.2,1947.2,1287.7,0 +16.4,5924.5,5924.5,3531.2,0 +20.6,8300.0,8300.0,5086.9,0 +24.8,9024.2,9024.2,5595.5,0 +29.1,9141.6,9182.3,5712.7,0 +33.4,9265.0,9265.0,5835.1,0 +37.7,9392.2,9412.2,5961.1,0 +42.0,9518.6,9520.8,6086.8,0 +46.3,9656.7,9659.1,6223.8,0 +50.7,9832.8,9870.5,6401.5,0 +55.2,10222.1,10224.4,6805.2,0 +59.6,10462.8,10465.2,7031.0,0 +63.9,10783.3,11065.8,7351.5,0 +68.3,11139.4,11141.8,7707.7,0 +72.8,11464.9,12392.8,8034.2,0 +77.3,11350.7,13392.9,7898.4,0 +82.1,13011.4,14847.8,8928.2,20718 +86.8,13079.5,14919.6,8961.7,36389 +91.6,13124.4,14961.1,8941.5,49019 +96.3,13170.3,14993.7,8946.1,58848 +101.0,13201.6,15017.0,8948.3,69565 +105.8,13219.0,15029.4,8952.4,79810 +110.5,13241.5,15043.7,8965.2,89771 +115.3,13268.6,15061.9,8980.3,102731 +120.0,13284.2,15081.1,8985.1,114516 +124.8,13299.1,15091.8,8994.3,123900 +129.6,13315.1,15100.2,9008.5,134838 +134.3,13341.6,15123.6,9029.0,147053 +139.1,13348.7,15134.5,9032.2,157359 +143.8,13359.0,15139.6,9042.2,169413 +148.7,13370.2,15144.2,9051.7,181616 +153.4,13372.3,15149.7,9050.8,191185 +158.0,13383.5,15156.1,9059.4,201788 +162.8,13381.9,15157.5,9056.0,211274 +167.4,13384.8,15158.8,9059.3,221701 +172.1,13390.8,15164.9,9065.2,233349 +176.9,13399.8,15168.9,9075.5,245749 +181.5,13402.8,15174.4,9079.1,256481 +186.3,13405.3,15175.5,9079.9,266129 +191.0,13404.2,15176.6,9079.5,275866 +195.8,13415.1,15180.6,9090.0,290080 +200.3,13412.9,15181.7,9088.1,297976 +205.1,13423.9,15182.8,9099.5,308553 +209.9,13422.9,15184.6,9098.8,321317 +214.6,13431.2,15189.9,9106.3,330584 +219.5,13436.6,15195.9,9111.0,344843 +224.2,13448.6,15203.3,9122.2,357156 +228.9,13448.4,15208.0,9122.1,371906 +233.5,13449.5,15210.9,9122.3,380820 +238.4,13453.7,15212.5,9126.8,387670 +243.2,13454.9,15215.0,9128.1,397915 +248.0,13451.0,15215.5,9123.9,407807 +252.6,13453.8,15215.9,9126.6,418125 +257.2,13458.8,15217.1,9131.5,429056 +262.1,13462.2,15220.9,9135.1,440859 +266.8,13466.1,15221.8,9137.6,450678 +271.4,13474.2,15224.3,9146.6,460379 +276.0,13473.3,15226.2,9146.8,470041 +280.6,13472.2,15227.6,9145.5,480025 +285.4,13477.4,15228.5,9150.6,491228 +290.0,13476.7,15230.6,9151.1,503166 +294.7,13477.5,15233.0,9151.1,515360 +299.3,13482.9,15234.6,9156.6,526673 +304.1,13479.8,15235.4,9153.4,540089 +308.8,13486.1,15236.2,9159.1,549369 +313.6,13482.0,15237.8,9155.0,559636 +318.3,13485.5,15237.8,9158.5,566057 +323.0,13484.8,15238.4,9157.6,579260 +327.7,13505.9,15243.9,9178.3,592677 +332.2,13513.3,15254.0,9185.6,601623 +336.9,13512.0,15254.7,9184.7,608152 +341.7,13514.5,15255.5,9186.7,620775 +346.6,13520.5,15258.9,9192.7,632442 +351.4,13520.6,15260.6,9192.4,641323 +356.0,13518.5,15261.3,9189.7,654532 +360.7,13519.1,15262.1,9190.8,666195 +365.5,13517.8,15262.8,9189.8,676089 +370.2,13520.0,15263.5,9192.1,689785 +375.1,13523.2,15263.9,9195.3,699720 +379.7,13524.5,15265.5,9197.3,709356 +384.6,13524.7,15265.5,9197.3,723096 +389.4,13526.2,15265.7,9198.8,732699 +394.1,13529.3,15266.2,9201.2,741401 +398.8,13528.5,15266.7,9201.0,751236 +403.7,13528.8,15267.5,9201.6,762812 +408.5,13530.8,15269.1,9202.5,772862 +413.2,13529.9,15269.7,9202.2,783132 +417.9,13573.4,15339.8,9245.2,790728 +422.6,13592.9,15367.0,9265.3,799612 +427.5,13595.7,15367.0,9268.2,809320 +432.2,13596.2,15367.0,9268.8,817055 +436.9,13597.6,15367.0,9270.5,827285 +441.7,13599.2,15367.0,9272.5,839098 +446.5,13605.3,15367.0,9277.9,852450 +451.3,13610.4,15367.0,9282.6,859939 +456.1,13609.4,15367.0,9282.3,868081 +461.0,13612.8,15367.0,9284.5,878321 +465.7,13613.8,15367.0,9285.5,886667 +470.5,13612.8,15367.0,9284.8,898439 +475.1,13615.5,15367.0,9287.3,906962 +480.0,13611.3,15367.0,9282.9,918009 +484.8,13612.7,15367.0,9284.5,935111 +489.7,13612.8,15367.0,9284.5,941761 +494.4,13621.3,15367.6,9293.0,948168 +499.1,13621.8,15367.7,9294.0,956550 +503.9,13624.4,15367.7,9297.0,965380 +508.9,13622.6,15367.7,9294.6,975856 +513.9,13622.8,15367.7,9294.3,994592 +518.5,13624.9,15367.7,9296.8,1001440 +523.3,13624.4,15367.7,9295.9,1014483 +528.1,13628.6,15367.7,9299.8,1014483 +533.2,13636.1,15367.7,9308.2,1023825 +537.9,13635.5,15367.7,9307.4,1045715 +542.8,13629.8,15367.7,9301.2,1050652 +547.6,13629.6,15367.8,9301.8,1061878 +552.7,13627.6,15367.9,9299.7,1071535 +557.5,13632.2,15368.8,9304.6,1072987 +562.2,13635.1,15368.8,9307.0,1091756 +566.9,13651.7,15368.8,9323.9,1100548 +571.7,13658.3,15368.8,9330.8,1108671 +576.5,13660.1,15370.4,9332.5,1113912 +581.4,13661.2,15370.9,9333.7,1135596 +586.3,13659.6,15370.9,9332.1,1143322 +591.1,13659.6,15370.9,9332.3,1143322 +596.1,13667.6,15373.6,9339.3,1169301 +600.9,13667.4,15375.0,9339.6,1175302 +605.8,13672.7,15378.9,9345.0,1181264 +610.5,13666.7,15379.6,9339.0,1187181 +615.3,13669.6,15380.0,9341.9,1200425 +620.3,13668.5,15381.3,9340.8,1209682 +625.2,13670.7,15381.3,9343.5,1228198 +629.8,13673.9,15382.3,9346.1,1233274 +634.7,13678.1,15382.8,9349.9,1241735 +639.4,13673.6,15383.9,9345.8,1248768 +644.0,13674.0,15383.9,9346.5,1267760 +649.0,13673.7,15383.9,9346.4,1273705 +653.7,13677.9,15383.9,9350.7,1278729 +658.7,13672.0,15383.9,9345.3,1292705 +663.5,13682.7,15387.9,9355.5,1298155 +668.3,13687.0,15390.7,9359.2,1320823 +673.1,13687.6,15392.7,9358.4,1323335 +677.8,13686.4,15395.0,9357.1,1327351 +682.7,13704.4,15404.7,9375.1,1338325 +687.6,13697.6,15405.5,9368.1,1361377 +692.3,13692.5,15405.6,9363.2,1361377 +697.1,13700.6,15406.9,9371.4,1371970 +701.8,13700.5,15407.0,9372.0,1375905 +706.6,13695.6,15407.0,9366.8,1396183 +711.6,13694.1,15407.0,9365.3,1396183 +716.9,13698.1,15407.0,9370.0,1404024 +721.7,13703.9,15407.6,9375.5,1417941 +726.3,13704.3,15408.5,9376.1,1423947 +731.3,13699.8,15408.6,9372.3,1430683 +736.1,13699.9,15408.8,9372.8,1449061 +740.7,13695.5,15408.9,9368.3,1449061 +745.5,13697.7,15408.9,9369.7,1449061 +750.5,13706.3,15409.0,9378.3,1453856 +755.3,13723.6,15419.0,9395.6,1453856 +760.3,13730.5,15426.6,9402.6,1453856 +765.1,13752.5,15445.4,9424.4,1459659 +769.8,13766.8,15458.5,9439.2,1465939 +774.6,13777.0,15467.1,9449.0,1465939 +779.4,13797.1,15485.4,9469.7,1470762 +784.3,13809.4,15494.5,9481.7,1476618 +789.2,13814.9,15502.4,9486.6,1476618 +794.1,13822.9,15509.1,9494.6,1548661 +799.0,13758.6,15510.3,9430.5,1548661 +803.7,13728.6,15510.3,9400.1,1548661 +808.5,13724.9,15510.3,9396.4,1548661 +813.2,13713.7,15510.3,9385.1,1563382 +818.1,13716.2,15510.3,9387.0,1563382 +822.9,13713.7,15510.3,9385.3,1587171 +827.6,13715.9,15510.3,9387.4,1588303 +832.4,13718.8,15510.3,9390.9,1610353 +837.3,13723.7,15510.3,9396.0,1615280 +842.0,13722.8,15510.3,9394.7,1636326 +846.9,13724.8,15510.3,9396.6,1638135 +851.6,13722.9,15510.3,9395.6,1640083 +856.4,13722.3,15510.3,9395.4,1660647 +861.3,13722.4,15510.3,9395.6,1664895 +866.0,13725.0,15510.3,9397.1,1684177 +870.7,13723.9,15510.3,9397.2,1686582 +875.4,13726.7,15510.3,9399.6,1704866 +880.2,13726.2,15510.3,9398.7,1708906 +885.0,13739.0,15510.3,9414.2,1729400 +889.6,13739.9,15510.3,9414.5,1729400 +894.3,13740.2,15510.3,9415.4,1746597 +899.0,13745.0,15510.3,9417.8,1748209 +903.9,13744.0,15510.3,9416.9,1767587 +908.6,13747.8,15510.3,9419.8,1767587 +913.2,13747.3,15510.3,9419.8,1784800 +918.1,13747.3,15510.3,9418.9,1789363 +922.7,13749.5,15510.3,9421.5,1810811 +927.4,13750.5,15510.3,9422.4,1810811 +932.0,13747.6,15510.3,9419.6,1810811 +936.8,13748.8,15510.3,9420.7,1834440 +941.5,13753.0,15510.3,9425.4,1834440 +946.1,13750.8,15510.3,9423.5,1852752 +951.0,13747.4,15510.3,9420.4,1866942 +955.8,13748.5,15510.3,9421.5,1867080 +960.5,13750.0,15510.3,9422.1,1886854 +965.2,13751.6,15510.3,9423.5,1886854 +970.0,13748.9,15510.3,9421.4,1902519 +974.7,13756.9,15510.3,9429.3,1902519 +979.4,13753.0,15510.3,9424.7,1918930 +984.1,13756.8,15510.3,9428.4,1918930 +988.8,13759.2,15510.3,9431.0,1934975 +993.5,13757.0,15510.3,9428.8,1937783 +998.2,13759.4,15510.3,9431.6,1955829 +1003.0,13761.8,15510.3,9434.3,1956597 +1007.6,13759.1,15510.3,9432.0,1972997 +1012.4,13759.5,15510.3,9431.4,1972997 +1017.2,13766.4,15510.3,9439.1,1993161 +1022.2,13772.8,15510.3,9445.2,1993161 +1027.0,13772.4,15510.3,9445.5,2010832 +1031.6,13769.1,15510.3,9441.5,2012217 +1036.4,13771.2,15510.3,9443.7,2031837 +1041.1,13769.4,15510.3,9441.5,2031837 +1045.8,13812.6,15510.3,9485.0,2049993 +1050.7,13814.2,15510.3,9486.9,2069767 +1055.5,13814.7,15510.3,9487.2,2069767 +1060.4,13811.8,15510.3,9483.8,2080150 +1065.2,13815.5,15510.3,9487.7,2080150 +1070.0,13815.0,15510.3,9488.0,2099391 +1074.9,13819.0,15510.3,9491.7,2099391 +1079.7,13817.1,15510.3,9489.8,2118559 +1084.5,13820.6,15510.3,9493.5,2118559 +1089.3,13822.1,15510.3,9493.8,2140730 +1094.1,13821.8,15510.3,9494.0,2140730 +1098.7,13825.2,15510.3,9497.1,2159369 +1103.4,13824.0,15510.3,9495.9,2159369 +1108.3,13828.9,15510.3,9501.2,2179255 +1113.2,13827.6,15510.3,9500.2,2179255 +1117.9,13829.0,15510.3,9501.7,2200162 +1122.9,13827.5,15510.3,9500.6,2221020 +1127.6,13829.2,15510.3,9502.2,2221020 +1132.2,13834.7,15510.7,9508.1,2236315 +1137.0,13834.9,15511.2,9507.5,2236315 +1141.8,13834.3,15511.5,9507.0,2254108 +1146.5,13837.5,15511.5,9510.4,2254108 +1151.2,13835.0,15511.5,9507.5,2271710 +1156.1,13834.4,15511.5,9506.6,2273632 +1160.8,13832.3,15511.5,9504.3,2273632 +1165.4,13835.3,15511.5,9506.4,2288006 +1169.9,13834.8,15512.1,9506.6,2304262 +1174.6,13832.7,15512.1,9504.7,2304262 +1179.3,13840.3,15512.6,9512.9,2322345 +1184.0,13841.7,15512.6,9514.4,2322345 +1188.6,13839.2,15512.7,9512.3,2337543 +1193.2,13842.3,15512.7,9513.4,2337543 +1197.9,13842.6,15513.4,9514.8,2349166 +1202.6,13840.2,15513.5,9512.1,2365928 +1207.3,13840.4,15513.5,9512.2,2365928 +1212.1,13842.3,15513.5,9514.3,2382789 +1216.7,13843.0,15513.5,9515.4,2382789 +1221.6,13842.5,15513.5,9514.5,2396794 +1226.3,13844.2,15513.5,9516.9,2415512 +1231.1,13842.7,15513.5,9514.3,2415512 +1235.9,13845.1,15513.5,9517.2,2421699 +1240.7,13842.1,15513.5,9513.9,2441319 +1245.5,13847.2,15513.5,9518.0,2441319 +1250.2,13848.5,15514.4,9519.0,2453304 +1254.9,13850.5,15514.7,9520.6,2453304 +1259.6,13850.5,15515.0,9519.6,2469795 +1264.5,13853.3,15515.1,9522.2,2490141 +1269.3,13853.0,15515.2,9521.4,2490141 +1274.0,13851.1,15515.6,9520.1,2499850 +1278.9,13850.4,15516.0,9518.9,2520148 +1283.7,13850.4,15516.2,9518.1,2520148 +1288.4,13857.7,15517.7,9525.8,2539407 +1293.2,13855.7,15517.7,9523.4,2539407 +1298.0,13855.6,15517.8,9523.7,2552428 +1302.9,13850.1,15517.8,9517.7,2570272 +1307.8,13853.6,15517.8,9521.5,2570272 +1312.6,13852.6,15519.3,9520.3,2585723 +1317.5,13856.5,15519.6,9523.8,2585723 +1322.3,13854.2,15519.6,9522.0,2601803 +1327.3,13853.1,15519.6,9520.9,2621771 +1332.1,13849.3,15519.6,9517.0,2621771 +1336.9,13864.5,15523.3,9531.8,2641605 +1341.6,13860.1,15523.6,9527.0,2641605 +1346.4,13859.8,15523.6,9526.7,2657209 +1351.1,13850.1,15523.6,9517.5,2676919 +1355.8,13853.5,15523.6,9520.8,2676919 +1360.6,13850.2,15523.6,9517.8,2688184 +1365.4,13850.3,15523.6,9517.6,2688184 +1370.1,13848.9,15523.6,9516.4,2704074 +1375.0,13850.1,15523.6,9517.5,2717542 +1379.6,13852.4,15523.6,9519.6,2717542 +1384.4,13852.8,15523.6,9520.3,2728253 +1389.2,13848.7,15523.6,9515.9,2747937 +1394.0,13848.2,15523.6,9515.5,2747937 +1398.7,13848.8,15523.6,9515.9,2759430 +1403.7,13849.3,15523.6,9516.6,2776972 +1408.5,13855.1,15523.6,9522.0,2776972 +1413.3,13853.4,15523.7,9520.7,2784518 +1418.1,13851.0,15524.3,9518.5,2800472 +1422.8,13849.9,15524.3,9517.3,2800472 +1427.5,13849.3,15524.3,9516.2,2815301 +1432.3,13847.1,15524.3,9514.0,2825359 +1437.0,13854.7,15524.3,9522.0,2825359 +1441.8,13857.5,15536.5,9524.9,2838486 +1446.6,13858.2,15575.9,9525.9,2838486 +1451.4,13853.9,15614.9,9521.0,2851150 +1456.2,13850.8,15614.9,9518.0,2865906 +1460.9,13849.1,15614.9,9516.5,2865906 +1465.7,13849.4,15614.9,9516.3,2879229 +1470.4,13851.3,15614.9,9518.2,2895939 +1475.1,13853.2,15614.9,9520.4,2895939 +1479.9,13851.8,15614.9,9519.0,2916575 +1484.6,13863.6,15614.9,9530.8,2934375 +1489.3,13859.1,15614.9,9526.9,2934375 +1494.0,13851.1,15614.9,9518.3,2939239 +1498.9,13852.3,15614.9,9519.0,2956839 +1503.7,13849.5,15614.9,9516.6,2956839 +1508.5,13856.5,15614.9,9524.1,2966757 +1513.4,13856.5,15614.9,9524.1,2986087 +1518.0,13861.6,15614.9,9528.6,2986087 +1522.8,13857.5,15614.9,9524.2,2997900 +1527.4,13855.1,15614.9,9522.0,3014162 +1532.3,13849.6,15614.9,9516.9,3014162 +1537.1,13853.7,15614.9,9521.1,3023391 +1541.9,13859.4,15614.9,9527.0,3041141 +1546.7,13859.4,15614.9,9526.8,3041141 +1551.4,13855.0,15614.9,9521.8,3052370 +1556.3,13854.2,15614.9,9520.9,3067482 +1561.1,13852.0,15614.9,9519.0,3067482 +1565.9,13851.0,15614.9,9517.9,3077229 +1570.8,13849.3,15614.9,9516.5,3095639 +1575.6,13852.6,15614.9,9519.6,3095639 +1580.3,13849.8,15614.9,9517.3,3109453 +1585.4,13851.4,15614.9,9518.8,3131187 +1590.2,13853.1,15614.9,9520.2,3131187 +1594.9,13853.8,15614.9,9521.0,3142700 +1599.6,13852.4,15614.9,9519.2,3157928 +1604.3,13853.8,15614.9,9521.3,3157928 +1608.9,13854.1,15614.9,9521.2,3170268 +1614.0,13852.2,15614.9,9519.4,3188670 +1618.7,13851.2,15614.9,9518.5,3188670 +1623.5,13849.9,15614.9,9517.0,3198749 +1628.3,13850.7,15614.9,9518.0,3214125 +1633.1,13850.0,15614.9,9517.0,3214125 +1637.9,13850.1,15614.9,9517.2,3224771 +1642.7,13851.5,15614.9,9518.4,3240859 +1647.5,13852.6,15614.9,9520.1,3240859 +1652.4,13852.3,15614.9,9519.4,3248989 +1657.2,13850.8,15614.9,9517.6,3265583 +1662.0,13851.2,15614.9,9518.2,3265583 +1667.0,13849.8,15614.9,9516.6,3270728 +1671.8,13853.5,15614.9,9521.0,3286602 +1676.9,13857.5,15614.9,9524.7,3302166 +1681.7,13854.0,15614.9,9520.9,3302166 +1686.4,13857.0,15614.9,9524.1,3303588 +1691.2,13856.7,15614.9,9523.8,3319102 +1696.0,13856.0,15614.9,9523.6,3334438 +1700.9,13853.6,15614.9,9521.2,3334438 +1705.7,13854.4,15614.9,9521.3,3336420 +1710.4,13852.0,15614.9,9518.7,3353776 +1715.3,13851.4,15614.9,9519.0,3368354 +1719.9,13851.8,15614.9,9519.0,3368354 +1724.8,13850.7,15614.9,9517.7,3369106 +1729.6,13849.9,15614.9,9516.8,3386278 +1734.5,13848.3,15614.9,9515.6,3399790 +1739.3,13849.8,15614.9,9517.1,3399790 +1744.2,13851.6,15614.9,9518.8,3401302 +1749.1,13850.5,15614.9,9518.1,3419236 +1754.3,13851.4,15614.9,9519.0,3433724 +1759.1,13850.6,15614.9,9517.5,3433724 +1764.0,13850.6,15614.9,9517.9,3435647 +1769.1,13853.4,15614.9,9520.4,3454859 +1774.0,13852.4,15614.9,9519.8,3454859 +1778.9,13852.5,15614.9,9519.8,3454859 +1783.9,13850.9,15614.9,9518.4,3472418 +1788.7,13852.5,15614.9,9519.5,3489862 +1793.7,13852.5,15614.9,9519.7,3502742 +1798.5,13855.2,15614.9,9522.0,3502742 +1803.3,13853.0,15614.9,9520.1,3502742 +1808.1,13854.6,15614.9,9522.0,3515943 +1813.0,13850.3,15614.9,9517.5,3526139 +1817.9,13850.2,15614.9,9517.1,3526139 +1822.7,13852.0,15614.9,9519.0,3528727 +1827.7,13852.2,15614.9,9519.1,3543935 +1832.6,13854.0,15614.9,9520.8,3558809 +1837.5,13853.5,15614.9,9521.3,3558809 +1842.3,13853.5,15614.9,9520.8,3560511 +1847.2,13854.1,15614.9,9520.8,3580413 +1852.1,13848.9,15614.9,9516.3,3594527 +1856.9,13849.9,15614.9,9517.3,3594527 +1861.8,13851.9,15614.9,9519.1,3594527 +1866.7,13853.8,15614.9,9521.1,3604826 +1871.6,13861.1,15614.9,9528.4,3623046 +1876.4,13860.2,15614.9,9527.1,3637250 +1881.3,13855.3,15614.9,9522.8,3637250 +1886.2,13856.6,15614.9,9524.2,3637250 +1891.0,13854.7,15614.9,9522.1,3652667 +1895.9,13855.3,15614.9,9522.4,3664801 +1900.8,13853.1,15614.9,9519.6,3664801 +1905.7,13856.6,15614.9,9523.9,3664801 +1910.6,13856.4,15614.9,9523.9,3683963 +1915.6,13856.0,15614.9,9523.3,3695373 +1920.6,13853.8,15614.9,9520.8,3695373 +1925.5,13854.1,15614.9,9521.3,3695373 +1930.2,13854.5,15614.9,9522.0,3708144 +1935.1,13851.8,15614.9,9519.3,3722680 +1940.0,13854.4,15614.9,9521.3,3722680 +1944.7,13849.9,15614.9,9516.6,3726297 +1949.6,13934.1,15614.9,9601.1,3740963 +1954.5,13935.2,15614.9,9602.2,3756479 +1959.3,13933.6,15614.9,9600.7,3756479 +1964.2,13932.5,15614.9,9600.2,3758225 +1969.1,13931.9,15614.9,9599.3,3774635 +1974.1,13934.9,15614.9,9602.5,3791087 +1979.0,13935.6,15614.9,9602.8,3791087 +1983.8,13935.2,15614.9,9602.1,3791087 +1988.6,13935.4,15614.9,9602.4,3807291 +1993.4,13931.9,15614.9,9599.1,3822381 +1998.2,13936.6,15614.9,9603.7,3822381 +2003.1,13936.1,15614.9,9603.3,3822381 +2008.0,13937.9,15614.9,9605.2,3837136 +2012.8,13936.3,15614.9,9604.0,3851508 +2017.7,13938.0,15614.9,9605.2,3861974 +2022.6,13938.7,15614.9,9606.1,3861974 +2027.5,13938.2,15614.9,9605.6,3862910 +2032.4,13938.7,15614.9,9605.9,3879222 +2037.2,13934.3,15614.9,9601.2,3893714 +2042.0,13936.4,15614.9,9603.6,3893714 +2046.9,13937.4,15614.9,9604.4,3895294 +2051.9,13934.2,15614.9,9601.5,3911344 +2056.7,13933.4,15614.9,9600.4,3926036 +2061.5,13932.3,15614.9,9599.8,3926036 +2066.2,13937.5,15614.9,9604.6,3926036 +2071.0,13933.7,15614.9,9601.0,3937273 +2075.9,13933.7,15614.9,9600.8,3951725 +2080.7,13932.7,15614.9,9600.1,3951725 +2085.5,13932.2,15614.9,9599.3,3951725 +2090.3,13935.6,15614.9,9602.5,3964216 +2095.2,13936.1,15614.9,9603.1,3979572 +2100.1,13936.2,15614.9,9603.4,3990780 +2105.1,13938.3,15614.9,9605.6,3990780 +2109.9,13937.5,15614.9,9604.6,3991297 +2114.8,13938.6,15614.9,9606.0,4004453 +2119.8,13936.9,15614.9,9604.2,4020543 +2124.9,13937.3,15614.9,9604.2,4020543 +2129.9,13937.3,15614.9,9604.1,4020987 +2134.9,13938.3,15614.9,9605.0,4037167 +2139.9,13937.7,15614.9,9604.9,4055377 +2144.9,13933.3,15614.9,9600.7,4055377 +2149.8,13937.0,15614.9,9604.3,4055377 +2154.7,13934.4,15614.9,9601.3,4071204 +2159.7,13941.4,15614.9,9608.6,4091038 +2164.7,13937.3,15614.9,9604.4,4091038 +2169.6,13935.6,15614.9,9602.6,4091038 +2174.5,13934.2,15614.9,9601.3,4101447 +2179.5,13934.4,15614.9,9602.0,4118695 +2184.6,13932.2,15614.9,9599.2,4134313 +2189.5,13934.3,15614.9,9601.6,4134313 +2194.3,13934.8,15614.9,9601.8,4134313 +2199.1,13935.2,15614.9,9602.6,4143437 +2204.0,13934.7,15614.9,9601.7,4158221 +2209.0,13934.8,15614.9,9602.0,4171659 +2214.0,13933.3,15614.9,9600.7,4171659 +2218.8,13936.0,15614.9,9603.0,4171659 +2223.7,13935.3,15614.9,9602.0,4184169 +2228.6,13936.1,15614.9,9602.7,4200085 +2233.5,13929.8,15614.9,9597.2,4200085 +2238.4,13936.5,15614.9,9603.4,4200085 +2243.4,13933.1,15614.9,9600.3,4211331 +2248.4,13938.7,15614.9,9605.8,4228107 +2253.4,13936.4,15614.9,9603.3,4240833 +2258.3,13935.7,15614.9,9602.4,4240833 +2263.3,13944.5,15614.9,9611.7,4241648 +2268.2,13940.2,15614.9,9607.5,4257204 +2273.0,13940.5,15614.9,9607.6,4275772 +2277.9,13937.7,15614.9,9604.7,4275772 +2282.9,13938.0,15614.9,9605.4,4275772 +2287.8,13938.3,15614.9,9605.4,4289807 +2292.7,13936.6,15614.9,9603.7,4305797 +2297.4,13934.0,15614.9,9601.2,4319899 +2302.4,13934.3,15614.9,9602.0,4319899 +2307.2,13936.2,15614.9,9603.4,4319899 +2312.1,13937.1,15614.9,9604.5,4336174 +2316.9,13941.4,15614.9,9608.2,4347048 +2321.8,13933.8,15614.9,9600.8,4362156 +2326.6,13935.5,15614.9,9602.3,4362156 +2331.5,13936.9,15614.9,9604.3,4362156 +2336.4,13934.7,15614.9,9602.0,4370878 +2341.4,13935.2,15614.9,9602.1,4388898 +2346.5,13935.0,15614.9,9601.9,4403028 +2351.4,13935.1,15614.9,9602.2,4403028 +2356.3,13933.7,15614.9,9601.0,4403028 +2361.2,13935.4,15614.9,9602.1,4418548 +2366.0,13936.9,15614.9,9603.5,4433748 +2370.8,13935.9,15614.9,9602.7,4433748 +2375.7,13933.3,15614.9,9600.3,4433748 +2380.5,13932.5,15614.9,9599.8,4446015 +2385.4,13932.8,15614.9,9600.0,4463241 +2390.2,13933.6,15614.9,9600.8,4478269 +2395.1,13936.4,15614.9,9603.8,4478269 +2400.0,13935.3,15614.9,9602.9,4481048 +2404.9,13936.5,15614.9,9604.2,4494604 +2410.0,13938.8,15614.9,9605.7,4510688 +2414.9,13937.6,15614.9,9604.9,4510688 +2419.7,13936.5,15614.9,9603.5,4510688 +2424.5,13933.9,15614.9,9600.8,4521250 +2429.4,13932.7,15614.9,9599.5,4539646 +2434.3,13931.3,15614.9,9598.0,4539646 +2439.2,13932.5,15614.9,9599.5,4539646 +2444.0,13930.8,15614.9,9597.3,4550779 +2448.9,13930.9,15614.9,9598.2,4567343 +2453.9,13934.1,15614.9,9601.7,4583611 +2458.7,13936.3,15614.9,9603.4,4583611 +2463.6,13941.8,15614.9,9609.2,4583611 +2468.4,13937.0,15614.9,9604.7,4599249 +2473.3,13934.7,15614.9,9601.7,4613085 +2478.1,13937.4,15614.9,9604.9,4627019 +2482.8,13936.8,15614.9,9604.9,4627019 +2487.6,13935.3,15614.9,9602.3,4627019 +2492.3,13933.8,15614.9,9601.3,4635971 +2497.1,13935.7,15614.9,9602.8,4653375 +2502.3,13935.6,15614.9,9603.1,4666267 +2507.3,13936.3,15614.9,9604.7,4666267 +2512.2,13934.4,15614.9,9601.6,4666267 +2517.1,13935.7,15614.9,9602.5,4674546 +2522.0,13935.4,15614.9,9602.7,4690242 +2526.8,13931.5,15614.9,9599.3,4705642 +2531.6,13946.4,15614.9,9615.8,4705642 +2536.4,13954.7,15619.1,9623.5,4713802 +2541.2,13945.6,15619.1,9614.4,4731112 +2546.2,13943.4,15619.1,9611.9,4749614 +2551.1,13933.7,15619.1,9602.3,4749614 +2555.9,13933.0,15619.1,9601.6,4749614 +2560.9,13931.3,15619.1,9598.7,4761867 +2565.5,13929.2,15619.1,9596.8,4773265 +2570.7,13968.5,15619.1,9637.1,4792937 +2575.6,14020.3,15625.1,9689.4,4794102 +2580.4,14025.8,15629.4,9694.2,4820762 +2585.3,14019.7,15631.1,9688.1,4857824 +2590.4,14015.7,15634.0,9683.7,4889042 +2595.3,14021.4,15634.0,9690.0,4919554 +2600.3,14028.7,15634.7,9697.1,4919554 +2605.3,14032.0,15634.7,9699.9,4919554 +2610.3,14019.3,15634.7,9687.4,4930007 +2615.4,14023.5,15634.7,9691.1,4961855 +2620.2,14021.3,15634.7,9688.7,4991107 +2624.9,14006.2,15634.7,9673.7,5012719 +2629.7,14011.0,15634.7,9678.6,5012719 +2634.6,14003.2,15634.7,9670.4,5012719 +2639.4,13985.0,15634.7,9652.7,5012719 +2644.2,13976.0,15634.7,9643.5,5028362 +2649.1,13971.7,15634.7,9639.5,5049574 +2653.8,13970.2,15634.7,9637.8,5049574 +2658.6,14000.3,15634.7,9667.9,5049574 +2663.6,13995.9,15634.7,9663.0,5049574 +2668.4,13985.9,15634.7,9652.5,5072387 +2673.3,13981.4,15634.7,9648.6,5091755 +2678.2,13986.2,15634.7,9653.5,5110555 +2683.2,14020.9,15634.7,9687.9,5110555 +2688.0,14000.4,15634.7,9667.6,5110555 +2692.8,13978.3,15634.7,9645.4,5126992 +2697.5,13973.4,15634.7,9640.7,5146444 +2702.2,13961.6,15634.7,9628.8,5162056 +2707.0,13960.0,15634.7,9626.9,5162056 +2711.9,13961.8,15634.7,9628.8,5162056 +2716.8,13962.9,15634.7,9629.8,5162056 +2721.6,13962.0,15634.7,9628.8,5175416 +2726.4,13960.4,15634.7,9627.5,5190140 +2731.2,13958.5,15634.7,9625.3,5205978 +2735.9,13960.0,15634.7,9627.2,5205978 +2740.8,13959.9,15634.7,9627.2,5205978 +2745.7,13959.4,15634.7,9626.7,5210719 +2750.4,13957.2,15634.7,9624.6,5229585 +2755.3,13956.7,15634.7,9624.0,5229585 +2760.1,13958.7,15634.7,9625.8,5229585 +2764.9,13958.5,15634.7,9625.7,5239811 +2769.7,13960.9,15634.7,9627.9,5258147 +2774.6,13959.0,15634.7,9625.7,5263755 +2779.4,13960.5,15634.7,9627.5,5284435 +2784.3,13966.9,15634.7,9633.9,5284435 +2789.3,13962.5,15634.7,9629.8,5284435 +2794.0,13960.7,15634.7,9627.8,5292582 +2798.8,13961.5,15634.7,9628.6,5306766 +2803.6,13959.3,15634.7,9626.3,5323968 +2808.4,13957.6,15634.7,9624.7,5323968 +2813.3,13960.8,15634.7,9627.4,5323968 +2818.2,13960.4,15634.7,9627.2,5323968 +2823.2,13963.7,15634.7,9630.6,5336572 +2828.3,13965.7,15634.7,9632.7,5350314 +2833.0,13960.1,15634.7,9627.1,5350314 +2837.8,13963.1,15634.7,9630.3,5350314 +2842.7,13957.4,15634.7,9624.6,5366126 +2847.7,13961.1,15634.7,9628.2,5374776 +2852.8,13960.3,15634.7,9627.3,5393450 +2857.4,13963.3,15634.7,9630.5,5393450 +2862.3,13957.4,15634.7,9624.9,5393450 +2867.1,13960.4,15634.7,9627.4,5407870 +2872.0,13956.2,15634.7,9623.3,5422358 +2876.7,13958.4,15634.7,9625.5,5436968 +2881.8,13959.4,15634.7,9626.5,5436968 +2886.6,13962.5,15634.7,9629.8,5436968 +2891.6,13962.7,15634.7,9629.6,5445992 +2896.5,13960.2,15634.7,9627.2,5461098 +2901.4,13958.5,15634.7,9625.1,5481232 +2906.3,13957.6,15634.7,9624.0,5496884 +2911.2,13957.4,15634.7,9624.1,5496884 +2916.2,13957.9,15634.7,9625.0,5496884 +2920.9,13959.1,15634.7,9626.1,5496884 +2926.0,13957.9,15634.7,9625.2,5507654 +2930.8,13956.6,15634.7,9623.4,5523888 +2935.4,13953.5,15634.7,9620.7,5523888 +2940.3,13958.2,15634.7,9625.7,5523888 +2945.1,13956.6,15634.7,9624.3,5536253 +2949.9,13957.1,15634.7,9624.6,5539885 +2954.7,13961.7,15634.7,9628.9,5561107 +2959.5,13957.5,15634.7,9625.0,5562781 +2964.6,13958.3,15634.7,9625.6,5562781 +2969.5,13962.3,15634.7,9629.5,5562781 +2974.4,13962.1,15634.7,9628.6,5578775 +2979.2,13955.3,15634.7,9621.6,5596845 +2984.2,13954.2,15634.7,9621.5,5609577 +2989.1,13958.4,15634.7,9625.4,5609577 +2993.9,13958.3,15634.7,9625.4,5609577 +2998.6,13958.3,15634.7,9625.5,5609577 +3003.6,13961.8,15634.7,9628.7,5614487 +3008.6,13969.4,15634.7,9636.6,5622849 +3013.4,13980.4,15634.7,9647.0,5628913 +3018.4,13987.5,15634.7,9654.2,5632653 +3023.2,13998.6,15634.7,9665.1,5635313 +3028.3,14016.2,15634.7,9683.1,5635313 +3033.1,14030.6,15634.7,9697.6,5635313 +3038.0,14037.7,15634.7,9704.8,5635313 +3042.8,14039.6,15635.4,9706.4,5674114 +3047.7,13985.9,15635.4,9652.8,5691314 +3052.6,13971.8,15635.4,9638.6,5706328 +3057.5,13959.0,15635.4,9626.2,5706328 +3062.3,13957.6,15635.4,9624.9,5706328 +3067.2,13960.1,15635.4,9627.0,5706328 +3072.0,13955.7,15635.4,9622.5,5714021 +3076.8,13965.0,15635.4,9631.9,5718353 +3081.7,13973.6,15635.4,9640.4,5723171 +3086.7,13986.0,15635.4,9653.1,5750977 +3091.5,13980.2,15635.4,9647.6,5750977 +3096.3,13969.8,15635.4,9636.4,5750977 +3101.0,13956.4,15635.4,9623.0,5750977 +3105.9,13957.2,15635.4,9624.1,5768831 +3110.8,13955.8,15635.4,9622.4,5781967 +3115.6,13955.4,15635.4,9622.7,5797925 +3120.5,13957.5,15635.4,9624.5,5797925 +3125.3,13955.1,15635.4,9622.3,5797925 +3130.1,13958.4,15635.4,9625.1,5805131 +3135.0,13953.4,15635.4,9620.2,5820583 +3139.7,13952.2,15635.4,9619.3,5835715 +3144.5,13952.5,15635.4,9619.6,5835715 +3149.3,13954.2,15635.4,9621.2,5835715 +3154.1,13951.3,15635.4,9618.5,5845812 +3158.9,13952.5,15635.4,9619.4,5864908 +3163.6,13951.2,15635.4,9618.3,5878214 +3168.5,13952.3,15635.4,9619.8,5878214 +3173.3,13956.5,15635.4,9623.8,5878214 +3178.0,13955.4,15635.4,9622.5,5883688 +3182.8,13958.3,15635.4,9625.7,5898184 +3187.5,13953.3,15635.4,9620.5,5914270 +3192.4,13953.7,15635.4,9620.8,5927980 +3197.2,13952.3,15635.4,9619.3,5927980 +3202.1,13955.1,15635.4,9621.9,5927980 +3206.9,13953.5,15635.4,9620.6,5929543 +3211.7,13956.5,15635.4,9623.7,5944169 +3216.6,13955.2,15635.4,9622.6,5960801 +3221.3,13956.3,15635.4,9622.6,5974879 +3226.1,13956.8,15635.4,9624.1,5974879 +3230.9,13956.2,15635.4,9623.2,5974879 +3235.6,13956.4,15635.4,9623.7,5984073 +3240.5,13956.6,15635.4,9623.4,5992123 +3245.3,13964.7,15635.4,9631.6,6012325 +3250.0,13960.4,15635.4,9627.3,6025631 +3254.9,13962.1,15635.4,9629.5,6025631 +3259.7,13959.6,15635.4,9626.9,6025631 +3264.5,13955.5,15635.4,9622.3,6032853 +3269.4,13956.2,15635.4,9623.2,6051073 +3274.3,13954.8,15635.4,9621.5,6065251 +3279.0,13967.2,15635.4,9634.5,6065251 +3283.9,13962.9,15635.4,9630.3,6065251 +3288.9,13965.6,15635.4,9632.7,6070155 +3293.7,13957.0,15635.4,9624.2,6088163 +3298.6,13958.8,15635.4,9626.1,6104343 +3303.4,13956.0,15635.4,9623.3,6120627 +3308.1,13954.3,15635.4,9621.0,6120627 +3312.9,13954.1,15635.4,9620.8,6120627 +3317.7,13954.9,15635.4,9621.6,6122504 +3322.6,13954.4,15635.4,9621.9,6141108 +3327.4,13958.8,15635.4,9626.3,6157238 +3332.2,13950.7,15635.4,9618.4,6157238 +3337.0,13955.3,15635.4,9622.4,6157238 +3341.8,13953.7,15635.4,9620.7,6160668 +3346.7,13956.0,15635.4,9623.1,6179148 +3351.5,13953.9,15635.4,9620.8,6195884 +3356.4,13957.6,15635.4,9624.7,6214430 +3361.1,13956.0,15635.4,9623.1,6227342 +3366.0,13956.9,15635.4,9624.0,6227342 +3370.9,13955.3,15635.4,9622.4,6227342 +3375.8,13961.8,15635.4,9628.7,6227342 +3380.8,13967.0,15635.4,9634.2,6234194 +3385.6,13968.1,15635.4,9634.7,6254732 +3390.4,13961.2,15635.4,9628.2,6268316 +3395.1,13959.1,15635.4,9626.5,6268316 +3399.9,13962.2,15635.4,9629.0,6268316 +3404.8,13961.0,15635.4,9627.7,6268316 +3409.6,13961.6,15635.4,9628.6,6284205 +3414.3,13962.9,15635.4,9629.7,6301429 +3419.0,13968.9,15635.4,9635.8,6317271 +3423.8,13963.5,15635.4,9630.6,6317271 +3428.7,13982.1,15635.4,9649.2,6317271 +3433.5,13985.1,15635.4,9652.2,6329562 +3438.2,13976.1,15635.4,9642.7,6344626 +3443.1,13961.4,15635.4,9628.1,6362310 +3447.7,13963.2,15635.4,9630.3,6377732 +3452.5,13960.9,15635.4,9627.7,6377732 +3457.5,13964.4,15635.4,9631.4,6377732 +3462.3,13960.2,15635.4,9627.5,6377732 +3467.2,13959.6,15635.4,9627.1,6390785 +3472.0,13958.0,15635.4,9624.9,6404863 +3476.8,13958.6,15635.4,9625.6,6421059 +3481.6,13955.2,15635.4,9621.8,6421059 +3486.5,13966.1,15635.4,9633.3,6421059 +3491.3,13963.6,15635.4,9631.2,6421059 +3496.2,13961.3,15635.4,9629.1,6436830 +3500.9,13959.7,15635.4,9626.5,6448642 +3505.6,13956.9,15635.4,9624.0,6463404 +3510.5,13955.3,15635.4,9622.2,6463404 +3515.3,13956.8,15635.4,9623.8,6463404 +3520.0,13957.1,15635.4,9624.4,6463404 +3524.8,13955.7,15635.4,9622.5,6474288 +3529.6,13958.0,15635.4,9625.0,6493784 +3534.5,13957.6,15635.4,9624.2,6510274 +3539.2,13954.3,15635.4,9621.3,6510274 +3544.1,13952.9,15635.4,9620.3,6510274 +3548.7,13958.0,15635.4,9624.7,6510274 +3553.5,13957.2,15635.4,9624.8,6526860 +3558.4,13960.0,15635.4,9627.3,6546032 +3563.2,13960.5,15635.4,9627.3,6561286 +3568.1,13962.8,15635.4,9629.1,6577878 +3572.9,13977.4,15635.4,9644.1,6577878 +3577.8,13976.7,15635.4,9643.6,6577878 +3582.5,13965.4,15635.4,9632.3,6577878 +3587.4,13966.9,15635.4,9634.0,6590436 +3592.3,13964.1,15635.4,9631.5,6607510 +3597.0,13960.6,15635.4,9627.9,6623952 +3601.8,13958.0,15635.4,9625.0,6623952 +3606.7,13956.1,15635.4,9624.0,6623952 +3611.6,13959.8,15635.4,9627.4,6623952 +3616.3,13958.5,15635.4,9625.6,6631422 +3621.3,13961.1,15635.4,9628.3,6636828 +3626.2,13970.0,15635.4,9637.4,6653332 +3631.1,13969.7,15635.4,9636.8,6666910 +3635.9,13967.3,15635.4,9634.0,6666910 +3640.9,13959.3,15635.4,9626.0,6666910 +3645.6,13956.5,15635.4,9623.9,6669402 +3650.3,13961.8,15635.4,9628.6,6686466 +3655.2,13964.8,15635.4,9631.6,6705218 +3660.0,13960.6,15635.4,9627.2,6717706 +3664.7,13957.8,15635.4,9625.0,6717706 +3669.5,14002.7,15636.8,9669.8,6717706 +3674.2,14028.4,15637.1,9695.6,6717706 +3679.0,14054.2,15638.0,9721.7,6730111 +3683.7,14054.2,15638.4,9721.1,6748771 +3688.7,14051.8,15638.4,9719.4,6764429 +3693.7,14050.3,15638.6,9717.3,6779479 +3698.6,14055.2,15639.7,9721.9,6779479 +3703.5,14054.6,15639.8,9721.4,6779479 +3708.4,14056.8,15640.5,9723.9,6779479 +3713.3,14059.3,15642.2,9726.4,6792044 +3718.3,14057.4,15642.6,9724.8,6808504 +3723.1,14056.7,15642.6,9723.4,6825312 +3728.0,14052.6,15642.6,9719.6,6825312 +3732.9,14052.8,15642.6,9719.8,6825312 +3737.7,14051.9,15642.6,9719.0,6825312 +3742.6,14048.2,15642.6,9715.1,6832436 +3747.5,14052.1,15642.6,9719.2,6843632 +3752.3,14060.4,15645.3,9727.5,6857058 +3757.0,14057.2,15645.4,9724.0,6871448 +3761.8,14052.0,15645.4,9718.9,6871448 +3766.6,14055.2,15645.4,9721.9,6871448 +3771.5,14052.9,15645.4,9719.9,6876580 +3776.4,14055.4,15645.4,9722.4,6894888 +3781.4,14058.1,15648.4,9725.0,6909234 +3786.2,14062.0,15650.4,9729.1,6923058 +3791.3,14055.1,15650.4,9721.7,6923058 +3796.0,14056.9,15650.4,9724.0,6923058 +3800.9,14058.8,15650.4,9726.1,6923058 +3805.8,14057.5,15650.4,9724.2,6936151 +3810.4,14054.9,15650.4,9721.8,6949891 +3815.2,14055.7,15650.4,9722.4,6968419 +3819.9,14052.4,15650.4,9719.1,6982029 +3824.8,14052.9,15650.4,9719.8,6982029 +3829.9,14053.4,15650.4,9720.5,6982029 +3834.7,14052.1,15650.4,9719.3,6982029 +3839.6,14057.2,15650.4,9724.2,6998527 +3844.6,14053.5,15650.4,9720.5,7011727 +3849.3,14055.9,15650.4,9722.3,7026549 +3854.1,14056.0,15650.4,9722.8,7026549 +3859.1,14061.8,15650.7,9729.1,7026549 +3864.1,14062.7,15650.8,9729.9,7026549 +3869.0,14060.8,15651.4,9727.5,7038150 +3873.7,14057.8,15651.4,9724.5,7046898 +3880.5,14062.3,15651.4,9728.9,7048822 +3885.5,14072.8,15652.7,9740.4,7056644 +3890.2,14077.8,15655.4,9744.2,7083770 +3895.1,14063.6,15655.6,9730.3,7083770 +3900.1,14056.6,15655.6,9723.4,7083770 +3904.9,14059.8,15655.6,9726.9,7083770 +3909.9,14058.8,15655.6,9725.9,7093708 +3914.7,14054.7,15655.6,9721.7,7107546 +3919.4,14060.3,15655.6,9727.1,7124508 +3924.2,14058.9,15655.6,9725.8,7140146 +3929.1,14057.7,15655.6,9724.4,7140146 +3933.9,14058.1,15655.6,9724.8,7140146 +3938.7,14056.0,15655.6,9722.6,7140146 +3943.6,14056.7,15655.6,9723.9,7147049 +3948.5,14054.7,15655.6,9721.6,7163615 +3953.4,14050.2,15655.6,9717.1,7168635 +3958.4,14051.4,15655.6,9718.1,7184727 +3963.5,14052.9,15655.6,9720.1,7194433 +3968.2,14055.4,15655.6,9722.7,7194433 +3973.1,14053.7,15655.6,9720.5,7194433 +3977.9,14053.0,15655.6,9719.3,7194433 +3982.6,14052.6,15655.6,9719.3,7203773 +3987.4,14053.7,15655.6,9720.7,7218543 +3992.1,14049.2,15655.6,9716.1,7233149 +3996.9,14209.6,15655.6,9876.4,7233149 +4001.7,14213.0,15655.6,9880.0,7233149 +4006.5,14211.6,15655.6,9878.3,7233149 +4011.5,14219.1,15655.6,9885.8,7240363 +4016.2,14221.9,15655.6,9889.1,7259649 +4021.1,14221.4,15655.6,9888.5,7274501 +4026.0,14216.0,15655.6,9882.8,7286659 +4030.8,14219.3,15655.6,9886.0,7286659 +4035.6,14217.6,15655.6,9884.5,7286659 +4040.5,14217.9,15655.6,9884.6,7287042 +4045.4,14219.8,15655.6,9887.4,7304608 +4050.2,14221.2,15655.6,9888.0,7321374 +4055.1,14224.2,15655.6,9890.6,7336862 +4060.0,14223.4,15655.6,9889.9,7336862 +4064.8,14224.0,15655.6,9890.9,7336862 +4069.5,14222.3,15655.6,9889.2,7336862 +4074.4,14222.3,15655.6,9889.3,7343691 +4079.3,14219.4,15655.6,9886.5,7360021 +4084.2,14222.8,15655.6,9890.0,7378957 +4089.2,14220.2,15655.6,9886.5,7392597 +4094.1,14214.8,15655.6,9881.5,7392597 +4098.9,14215.2,15655.6,9881.8,7392597 +4103.8,14219.4,15655.6,9886.2,7392597 +4108.7,14218.6,15655.6,9885.6,7402128 +4113.6,14217.3,15655.6,9884.3,7420102 +4118.4,14217.1,15655.6,9884.1,7435152 +4123.4,14216.3,15655.6,9883.3,7450558 +4128.2,14218.4,15655.6,9885.6,7450558 +4133.0,14217.1,15655.6,9884.1,7450558 +4137.8,14215.9,15655.6,9882.5,7450558 +4142.6,14211.0,15655.6,9878.2,7458600 +4147.4,14214.2,15655.6,9880.9,7475700 +4152.3,14216.5,15655.6,9883.5,7490084 +4157.2,14213.3,15655.6,9881.0,7503116 +4161.9,14216.2,15655.6,9883.3,7503116 +4166.8,14215.5,15655.6,9882.2,7503116 +4171.7,14220.8,15655.6,9887.1,7505279 +4176.5,14214.3,15655.6,9881.1,7515399 +4181.4,14214.7,15655.6,9881.6,7534233 +4186.3,14212.5,15655.6,9878.3,7547075 +4191.1,14213.4,15655.6,9880.2,7563317 +4195.8,14215.4,15655.6,9882.6,7563317 +4200.8,14216.1,15655.6,9883.0,7563317 +4205.7,14215.8,15655.6,9882.9,7563317 +4210.3,14219.2,15655.6,9885.8,7573915 +4215.3,14218.5,15655.6,9885.4,7590285 +4220.2,14217.1,15655.6,9884.0,7604777 +4225.1,14220.9,15655.8,9887.7,7604777 +4230.0,14224.3,15655.8,9891.2,7604777 +4234.7,14223.0,15655.8,9889.5,7604777 +4239.5,14223.0,15655.8,9889.5,7614183 +4244.3,14219.6,15655.8,9886.3,7627817 +4249.1,14215.6,15655.8,9882.2,7643931 +4254.0,14217.8,15655.8,9884.3,7660271 +4258.9,14215.1,15655.8,9881.5,7660271 +4263.7,14212.7,15655.8,9880.0,7660271 +4268.5,14214.2,15655.8,9880.9,7660271 +4273.4,14215.4,15655.8,9882.1,7662323 +4278.3,14215.9,15655.8,9883.0,7678163 +4283.2,14215.1,15655.8,9881.9,7692503 +4288.0,14219.8,15655.8,9886.4,7709543 +4292.8,14219.8,15655.8,9886.1,7709543 +4297.6,14218.7,15655.8,9885.6,7709543 +4302.5,14217.2,15655.8,9884.0,7709543 +4307.4,14213.3,15655.8,9880.2,7722107 +4312.2,14216.8,15655.8,9884.2,7738471 +4317.0,14209.6,15655.8,9876.5,7751589 +4321.8,14211.8,15655.8,9878.1,7765161 +4326.7,14220.0,15655.8,9886.8,7765161 +4331.5,14218.4,15655.8,9885.0,7765161 +4336.4,14214.8,15655.8,9882.0,7765161 +4341.1,14216.8,15655.8,9883.4,7776802 +4345.9,14214.9,15655.8,9881.9,7792618 +4350.7,14215.9,15655.8,9882.8,7808326 +4355.5,14212.3,15655.8,9878.9,7821926 +4360.2,14214.9,15655.8,9881.8,7833114 +4365.0,14211.4,15655.8,9877.8,7833114 +4369.9,14222.4,15655.8,9889.2,7833114 +4374.8,14220.1,15655.8,9886.9,7833114 +4379.7,14222.9,15655.8,9889.7,7844200 +4384.6,14217.0,15655.8,9884.1,7858578 +4389.3,14217.8,15655.8,9885.1,7875968 +4394.1,14215.7,15655.8,9882.3,7889414 +4398.8,14214.8,15655.8,9881.2,7889414 +4403.7,14214.2,15655.8,9881.8,7889414 +4408.5,14213.6,15655.8,9880.5,7889414 +4413.3,14213.6,15655.8,9880.2,7889414 +4418.2,14217.2,15655.8,9883.6,7906652 +4423.0,14220.2,15655.8,9884.2,7924686 +4427.9,14219.1,15655.8,9886.3,7942356 +4432.6,14219.6,15655.8,9886.5,7955506 +4437.5,14217.6,15655.8,9884.4,7955506 +4442.2,14214.1,15655.8,9881.6,7955506 +4447.1,14215.0,15655.8,9882.0,7955506 +4451.9,14215.3,15655.8,9882.0,7957722 +4456.7,14230.5,15657.3,9896.7,7975006 +4461.6,14219.6,15657.3,9886.4,7992922 +4466.4,14225.3,15657.3,9892.3,8008694 +4471.2,14222.3,15657.3,9889.5,8022808 +4476.1,14218.0,15657.3,9884.5,8022808 +4481.0,14215.6,15657.3,9882.6,8022808 +4485.9,14218.0,15657.3,9885.4,8022808 +4490.8,14218.2,15657.3,9885.0,8023317 +4495.7,14216.5,15657.3,9882.6,8040067 +4500.6,14215.3,15657.3,9882.2,8055871 +4505.5,14213.6,15657.3,9880.6,8070681 +4510.3,14217.2,15657.3,9884.7,8070681 +4515.2,14213.7,15657.3,9880.7,8070681 +4519.9,14214.2,15657.3,9881.8,8070681 +4524.8,14212.1,15657.3,9878.7,8078460 +4529.6,14213.7,15657.3,9880.4,8095096 +4534.6,14211.7,15657.3,9878.4,8109586 +4539.5,14211.0,15657.3,9878.2,8122434 +4544.3,14214.1,15657.3,9881.4,8122434 +4549.4,14219.7,15657.3,9887.1,8122434 +4554.2,14218.5,15657.3,9885.2,8122434 +4559.1,14217.4,15657.3,9884.1,8133315 +4563.9,14215.8,15657.3,9882.5,8148161 +4568.6,14215.9,15657.3,9882.6,8166225 +4573.5,14215.4,15657.3,9882.2,8180959 +4578.2,14212.8,15657.3,9879.1,8192363 +4583.2,14217.0,15657.3,9883.8,8192363 +4588.0,14217.4,15657.3,9884.3,8192363 +4592.7,14214.4,15657.3,9880.8,8192363 +4597.6,14216.2,15657.3,9882.7,8198568 +4602.4,14213.7,15657.3,9880.5,8215172 +4607.2,14214.9,15657.3,9881.9,8230080 +4612.0,14213.7,15657.3,9880.7,8246610 +4617.0,14211.1,15657.3,9878.1,8246610 +4621.8,14209.6,15657.3,9876.4,8246610 +4626.6,14214.3,15657.3,9881.6,8246610 +4631.4,14218.2,15657.3,9884.9,8251600 +4636.2,14222.8,15657.6,9889.3,8268848 +4641.0,14221.3,15657.6,9888.1,8283328 +4646.0,14215.0,15657.6,9882.0,8299976 +4650.8,14216.0,15657.6,9882.9,8299976 +4655.6,14216.0,15657.6,9882.8,8299976 +4660.4,14213.5,15657.6,9880.2,8299976 +4665.3,14215.9,15657.6,9882.6,8300180 +4670.1,14215.1,15657.6,9881.8,8315158 +4674.9,14216.2,15657.6,9883.1,8331982 +4679.7,14215.7,15657.6,9882.3,8348470 +4684.5,14217.4,15657.6,9884.1,8361722 +4689.3,14216.6,15657.6,9883.2,8361722 +4694.3,14214.3,15657.6,9880.9,8361722 +4699.1,14214.9,15657.6,9881.4,8361722 +4703.9,14213.1,15657.6,9880.0,8369191 +4708.8,14216.2,15657.6,9883.0,8388983 +4713.6,14216.5,15657.6,9883.0,8404203 +4718.5,14217.7,15657.6,9884.4,8418535 +4723.4,14213.9,15657.6,9881.0,8431723 +4728.2,14214.8,15657.6,9881.1,8431723 +4733.2,14211.8,15657.6,9878.2,8431723 +4738.1,14213.4,15657.6,9880.5,8431723 +4742.9,14212.8,15657.6,9879.6,8431723 +4747.7,14212.2,15657.6,9879.0,8446874 +4752.5,14213.2,15657.6,9879.6,8460888 +4757.3,14220.7,15657.6,9887.1,8479028 +4762.2,14219.8,15657.6,9886.2,8490470 +4767.0,14217.1,15657.6,9884.2,8490470 +4771.9,14216.0,15657.6,9882.8,8490470 +4776.7,14214.0,15657.6,9880.9,8490470 +4781.6,14213.9,15657.6,9880.9,8494586 +4786.4,14211.4,15657.6,9878.1,8508396 +4791.2,14214.6,15657.6,9881.2,8526836 +4795.9,14210.7,15657.6,9877.5,8539830 +4800.8,14211.1,15657.6,9877.7,8554024 +4805.7,14212.2,15657.6,9878.9,8554024 +4810.5,14211.1,15657.6,9878.4,8554024 +4815.3,14211.2,15657.6,9878.4,8554024 +4820.2,14215.1,15657.6,9881.9,8557181 +4825.0,14216.4,15657.6,9883.5,8574001 +4829.9,14214.0,15657.6,9881.0,8591127 +4834.7,14212.9,15657.6,9879.5,8603775 +4839.5,14211.1,15657.6,9878.1,8603775 +4844.1,14214.9,15657.6,9881.4,8603775 +4849.0,14211.5,15657.6,9878.0,8603775 +4853.8,14212.0,15657.6,9878.8,8605446 +4858.6,14211.7,15657.6,9878.7,8622624 +4863.4,14211.8,15657.6,9878.6,8636338 +4868.2,14211.9,15657.6,9879.0,8652346 +4873.0,14212.6,15657.6,9879.7,8665322 +4877.8,14207.8,15657.6,9874.7,8665322 +4882.6,14212.8,15657.6,9879.7,8665322 +4887.5,14213.6,15657.6,9880.8,8665322 +4892.3,14214.1,15657.6,9881.0,8671792 +4897.2,14216.3,15657.6,9882.9,8674708 +4902.1,14226.5,15657.6,9893.0,8680672 +4907.2,14234.4,15657.9,9901.6,8702546 +4912.1,14237.9,15659.2,9904.1,8715762 +4916.8,14223.8,15659.4,9890.4,8730240 +4921.6,14210.8,15659.4,9877.3,8730240 +4926.5,14211.2,15659.4,9878.2,8730240 +4931.4,14214.8,15659.4,9881.7,8730240 +4936.7,14213.2,15659.4,9880.2,8730240 +4941.7,14211.4,15659.4,9878.2,8744971 +4946.6,14211.3,15659.4,9877.9,8763853 +4951.3,14209.7,15659.4,9876.6,8774347 +4956.3,14213.1,15659.4,9880.0,8785853 +4961.1,14211.6,15659.4,9878.8,8785853 +4965.9,14211.9,15659.4,9879.1,8785853 +4970.8,14210.4,15659.4,9877.6,8785853 +4975.7,14211.4,15659.4,9877.4,8791607 +4980.6,14214.5,15659.4,9881.2,8808857 +4985.8,14213.8,15659.4,9880.5,8825241 +4990.7,14214.5,15659.4,9881.3,8840713 +4995.5,14216.7,15659.4,9882.9,8853673 +5000.6,14213.5,15659.4,9880.6,8853673 +5005.4,14218.2,15659.4,9885.2,8853673 +5010.0,14221.3,15659.4,9888.6,8853673 +5014.9,14220.6,15659.4,9887.7,8853673 +5019.8,14217.4,15659.4,9884.3,8869227 +5024.6,14216.8,15659.4,9883.6,8886715 +5029.4,14211.6,15659.4,9878.5,8901915 +5034.8,14212.8,15659.4,9878.9,8916439 +5039.7,14211.3,15659.4,9878.0,8916439 +5044.6,14214.9,15659.4,9882.5,8916439 +5049.5,14210.5,15659.4,9878.4,8916439 +5054.4,14212.5,15659.4,9880.2,8916439 +5059.3,14217.3,15659.4,9885.1,8918018 +5064.2,14227.9,15659.4,9895.4,8922372 +5069.2,14242.7,15661.3,9910.5,8927578 +5074.1,14251.8,15665.1,9919.1,8936796 +5079.0,14258.0,15670.9,9925.9,8943854 +5083.9,14266.5,15675.8,9933.8,8943854 +5088.9,14275.4,15684.3,9943.2,8943854 +5093.7,14286.0,15693.7,9953.6,8943854 +5098.7,14302.7,15707.9,9969.9,8943854 +5103.7,14311.3,15718.0,9978.3,8943854 +5108.5,14317.0,15722.3,9984.7,8943854 +5113.4,14331.4,15737.0,9998.6,8943854 +5118.3,14343.3,15744.8,10010.7,8943854 +5123.3,14359.7,15759.8,10026.7,8946964 +5128.2,14365.4,15766.5,10032.5,8952970 +5133.1,14381.0,15777.6,10048.2,8959188 +5137.9,14393.5,15790.3,10061.2,8959188 +5142.9,14401.6,15799.0,10068.5,8959188 +5147.7,14413.4,15808.2,10080.2,8959188 +5152.7,14426.6,15820.6,10093.7,8959188 +5157.6,14443.0,15833.3,10109.6,8969076 +5162.5,14438.2,15839.2,10105.4,8974736 +5167.5,14434.1,15840.1,10101.1,8980470 +5172.4,14434.4,15840.1,10101.4,8987472 +5177.2,14443.2,15841.0,10110.2,8991780 +5182.3,14451.6,15844.1,10118.7,8997578 +5187.3,14462.0,15848.4,10129.0,8997578 +5192.4,14463.0,15852.3,10130.6,8997578 +5197.7,14471.7,15857.8,10138.8,9018453 +5204.7,14360.3,15858.6,10028.1,9131707 +5213.0,14225.6,15858.6,9890.1,9160583 +5217.9,14220.3,15858.6,9888.8,9176985 +5223.1,14218.9,15858.6,9886.9,9192221 +5228.1,14218.6,15858.6,9886.5,9192221 +5233.0,14222.0,15858.6,9889.6,9192221 +5238.3,14222.0,15858.6,9889.7,9192221 +5244.3,14221.2,15858.6,9889.1,9192221 +5249.4,14217.2,15858.6,9885.0,9192221 +5254.6,14217.5,15858.6,9885.3,9192221 +5259.9,14215.9,15858.6,9884.0,9192221 +5265.1,14220.4,15858.6,9889.4,9205072 +5270.4,14220.2,15858.6,9888.1,9220252 +5275.9,14219.3,15858.6,9887.4,9220252 +5281.0,14220.4,15858.6,9888.2,9220252 +5286.1,14220.1,15858.6,9888.3,9220252 +5291.2,14214.5,15858.6,9882.7,9220252 +5296.8,14215.4,15858.6,9883.8,9225886 +5301.9,14213.2,15858.6,9881.2,9242414 +5307.1,14215.1,15858.6,9883.0,9256060 +5312.3,14212.7,15858.6,9880.0,9268980 +5317.5,14214.4,15858.6,9882.9,9268980 +5322.9,14213.5,15858.6,9881.5,9268980 +5328.0,14213.9,15858.6,9881.7,9268980 +5333.0,14212.5,15858.6,9879.8,9276201 +5338.1,14214.4,15858.6,9882.5,9294287 +5343.2,14212.1,15858.6,9880.5,9311591 +5348.3,14215.4,15858.6,9883.9,9322385 +5353.5,14214.6,15858.6,9883.3,9332769 +5358.3,14215.4,15858.6,9883.3,9332769 +5363.1,14220.4,15858.6,9888.4,9332769 +5367.9,14220.2,15858.6,9887.8,9332769 +5372.8,14214.3,15858.6,9882.1,9332769 +5377.7,14214.5,15858.6,9882.8,9343997 +5382.5,14210.6,15858.6,9878.9,9357121 +5387.3,14217.0,15858.6,9885.3,9373109 +5392.1,14213.9,15858.6,9881.9,9384583 +5397.0,14214.4,15858.6,9882.6,9397623 +5401.8,14229.5,15858.6,9898.0,9397623 +5406.7,14225.7,15858.6,9893.9,9397623 +5411.4,14218.4,15858.6,9886.2,9397623 +5416.5,14219.3,15858.6,9887.6,9397623 +5421.2,14214.3,15858.6,9882.3,9405841 +5426.2,14214.3,15858.6,9882.7,9420621 +5430.9,14211.9,15858.6,9880.4,9436213 +5435.8,14215.7,15858.6,9883.6,9450083 +5440.6,14212.6,15858.6,9880.5,9460957 +5445.4,14215.6,15858.6,9884.1,9460957 +5450.2,14233.9,15858.6,9901.8,9460957 +5455.1,14231.5,15858.6,9899.3,9460957 +5459.9,14223.0,15858.6,9890.8,9460957 +5464.8,14218.8,15858.6,9888.1,9477301 +5469.6,14217.9,15858.6,9885.9,9494509 +5474.4,14222.3,15858.6,9890.5,9510013 +5479.2,14220.6,15858.6,9888.7,9522843 +5483.9,14216.5,15858.6,9884.0,9522843 +5488.8,14218.4,15858.6,9886.2,9522843 +5493.5,14218.1,15858.6,9886.4,9522843 +5498.4,14228.6,15858.6,9897.2,9525635 +5503.3,14217.1,15858.6,9885.1,9541915 +5508.1,14218.4,15858.6,9886.4,9556449 +5513.0,14218.7,15858.6,9886.7,9573463 +5517.8,14217.4,15858.6,9885.1,9588281 +5522.7,14214.2,15858.6,9882.0,9599269 +5527.7,14216.7,15858.6,9884.7,9599269 +5532.6,14217.4,15858.6,9885.4,9599269 +5537.4,14224.1,15858.6,9892.0,9599269 +5542.3,14224.5,15858.6,9893.1,9599269 +5547.0,14224.0,15858.6,9892.3,9599269 +5551.9,14217.7,15858.6,9885.6,9614673 +5556.8,14216.7,15858.6,9884.4,9632149 +5561.6,14214.0,15858.6,9881.8,9648995 +5566.6,14218.6,15858.6,9886.6,9663187 +5571.4,14214.1,15858.6,9882.2,9663187 +5576.2,14213.8,15858.6,9881.7,9663187 +5581.1,14212.8,15858.6,9880.6,9663187 +5586.0,14217.8,15858.6,9886.2,9663187 +5590.8,14218.3,15858.6,9886.3,9675940 +5595.7,14218.7,15858.6,9887.1,9694180 +5600.5,14217.3,15858.6,9885.6,9711158 +5605.4,14216.0,15858.6,9884.2,9727020 +5610.2,14217.3,15858.6,9886.1,9742370 +5615.0,14216.4,15858.6,9884.3,9742370 +5619.9,14214.6,15858.6,9882.5,9742370 +5624.7,14215.4,15858.6,9883.2,9742370 +5629.5,14214.3,15858.6,9882.2,9742370 +5634.4,14215.7,15858.6,9883.5,9753908 +5639.3,14221.5,15858.6,9889.7,9771054 +5644.2,14217.4,15858.6,9885.4,9790978 +5649.2,14216.4,15858.6,9884.3,9807830 +5654.1,14212.9,15858.6,9881.0,9821202 +5658.9,14214.0,15858.6,9881.9,9821202 +5663.8,14214.3,15858.6,9882.7,9821202 +5668.7,14216.6,15858.6,9884.1,9821202 +5673.6,14217.1,15858.6,9885.7,9821202 +5678.6,14219.8,15858.6,9887.8,9834115 +5683.6,14216.5,15858.6,9884.5,9851905 +5688.4,14218.8,15858.6,9886.9,9869679 +5693.5,14213.5,15858.6,9881.5,9884989 +5698.4,14214.3,15858.6,9882.5,9898381 +5703.2,14212.0,15858.6,9879.9,9898381 +5708.0,14216.5,15858.6,9884.7,9898381 +5712.9,14213.8,15858.6,9881.7,9898381 +5717.8,14215.3,15858.6,9883.8,9898381 +5722.7,14214.5,15858.6,9882.6,9902181 +5727.4,14214.5,15858.6,9882.4,9920511 +5732.3,14214.4,15858.6,9882.4,9933691 +5737.1,14212.4,15858.6,9881.2,9946547 +5742.0,14210.9,15858.6,9878.8,9959585 +5746.8,14212.9,15858.6,9880.7,9959585 +5751.5,14213.1,15858.6,9880.8,9959585 +5756.5,14215.6,15858.6,9883.6,9959585 +5761.4,14216.4,15858.6,9884.5,9959585 +5766.3,14216.0,15858.6,9884.3,9973646 +5771.0,14219.2,15858.6,9886.7,9988594 +5775.8,14216.6,15858.6,9884.8,10005888 +5780.7,14214.0,15858.6,9881.8,10020742 +5785.5,14213.6,15858.6,9881.1,10035854 +5790.4,14214.3,15858.6,9882.0,10035854 +5795.1,14213.3,15858.6,9881.4,10035854 +5800.0,14219.9,15858.6,9888.2,10035854 +5804.8,14219.8,15858.6,9888.7,10035854 +5809.6,14215.6,15858.6,9884.1,10043499 +5814.5,14214.6,15858.6,9883.1,10060735 +5819.4,14210.3,15858.6,9879.2,10077583 +5824.3,14217.8,15858.6,9885.9,10089485 +5829.1,14215.1,15858.6,9883.1,10102281 +5833.9,14214.5,15858.6,9882.3,10113729 +5838.8,14218.5,15858.6,9886.4,10113729 +5843.7,14213.5,15858.6,9881.4,10113729 +5848.6,14213.9,15858.6,9881.8,10113729 +5853.5,14210.4,15858.6,9878.2,10113729 +5858.3,14215.9,15858.6,9884.0,10121557 +5863.3,14213.2,15858.6,9881.2,10135053 +5868.2,14217.6,15858.6,9885.9,10151179 +5873.1,14259.6,15858.6,9928.6,10174227 +5878.0,14274.2,15858.6,9942.2,10201399 +5882.9,14279.7,15858.6,9947.5,10201399 +5887.7,14278.3,15858.6,9946.7,10201399 +5892.6,14270.1,15858.6,9938.3,10211528 +5897.4,14269.7,15858.6,9937.4,10232802 +5902.3,14270.7,15858.6,9939.1,10267690 +5907.2,14271.1,15858.6,9938.9,10293742 +5912.1,14280.0,15858.6,9948.2,10317422 +5917.0,14271.3,15858.6,9939.5,10340826 +5921.8,14270.5,15858.6,9938.7,10368922 +5926.7,14270.7,15858.6,9939.0,10388626 +5931.6,14272.8,15858.6,9941.0,10388626 +5936.5,14283.7,15858.6,9952.2,10388626 +5941.4,14281.8,15858.6,9950.2,10388626 +5946.3,14283.1,15858.6,9950.9,10388626 +5951.2,14280.2,15858.6,9948.7,10393296 +5955.9,14266.3,15858.6,9934.2,10410206 +5960.8,14278.0,15858.6,9945.9,10444004 +5965.7,14275.1,15858.6,9942.7,10467610 +5970.8,14274.8,15858.6,9943.3,10489190 +5975.7,14276.9,15858.6,9945.3,10512100 +5980.5,14271.9,15858.6,9939.8,10512100 +5985.3,14284.1,15858.6,9952.2,10512100 +5990.2,14263.2,15858.6,9931.7,10512100 +5995.0,14269.0,15858.6,9937.1,10512100 +5999.9,14266.6,15858.6,9934.9,10512100 +6004.8,14264.1,15858.6,9932.0,10529358 +6009.6,14267.6,15858.6,9935.5,10556206 +6014.3,14278.4,15858.6,9946.7,10577518 +6019.2,14272.5,15858.6,9940.1,10608930 +6024.1,14265.7,15858.6,9933.5,10633432 +6028.9,14264.9,15858.6,9933.4,10653840 +6033.8,14264.7,15858.6,9932.2,10673486 +6038.6,14264.6,15858.6,9932.3,10673486 +6043.5,14277.4,15858.6,9945.7,10673486 +6048.4,14278.2,15858.6,9946.7,10673486 +6053.3,14280.1,15858.6,9948.0,10673486 +6058.3,14270.8,15858.6,9938.2,10673486 +6063.1,14275.8,15858.6,9943.8,10685621 +6068.1,14266.3,15858.6,9934.0,10713437 +6072.8,14263.4,15858.6,9931.5,10731341 +6077.8,14270.6,15858.6,9938.2,10757247 +6082.5,14264.4,15858.6,9932.5,10773743 +6087.6,14263.0,15858.6,9931.4,10798375 +6092.3,14266.8,15858.6,9934.2,10798375 +6097.3,14265.4,15858.6,9933.5,10798375 +6102.1,14264.0,15858.6,9931.5,10798375 +6106.8,14277.5,15858.6,9945.4,10798375 +6111.7,14292.9,15858.6,9961.3,10798375 +6116.6,14310.6,15858.6,9977.9,10798375 +6121.6,14294.8,15858.6,9962.5,10806259 +6126.6,14266.3,15858.6,9933.7,10820519 +6131.5,14263.4,15858.6,9931.0,10841911 +6136.2,14260.5,15858.6,9928.5,10859677 +6141.1,14262.9,15858.6,9931.2,10882279 +6145.8,14259.7,15858.6,9928.0,10902521 +6150.7,14254.9,15858.6,9923.1,10925589 +6155.6,14251.1,15858.6,9919.0,10925589 +6160.3,14257.2,15858.6,9924.9,10925589 +6165.1,14262.8,15858.6,9930.6,10925589 +6170.1,14262.6,15858.6,9930.4,10925589 +6174.9,14259.1,15858.6,9926.8,10925589 +6179.7,14260.9,15858.6,9929.2,10940337 +6184.6,14260.8,15858.6,9928.0,10963295 +6189.5,14262.1,15858.6,9930.0,10990471 +6194.5,14262.2,15858.6,9930.0,11011059 +6199.3,14262.9,15858.6,9931.0,11026165 +6204.2,14256.0,15858.6,9923.5,11047783 +6209.0,14261.4,15858.6,9929.4,11047783 +6213.9,14258.9,15858.6,9926.7,11047783 +6218.9,14260.5,15858.6,9928.1,11047783 +6223.6,14257.6,15858.6,9925.5,11047783 +6228.6,14255.5,15858.6,9923.6,11047783 +6233.4,14258.5,15858.6,9926.2,11057815 +6238.3,14256.1,15858.6,9924.1,11082807 +6243.3,14254.1,15858.6,9921.8,11106829 +6248.2,14254.1,15858.6,9922.7,11126147 +6253.1,14256.8,15858.6,9924.6,11147069 +6258.0,14252.2,15858.6,9920.2,11164493 +6262.8,14255.2,15858.6,9922.9,11164493 +6267.7,14256.2,15858.6,9924.4,11164493 +6272.5,14257.4,15858.6,9925.7,11164493 +6277.3,14254.6,15858.6,9922.7,11164493 +6282.2,14256.6,15858.6,9924.3,11164493 +6287.0,14252.1,15858.6,9919.8,11171685 +6291.8,14250.5,15858.6,9918.7,11193989 +6296.6,14254.2,15858.6,9923.0,11216179 +6301.3,14263.2,15858.6,9930.7,11237573 +6306.2,14257.5,15858.6,9925.5,11257041 +6311.0,14257.4,15858.6,9925.3,11273257 +6315.7,14255.3,15858.6,9923.4,11273257 +6320.5,14254.3,15858.6,9921.7,11273257 +6325.2,14248.8,15858.6,9916.3,11273257 +6330.0,14252.1,15858.6,9919.9,11273257 +6334.9,14254.3,15858.6,9921.9,11273257 +6339.8,14259.3,15858.6,9927.2,11287614 +6344.7,14256.1,15858.6,9920.1,11307494 +6349.5,14251.0,15858.6,9919.1,11332362 +6354.3,14251.2,15858.6,9919.5,11351752 +6359.2,14256.0,15858.6,9923.7,11371618 +6364.1,14262.5,15858.6,9930.7,11391226 +6368.8,14256.2,15858.6,9923.9,11404612 +6373.6,14260.1,15858.6,9927.9,11404612 +6378.4,14254.1,15858.6,9921.6,11404612 +6383.1,14249.8,15858.6,9917.3,11404612 +6388.0,14245.3,15858.6,9913.4,11404612 +6392.8,14252.2,15858.6,9920.2,11404612 +6397.7,14250.8,15858.6,9918.8,11415714 +6402.6,14252.1,15858.6,9919.9,11437062 +6407.4,14254.8,15858.6,9922.4,11459380 +6412.2,14252.9,15858.6,9920.5,11482756 +6416.9,14248.3,15858.6,9916.1,11503460 +6421.7,14252.7,15858.6,9920.8,11519912 +6426.6,14255.5,15858.6,9923.6,11519912 +6431.4,14254.0,15858.6,9922.0,11519912 +6436.3,14250.6,15858.6,9918.8,11519912 +6441.2,14248.6,15858.6,9916.5,11519912 +6446.1,14245.8,15858.6,9914.1,11519912 +6451.0,14249.2,15858.6,9917.4,11523434 +6455.7,14244.0,15858.6,9911.8,11543062 +6460.6,14247.2,15858.6,9914.9,11566462 +6465.3,14245.2,15858.6,9913.5,11587142 +6470.2,14247.4,15858.6,9915.2,11604150 +6475.2,14245.0,15858.6,9913.5,11623148 +6479.9,14251.4,15858.6,9919.2,11642232 +6484.9,14247.1,15858.6,9915.0,11642232 +6489.7,14249.4,15858.6,9917.6,11642232 +6494.7,14238.8,15858.6,9906.6,11642232 +6499.7,14248.8,15858.6,9916.7,11642232 +6504.6,14251.3,15858.6,9919.6,11642232 +6509.4,14248.5,15858.6,9916.5,11642232 +6514.3,14249.6,15858.6,9917.1,11642232 +6519.2,14252.6,15858.6,9920.3,11660300 +6523.9,14249.4,15858.6,9917.6,11679204 +6528.8,14249.3,15858.6,9917.1,11700314 +6533.9,14247.9,15858.6,9915.4,11721650 +6538.9,14247.9,15858.6,9916.0,11736868 +6543.7,14248.3,15858.6,9916.1,11754630 +6548.6,14246.6,15858.6,9914.7,11754630 +6553.4,14246.1,15858.6,9914.4,11754630 +6558.2,14246.9,15858.6,9915.2,11754630 +6563.1,14247.2,15858.6,9915.6,11754630 +6568.1,14246.0,15858.6,9914.8,11754630 +6572.9,14246.4,15858.6,9914.3,11760095 +6577.8,14247.2,15858.6,9915.9,11780467 +6582.6,14250.1,15858.6,9918.2,11802347 +6587.3,14245.9,15858.6,9914.2,11822077 +6592.0,14248.5,15858.6,9915.9,11839223 +6596.9,14241.1,15858.6,9909.3,11856887 +6601.7,14244.3,15858.6,9912.2,11856887 +6606.6,14246.0,15858.6,9913.6,11856887 +6611.4,14249.3,15858.6,9917.2,11856887 +6616.1,14245.7,15858.6,9913.6,11856887 +6620.9,14249.5,15858.6,9917.4,11856887 +6625.7,14248.0,15858.6,9916.0,11867460 +6630.5,14250.0,15858.6,9918.0,11887038 +6635.3,14246.6,15858.6,9915.3,11908238 +6640.1,14250.1,15858.6,9918.1,11925448 +6644.9,14244.5,15858.6,9912.9,11943038 +6649.7,14246.5,15858.6,9914.3,11960242 +6654.5,14245.7,15858.6,9913.8,11979330 +6659.5,14242.9,15858.6,9912.0,11979330 +6664.2,14246.7,15858.6,9914.4,11979330 +6669.0,14246.9,15858.6,9914.6,11979330 +6673.9,14247.2,15858.6,9915.0,11979330 +6678.7,14246.5,15858.6,9914.5,11979330 +6683.7,14243.4,15858.6,9911.8,11989529 +6688.5,14247.4,15858.6,9915.1,12006807 +6693.2,14241.1,15858.6,9909.2,12023427 +6698.0,14239.8,15858.6,9907.2,12039937 +6702.9,14244.0,15858.6,9912.0,12060381 +6707.6,14246.6,15858.6,9914.7,12076119 +6712.5,14247.4,15858.6,9914.9,12094489 +6717.3,14246.5,15858.6,9913.7,12094489 +6722.1,14246.2,15858.6,9914.2,12094489 +6727.1,14240.7,15858.6,9909.0,12094489 +6731.8,14243.1,15858.6,9911.5,12094489 +6736.7,14237.1,15858.6,9905.3,12094489 +6741.6,14240.0,15858.6,9908.3,12098101 +6746.3,14241.2,15858.6,9909.2,12119401 +6751.2,14245.0,15858.6,9913.2,12140853 +6756.0,14241.3,15858.6,9909.1,12156955 +6760.8,14249.1,15858.6,9917.5,12176509 +6765.6,14240.9,15858.6,9909.0,12191777 +6770.5,14239.7,15858.6,9907.7,12212329 +6775.5,14243.7,15858.6,9911.3,12212329 +6780.1,14248.2,15858.6,9916.2,12212329 +6785.1,14245.4,15858.6,9913.5,12212329 +6789.9,14245.5,15858.6,9914.0,12212329 +6794.8,14242.3,15858.6,9910.1,12212329 +6799.6,14247.8,15858.6,9915.6,12216190 +6804.4,14247.3,15858.6,9915.2,12234232 +6809.2,14243.7,15858.6,9911.7,12255088 +6814.1,14244.5,15858.6,9912.3,12272384 +6818.9,14245.1,15858.6,9913.0,12289556 +6823.9,14244.0,15858.6,9911.6,12306018 +6828.7,14245.5,15858.6,9913.6,12325038 +6833.6,14245.5,15858.6,9913.2,12325038 +6838.5,14243.2,15858.6,9911.5,12325038 +6843.4,14244.7,15858.6,9912.9,12325038 +6848.2,14243.0,15858.6,9910.3,12325038 +6853.1,14240.3,15858.6,9907.9,12325038 +6858.0,14239.4,15858.6,9907.2,12325225 +6862.9,14240.2,15858.6,9908.0,12345497 +6867.9,14237.3,15858.6,9905.3,12360867 +6872.7,14243.4,15858.6,9911.3,12382017 +6877.5,14239.6,15858.6,9907.7,12401211 +6882.3,14239.8,15858.6,9907.9,12417275 +6887.2,14239.8,15858.6,9907.6,12432459 +6892.1,14239.4,15858.6,9907.6,12432459 +6896.9,14241.3,15858.6,9909.4,12432459 +6901.8,14241.7,15858.6,9910.3,12432459 +6906.6,14242.5,15858.6,9909.5,12432459 +6911.5,14248.3,15858.6,9916.8,12432459 +6916.4,14255.2,15858.6,9923.1,12432459 +6921.3,14257.5,15858.6,9925.5,12434731 +6926.2,14249.8,15858.6,9917.2,12453841 +6931.1,14252.1,15858.6,9920.2,12474437 +6936.1,14249.8,15858.6,9917.9,12493707 +6940.8,14247.1,15858.6,9915.0,12510665 +6945.6,14243.4,15858.6,9911.4,12525269 +6950.4,14241.3,15858.6,9909.0,12541353 +6955.3,14279.7,15858.6,9947.0,12541353 +6960.2,14326.2,15858.6,9994.3,12541353 +6965.2,14321.5,15858.6,9989.3,12541353 +6970.1,14337.9,15858.6,10004.9,12577301 +6975.1,14316.6,15858.6,9984.0,12605809 +6980.1,14313.5,15858.6,9981.0,12644675 +6985.0,14308.0,15858.6,9976.0,12676979 +6990.0,14312.2,15858.6,9980.0,12721271 +6994.8,14308.5,15858.6,9975.7,12746613 +6999.7,14307.3,15858.6,9975.0,12772783 +7004.6,14311.1,15858.6,9978.9,12806583 +7009.5,14299.5,15858.6,9967.4,12834323 +7014.4,14303.8,15858.6,9971.5,12867743 +7019.1,14306.2,15858.6,9974.5,12867743 +7024.0,14306.2,15858.6,9974.3,12867743 +7028.8,14300.3,15858.6,9968.6,12867743 +7033.8,14298.2,15858.6,9966.0,12867743 +7038.7,14303.1,15858.6,9970.9,12867743 +7043.5,14305.1,15858.6,9973.0,12867743 +7048.4,14302.9,15858.6,9969.4,12867743 +7053.2,14291.2,15858.6,9959.7,12899404 +7058.1,14296.0,15858.6,9963.7,12924796 +7063.0,14295.6,15858.6,9963.4,12950670 +7068.0,14298.6,15858.6,9966.8,12987774 +7073.0,14296.3,15858.6,9964.5,13014362 +7077.7,14292.3,15858.6,9960.5,13038682 +7082.6,14288.7,15858.6,9956.8,13062444 +7087.4,14290.4,15858.6,9957.8,13088128 +7092.4,14290.8,15858.6,9958.8,13088128 +7097.2,14291.9,15858.6,9960.0,13088128 +7102.1,14286.9,15858.6,9954.4,13088128 +7106.9,14288.1,15858.6,9955.4,13088128 +7111.8,14280.9,15858.6,9948.8,13088128 +7116.6,14287.3,15858.6,9955.5,13088128 +7121.6,14313.0,15858.6,9981.0,13088128 +7126.6,14308.3,15858.6,9975.8,13088128 +7131.4,14311.6,15858.6,9979.2,13088128 +7136.6,14279.1,15858.6,9946.6,13108231 +7141.6,14282.2,15858.6,9950.9,13132165 +7146.7,14267.0,15858.6,9935.1,13147795 +7151.5,14284.9,15858.6,9953.5,13174817 +7156.4,14281.2,15858.6,9949.4,13193337 +7161.3,14278.8,15858.6,9947.0,13212141 +7166.4,14280.4,15858.6,9948.4,13233227 +7171.2,14278.7,15858.6,9946.9,13251057 +7176.2,14274.8,15858.6,9942.4,13251057 +7181.1,14278.2,15858.6,9947.2,13251057 +7186.1,14290.4,15858.6,9958.3,13251057 +7191.0,14291.2,15858.6,9958.9,13251057 +7195.9,14285.2,15858.6,9953.1,13251057 +7200.7,14284.9,15858.6,9953.1,13251057 +7205.4,14278.7,15858.6,9946.5,13251057 +7210.3,14278.3,15858.6,9945.8,13259680 +7215.1,14274.1,15858.6,9942.1,13281386 +7220.2,14279.5,15858.6,9947.7,13303390 +7224.9,14276.3,15858.6,9944.5,13330800 +7229.9,14280.4,15858.6,9949.2,13353920 +7234.7,14273.4,15858.6,9941.0,13369204 +7239.6,14276.1,15858.6,9943.9,13392482 +7244.4,14273.5,15858.6,9941.9,13414828 +7249.1,14274.7,15858.6,9942.2,13414828 +7254.0,14275.4,15858.6,9943.6,13414828 +7259.0,14275.4,15858.6,9943.8,13414828 +7264.0,14279.8,15858.6,9947.5,13414828 +7268.8,14284.0,15858.6,9951.9,13414828 +7273.8,14279.2,15858.6,9947.5,13414828 +7278.7,14285.5,15858.6,9953.3,13414828 +7283.5,14311.0,15858.6,9979.2,13414828 +7288.6,14298.3,15858.6,9965.9,13421340 +7293.5,14281.4,15858.6,9949.1,13443308 +7298.5,14277.2,15858.6,9945.1,13466562 +7303.4,14277.2,15858.6,9945.1,13489300 +7308.2,14281.3,15858.6,9949.4,13510084 +7313.0,14274.6,15858.6,9942.5,13532370 +7317.9,14275.5,15858.6,9943.2,13550746 +7322.7,14285.7,15858.6,9953.9,13575158 +7327.5,14279.7,15858.6,9947.5,13575158 +7332.4,14281.4,15858.6,9950.0,13575158 +7337.3,14277.6,15858.6,9945.7,13575158 +7342.1,14277.9,15858.6,9945.8,13575158 +7347.0,14274.2,15858.6,9941.8,13575158 +7352.0,14273.1,15858.6,9940.6,13575158 +7356.9,14272.0,15858.6,9940.2,13579774 +7361.7,14272.9,15858.6,9940.6,13602348 +7366.5,14278.6,15858.6,9946.5,13631222 +7371.4,14273.7,15858.6,9941.0,13653470 +7376.2,14275.7,15858.6,9943.7,13676042 +7381.0,14274.6,15858.6,9942.3,13696662 +7385.7,14272.1,15858.6,9940.7,13718024 +7390.5,14268.7,15858.6,9937.2,13732118 +7395.5,14271.7,15858.6,9939.2,13732118 +7400.4,14283.9,15858.6,9952.1,13732118 +7405.4,14284.9,15858.6,9953.5,13732118 +7410.3,14279.7,15858.6,9947.6,13732118 +7415.3,14277.1,15858.6,9944.9,13732118 +7420.1,14275.1,15858.6,9943.0,13732118 +7425.0,14272.1,15858.6,9939.4,13738713 +7429.7,14272.2,15858.6,9940.3,13764491 +7434.6,14270.0,15858.6,9937.9,13788863 +7439.3,14270.8,15858.6,9938.4,13808429 +7444.2,14273.6,15858.6,9941.2,13829061 +7449.0,14272.3,15858.6,9940.0,13845449 +7453.9,14269.5,15858.6,9937.3,13862443 +7458.9,14272.7,15858.6,9940.0,13884115 +7463.9,14274.5,15858.6,9942.7,13884115 +7468.9,14275.2,15858.6,9942.8,13884115 +7473.8,14275.7,15858.6,9944.0,13884115 +7478.5,14293.5,15858.6,9961.7,13884115 +7483.5,14285.3,15858.6,9953.2,13884115 +7488.4,14275.2,15858.6,9942.7,13884115 +7493.2,14268.5,15858.6,9936.0,13884115 +7498.0,14265.1,15858.6,9933.0,13896750 +7502.8,14266.5,15858.6,9934.3,13920442 +7507.7,14268.0,15858.6,9936.3,13943922 +7512.6,14265.7,15858.6,9933.8,13964776 +7517.3,14258.6,15858.6,9927.0,13982852 +7522.2,14261.7,15858.6,9930.5,14003028 +7527.1,14262.5,15858.6,9930.0,14022874 +7531.9,14267.0,15858.6,9935.9,14022874 +7536.9,14265.9,15858.6,9933.6,14022874 +7541.7,14265.8,15858.6,9933.5,14022874 +7546.5,14273.3,15858.6,9941.6,14022874 +7551.3,14276.5,15858.6,9945.0,14022874 +7556.1,14277.9,15858.6,9945.7,14022874 +7561.0,14278.3,15858.6,9946.1,14035821 +7565.7,14273.9,15858.6,9941.5,14049959 +7570.6,14277.6,15858.6,9945.7,14074551 +7575.6,14270.5,15858.6,9938.6,14093563 +7580.6,14270.8,15858.6,9938.7,14117431 +7585.4,14287.0,15858.6,9954.9,14136553 +7590.2,14300.3,15858.6,9967.9,14156395 +7595.1,14295.9,15858.6,9964.2,14174599 +7600.0,14309.8,15858.6,9977.9,14174599 +7604.9,14355.3,15858.6,10024.1,14174599 +7609.9,14370.8,15858.6,10039.0,14174599 +7614.8,14386.6,15858.6,10054.4,14174599 +7619.9,14385.8,15858.6,10053.8,14174599 +7624.7,14383.5,15858.6,10050.7,14174599 +7629.5,14381.1,15858.6,10048.9,14176415 +7634.3,14380.2,15858.6,10047.7,14194891 +7639.1,14384.7,15858.6,10052.2,14213359 +7644.2,14380.5,15858.6,10048.3,14235991 +7649.0,14381.1,15858.6,10049.3,14254923 +7653.8,14381.3,15858.6,10049.8,14276853 +7658.6,14381.4,15858.6,10048.8,14292277 +7663.6,14379.3,15858.6,10047.3,14309053 +7668.6,14377.6,15858.6,10045.9,14329745 +7673.6,14377.8,15858.6,10045.1,14329745 +7678.7,14381.3,15858.6,10049.1,14329745 +7683.6,14380.6,15858.6,10048.9,14329745 +7688.4,14377.7,15858.6,10045.8,14329745 +7693.3,14377.4,15858.6,10045.3,14329745 +7698.3,14381.5,15858.6,10049.5,14329745 +7703.2,14383.8,15858.6,10052.0,14332251 +7708.0,14383.5,15858.6,10051.1,14353121 +7712.8,14382.9,15858.6,10050.4,14374437 +7717.5,14379.2,15858.6,10046.5,14390625 +7722.3,14381.5,15858.6,10049.5,14412075 +7727.1,14379.2,15858.6,10047.1,14430677 +7731.9,14376.9,15858.6,10044.7,14450789 +7736.8,14375.6,15858.6,10043.2,14468037 +7741.6,14375.5,15858.6,10043.6,14484449 +7746.4,14379.8,15858.6,10047.1,14501855 +7751.3,14382.9,15858.6,10050.6,14501855 +7756.0,14382.3,15858.6,10050.0,14501855 +7760.8,14382.4,15858.6,10050.3,14501855 +7765.6,14378.1,15858.6,10046.3,14501855 +7770.6,14395.1,15858.6,10062.5,14501855 +7775.7,14405.7,15858.6,10074.1,14501855 +7780.6,14423.1,15858.6,10090.5,14501855 +7785.4,14435.1,15858.6,10103.5,14501855 +7790.4,14420.4,15858.6,10088.5,14504864 +7795.4,14388.1,15858.6,10055.8,14528616 +7800.3,14383.1,15858.6,10051.2,14534584 +7805.3,14371.8,15858.6,10039.4,14548070 +7810.2,14374.6,15858.6,10041.6,14561858 +7815.0,14364.1,15858.6,10031.6,14577046 +7819.8,14367.2,15858.6,10035.1,14590836 +7824.5,14370.5,15858.6,10038.7,14609330 +7829.2,14373.4,15858.6,10041.4,14609330 +7834.0,14371.7,15858.6,10039.3,14609330 +7838.8,14377.6,15858.6,10043.4,14609330 +7843.5,14373.0,15858.6,10040.2,14609330 +7848.4,14376.2,15858.6,10044.3,14609330 +7853.1,14373.6,15858.6,10041.6,14609330 +7857.9,14373.7,15858.6,10041.5,14609330 +7862.7,14374.6,15858.6,10042.9,14630110 +7867.5,14376.2,15858.6,10044.3,14652766 +7872.4,14376.0,15858.6,10043.8,14670400 +7877.3,14374.3,15858.6,10042.5,14691034 +7882.3,14376.9,15858.6,10045.0,14710870 +7887.3,14378.3,15858.6,10046.1,14730032 +7892.4,14376.8,15858.6,10044.7,14746810 +7897.2,14375.7,15858.6,10043.7,14766102 +7902.1,14380.5,15858.6,10049.4,14766102 +7906.8,14377.4,15858.6,10045.0,14766102 +7911.6,14376.9,15858.6,10044.5,14766102 +7916.4,14376.2,15858.6,10043.9,14766102 +7921.3,14375.2,15858.6,10043.0,14766102 +7926.1,14380.6,15858.6,10048.5,14766102 +7931.0,14378.8,15858.6,10047.1,14779357 +7935.8,14377.7,15858.6,10045.5,14802011 +7940.6,14377.8,15858.6,10045.5,14821779 +7945.5,14381.1,15858.6,10048.1,14845467 +7950.3,14379.5,15858.6,10047.5,14867123 +7955.1,14377.9,15858.6,10045.4,14889925 +7959.9,14374.0,15858.6,10042.0,14905981 +7964.8,14376.6,15858.6,10044.4,14927761 +7969.7,14375.8,15858.6,10043.4,14948105 +7974.5,14378.1,15858.6,10045.8,14948105 +7979.4,14375.9,15858.6,10043.4,14948105 +7984.2,14387.3,15858.6,10055.0,14948105 +7989.1,14380.8,15858.6,10048.8,14948105 +7993.9,14376.5,15858.6,10044.3,14948105 +7998.7,14371.6,15858.6,10039.5,14948105 +8003.6,14375.4,15858.6,10043.3,14948105 +8008.5,14377.1,15858.6,10044.7,14965573 +8013.2,14376.2,15858.6,10043.9,14985467 +8018.0,14375.4,15858.6,10043.6,15010099 +8022.8,14373.9,15858.6,10042.1,15030411 +8027.7,14375.3,15858.6,10043.1,15048811 +8032.6,14371.0,15858.6,10039.0,15068993 +8037.4,14371.9,15858.6,10039.8,15092329 +8042.3,14374.2,15858.6,10041.7,15092329 +8047.2,14376.3,15858.6,10043.9,15092329 +8052.0,14372.2,15858.6,10039.8,15092329 +8056.9,14370.1,15858.6,10037.8,15092329 +8061.7,14374.4,15858.6,10042.2,15092329 +8066.5,14372.3,15858.6,10039.8,15092329 +8071.3,14375.6,15858.6,10043.5,15092329 +8076.1,14376.4,15858.6,10044.6,15102891 +8080.9,14373.2,15858.6,10040.8,15122587 +8085.7,14371.5,15858.6,10039.9,15141893 +8090.5,14371.6,15858.6,10039.7,15164903 +8095.3,14371.4,15858.6,10039.0,15184529 +8100.1,14370.3,15858.6,10038.8,15207099 +8105.0,14370.7,15858.6,10038.4,15222001 +8109.9,14373.2,15858.6,10041.1,15240185 +8114.7,14371.5,15858.6,10039.0,15259157 +8119.6,14372.5,15858.6,10039.9,15279947 +8124.4,14373.3,15858.6,10041.2,15279947 +8129.3,14373.8,15858.6,10041.8,15279947 +8134.0,14374.5,15858.6,10042.4,15279947 +8138.9,14371.5,15858.6,10039.7,15279947 +8143.8,14368.3,15858.6,10036.5,15279947 +8148.5,14368.2,15858.6,10036.4,15279947 +8153.3,14365.2,15858.6,10032.8,15279947 +8158.1,14367.4,15858.6,10035.0,15281651 +8163.0,14370.3,15858.6,10038.0,15302435 +8167.9,14370.2,15858.6,10037.7,15321603 +8172.7,14369.0,15858.6,10036.9,15342803 +8177.6,14373.0,15858.6,10040.8,15363313 +8182.4,14368.6,15858.6,10035.8,15382493 +8187.4,14367.7,15858.6,10035.8,15398401 +8192.3,14367.7,15858.6,10035.3,15415293 +8197.0,14366.3,15858.6,10034.5,15434965 +8201.9,14369.6,15858.6,10037.9,15434965 +8206.7,14368.0,15858.6,10035.8,15434965 +8211.6,14367.6,15858.6,10035.3,15434965 +8216.4,14371.3,15858.6,10038.7,15434965 +8221.1,14371.3,15858.6,10039.0,15434965 +8225.9,14370.8,15858.6,10039.1,15434965 +8230.6,14371.0,15858.6,10038.4,15434965 +8235.3,14371.1,15858.6,10039.1,15434965 +8240.1,14373.0,15858.6,10041.1,15443455 +8245.0,14373.3,15858.6,10041.2,15467225 +8249.9,14374.7,15858.6,10042.9,15487499 +8254.7,14367.9,15858.6,10036.6,15507849 +8259.6,14372.0,15858.6,10039.5,15528569 +8264.3,14377.3,15858.6,10045.2,15547035 +8269.2,14374.4,15858.6,10042.5,15567175 +8274.1,14369.8,15858.6,10038.4,15582777 +8279.0,14367.8,15858.6,10035.9,15601021 +8283.8,14366.7,15858.6,10035.3,15615641 +8288.5,14367.6,15858.6,10035.7,15615641 +8293.4,14371.6,15858.6,10039.3,15615641 +8298.2,14368.0,15858.6,10035.8,15615641 +8303.0,14368.9,15858.6,10037.1,15615641 +8307.9,14368.9,15858.6,10037.0,15615641 +8312.7,14368.8,15858.6,10037.0,15615641 +8317.6,14368.4,15858.6,10036.4,15615641 +8322.5,14369.1,15858.6,10037.2,15615641 +8327.3,14367.2,15858.6,10035.3,15615641 +8332.1,14369.1,15858.6,10036.7,15638501 +8336.9,14364.8,15858.6,10032.6,15661171 +8341.9,14367.2,15858.6,10035.2,15680201 +8346.8,14363.0,15858.6,10030.6,15698313 +8351.7,14366.7,15858.6,10035.1,15714805 +8356.5,14363.0,15858.6,10030.8,15730325 +8361.2,14362.1,15858.6,10030.6,15748073 +8365.9,14367.1,15858.6,10035.3,15764033 +8370.6,14365.6,15858.6,10033.3,15764033 +8375.5,14366.8,15858.6,10034.7,15764033 +8380.4,14368.6,15858.6,10036.8,15764033 +8385.1,14365.3,15858.6,10033.1,15764033 +8390.0,14364.5,15858.6,10032.3,15764033 +8394.7,14369.9,15858.6,10038.0,15764033 +8399.6,14369.4,15858.6,10037.1,15764033 +8404.4,14367.8,15858.6,10035.7,15767222 +8409.1,14365.6,15858.6,10033.5,15782442 +8413.9,14367.5,15858.6,10035.6,15799426 +8418.8,14364.8,15858.6,10032.9,15824102 +8423.6,14365.0,15858.6,10033.2,15844474 +8428.4,14363.6,15858.6,10031.8,15863938 +8433.3,14364.2,15858.6,10032.3,15880194 +8438.1,14362.7,15858.6,10030.7,15892686 +8443.1,14363.9,15858.6,10031.8,15912956 +8448.1,14365.6,15858.6,10033.2,15928916 +8452.9,14366.4,15858.6,10033.6,15928916 +8457.8,14363.8,15858.6,10031.4,15928916 +8462.6,14362.9,15858.6,10030.7,15928916 +8467.4,14360.9,15858.6,10029.3,15928916 +8472.2,14365.9,15858.6,10033.5,15928916 +8477.0,14368.9,15858.6,10037.4,15928916 +8481.7,14368.6,15858.6,10036.6,15928916 +8486.5,14366.4,15858.6,10034.3,15928916 +8491.3,14366.2,15858.6,10034.3,15944748 +8496.3,14363.9,15858.6,10031.9,15965866 +8501.1,14365.6,15858.6,10033.2,15981300 +8505.9,14367.8,15858.6,10035.9,16001588 +8510.8,14363.2,15858.6,10030.7,16019272 +8515.5,14363.8,15858.6,10032.1,16037438 +8520.3,14362.8,15858.6,10031.2,16055020 +8525.3,14362.7,15858.6,10030.4,16071968 +8530.1,14362.6,15858.6,10030.9,16071968 +8534.8,14368.4,15858.6,10036.0,16071968 +8539.7,14363.0,15858.6,10031.4,16071968 +8544.6,14363.1,15858.6,10031.2,16071968 +8549.4,14362.1,15858.6,10030.0,16071968 +8554.1,14366.3,15858.6,10033.9,16071968 +8558.9,14366.6,15858.6,10034.3,16071968 +8563.6,14366.3,15858.6,10034.2,16074670 +8568.4,14366.6,15858.6,10034.6,16097044 +8573.2,14367.8,15858.6,10035.5,16114698 +8578.0,14365.7,15858.6,10033.1,16138422 +8582.8,14365.2,15858.6,10033.1,16156650 +8587.5,14364.7,15858.6,10032.0,16176326 +8592.3,14361.7,15858.6,10029.6,16189456 +8597.2,14360.4,15858.6,10028.5,16204950 +8602.0,14364.2,15858.6,10032.9,16224674 +8606.9,14361.7,15858.6,10029.5,16224674 +8611.8,14368.0,15858.6,10035.6,16224674 +8616.7,14364.7,15858.6,10032.4,16224674 +8621.6,14364.9,15858.6,10032.2,16224674 +8626.4,14363.9,15858.6,10032.1,16224674 +8631.3,14364.8,15858.6,10033.4,16224674 +8636.1,14365.7,15858.6,10033.8,16224674 +8640.8,14364.0,15858.6,10032.0,16235221 +8645.6,14362.0,15858.6,10029.9,16255001 +8650.4,14363.8,15858.6,10032.2,16273823 +8655.1,14363.4,15858.6,10031.4,16290301 +8659.9,14360.4,15858.6,10028.1,16311481 +8664.6,14364.9,15858.6,10032.9,16329835 +8669.3,14360.0,15858.6,10027.9,16344343 +8674.2,14360.0,15858.6,10028.6,16360211 +8678.8,14967.0,15858.6,10653.8,16366517 +8683.7,14687.1,15858.6,10354.7,16382877 +8688.5,14687.4,15858.6,10355.6,16398711 +8693.4,14691.8,15858.6,10359.4,16398711 +8698.2,14693.0,15858.6,10360.7,16398711 +8703.1,14701.8,15858.6,10369.7,16398711 +8707.9,14699.8,15858.6,10367.4,16398711 +8712.6,14695.2,15858.6,10363.1,16398711 +8717.4,14687.7,15858.6,10355.6,16398711 +8722.2,14687.9,15858.6,10356.1,16398711 +8727.2,14687.5,15858.6,10355.6,16398711 +8732.0,14688.4,15858.6,10356.5,16410758 +8736.8,14688.8,15858.6,10356.8,16432880 +8741.6,14687.5,15858.6,10355.3,16448964 +8746.6,14694.1,15858.6,10361.5,16470988 +8751.4,14689.4,15858.6,10357.4,16488698 +8756.2,14690.3,15858.6,10357.9,16503886 +8761.0,14689.2,15858.6,10357.3,16519928 +8765.8,14686.5,15858.6,10354.3,16533008 +8770.7,14688.1,15858.6,10356.0,16549858 +8775.6,14691.2,15858.6,10359.2,16549858 +8780.5,14689.5,15858.6,10357.6,16549858 +8785.4,14688.4,15858.6,10357.0,16549858 +8790.2,14686.1,15858.6,10354.0,16549858 +8795.0,14685.0,15858.6,10353.3,16549858 +8799.8,14684.6,15858.6,10352.3,16549858 +8804.8,14672.2,15858.6,10340.2,16549858 +8809.6,14658.4,15858.6,10326.2,16549858 +8814.4,14658.8,15858.6,10326.7,16567468 +8819.1,14644.9,15858.6,10312.8,16589124 +8824.0,14630.2,15858.6,10297.0,16606776 +8828.8,14597.5,15858.6,10265.6,16624124 +8833.7,14568.6,15858.6,10236.0,16644892 +8838.6,14567.4,15858.6,10235.3,16659278 +8843.5,14566.4,15858.6,10234.4,16675546 +8848.3,14566.2,15858.6,10233.8,16691730 +8853.3,14569.3,15858.6,10237.4,16708242 +8858.2,14573.2,15858.6,10241.2,16708242 +8863.0,14575.2,15858.6,10242.8,16708242 +8867.9,14572.1,15858.6,10239.9,16708242 +8872.7,14569.9,15858.6,10238.0,16708242 +8877.7,14571.0,15858.6,10239.2,16708242 +8882.5,14572.0,15858.6,10240.0,16708242 +8887.4,14570.1,15858.6,10238.2,16708242 +8892.2,14574.8,15858.6,10242.6,16708242 +8897.1,14572.6,15858.6,10239.9,16715128 +8902.0,14570.1,15858.6,10237.9,16736266 +8906.9,14569.7,15858.6,10237.4,16748486 +8911.8,14567.9,15858.6,10235.3,16767334 +8916.8,14569.7,15858.6,10237.3,16788772 +8921.7,14567.5,15858.6,10235.3,16805082 +8926.5,14570.2,15858.6,10237.8,16819730 +8931.2,14570.0,15858.6,10237.5,16836838 +8935.9,14569.5,15858.6,10238.2,16854178 +8940.8,14568.3,15858.6,10236.3,16854178 +8945.7,14571.0,15858.6,10239.1,16854178 +8950.7,14570.3,15858.6,10237.7,16854178 +8955.6,14570.3,15858.6,10238.3,16854178 +8960.4,14577.5,15858.6,10245.3,16854178 +8965.3,14577.4,15858.6,10245.2,16854178 +8970.0,14576.6,15858.6,10244.3,16854178 +8974.9,14577.3,15858.6,10244.5,16854178 +8979.8,14579.2,15858.6,10247.2,16867010 +8984.6,14575.4,15858.6,10243.1,16884482 +8989.5,14573.2,15858.6,10241.3,16906424 +8994.4,14569.9,15858.6,10238.0,16924056 +8999.2,14572.0,15858.6,10240.0,16940308 +9004.0,14574.8,15858.6,10243.4,16958348 +9008.9,14573.3,15858.6,10241.5,16973578 +9013.7,14567.9,15858.6,10236.2,16986934 +9018.6,14572.1,15858.6,10240.6,17003428 +9023.4,14569.5,15858.6,10237.5,17003428 +9028.2,14573.8,15858.6,10241.8,17003428 +9033.1,14571.7,15858.6,10240.5,17003428 +9037.9,14569.2,15858.6,10237.1,17003428 +9042.7,14570.9,15858.6,10238.3,17003428 +9047.6,14569.9,15858.6,10237.8,17003428 +9052.4,14570.0,15858.6,10238.3,17003428 +9057.3,14570.1,15858.6,10237.8,17003428 +9062.1,14566.8,15858.6,10234.7,17020305 +9067.0,14567.6,15858.6,10235.3,17038817 +9071.9,14568.1,15858.6,10235.9,17059695 +9076.7,14568.6,15858.6,10236.7,17077931 +9081.6,14573.5,15858.6,10241.0,17096199 +9086.4,14573.5,15858.6,10241.4,17118375 +9091.2,14571.3,15858.6,10239.3,17134019 +9096.0,14572.6,15858.6,10240.8,17150501 +9100.8,14575.3,15858.6,10243.2,17169753 +9105.8,14578.4,15858.6,10246.6,17169753 +9110.7,14579.4,15858.6,10247.6,17169753 +9115.5,14573.5,15858.6,10241.6,17169753 +9120.4,14576.2,15858.6,10242.4,17169753 +9125.3,14575.0,15858.6,10242.7,17169753 +9130.0,14577.3,15858.6,10244.2,17169753 +9134.8,14577.5,15858.6,10245.6,17169753 +9139.5,14573.8,15858.6,10241.5,17169753 +9144.4,14570.1,15858.6,10236.9,17182730 +9149.2,14570.8,15858.6,10237.7,17201704 +9154.1,14567.7,15858.6,10235.8,17220156 +9158.9,14572.7,15858.6,10240.7,17239222 +9163.7,14568.0,15858.6,10235.5,17257180 +9168.6,14574.4,15858.6,10242.1,17274778 +9173.3,14575.0,15858.6,10243.5,17292476 +9178.1,14576.9,15858.6,10244.6,17311288 +9183.1,14572.3,15858.6,10240.1,17328522 +9187.9,14573.8,15858.6,10241.7,17328522 +9192.7,14573.4,15858.6,10241.3,17328522 +9197.5,14574.4,15858.6,10242.1,17328522 +9202.3,14571.9,15858.6,10239.4,17328522 +9207.2,14574.2,15858.6,10241.5,17328522 +9212.0,14572.8,15858.6,10240.9,17328522 +9216.9,14575.0,15858.6,10242.8,17328522 +9221.8,14574.4,15858.6,10242.0,17328522 +9226.7,14574.8,15858.6,10242.4,17344766 +9231.7,14570.0,15858.6,10238.0,17364206 +9236.6,14569.8,15858.6,10236.5,17381562 +9241.7,14569.2,15858.6,10237.0,17401676 +9246.7,14568.6,15858.6,10236.3,17414730 +9251.6,14580.9,15858.6,10249.0,17421262 +9256.4,14596.7,15858.6,10264.6,17448260 +9261.3,14600.1,15858.6,10266.8,17466574 +9266.6,14580.5,15858.6,10247.8,17485474 +9271.4,14574.9,15858.6,10242.2,17502884 +9276.3,14567.6,15858.6,10235.3,17502884 +9281.1,14568.6,15858.6,10236.8,17502884 +9285.9,14568.8,15858.6,10236.9,17502884 +9290.7,14568.7,15858.6,10236.8,17502884 +9295.6,14575.9,15858.6,10243.8,17502884 +9300.5,14572.6,15858.6,10240.9,17502884 +9305.3,14574.5,15858.6,10242.1,17502884 +9310.2,14576.1,15858.6,10243.5,17502884 +9314.9,14573.6,15858.6,10241.6,17505174 +9319.6,14573.6,15858.6,10242.1,17524060 +9324.4,14571.2,15858.6,10238.6,17543402 +9329.3,14577.0,15858.6,10245.2,17562650 +9334.1,14568.6,15858.6,10236.8,17579172 +9339.1,14568.6,15858.6,10236.4,17598746 +9343.8,14569.8,15858.6,10237.7,17613850 +9348.9,14568.9,15858.6,10237.3,17630996 +9353.8,14569.5,15858.6,10236.8,17646670 +9358.8,14566.8,15858.6,10234.5,17666604 +9363.6,14565.5,15858.6,10233.4,17682844 +9368.4,14567.0,15858.6,10234.8,17682844 +9373.4,14569.2,15858.6,10236.7,17682844 +9378.2,14568.2,15858.6,10236.3,17682844 +9383.0,14570.8,15858.6,10239.2,17682844 +9388.0,14572.2,15858.6,10240.3,17682844 +9392.8,14570.4,15858.6,10238.1,17682844 +9397.7,14568.5,15858.6,10235.7,17682844 +9402.5,14571.6,15858.6,10239.2,17682844 +9407.4,14571.1,15858.6,10238.8,17683901 +9412.3,14569.7,15858.6,10237.6,17701713 +9417.2,14568.8,15858.6,10236.4,17722261 +9422.0,14572.9,15858.6,10240.7,17739863 +9426.9,14569.3,15858.6,10236.8,17758319 +9431.7,14568.7,15858.6,10236.3,17777015 +9436.6,14567.0,15858.6,10234.9,17792589 +9441.5,14571.2,15858.6,10238.9,17809661 +9446.5,14568.5,15858.6,10236.1,17825559 +9451.3,14567.6,15858.6,10235.6,17846189 +9456.0,14568.6,15858.6,10236.6,17846189 +9460.8,14570.4,15858.6,10238.7,17846189 +9465.7,14569.9,15858.6,10237.7,17846189 +9470.6,14568.3,15858.6,10236.2,17846189 +9475.4,14567.1,15858.6,10234.6,17846189 +9480.3,14566.2,15858.6,10234.1,17846189 +9485.2,14568.3,15858.6,10236.3,17846189 +9489.9,14569.7,15858.6,10238.3,17849797 +9494.7,14572.4,15858.6,10240.3,17870021 +9499.6,14569.8,15858.6,10238.3,17889773 +9504.4,14572.9,15858.6,10240.7,17912075 +9509.3,14567.7,15858.6,10235.3,17931267 +9514.1,14564.5,15858.6,10231.9,17950959 +9518.8,14566.1,15858.6,10233.8,17965685 +9523.6,14563.6,15858.6,10231.3,17984449 +9528.3,14566.1,15858.6,10234.0,18000191 +9533.2,14561.6,15858.6,10229.2,18015693 +9537.9,14563.0,15858.6,10230.7,18031705 +9542.9,14562.8,15858.6,10230.4,18031705 +9547.9,14565.4,15858.6,10232.9,18031705 +9552.8,14567.2,15858.6,10234.7,18031705 +9557.7,14566.5,15858.6,10234.3,18031705 +9562.7,14569.4,15858.6,10236.9,18031705 +9567.6,14575.9,15858.6,10243.9,18031705 +9572.6,14597.0,15858.6,10265.0,18031705 +9577.6,14607.1,15858.6,10275.1,18031705 +9582.5,14624.3,15858.6,10291.8,18031705 +9587.3,14639.8,15858.6,10307.3,18031705 +9592.2,14663.8,15858.6,10331.4,18039207 +9597.0,14629.1,15858.6,10296.9,18060555 +9602.0,14583.1,15858.6,10251.2,18069227 +9606.9,14592.6,15858.6,10260.9,18081699 +9611.9,14584.7,15858.6,10252.2,18107997 +9616.8,14584.3,15858.6,10251.8,18118653 +9621.6,14567.9,15858.6,10235.7,18138697 +9626.6,14570.2,15858.6,10238.6,18155363 +9631.5,14566.7,15858.6,10235.3,18169963 +9636.4,14570.1,15858.6,10238.2,18188037 +9641.2,14566.7,15858.6,10234.6,18203701 +9646.2,14567.5,15858.6,10235.3,18203701 +9651.0,14564.7,15858.6,10232.6,18203701 +9655.9,14564.1,15858.6,10232.5,18203701 +9660.8,14563.8,15858.6,10232.0,18203701 +9665.7,14567.3,15858.6,10235.3,18203701 +9670.7,14567.1,15858.6,10234.3,18203701 +9675.6,14568.8,15858.6,10236.4,18203701 +9680.6,14567.1,15858.6,10235.2,18203701 +9685.4,14569.6,15858.6,10237.2,18211352 +9690.3,14567.9,15858.6,10235.7,18230836 +9695.1,14567.1,15858.6,10235.3,18247312 +9700.0,14566.5,15858.6,10234.0,18266502 +9704.9,14568.8,15858.6,10237.2,18285758 +9709.6,14563.6,15858.6,10231.2,18303326 +9714.6,14564.4,15858.6,10232.2,18317510 +9719.6,14562.9,15858.6,10230.5,18334048 +9724.5,14564.2,15858.6,10231.6,18350794 +9729.5,14565.7,15858.6,10233.1,18367618 +9734.3,14570.0,15858.6,10238.1,18367618 +9739.2,14574.3,15858.6,10242.2,18367618 +9743.9,14572.6,15858.6,10241.2,18367618 +9748.7,14579.6,15858.6,10247.9,18367618 +9753.6,14598.9,15858.6,10266.9,18367618 +9758.6,14590.3,15858.6,10257.8,18367618 +9763.5,14575.5,15858.6,10243.1,18367618 +9768.5,14569.2,15858.6,10236.6,18367618 +9773.3,14565.1,15858.6,10233.3,18367618 +9778.3,14565.1,15858.6,10233.2,18386257 +9783.1,14568.2,15858.6,10236.0,18402821 +9787.9,14565.0,15858.6,10233.2,18421043 +9792.7,14563.5,15858.6,10231.7,18440405 +9797.5,14566.9,15858.6,10235.3,18457151 +9802.6,14568.0,15858.6,10235.7,18475243 +9807.8,14568.7,15858.6,10236.7,18495457 +9812.8,14565.4,15858.6,10233.8,18509259 +9817.6,14565.2,15858.6,10233.2,18525345 +9822.7,14564.9,15858.6,10232.8,18525345 +9827.6,14567.7,15858.6,10235.5,18525345 +9832.5,14568.2,15858.6,10236.4,18525345 +9837.3,14567.1,15858.6,10235.2,18525345 +9842.1,14566.6,15858.6,10234.5,18525345 +9846.9,14566.0,15858.6,10234.4,18525345 +9851.7,14564.9,15858.6,10233.1,18525345 +9856.6,14564.3,15858.6,10232.6,18525345 +9861.4,14564.9,15858.6,10232.3,18539698 +9866.2,14564.5,15858.6,10232.5,18558992 +9871.0,14560.9,15858.6,10228.6,18572206 +9876.0,14563.5,15858.6,10231.1,18592022 +9880.9,14562.0,15858.6,10229.3,18610202 +9885.8,14564.2,15858.6,10232.5,18629332 +9890.6,14562.1,15858.6,10230.2,18645458 +9895.5,14565.8,15858.6,10233.7,18662434 +9900.3,14561.6,15858.6,10229.8,18679656 +9905.2,14563.1,15858.6,10230.7,18692376 +9910.0,14562.4,15858.6,10229.9,18692376 +9914.8,14563.5,15858.6,10230.9,18692376 +9919.6,14562.4,15858.6,10229.9,18692376 +9924.5,14560.9,15858.6,10229.1,18692376 +9929.3,14575.2,15858.6,10242.8,18692376 +9934.3,14599.1,15858.6,10266.5,18692376 +9939.2,14590.2,15858.6,10258.6,18692376 +9943.9,14578.1,15858.6,10246.1,18692376 +9948.7,14568.9,15858.6,10236.4,18692376 +9953.7,14565.1,15858.6,10232.6,18701734 +9958.4,14570.2,15858.6,10237.6,18712510 +9963.3,14570.1,15858.6,10238.1,18735880 +9968.1,14573.2,15858.6,10241.2,18757608 +9972.9,14588.7,15858.6,10256.5,18775990 +9977.7,14588.4,15858.6,10256.4,18793812 +9982.5,14605.1,15858.6,10272.7,18811392 +9987.4,14603.1,15858.6,10271.3,18831526 +9992.4,14604.7,15858.6,10272.1,18845330 +9997.3,14600.2,15858.6,10267.9,18859966 +10002.1,14600.7,15858.6,10268.4,18859966 +10007.0,14604.1,15858.6,10272.1,18859966 +10011.8,14603.4,15858.6,10271.5,18859966 +10016.8,14602.0,15858.6,10269.7,18859966 +10021.6,14601.7,15858.6,10269.2,18859966 +10026.4,14605.2,15858.6,10273.2,18859966 +10031.3,14605.1,15858.6,10273.4,18859966 +10036.1,14605.2,15858.6,10273.1,18859966 +10041.1,14600.7,15858.6,10269.0,18859966 +10045.9,14603.3,15858.6,10271.0,18864762 +10050.9,14603.0,15858.6,10270.9,18887572 +10055.8,14605.6,15858.6,10273.2,18905646 +10060.8,14605.2,15858.6,10272.9,18925550 +10065.6,14600.2,15858.6,10268.2,18941526 +10070.4,14606.6,15858.6,10274.4,18959184 +10075.3,14604.8,15858.6,10273.1,18978724 +10080.4,14602.3,15858.6,10269.6,18997842 +10085.3,14599.0,15858.6,10267.6,19011924 +10090.1,14599.6,15858.6,10267.8,19026508 +10094.9,14601.7,15858.6,10269.5,19045148 +10099.9,14603.7,15858.6,10272.1,19045148 +10104.8,14609.7,15858.6,10278.3,19045148 +10109.9,14618.3,15858.6,10285.6,19045148 +10114.8,14626.0,15858.6,10293.8,19045148 +10119.8,14639.7,15858.6,10307.2,19045148 +10124.8,14657.2,15858.6,10325.2,19045148 +10129.6,14649.9,15858.6,10318.0,19045148 +10134.5,14610.6,15858.6,10278.9,19045148 +10139.3,14609.7,15858.6,10277.3,19045148 +10144.1,14600.1,15858.6,10267.4,19045148 +10148.9,14602.3,15858.6,10269.8,19046717 +10153.7,14602.5,15858.6,10270.9,19062031 +10158.7,14606.8,15858.6,10275.2,19084263 +10163.7,14605.5,15858.6,10273.2,19105169 +10168.5,14605.2,15858.6,10273.2,19121481 +10173.3,14606.3,15858.6,10273.8,19138821 +10178.2,14602.9,15858.6,10270.5,19151237 +10183.0,14601.1,15858.6,10268.9,19167667 +10188.0,14598.8,15858.6,10267.6,19184811 +10192.8,14605.0,15858.6,10273.2,19184811 +10197.7,14602.8,15858.6,10271.3,19184811 +10202.7,14602.3,15858.6,10270.2,19184811 +10207.6,14608.5,15858.6,10276.3,19184811 +10212.5,14614.6,15858.6,10282.8,19184811 +10217.3,14631.4,15858.6,10299.7,19184811 +10222.4,14645.0,15859.1,10313.6,19184811 +10227.4,14644.3,15859.1,10312.0,19184811 +10232.2,14621.7,15859.1,10289.4,19184811 +10237.1,14614.1,15859.1,10283.1,19195418 +10242.2,14604.2,15859.1,10272.8,19214128 +10247.0,14603.8,15859.1,10272.2,19232616 +10251.9,14600.9,15859.1,10268.5,19252326 +10256.9,14603.2,15859.1,10271.8,19269724 +10261.8,14603.4,15859.1,10271.9,19288418 +10266.5,14605.0,15859.1,10272.5,19306104 +10271.4,14604.8,15859.1,10272.8,19321448 +10276.5,14601.8,15859.1,10269.6,19334104 +10281.4,14601.9,15859.1,10270.0,19349542 +10286.4,14603.6,15859.1,10271.4,19364284 +10291.3,14601.4,15859.1,10269.4,19364284 +10296.2,14605.7,15859.1,10273.2,19364284 +10301.1,14601.0,15859.1,10269.8,19364284 +10306.1,14607.7,15859.1,10275.9,19364284 +10311.2,14609.4,15859.1,10277.3,19364284 +10316.0,14621.5,15859.1,10289.7,19364284 +10321.0,14623.4,15859.1,10291.4,19364284 +10325.9,14628.9,15859.1,10296.6,19364284 +10331.0,14631.9,15859.1,10299.2,19364284 +10335.9,14633.5,15859.1,10301.0,19364284 +10340.8,14636.4,15859.1,10302.8,19364284 +10345.7,14641.3,15859.1,10309.2,19364284 +10350.7,14640.5,15859.1,10308.1,19371351 +10355.5,14611.3,15859.1,10279.1,19391201 +10360.4,14609.4,15859.1,10277.3,19407741 +10365.4,14606.2,15859.1,10274.4,19425639 +10370.3,14609.2,15859.1,10277.5,19446589 +10375.3,14603.4,15859.1,10271.2,19461325 +10380.2,14601.0,15859.1,10269.2,19476637 +10385.1,14598.9,15859.1,10266.8,19489859 +10389.9,14600.8,15859.1,10269.1,19506501 +10394.7,14598.9,15859.1,10266.7,19518729 +10399.7,14659.9,15862.0,10328.3,19518729 +10404.6,14717.2,15864.6,10384.2,19518729 +10409.4,14709.8,15864.6,10378.2,19518729 +10414.3,14679.5,15865.7,10347.5,19518729 +10419.1,14689.2,15865.7,10357.6,19525953 +10423.9,14690.8,15865.7,10358.7,19559363 +10428.7,14683.0,15865.7,10351.0,19590863 +10433.6,14677.4,15865.7,10345.3,19615733 +10438.6,14664.3,15865.7,10332.5,19661733 +10443.5,14676.8,15865.7,10345.5,19692109 +10448.5,14679.0,15865.7,10346.8,19726545 +10453.4,14670.7,15865.7,10339.7,19750579 +10458.3,14674.0,15865.7,10341.9,19785309 +10463.1,14677.4,15865.7,10345.7,19817185 +10468.0,14670.7,15865.7,10339.6,19847823 +10472.8,14668.8,15865.7,10336.9,19883777 +10477.7,14655.6,15865.7,10323.9,19910865 +10482.8,14658.5,15865.7,10326.1,19936775 +10487.7,14663.8,15865.7,10332.7,19967979 +10492.6,14650.8,15865.7,10319.5,19993519 +10497.5,14665.7,15866.9,10333.5,19993519 +10502.3,14662.2,15868.9,10330.2,19993519 +10507.3,14646.3,15868.9,10314.0,19993519 +10512.3,14653.5,15868.9,10322.3,19993519 +10517.3,14653.3,15868.9,10321.2,19993519 +10522.2,14650.7,15868.9,10317.8,19993519 +10527.2,14644.8,15868.9,10312.6,19993519 +10532.1,14650.9,15868.9,10319.2,19993519 +10537.0,14648.2,15868.9,10316.2,19993519 +10541.9,14649.5,15868.9,10317.4,19993519 +10546.8,14649.2,15868.9,10317.3,20009615 +10551.7,14650.5,15868.9,10318.4,20038889 +10556.7,14661.4,15868.9,10329.0,20067641 +10561.7,14660.4,15868.9,10328.6,20099681 +10566.6,14645.6,15868.9,10312.6,20127835 +10571.6,14642.7,15868.9,10310.2,20155671 +10576.4,14640.0,15868.9,10308.1,20177071 +10581.4,14640.3,15868.9,10307.9,20200541 +10586.2,14631.3,15868.9,10299.7,20220525 +10591.1,14634.4,15868.9,10302.6,20247409 +10596.2,14634.1,15868.9,10301.8,20266685 +10601.1,14636.4,15868.9,10304.5,20284251 +10606.0,14642.5,15868.9,10310.2,20312417 +10611.0,14635.7,15868.9,10302.9,20335579 +10616.0,14641.6,15868.9,10309.4,20335579 +10620.9,14641.0,15868.9,10308.4,20335579 +10625.7,14637.6,15868.9,10305.7,20335579 +10630.6,14653.5,15868.9,10321.3,20335579 +10635.8,14667.6,15868.9,10336.0,20335579 +10640.8,14684.9,15868.9,10352.4,20335579 +10645.7,14665.6,15868.9,10333.2,20335579 +10650.6,14652.3,15868.9,10320.4,20335579 +10655.5,14638.9,15868.9,10306.7,20335579 +10660.4,14643.3,15868.9,10311.1,20335579 +10665.4,14640.3,15868.9,10308.3,20335579 +10670.3,14651.2,15868.9,10318.9,20335579 +10675.2,14639.1,15868.9,10306.9,20335579 +10680.3,14633.7,15868.9,10301.5,20337597 +10685.2,14623.2,15868.9,10291.0,20340205 +10690.3,14651.0,15868.9,10319.5,20351311 +10695.2,14666.5,15869.1,10334.6,20395235 +10700.0,14664.8,15869.3,10332.4,20422661 +10704.8,14646.8,15869.3,10315.3,20440945 +10709.7,14642.6,15869.3,10310.6,20467433 +10714.6,14635.7,15869.3,10304.2,20493025 +10719.6,14634.9,15869.3,10301.9,20514785 +10724.5,14624.5,15869.3,10292.3,20536065 +10729.5,14624.3,15869.3,10292.4,20558767 +10734.6,14635.7,15869.3,10304.1,20581775 +10739.6,14644.2,15869.3,10312.0,20581775 +10744.6,14638.7,15869.3,10306.8,20581775 +10749.5,14631.9,15869.3,10299.6,20581775 +10754.4,14626.1,15869.3,10292.5,20581775 +10759.3,14619.0,15869.3,10287.4,20581775 +10764.1,14622.6,15869.3,10290.2,20581775 +10769.0,14642.7,15869.3,10310.6,20581775 +10773.9,14648.3,15869.3,10315.8,20581775 +10778.7,14644.4,15869.3,10312.1,20581775 +10783.6,14630.8,15869.3,10298.8,20581775 +10788.5,14629.2,15869.3,10296.7,20581775 +10793.3,14622.2,15869.3,10290.5,20585914 +10798.3,14626.4,15869.3,10294.2,20611022 +10803.2,14623.4,15869.3,10291.3,20635640 +10808.2,14620.6,15869.3,10288.5,20656938 +10813.1,14623.9,15869.3,10291.5,20682088 +10818.1,14628.0,15869.3,10296.3,20700126 +10822.8,14624.9,15869.3,10293.1,20722944 +10827.8,14619.8,15869.3,10287.7,20741030 +10832.8,14624.5,15869.3,10293.4,20761098 +10837.8,14614.2,15869.3,10282.3,20779394 +10842.7,14614.5,15869.3,10282.0,20797378 +10847.5,14616.0,15869.3,10283.7,20815180 +10852.3,14613.0,15869.3,10280.5,20815180 +10857.2,14613.8,15869.3,10281.5,20815180 +10862.1,14630.8,15869.3,10298.8,20815180 +10866.9,14648.8,15869.3,10316.6,20815180 +10871.9,14653.4,15869.3,10321.3,20815180 +10876.8,14635.2,15869.3,10302.9,20815180 +10881.6,14635.2,15869.3,10303.2,20815180 +10886.4,14620.3,15869.3,10287.8,20815180 +10891.2,14626.4,15869.3,10294.7,20815180 +10896.0,14614.7,15869.3,10283.0,20815180 +10900.7,14616.8,15869.3,10285.2,20815180 +10905.6,14612.4,15869.3,10279.9,20815180 +10910.7,14610.4,15869.3,10278.3,20825008 +10915.6,14603.8,15869.3,10271.7,20844690 +10920.5,14620.6,15869.3,10288.6,20864710 +10925.3,14615.5,15869.3,10283.0,20883252 +10930.2,14616.9,15869.3,10284.5,20906076 +10935.3,14613.8,15869.3,10281.5,20926844 +10940.5,14614.5,15869.3,10282.4,20946108 +10945.4,14610.7,15869.3,10278.6,20964516 +10950.5,14612.5,15869.3,10280.5,20982992 +10955.4,14608.1,15869.3,10275.8,21002788 +10960.3,14611.3,15869.3,10278.4,21002788 +10965.4,14610.9,15869.3,10278.3,21002788 +10970.3,14610.0,15869.3,10277.6,21002788 +10975.1,14604.8,15869.3,10272.8,21002788 +10980.0,14609.4,15869.3,10277.3,21002788 +10984.9,14606.4,15869.3,10274.5,21002788 +10989.9,14610.4,15869.3,10277.8,21002788 +10994.7,14608.0,15869.3,10275.8,21002788 +10999.7,14608.4,15869.3,10276.0,21002788 +11004.5,14608.1,15869.3,10276.2,21002788 +11009.3,14605.8,15869.3,10273.8,21003088 +11014.2,14606.6,15869.3,10274.5,21024036 +11019.1,14607.7,15869.3,10276.1,21040888 +11024.0,14604.5,15869.3,10273.1,21061368 +11028.8,14606.7,15869.3,10274.6,21071072 +11033.5,14595.0,15869.3,10262.9,21071072 +11038.4,14586.6,15869.3,10254.5,21071072 +11043.2,14579.8,15869.3,10247.7,21071072 +11047.9,14576.9,15869.3,10244.8,21071072 +11052.7,14576.8,15869.3,10244.7,21071072 +11057.5,14576.5,15869.3,10244.4,21071072 +11062.4,14576.5,15869.3,10244.4,21071072 +11067.2,14576.4,15869.3,10244.3,21071072 +11071.9,14576.2,15869.3,10244.1,21071072 +11076.6,14576.0,15869.3,10243.9,21071072 +11081.3,14575.9,15869.3,10243.8,21071072 +11086.0,14575.7,15869.3,10243.7,21071072 +11090.8,14575.6,15869.3,10243.6,21071072 +11095.6,14575.6,15869.3,10243.6,21071072 +11100.3,14575.6,15869.3,10243.5,21071072 +11105.2,14575.5,15869.3,10243.5,21071072 +11109.9,14601.0,15869.3,10269.1,21071072 +11114.6,14633.7,15869.3,10301.7,21071072 +11119.4,14665.9,15869.3,10333.8,21071072 +11124.2,14695.8,15869.3,10364.0,21071072 +11128.9,14763.5,15869.3,10433.6,21071072 +11133.8,14876.5,15869.3,10545.9,21071072 +11138.5,14982.2,15869.3,10650.0,21071072 +11143.3,14765.1,15869.3,10432.7,21071072 +11148.1,14578.5,15869.3,10246.1,21071072 +11152.9,14578.4,15869.3,10246.0,21071072 +11157.8,14578.3,15869.3,10246.0,21071072 +11162.8,14578.2,15869.3,10245.8,21071072 +11167.6,14578.1,15869.3,10245.8,21071072 +11172.3,14577.9,15869.3,10245.6,21071072 +11176.9,14577.9,15869.3,10245.5,21071072 +11181.7,14577.7,15869.3,10245.3,21071072 +11186.4,14608.4,15869.3,10276.2,21071072 +11191.2,14642.1,15869.3,10310.0,21071072 +11195.9,14673.4,15869.3,10341.3,21071072 +11200.7,14702.4,15869.3,10370.7,21071072 +11205.5,14782.6,15869.3,10452.2,21071072 +11210.2,14890.3,15869.3,10558.7,21071072 diff --git a/benchmarks/realtime-data-indexing/results/30gb-processes.jsonl.gz b/benchmarks/realtime-data-indexing/results/30gb-processes.jsonl.gz new file mode 100644 index 0000000..7ace5a2 Binary files /dev/null and b/benchmarks/realtime-data-indexing/results/30gb-processes.jsonl.gz differ diff --git a/benchmarks/realtime-data-indexing/results/30gb-summary.json b/benchmarks/realtime-data-indexing/results/30gb-summary.json new file mode 100644 index 0000000..31b9a07 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/30gb-summary.json @@ -0,0 +1,283 @@ +{ + "dataset": "30gb", + "corpus_mb": 30000.0, + "documents": 9202620, + "chunks": 20933962, + "max_observed_chunks": 21071072, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count", + "indexing_seconds": 11028.8, + "throughput_mb_per_min": 163.21, + "peak_rss_mb": 14982.2, + "peak_pss_mb": 10653.8, + "peak_processes": [ + { + "pid": 1, + "worker_rank": null, + "VmHWM": 990236, + "VmRSS": 990064, + "Pss": 626430, + "Private_Clean": 308, + "Private_Dirty": 583628, + "mapping_groups": { + "embedding": { + "Pss": 50121, + "private_kb": 20004 + }, + "other": { + "Pss": 575784, + "private_kb": 563932 + } + } + }, + { + "pid": 131, + "worker_rank": null, + "VmHWM": 183180, + "VmRSS": 183064, + "Pss": 106999, + "Private_Clean": 48, + "Private_Dirty": 98500, + "mapping_groups": { + "embedding": { + "Pss": 0, + "private_kb": 0 + }, + "other": { + "Pss": 106783, + "private_kb": 98548 + } + } + }, + { + "pid": 146, + "worker_rank": 0, + "VmHWM": 5524492, + "VmRSS": 4688216, + "Pss": 4206433, + "Private_Clean": 672, + "Private_Dirty": 4144476, + "mapping_groups": { + "embedding": { + "Pss": 63910, + "private_kb": 19844 + }, + "other": { + "Pss": 3837409, + "private_kb": 3820708 + } + } + }, + { + "pid": 147, + "worker_rank": 1, + "VmHWM": 1366804, + "VmRSS": 1354812, + "Pss": 853678, + "Private_Clean": 352, + "Private_Dirty": 791896, + "mapping_groups": { + "embedding": { + "Pss": 64360, + "private_kb": 20012 + }, + "other": { + "Pss": 787846, + "private_kb": 771272 + } + } + }, + { + "pid": 148, + "worker_rank": 2, + "VmHWM": 1363468, + "VmRSS": 1352284, + "Pss": 850974, + "Private_Clean": 428, + "Private_Dirty": 789228, + "mapping_groups": { + "embedding": { + "Pss": 63891, + "private_kb": 19844 + }, + "other": { + "Pss": 786818, + "private_kb": 770068 + } + } + }, + { + "pid": 149, + "worker_rank": 3, + "VmHWM": 1362156, + "VmRSS": 1352776, + "Pss": 851694, + "Private_Clean": 0, + "Private_Dirty": 790560, + "mapping_groups": { + "embedding": { + "Pss": 63910, + "private_kb": 19844 + }, + "other": { + "Pss": 787268, + "private_kb": 770716 + } + } + }, + { + "pid": 150, + "worker_rank": 4, + "VmHWM": 1359792, + "VmRSS": 1348816, + "Pss": 851688, + "Private_Clean": 1548, + "Private_Dirty": 787716, + "mapping_groups": { + "embedding": { + "Pss": 63957, + "private_kb": 19880 + }, + "other": { + "Pss": 787206, + "private_kb": 769384 + } + } + }, + { + "pid": 151, + "worker_rank": 5, + "VmHWM": 1361732, + "VmRSS": 1351420, + "Pss": 855209, + "Private_Clean": 2032, + "Private_Dirty": 789780, + "mapping_groups": { + "embedding": { + "Pss": 65469, + "private_kb": 20428 + }, + "other": { + "Pss": 789216, + "private_kb": 771384 + } + } + }, + { + "pid": 152, + "worker_rank": 6, + "VmHWM": 1358432, + "VmRSS": 1349080, + "Pss": 851454, + "Private_Clean": 720, + "Private_Dirty": 789072, + "mapping_groups": { + "embedding": { + "Pss": 65041, + "private_kb": 20148 + }, + "other": { + "Pss": 785630, + "private_kb": 769388 + } + } + }, + { + "pid": 153, + "worker_rank": 7, + "VmHWM": 1368916, + "VmRSS": 1355724, + "Pss": 854924, + "Private_Clean": 60, + "Private_Dirty": 793968, + "mapping_groups": { + "embedding": { + "Pss": 63874, + "private_kb": 19868 + }, + "other": { + "Pss": 790534, + "private_kb": 774160 + } + } + } + ], + "peak_elapsed_s": 8678.846100345254, + "jemalloc_conf": "background_thread:true", + "omp_threads": 12, + "indexer_cpu_seconds": 99590.4, + "avg_parallelism": 9.0, + "vector_db_disk_mb": 49738.0, + "qdrant_image_id": "sha256:0ad2e23181e5646b286253c04cea23f07040ca313cc377d9fa7b21db184b6406", + "vector_db_mb_per_corpus_mb": 1.66, + "retrieval_accuracy": "14/20", + "embedder": "sentence-transformers/static-retrieval-mrl-en-v1 (local, 256 dims)", + "approx_text_pages": 15728646, + "accuracy_misses": [ + { + "expected": "0000/6000-Tom-Tango.txt", + "query": "Tom Tango: Tom Tango and \"TangoTiger\" are aliases used online by a baseball sabermetrics and ice hockey statistics analy", + "got": [ + "0240/2077-Tango-disambiguation.txt", + "0240/2077-Tango-disambiguation.txt", + "0543/5737-Tango.txt", + "0598/4990-History-of-the-tango.txt", + "0295/8960-Histoire-du-Tango.txt" + ] + }, + { + "expected": "0001/6000-Ricky-Fuji.txt", + "query": "Ricky Fuji: (born September 27, 1965) better known under his ring name is a Japanese professional wrestler currently si", + "got": [ + "0592/7826-Kazuyuki-Fujita.txt", + "0369/6383-Sho-wrestler.txt", + "0592/1210-Tatsumi-Fujinami.txt", + "0454/6595-Daiki-Inaba.txt", + "0321/0463-Jeff-Cobb.txt" + ] + }, + { + "expected": "0003/2000-Theroigne-de-Mericourt.txt", + "query": "Theroigne de Mericourt: Anne-Jos\u00e8phe Th\u00e9roigne de M\u00e9ricourt (born Anne-Jos\u00e8phe Terwagne; 13 August 1762 \u2013 8 June 1817) w", + "got": [ + "0886/9728-Anne-Jos-phe-Th-roigne-de-M-ricourt.txt", + "0886/9728-Anne-Jos-phe-Th-roigne-de-M-ricourt.txt", + "0833/1829-Charles-III-de-Rohan.txt", + "0886/9728-Anne-Jos-phe-Th-roigne-de-M-ricourt.txt", + "0023/8794-Jean-Baptiste-Jacques-Augustin.txt" + ] + }, + { + "expected": "0003/6000-Black-capped-fruit-bat.txt", + "query": "Black-capped fruit bat: The black-capped fruit bat (Chironax melanocephalus) is a species of megabat in the monotypic ge", + "got": [ + "0551/7828-Leaf-nosed-bat.txt", + "0033/2843-Black-winged-little-yellow-bat.txt", + "0083/8702-Wrinkle-faced-bat.txt", + "0046/2629-Antillean-fruit-eating-bat.txt", + "0100/5548-List-of-bats.txt" + ] + }, + { + "expected": "0005/6000-Jamie-Mueller.txt", + "query": "Jamie Mueller: Jamie F.", + "got": [ + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0623/5680-Jamie-Oliver.txt", + "0000/6975-Jamie.txt", + "0039/1332-Jamieson.txt" + ] + }, + { + "expected": "0005/8000-Ruhleben-internment-camp.txt", + "query": "Ruhleben internment camp: Ruhleben internment camp was a civilian detention camp in Germany during World War I.", + "got": [ + "0809/3391-S-ok-w.txt", + "0639/3929-R-hle-Familienname.txt", + "0638/7530-Ruhl.txt", + "0676/0506-Kohlenw-sche.txt", + "0849/5679-Ruhden.txt" + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/3gb-compose.json b/benchmarks/realtime-data-indexing/results/3gb-compose.json new file mode 100644 index 0000000..de20431 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/3gb-compose.json @@ -0,0 +1,170 @@ +{ + "name": "serviette-consistency-bench", + "networks": { + "default": { + "name": "serviette-consistency-bench_default", + "ipam": {} + } + }, + "services": { + "indexer": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "indexer", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OMP_NUM_THREADS": "12", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/3gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + }, + "qdrant": { + "command": null, + "entrypoint": null, + "image": "qdrant/qdrant:latest", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 6333, + "published": "8301", + "protocol": "tcp" + } + ] + }, + "server": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "server", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 8000, + "published": "8300", + "protocol": "tcp" + } + ], + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/3gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + } + }, + "x-serviette": { + "build": { + "context": "../..", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "PV3L-4L4N-TF7R-JWM7-LKVK-PX3E-FHTX-KMNC", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "volumes": [ + "~/.cache/huggingface:/hf-cache:ro", + "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/3gb/docs:/data/docs:ro", + "./config.yaml:/bench/config.yaml:ro" + ] + } +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/3gb-expected-output.json b/benchmarks/realtime-data-indexing/results/3gb-expected-output.json new file mode 100644 index 0000000..26bd54b --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/3gb-expected-output.json @@ -0,0 +1,8 @@ +{ + "documents": 841890, + "exact_chunks": 1888846, + "corpus_bytes": 3145728437, + "dataset": "3gb", + "elapsed_seconds": 132.3, + "source": "Frozen corpus files; UTF-8 decode and actual build_xpack_splitter, token_count=512; executed in immutable serviette-bench image" +} diff --git a/benchmarks/realtime-data-indexing/results/3gb-memory.csv b/benchmarks/realtime-data-indexing/results/3gb-memory.csv new file mode 100644 index 0000000..4af1c53 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/3gb-memory.csv @@ -0,0 +1,239 @@ +elapsed_s,rss_mb,hwm_mb,pss_mb,chunks +11.3,1160.8,1160.8,916.3,0 +14.9,2899.9,2903.6,1955.7,0 +19.3,7607.2,7607.2,4830.1,0 +23.8,8978.5,8978.5,5580.5,0 +28.0,9240.6,9362.9,5824.7,18 +32.8,10887.3,10890.9,6841.0,20741 +37.2,10949.6,10971.7,6870.4,36392 +41.8,10955.4,10997.4,6829.8,46668 +46.5,10994.7,11011.9,6833.0,55937 +51.3,11016.8,11029.7,6844.0,66962 +55.9,11076.1,11080.9,6854.5,77132 +60.5,11107.6,11110.5,6866.1,86724 +64.9,11116.2,11128.0,6860.2,94219 +69.6,11143.3,11147.7,6880.4,104560 +74.2,11152.9,11160.6,6885.3,114310 +79.0,11173.5,11175.5,6895.8,126681 +83.4,11179.7,11185.5,6899.0,135915 +88.0,11196.9,11198.8,6912.2,146506 +92.7,11206.7,11210.9,6916.5,157506 +97.2,11223.6,11227.7,6932.4,167008 +101.8,11224.1,11233.9,6932.6,176748 +106.3,11230.3,11237.9,6934.8,189840 +110.8,11237.6,11244.1,6943.1,199141 +115.4,11243.5,11248.3,6948.7,208275 +120.1,11254.3,11255.7,6957.8,222485 +124.8,11257.0,11261.6,6959.0,231855 +129.4,11260.0,11263.2,6961.7,240697 +133.9,11262.4,11267.9,6964.4,247514 +138.4,11266.7,11271.7,6969.6,257898 +142.9,11265.5,11273.8,6968.3,268467 +147.7,11274.6,11276.6,6978.0,280395 +152.4,11278.5,11283.9,6981.7,290462 +157.1,11278.2,11285.6,6981.3,298950 +161.7,11290.3,11295.3,6993.2,308742 +166.4,11287.4,11296.4,6990.4,317720 +171.1,11291.5,11298.4,6994.1,330935 +175.8,11298.8,11304.8,7001.0,340998 +180.4,11310.0,11312.3,7012.1,350989 +185.1,11309.0,11315.5,7010.4,365625 +189.6,11315.4,11318.9,7015.6,373867 +194.4,11313.1,11323.2,7014.5,381916 +198.9,11322.0,11325.7,7021.6,392351 +203.6,11315.6,11326.7,7015.3,401283 +208.1,11318.2,11327.0,7018.3,410004 +212.7,11320.6,11328.4,7021.1,420449 +217.3,11321.6,11328.6,7022.1,429277 +221.7,11323.6,11329.6,7023.8,438905 +226.5,11327.1,11331.7,7028.6,450295 +231.1,11327.6,11332.4,7029.6,459984 +235.6,11330.7,11334.3,7032.2,470147 +240.5,11334.2,11336.7,7035.8,480050 +245.3,11339.2,11340.9,7040.9,492249 +249.8,11341.8,11343.5,7043.7,502348 +254.5,11338.7,11346.1,7040.5,512072 +259.0,11346.1,11349.4,7046.9,521852 +263.6,11347.9,11353.2,7048.5,539449 +268.4,11353.1,11355.7,7053.4,551332 +273.2,11347.7,11357.3,7048.8,552192 +278.0,11349.6,11358.0,7050.6,561555 +282.6,11349.1,11360.1,7050.0,576253 +287.2,11373.1,11384.8,7073.7,589667 +291.9,11366.2,11385.7,7067.2,597436 +296.7,11373.5,11390.4,7074.3,607606 +301.1,11373.8,11391.2,7074.0,616011 +305.5,11372.3,11391.3,7073.3,619952 +310.1,11378.0,11392.1,7078.5,632937 +314.8,11375.3,11392.7,7075.9,637846 +319.3,11377.1,11393.0,7078.1,648567 +324.1,11377.3,11393.6,7078.7,658660 +328.8,11380.7,11394.7,7082.7,669802 +333.4,11381.4,11395.5,7083.2,681259 +337.9,11388.1,11397.5,7089.8,689211 +342.7,11387.9,11399.4,7089.5,701460 +347.4,11396.5,11402.8,7098.0,710595 +351.9,11392.8,11404.3,7094.3,720024 +356.7,11392.8,11404.3,7094.3,736769 +361.1,11391.1,11404.4,7092.3,744296 +365.8,11391.7,11405.0,7092.8,752269 +370.5,11391.2,11405.4,7092.2,760510 +375.1,11391.6,11405.5,7091.5,771452 +379.8,11393.7,11406.4,7093.9,779160 +384.5,11439.6,11485.5,7139.8,787073 +389.1,11463.1,11513.1,7163.3,798157 +393.8,11457.7,11513.1,7157.7,807810 +398.4,11461.0,11513.1,7160.5,814370 +403.2,11460.9,11513.1,7161.0,821909 +407.9,11467.3,11513.1,7167.2,838357 +412.7,11469.2,11513.1,7169.7,851815 +417.5,11473.4,11513.1,7174.0,859976 +422.1,11472.3,11513.1,7172.8,871509 +426.9,11475.0,11513.1,7175.3,883756 +431.7,11472.2,11513.1,7172.8,891845 +436.4,11473.8,11513.1,7174.5,900794 +441.0,11475.4,11513.1,7175.5,909430 +445.6,11480.3,11513.1,7181.6,917407 +450.2,11483.7,11513.1,7184.5,925360 +454.7,11481.6,11513.1,7183.1,942708 +459.1,11485.6,11513.1,7186.6,951612 +463.9,11480.3,11513.1,7181.1,960737 +468.6,11484.6,11513.1,7186.0,968062 +473.1,11485.6,11513.1,7186.9,977659 +477.6,11489.9,11513.1,7190.8,982681 +482.4,11492.7,11513.1,7192.9,990592 +487.2,11495.3,11513.1,7194.6,1012458 +491.9,11495.4,11513.1,7195.6,1022736 +496.3,11500.0,11515.1,7200.2,1023083 +500.8,11501.5,11517.9,7201.7,1035372 +505.5,11501.9,11517.9,7202.2,1042885 +510.0,11501.9,11518.4,7202.6,1059289 +514.8,11503.2,11518.4,7203.5,1065844 +519.4,11504.9,11518.4,7205.0,1074254 +523.9,11501.2,11518.4,7201.6,1074993 +528.4,11521.7,11554.7,7222.1,1096107 +533.1,11522.7,11554.9,7224.0,1106898 +537.7,11524.5,11554.9,7225.9,1111947 +542.2,11525.8,11554.9,7226.4,1121117 +546.8,11528.6,11555.0,7229.8,1126673 +551.4,11528.1,11555.1,7229.3,1146695 +556.1,11529.0,11555.3,7229.7,1153475 +560.5,11530.7,11556.2,7231.5,1159876 +565.1,11531.5,11556.8,7232.1,1164050 +569.7,11542.9,11563.6,7243.9,1187562 +574.1,11536.1,11564.1,7237.4,1192732 +578.7,11538.9,11565.7,7239.4,1196356 +583.5,11539.4,11566.4,7239.8,1202365 +588.2,11541.6,11567.4,7242.0,1221099 +592.9,11546.2,11570.9,7246.7,1231909 +597.7,11551.5,11574.0,7250.8,1236871 +602.7,11551.6,11576.3,7250.9,1259105 +607.4,11552.1,11576.6,7252.0,1260047 +611.8,11549.0,11576.6,7249.1,1264513 +616.3,11543.7,11576.6,7244.5,1282453 +621.0,11548.6,11576.6,7249.0,1291133 +625.5,11546.7,11576.6,7247.1,1292704 +630.1,11550.2,11576.6,7250.8,1314398 +634.9,11553.4,11577.5,7253.7,1319330 +639.6,11554.2,11578.0,7254.8,1323444 +644.2,11576.3,11594.9,7277.3,1350332 +648.9,11580.7,11599.2,7279.7,1350332 +653.8,11571.6,11601.3,7270.8,1354735 +658.2,11571.7,11601.6,7270.9,1373737 +662.9,11565.8,11601.6,7264.8,1375835 +667.5,11567.1,11601.6,7266.5,1392877 +672.2,11562.8,11601.6,7262.0,1400375 +677.0,11567.1,11601.6,7266.6,1400375 +681.8,11565.4,11601.6,7265.3,1420814 +686.5,11567.0,11601.6,7266.7,1428100 +691.1,11564.2,11601.6,7263.8,1428100 +695.7,11567.4,11601.6,7267.4,1450795 +700.3,11566.0,11601.6,7266.3,1450795 +704.7,11566.0,11601.6,7265.7,1464920 +709.3,11571.6,11601.7,7271.2,1467374 +714.1,11570.2,11601.8,7270.4,1488652 +718.8,11575.0,11602.2,7274.7,1491955 +723.4,11575.2,11602.3,7275.6,1496638 +728.1,11579.7,11602.5,7279.9,1516496 +732.6,11575.5,11602.8,7276.0,1520942 +737.4,11580.1,11602.8,7280.8,1526732 +741.8,11577.9,11603.9,7277.7,1547572 +746.5,11581.6,11604.6,7282.2,1547572 +751.3,11580.2,11605.1,7279.9,1565919 +756.0,11582.2,11605.2,7282.5,1570684 +760.7,11582.9,11606.0,7283.0,1589420 +765.2,11583.9,11606.3,7283.8,1589420 +769.7,11585.7,11606.6,7286.3,1604402 +774.0,11583.5,11606.7,7284.1,1606124 +778.6,11588.1,11608.8,7288.7,1627370 +783.2,11583.0,11608.8,7283.7,1627370 +787.9,11585.7,11608.9,7285.9,1645836 +792.4,11585.3,11608.9,7286.3,1646953 +796.7,11591.7,11611.7,7292.5,1663793 +801.2,11590.9,11612.9,7291.5,1663793 +805.8,11594.6,11613.1,7295.6,1667788 +810.5,11598.9,11618.2,7299.6,1687472 +815.2,11594.7,11618.4,7295.7,1688387 +819.6,11598.5,11618.6,7299.9,1708453 +824.5,11601.4,11622.9,7305.5,1708453 +829.2,11608.6,11627.2,7310.6,1723222 +833.9,11603.9,11627.5,7305.8,1723222 +838.4,11611.7,11630.0,7312.7,1744220 +843.1,11611.5,11632.8,7311.5,1746519 +847.7,11612.6,11633.3,7312.6,1764249 +852.4,11613.3,11633.6,7313.0,1764633 +857.1,11614.0,11635.7,7313.1,1785483 +861.7,11614.4,11636.2,7313.6,1785483 +866.4,11616.7,11638.4,7316.0,1802190 +871.1,11619.6,11642.0,7319.0,1804507 +875.7,11616.7,11642.5,7316.3,1819737 +880.6,11620.9,11642.5,7321.2,1823147 +885.0,11620.7,11642.6,7321.3,1841187 +889.7,11624.7,11643.9,7325.7,1843276 +894.3,11619.2,11644.0,7319.9,1856156 +899.1,11618.2,11644.0,7319.2,1856156 +903.6,11617.3,11644.2,7318.4,1867951 +908.1,11621.7,11644.3,7323.6,1867951 +912.5,11621.8,11644.7,7322.6,1884540 +917.2,11620.5,11645.2,7320.9,1885622 +921.9,11620.1,11645.3,7320.5,1895558 +926.4,11639.3,11672.3,7339.6,1895558 +931.1,11636.2,11676.3,7336.5,1895558 +935.7,11651.3,11676.3,7351.8,1895558 +940.2,11623.2,11676.3,7323.9,1895558 +944.9,11615.6,11676.3,7316.0,1895558 +949.5,11617.7,11676.3,7318.1,1895558 +954.0,11622.3,11676.3,7322.7,1895558 +958.5,11624.7,11676.3,7325.1,1895558 +963.1,11641.7,11676.3,7342.4,1895558 +967.5,11623.8,11676.3,7324.3,1895558 +972.0,11612.2,11676.3,7312.4,1895558 +976.3,11617.5,11676.3,7317.7,1895558 +981.0,11617.5,11676.3,7317.7,1895558 +985.7,11640.4,11676.3,7341.0,1895558 +990.5,11625.1,11676.3,7325.5,1895558 +994.9,11612.6,11676.3,7313.0,1895558 +999.6,11614.2,11676.3,7314.4,1895558 +1004.1,11609.6,11676.3,7309.8,1895558 +1008.8,11620.8,11676.3,7321.0,1895558 +1013.4,11635.2,11676.3,7335.6,1895558 +1018.1,11622.9,11676.3,7323.2,1895558 +1022.9,11617.7,11676.3,7318.2,1895558 +1027.2,11616.2,11676.3,7316.4,1895558 +1031.7,11615.0,11676.3,7315.2,1895558 +1036.3,11608.7,11676.3,7308.9,1895558 +1041.1,11623.9,11676.3,7324.1,1895558 +1045.6,11636.0,11676.3,7336.5,1895558 +1050.0,11618.8,11676.3,7319.3,1895558 +1054.5,11611.4,11676.3,7311.5,1895558 +1059.2,11614.7,11676.3,7314.9,1895558 +1063.7,11621.6,11676.3,7321.8,1895558 +1068.4,11619.3,11676.3,7319.5,1895558 +1073.0,11631.3,11676.3,7331.9,1895558 +1077.5,11618.8,11676.3,7319.1,1895558 +1081.9,11616.5,11676.3,7316.7,1895558 +1086.4,11613.1,11676.3,7313.3,1895558 +1091.2,11642.1,11676.3,7342.4,1895558 +1095.9,11620.9,11676.3,7321.7,1895558 +1100.2,11615.0,11676.3,7315.3,1895558 +1104.8,11609.0,11676.3,7309.2,1895558 diff --git a/benchmarks/realtime-data-indexing/results/3gb-processes.jsonl.gz b/benchmarks/realtime-data-indexing/results/3gb-processes.jsonl.gz new file mode 100644 index 0000000..2c291ad Binary files /dev/null and b/benchmarks/realtime-data-indexing/results/3gb-processes.jsonl.gz differ diff --git a/benchmarks/realtime-data-indexing/results/3gb-summary.json b/benchmarks/realtime-data-indexing/results/3gb-summary.json new file mode 100644 index 0000000..c02e552 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/3gb-summary.json @@ -0,0 +1,249 @@ +{ + "dataset": "3gb", + "corpus_mb": 3000.0, + "documents": 841890, + "chunks": 1888846, + "max_observed_chunks": 1895558, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count", + "indexing_seconds": 921.9, + "throughput_mb_per_min": 195.26, + "peak_rss_mb": 11651.3, + "peak_pss_mb": 7351.8, + "peak_processes": [ + { + "pid": 1, + "worker_rank": null, + "VmHWM": 1025036, + "VmRSS": 1025036, + "Pss": 660185, + "Private_Clean": 220, + "Private_Dirty": 617688, + "mapping_groups": { + "embedding": { + "Pss": 49740, + "private_kb": 19844 + }, + "other": { + "Pss": 609922, + "private_kb": 598064 + } + } + }, + { + "pid": 131, + "worker_rank": null, + "VmHWM": 199100, + "VmRSS": 199100, + "Pss": 123689, + "Private_Clean": 64, + "Private_Dirty": 114400, + "mapping_groups": { + "embedding": { + "Pss": 0, + "private_kb": 0 + }, + "other": { + "Pss": 123477, + "private_kb": 114464 + } + } + }, + { + "pid": 159, + "worker_rank": 0, + "VmHWM": 1637576, + "VmRSS": 1637428, + "Pss": 1139125, + "Private_Clean": 828, + "Private_Dirty": 1074948, + "mapping_groups": { + "embedding": { + "Pss": 50787, + "private_kb": 19876 + }, + "other": { + "Pss": 1087818, + "private_kb": 1055900 + } + } + }, + { + "pid": 160, + "worker_rank": 1, + "VmHWM": 1294400, + "VmRSS": 1291236, + "Pss": 802754, + "Private_Clean": 1520, + "Private_Dirty": 740340, + "mapping_groups": { + "embedding": { + "Pss": 48696, + "private_kb": 19852 + }, + "other": { + "Pss": 753542, + "private_kb": 722008 + } + } + }, + { + "pid": 161, + "worker_rank": 2, + "VmHWM": 1302396, + "VmRSS": 1299492, + "Pss": 803531, + "Private_Clean": 472, + "Private_Dirty": 737176, + "mapping_groups": { + "embedding": { + "Pss": 53353, + "private_kb": 19848 + }, + "other": { + "Pss": 749662, + "private_kb": 717800 + } + } + }, + { + "pid": 162, + "worker_rank": 3, + "VmHWM": 1301700, + "VmRSS": 1297924, + "Pss": 799329, + "Private_Clean": 68, + "Private_Dirty": 736416, + "mapping_groups": { + "embedding": { + "Pss": 50167, + "private_kb": 19844 + }, + "other": { + "Pss": 748644, + "private_kb": 716640 + } + } + }, + { + "pid": 163, + "worker_rank": 4, + "VmHWM": 1301564, + "VmRSS": 1297204, + "Pss": 799480, + "Private_Clean": 352, + "Private_Dirty": 737080, + "mapping_groups": { + "embedding": { + "Pss": 49783, + "private_kb": 19844 + }, + "other": { + "Pss": 749179, + "private_kb": 717588 + } + } + }, + { + "pid": 164, + "worker_rank": 5, + "VmHWM": 1301196, + "VmRSS": 1297240, + "Pss": 802992, + "Private_Clean": 788, + "Private_Dirty": 736188, + "mapping_groups": { + "embedding": { + "Pss": 53088, + "private_kb": 20188 + }, + "other": { + "Pss": 749392, + "private_kb": 716788 + } + } + }, + { + "pid": 165, + "worker_rank": 6, + "VmHWM": 1302524, + "VmRSS": 1298364, + "Pss": 798797, + "Private_Clean": 88, + "Private_Dirty": 736192, + "mapping_groups": { + "embedding": { + "Pss": 49759, + "private_kb": 19844 + }, + "other": { + "Pss": 748518, + "private_kb": 716436 + } + } + }, + { + "pid": 166, + "worker_rank": 7, + "VmHWM": 1291068, + "VmRSS": 1287936, + "Pss": 798411, + "Private_Clean": 668, + "Private_Dirty": 735888, + "mapping_groups": { + "embedding": { + "Pss": 49272, + "private_kb": 20140 + }, + "other": { + "Pss": 748626, + "private_kb": 716416 + } + } + } + ], + "peak_elapsed_s": 935.7114451080561, + "jemalloc_conf": "background_thread:true", + "omp_threads": 12, + "indexer_cpu_seconds": 9358.7, + "avg_parallelism": 10.2, + "vector_db_disk_mb": 5010.6, + "vector_db_mb_per_corpus_mb": 1.67, + "retrieval_accuracy": "17/20", + "embedder": "sentence-transformers/static-retrieval-mrl-en-v1 (local, 256 dims)", + "approx_text_pages": 1572864, + "accuracy_misses": [ + { + "expected": "0003/6000-Black-capped-fruit-bat.txt", + "query": "Black-capped fruit bat: The black-capped fruit bat (Chironax melanocephalus) is a species of megabat in the monotypic ge", + "got": [ + "0033/2843-Black-winged-little-yellow-bat.txt", + "0083/8702-Wrinkle-faced-bat.txt", + "0046/2629-Antillean-fruit-eating-bat.txt", + "0083/8676-Honduran-fruit-eating-bat.txt", + "0083/9282-Hairy-little-fruit-bat.txt" + ] + }, + { + "expected": "0004/4000-Fat-tailed-distribution.txt", + "query": "Fat-tailed distribution: A fat-tailed distribution is a probability distribution that exhibits a large skewness or kurto", + "got": [ + "0045/5196-Heavy-tailed-distribution.txt", + "0045/5196-Heavy-tailed-distribution.txt", + "0075/4783-Truncated-normal-distribution.txt", + "0056/6873-Normal-gamma-distribution.txt", + "0064/4419-Shape-of-a-probability-distribution.txt" + ] + }, + { + "expected": "0005/6000-Jamie-Mueller.txt", + "query": "Jamie Mueller: Jamie F.", + "got": [ + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0039/1332-Jamieson.txt", + "0000/6975-Jamie.txt" + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/50gb-compose.json b/benchmarks/realtime-data-indexing/results/50gb-compose.json new file mode 100644 index 0000000..2ac0c2d --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/50gb-compose.json @@ -0,0 +1,170 @@ +{ + "name": "serviette-consistency-bench", + "networks": { + "default": { + "name": "serviette-consistency-bench_default", + "ipam": {} + } + }, + "services": { + "indexer": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "indexer", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OMP_NUM_THREADS": "12", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/50gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + }, + "qdrant": { + "command": null, + "entrypoint": null, + "image": "qdrant/qdrant:latest", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 6333, + "published": "8301", + "protocol": "tcp" + } + ] + }, + "server": { + "build": { + "context": "/home/sergey/serviette", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "command": [ + "python", + "-m", + "serviette.cli", + "server", + "--config", + "/bench/config.yaml" + ], + "depends_on": { + "qdrant": { + "condition": "service_started", + "required": true + } + }, + "entrypoint": null, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "networks": { + "default": null + }, + "ports": [ + { + "mode": "ingress", + "host_ip": "127.0.0.1", + "target": 8000, + "published": "8300", + "protocol": "tcp" + } + ], + "volumes": [ + { + "type": "bind", + "source": "/home/sergey/.cache/huggingface", + "target": "/hf-cache", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/50gb/docs", + "target": "/data/docs", + "read_only": true, + "bind": {} + }, + { + "type": "bind", + "source": "/home/sergey/serviette/benchmarks/realtime-data-indexing/config.yaml", + "target": "/bench/config.yaml", + "read_only": true, + "bind": {} + } + ] + } + }, + "x-serviette": { + "build": { + "context": "../..", + "dockerfile": "benchmarks/realtime-data-indexing/Dockerfile" + }, + "environment": { + "HF_HOME": "/hf-cache", + "HF_HUB_OFFLINE": "1", + "MALLOC_ARENA_MAX": "", + "OPENAI_API_KEY": "x", + "PATHWAY_LICENSE_KEY": "PV3L-4L4N-TF7R-JWM7-LKVK-PX3E-FHTX-KMNC", + "_RJEM_MALLOC_CONF": "background_thread:true" + }, + "image": "serviette-bench", + "volumes": [ + "~/.cache/huggingface:/hf-cache:ro", + "/mnt/scratch/sergey/serviette-consistency-20261012/datasets/50gb/docs:/data/docs:ro", + "./config.yaml:/bench/config.yaml:ro" + ] + } +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/50gb-expected-output.json b/benchmarks/realtime-data-indexing/results/50gb-expected-output.json new file mode 100644 index 0000000..b99c130 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/50gb-expected-output.json @@ -0,0 +1,10 @@ +{ + "documents": 17083603, + "exact_chunks": 37472786, + "corpus_bytes": 52428765709, + "dataset": "50gb", + "elapsed_seconds": 366.1, + "source": "Frozen corpus files; UTF-8 decode and actual build_xpack_splitter, token_count=512; executed in immutable serviette-bench image", + "image": "sha256:84b80c81426ebd36d7a1b9635d536c1ed8d05df4079e199f9924d3dbe4839c81", + "workers": 48 +} diff --git a/benchmarks/realtime-data-indexing/results/50gb-memory.csv b/benchmarks/realtime-data-indexing/results/50gb-memory.csv new file mode 100644 index 0000000..2c2ef4f --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/50gb-memory.csv @@ -0,0 +1,4463 @@ +elapsed_s,rss_mb,hwm_mb,pss_mb,chunks +12.5,2585.5,2585.6,1723.1,0 +16.4,6424.5,6424.5,3954.5,0 +20.6,8696.0,8696.0,5484.7,0 +24.9,10245.1,10295.4,6814.0,0 +29.0,10343.1,10343.1,6912.0,0 +33.2,10445.0,10448.3,7012.5,0 +37.7,10546.9,10550.1,7114.2,0 +42.0,10645.7,10658.3,7213.1,0 +46.2,10753.8,10766.4,7321.3,0 +50.3,10858.3,10870.9,7425.8,0 +54.4,10958.5,11106.0,7525.9,0 +58.6,11060.8,11106.1,7628.0,0 +62.7,11170.2,11182.9,7737.3,0 +66.8,11279.2,11291.8,7846.5,0 +70.9,11383.5,11396.1,7950.7,0 +75.0,11481.6,11494.2,8048.7,0 +79.2,11581.6,11594.3,8148.8,0 +83.4,11732.5,12026.9,8299.4,0 +87.8,11992.7,12026.9,8559.8,0 +92.0,12271.9,12337.8,8838.8,0 +96.2,12538.9,13001.3,9105.9,0 +100.5,12803.9,13001.3,9370.9,0 +104.6,13077.0,13089.7,9644.1,0 +108.8,13336.2,14334.7,9903.1,0 +113.1,13607.4,14334.7,10176.3,0 +117.4,13886.5,14334.7,10455.4,0 +121.8,14177.7,14334.7,10744.6,0 +126.0,14464.5,14477.2,11031.4,0 +130.3,14730.5,14743.2,11297.4,0 +134.4,15354.9,16995.3,11949.8,0 +139.2,16218.2,20063.4,12216.6,10894 +143.9,16567.7,20351.3,12416.3,29985 +148.6,16566.3,20378.7,12373.2,41872 +153.1,16578.4,20388.3,12362.8,54042 +157.9,16592.4,20390.6,12368.5,64102 +162.4,16633.5,20410.3,12372.2,74605 +167.3,16648.4,20422.1,12380.2,87094 +172.1,16674.2,20436.9,12395.7,99588 +176.7,16701.5,20456.1,12413.6,107535 +181.6,16705.6,20464.7,12419.2,119318 +186.3,16723.6,20477.3,12432.1,131558 +190.9,16739.6,20492.7,12442.4,141766 +195.7,16760.3,20508.7,12459.8,154222 +200.2,16763.1,20517.8,12460.8,166351 +205.0,16785.1,20525.9,12477.8,179976 +209.9,16788.3,20530.8,12479.8,190370 +214.4,16783.6,20531.6,12475.0,190447 +219.1,16791.8,20539.7,12478.8,204390 +223.9,16799.2,20543.6,12486.9,214435 +228.5,16806.1,20547.7,12493.2,227802 +233.2,16805.3,20548.3,12492.1,238354 +237.8,16810.2,20549.6,12496.9,249489 +242.6,16822.3,20558.1,12507.6,259867 +247.3,16830.2,20565.6,12513.5,271751 +251.8,16830.0,20571.3,12513.0,279764 +256.7,16838.5,20573.6,12521.2,284650 +261.3,16834.0,20574.4,12516.8,297705 +265.8,16850.4,20574.6,12533.3,308508 +270.3,16847.5,20576.2,12529.4,318896 +274.9,16846.1,20576.7,12528.7,329245 +279.6,16849.9,20578.0,12532.5,344389 +284.1,16856.2,20583.3,12538.8,355231 +288.9,16853.1,20584.4,12535.6,365827 +293.7,16853.7,20585.3,12536.1,379934 +298.6,16860.9,20589.0,12542.8,385409 +303.4,16864.9,20590.9,12546.4,397032 +308.3,16866.3,20591.4,12547.9,406718 +313.3,16861.0,20591.6,12542.8,416682 +317.8,16870.7,20594.6,12552.0,427486 +322.5,16871.2,20596.6,12552.5,437252 +327.9,16870.8,20597.6,12551.7,447976 +333.8,16869.2,20598.6,12550.7,451958 +339.7,16869.2,20598.7,12550.7,464610 +346.1,16868.4,20598.7,12549.4,464610 +352.5,16871.2,20599.5,12552.9,473596 +358.8,16872.4,20599.6,12553.6,473596 +365.1,16870.8,20600.8,12551.5,481902 +371.8,16872.7,20600.8,12553.8,481902 +378.6,16877.0,20601.0,12557.5,488578 +387.7,16878.4,20601.4,12559.3,491767 +394.9,16882.0,20604.1,12563.1,499749 +401.6,16877.1,20604.5,12557.5,499749 +408.0,16882.0,20606.0,12562.3,504549 +414.5,16878.8,20606.5,12559.3,508212 +421.1,16877.5,20606.5,12557.6,512152 +427.6,16881.8,20606.6,12561.5,518237 +434.1,16879.0,20606.8,12558.9,520462 +440.4,16880.9,20607.8,12560.7,525732 +445.6,16889.3,20611.6,12569.5,532788 +450.7,16892.3,20615.1,12572.5,545731 +456.3,16895.2,20615.9,12574.5,549160 +461.5,16891.0,20616.6,12571.0,552957 +466.9,16893.6,20617.2,12573.4,560788 +471.7,16900.5,20620.2,12580.6,574584 +476.7,16913.9,20623.8,12593.6,584089 +481.4,16912.6,20625.4,12593.2,591666 +486.9,16917.5,20628.9,12597.5,602151 +492.1,16911.1,20629.9,12591.4,606140 +497.1,16913.0,20630.1,12593.2,620086 +502.7,16908.0,20630.2,12588.3,624560 +508.3,16913.5,20631.7,12593.1,627855 +513.8,16909.1,20631.7,12589.5,640221 +518.8,16914.3,20631.7,12593.9,643918 +524.1,16912.7,20631.7,12592.0,651424 +529.7,16913.8,20631.7,12593.2,656405 +536.0,16915.4,20632.7,12594.8,666643 +542.7,16909.7,20633.0,12589.2,668186 +549.4,16908.2,20633.0,12588.1,676761 +555.5,16914.1,20633.0,12594.3,676919 +561.1,16914.5,20633.0,12594.3,681514 +567.6,16916.2,20633.0,12596.2,690198 +573.6,16912.8,20633.0,12592.4,690198 +580.1,16916.2,20633.0,12595.8,697609 +587.3,16917.8,20633.2,12597.5,697609 +593.7,16915.4,20633.8,12595.6,701142 +600.7,16916.4,20633.8,12596.3,710968 +607.4,16916.9,20634.1,12596.7,710968 +614.6,16918.3,20634.1,12598.4,714733 +621.6,16915.5,20634.2,12595.8,723993 +629.7,16919.5,20635.1,12599.0,723993 +638.2,16922.9,20635.6,12602.3,733215 +645.4,16922.9,20635.9,12603.6,733360 +651.0,16920.6,20637.3,12600.3,738698 +655.9,16924.0,20637.4,12604.0,747904 +661.5,16923.5,20637.6,12603.1,750538 +667.0,16923.5,20637.6,12603.1,765554 +671.8,16922.6,20637.6,12602.3,771349 +676.6,16924.9,20637.8,12604.7,780208 +681.4,16954.6,20681.9,12634.4,787559 +686.1,16991.5,20738.2,12671.4,794450 +690.8,16989.2,20738.2,12669.0,799197 +696.5,16988.5,20738.2,12668.5,809698 +701.6,16986.1,20738.2,12665.3,810468 +707.3,16989.1,20738.2,12668.7,821724 +713.2,16980.1,20738.2,12661.0,821724 +718.9,16984.5,20738.2,12664.4,828950 +724.9,16986.4,20738.2,12666.1,831317 +730.0,16997.4,20738.2,12677.6,847244 +734.6,16998.9,20738.2,12679.0,856716 +739.4,16996.7,20738.2,12677.5,868903 +744.1,16996.8,20738.2,12677.5,880570 +749.0,16999.9,20738.2,12680.6,887685 +753.6,16997.7,20738.2,12678.2,895315 +758.5,17000.2,20738.2,12680.5,906968 +763.1,17000.0,20738.2,12680.0,914297 +767.5,17003.6,20738.2,12683.2,928664 +772.0,17003.3,20738.2,12683.3,938486 +776.8,17007.4,20738.2,12687.4,947734 +781.5,17009.4,20738.2,12689.0,953689 +786.4,17011.5,20738.2,12691.8,962234 +790.9,17009.7,20738.2,12689.9,981138 +795.6,17011.0,20738.2,12690.8,990877 +800.4,17007.9,20738.2,12687.4,995674 +804.9,17007.4,20738.2,12687.6,1002359 +809.7,17009.1,20738.2,12688.9,1020045 +814.7,17010.6,20738.2,12690.7,1027555 +819.3,17010.5,20738.2,12690.9,1035339 +824.3,17013.1,20738.2,12693.6,1040112 +829.2,17011.0,20738.2,12691.8,1051170 +833.8,17016.4,20738.2,12696.7,1069486 +838.5,17017.6,20738.2,12697.0,1077912 +843.4,17016.8,20738.2,12697.2,1086605 +848.5,17030.9,20738.2,12711.3,1093736 +853.6,17037.7,20738.2,12718.5,1098598 +858.4,17041.0,20738.2,12721.3,1118234 +863.3,17042.5,20738.2,12722.3,1127296 +868.2,17045.2,20738.2,12725.1,1136020 +873.0,17040.1,20738.2,12719.8,1140087 +877.5,17045.4,20738.2,12725.3,1159133 +882.4,17041.7,20738.2,12721.6,1167733 +887.1,17054.5,20739.2,12734.1,1170276 +891.9,17048.7,20739.5,12728.1,1177187 +896.7,17051.9,20739.5,12731.5,1198303 +901.4,17043.2,20739.5,12723.2,1207583 +906.1,17044.2,20739.5,12724.4,1212826 +910.8,17050.8,20739.5,12731.0,1218662 +915.7,17052.1,20739.5,12732.0,1240416 +920.3,17051.6,20739.5,12732.0,1249567 +925.0,17049.5,20739.5,12730.1,1249567 +929.8,17047.4,20739.5,12727.6,1269028 +935.0,17049.7,20739.6,12730.7,1278002 +939.9,17073.0,20752.5,12753.1,1280029 +944.6,17072.5,20753.0,12753.0,1296453 +949.8,17068.5,20753.3,12748.8,1303089 +954.9,17072.7,20753.7,12753.1,1321077 +959.6,17075.0,20756.3,12756.6,1326278 +964.3,17079.3,20757.9,12761.0,1328476 +969.0,17097.3,20772.6,12777.9,1350738 +973.7,17088.8,20772.7,12768.9,1352429 +978.5,17090.7,20772.8,12771.2,1370915 +983.3,17083.5,20772.8,12764.4,1373428 +988.2,17083.9,20772.8,12765.1,1376888 +992.9,17086.5,20772.8,12767.7,1394842 +997.8,17085.8,20772.8,12767.3,1401873 +1002.7,17082.9,20772.8,12764.7,1403415 +1007.5,17088.1,20773.2,12769.9,1423603 +1012.0,17084.2,20773.2,12766.8,1425704 +1016.9,17088.5,20773.2,12770.7,1429280 +1021.7,17089.9,20773.2,12772.3,1449054 +1026.6,17091.6,20773.3,12773.0,1453132 +1031.4,17085.8,20773.3,12767.5,1468044 +1036.0,17090.2,20773.3,12772.5,1475796 +1040.7,17090.1,20773.3,12772.9,1475796 +1045.4,17092.6,20773.3,12774.7,1491391 +1050.0,17091.2,20773.4,12773.7,1495249 +1054.6,17103.4,20778.9,12785.3,1516383 +1059.1,17099.4,20779.6,12782.0,1516383 +1063.7,17100.9,20779.6,12782.9,1535950 +1068.3,17102.3,20780.1,12784.1,1536474 +1073.0,17103.2,20783.7,12784.6,1556504 +1077.5,17102.2,20784.0,12784.3,1556504 +1082.2,17101.3,20784.0,12783.6,1572730 +1086.7,17107.6,20785.7,12790.4,1575508 +1091.5,17106.4,20786.2,12788.6,1579362 +1096.3,17107.1,20786.3,12789.3,1599874 +1100.8,17102.6,20786.3,12784.7,1599874 +1105.5,17111.8,20789.6,12794.0,1617700 +1110.4,17111.2,20790.2,12793.5,1617700 +1115.0,17109.1,20790.3,12791.6,1636506 +1119.7,17107.6,20790.3,12790.5,1636506 +1124.5,17108.3,20790.3,12791.9,1655216 +1129.4,17108.8,20790.3,12791.7,1655216 +1133.9,17109.8,20790.5,12793.2,1670834 +1138.6,17111.1,20790.8,12794.3,1690752 +1143.4,17117.1,20794.2,12800.2,1691942 +1148.0,17117.6,20794.5,12800.4,1705472 +1152.7,17118.1,20794.9,12800.7,1705494 +1157.3,17126.9,20802.8,12813.2,1719034 +1162.1,17125.5,20803.7,12810.5,1721650 +1166.8,17129.8,20803.9,12814.4,1737728 +1171.5,17126.7,20805.4,12810.1,1739984 +1176.2,17132.7,20809.4,12815.2,1762038 +1180.7,17130.3,20809.6,12812.3,1762038 +1185.4,17128.7,20810.2,12810.5,1776906 +1190.3,17127.5,20810.2,12809.9,1777736 +1195.1,17130.2,20810.2,12812.8,1791650 +1200.1,17131.9,20812.2,12814.0,1793457 +1204.7,17136.6,20812.6,12819.0,1813103 +1209.4,17137.3,20814.8,12819.7,1815309 +1213.8,17146.3,20819.9,12829.1,1824419 +1218.6,17143.5,20820.8,12825.6,1824419 +1224.0,17138.9,20821.5,12821.1,1834842 +1229.8,17136.3,20821.5,12818.4,1844486 +1236.1,17136.8,20821.5,12819.1,1844486 +1242.3,17129.0,20821.5,12811.2,1844605 +1248.8,17127.0,20821.5,12809.5,1851021 +1257.5,17137.4,20821.6,12819.4,1851021 +1265.7,17138.1,20821.8,12820.4,1858509 +1272.1,17138.6,20822.8,12821.2,1863573 +1279.2,17140.2,20823.7,12821.7,1863573 +1286.2,17140.6,20823.7,12822.8,1870411 +1295.1,17141.0,20823.9,12822.7,1870411 +1303.0,17140.6,20824.3,12822.2,1877685 +1309.5,17140.9,20824.7,12822.7,1885405 +1316.3,17143.0,20824.9,12825.1,1885405 +1322.8,17144.9,20825.1,12825.9,1896144 +1331.5,17146.9,20825.8,12828.5,1908346 +1340.5,17144.5,20826.6,12825.8,1908346 +1346.6,17149.0,20827.2,12830.6,1914601 +1351.7,17148.8,20827.5,12830.3,1914601 +1356.7,17154.4,20829.0,12836.2,1923552 +1362.1,17149.6,20829.1,12831.3,1934548 +1367.3,17154.8,20829.1,12837.0,1934548 +1371.9,17154.5,20829.5,12836.4,1948324 +1377.1,17159.3,20831.3,12841.0,1950455 +1382.1,17154.4,20832.2,12835.4,1965175 +1386.7,17154.5,20832.2,12836.2,1968078 +1391.4,17154.9,20833.9,12836.8,1982588 +1396.1,17161.9,20835.3,12843.2,1985452 +1400.8,17161.2,20837.1,12842.8,2003660 +1405.6,17162.8,20837.7,12844.6,2003660 +1410.5,17159.4,20838.7,12841.0,2017704 +1415.5,17158.2,20838.7,12840.0,2017704 +1420.3,17159.9,20838.7,12841.9,2034695 +1425.6,17200.3,20838.7,12881.8,2037151 +1430.4,17199.0,20838.7,12880.5,2053423 +1435.1,17199.8,20838.7,12881.8,2068213 +1439.8,17201.6,20838.7,12883.9,2069720 +1444.8,17205.9,20841.9,12888.3,2086798 +1449.7,17208.3,20842.9,12890.3,2087869 +1454.3,17208.0,20843.0,12890.3,2107173 +1458.9,17209.0,20844.3,12890.9,2108575 +1463.9,17208.6,20846.7,12890.5,2125161 +1468.6,17209.0,20847.0,12890.9,2125161 +1473.7,17214.0,20851.5,12896.5,2144088 +1478.7,17214.8,20851.9,12897.5,2144088 +1485.0,17210.7,20852.2,12892.9,2151954 +1491.9,17209.3,20853.1,12891.5,2161252 +1500.7,17207.8,20853.1,12890.3,2161252 +1507.9,17209.0,20853.1,12891.7,2161897 +1514.9,17207.4,20853.1,12889.9,2169105 +1521.6,17209.7,20853.1,12891.6,2169105 +1528.9,17208.1,20853.1,12890.0,2174078 +1537.2,17212.1,20853.1,12893.2,2186174 +1544.2,17211.4,20853.1,12892.6,2190774 +1550.9,17212.8,20853.4,12894.3,2200362 +1559.7,17216.9,20855.3,12898.0,2200362 +1568.2,17216.6,20855.3,12898.3,2203015 +1576.2,17213.5,20855.3,12895.5,2211199 +1583.8,17212.3,20855.3,12894.4,2211199 +1590.6,17213.0,20855.9,12894.0,2211199 +1596.2,17217.8,20856.1,12899.2,2227038 +1601.6,17226.2,20859.6,12907.4,2227038 +1607.2,17226.3,20860.1,12907.5,2238922 +1612.4,17216.3,20860.2,12897.4,2249126 +1617.3,17219.4,20860.2,12900.7,2249126 +1622.0,17220.1,20860.2,12901.0,2262912 +1627.1,17221.6,20860.8,12903.0,2276990 +1632.1,17220.1,20860.9,12901.6,2276990 +1637.1,17222.1,20860.9,12903.8,2286007 +1642.4,17221.5,20861.2,12903.4,2286007 +1647.4,17224.6,20861.2,12906.4,2296131 +1652.9,17220.0,20861.3,12901.4,2306327 +1658.1,17223.5,20861.3,12905.0,2306327 +1663.7,17225.9,20863.0,12907.2,2311188 +1669.0,17224.2,20863.1,12905.9,2320134 +1674.1,17227.0,20863.7,12909.0,2320134 +1678.5,17226.5,20863.7,12908.6,2333740 +1683.3,17230.3,20864.3,12912.6,2351830 +1688.1,17232.7,20865.6,12914.7,2352726 +1692.5,17233.5,20866.3,12915.0,2367750 +1697.2,17235.0,20867.7,12915.7,2367750 +1701.8,17232.5,20868.0,12913.8,2380475 +1706.4,17232.2,20868.2,12913.1,2391823 +1711.3,17230.2,20868.3,12910.8,2391823 +1715.7,17231.4,20868.3,12912.3,2404037 +1720.3,17232.3,20868.9,12912.6,2422535 +1725.1,17231.2,20868.9,12912.0,2422535 +1729.7,17231.2,20868.9,12912.4,2437855 +1734.5,17230.1,20868.9,12910.4,2437855 +1739.2,17230.6,20868.9,12910.5,2449285 +1743.8,17234.9,20868.9,12914.3,2468451 +1748.7,17231.3,20868.9,12911.0,2468451 +1753.4,17237.1,20870.3,12917.3,2476139 +1758.0,17237.4,20870.9,12916.9,2494943 +1762.7,17235.6,20871.0,12915.6,2494943 +1767.4,17238.5,20872.4,12917.6,2509023 +1772.2,17233.0,20872.4,12912.4,2509023 +1776.7,17237.0,20872.4,12916.3,2524775 +1781.2,17234.2,20872.4,12913.7,2541459 +1786.0,17238.4,20872.5,12917.9,2541459 +1790.6,17234.7,20872.5,12914.1,2551972 +1795.1,17234.8,20872.5,12914.2,2567830 +1799.7,17236.4,20873.6,12915.4,2567830 +1804.2,17235.1,20873.9,12914.4,2578605 +1809.1,17232.4,20873.9,12912.2,2597631 +1813.8,17235.0,20873.9,12914.6,2597631 +1818.4,17239.4,20874.4,12918.6,2611054 +1823.0,17235.8,20874.4,12915.0,2611054 +1827.5,17238.8,20874.5,12918.3,2625985 +1832.2,17241.9,20876.6,12921.6,2645953 +1836.9,17240.7,20876.7,12920.4,2645953 +1841.8,17238.8,20876.7,12918.6,2656593 +1846.3,17239.3,20876.7,12918.8,2673321 +1851.0,17236.6,20876.7,12915.9,2673321 +1855.5,17236.0,20876.7,12915.3,2684372 +1860.3,17236.3,20876.8,12916.0,2703768 +1865.0,17238.4,20877.4,12917.7,2703768 +1869.4,17233.6,20877.5,12912.4,2706745 +1874.0,17234.0,20877.5,12913.0,2724865 +1878.6,17234.7,20877.5,12913.6,2724865 +1883.1,17236.1,20877.8,12915.1,2735688 +1887.7,17236.0,20877.8,12915.4,2754496 +1892.4,17233.1,20877.8,12912.5,2769462 +1897.2,17233.1,20877.8,12912.4,2769462 +1901.7,17234.5,20877.8,12913.9,2783063 +1906.5,17242.6,20879.0,12922.3,2797845 +1911.3,17235.2,20879.1,12915.2,2797845 +1916.2,17238.8,20879.1,12918.4,2804127 +1920.7,17237.0,20879.1,12916.1,2822649 +1925.2,17238.3,20879.1,12917.2,2822649 +1929.8,17246.6,20884.0,12925.8,2826071 +1934.3,17245.9,20911.0,12924.8,2841625 +1939.0,17239.1,20924.2,12917.9,2848779 +1943.4,17237.6,20976.4,12916.6,2848779 +1948.2,17236.9,20976.4,12915.6,2856010 +1952.7,17233.8,20976.4,12912.6,2873766 +1957.5,17233.9,20976.4,12912.6,2873766 +1962.2,17231.3,20976.4,12910.2,2884012 +1966.6,17232.7,20976.4,12911.7,2902346 +1971.3,17232.8,20976.4,12911.7,2902346 +1975.9,17247.4,20976.4,12926.3,2913584 +1980.7,17242.2,20976.4,12921.0,2923706 +1985.3,17242.8,20976.4,12921.4,2923706 +1990.0,17236.0,20976.4,12914.0,2930963 +1994.7,17236.9,20976.4,12915.9,2945061 +1999.3,17236.8,20976.4,12915.6,2945061 +2003.9,17240.5,20976.4,12919.2,2961379 +2008.5,17242.0,20976.4,12920.5,2982195 +2013.3,17241.9,20976.4,12920.9,2999859 +2018.0,17242.7,20976.4,12921.1,2999859 +2022.7,17243.1,20976.4,12921.7,3007540 +2027.5,17238.8,20976.4,12917.1,3024156 +2032.0,17238.5,20976.4,12917.0,3024156 +2036.8,17236.1,20976.4,12914.7,3036417 +2041.6,17236.8,20976.4,12915.6,3053197 +2046.4,17238.2,20976.4,12916.6,3053197 +2051.0,17235.8,20976.4,12914.4,3059571 +2055.8,17237.4,20976.4,12916.1,3073031 +2060.6,17235.5,20976.4,12914.0,3073031 +2065.2,17237.1,20976.4,12915.6,3085138 +2070.0,17239.6,20976.4,12917.9,3100668 +2074.8,17238.2,20976.4,12916.9,3100668 +2079.6,17237.2,20976.4,12915.3,3113122 +2084.3,17237.4,20976.4,12915.6,3131270 +2088.9,17239.5,20976.4,12918.1,3131270 +2093.5,17238.4,20976.4,12917.1,3136925 +2098.1,17238.1,20976.4,12916.3,3151305 +2102.6,17237.8,20976.4,12915.9,3151305 +2107.0,17238.0,20976.4,12916.4,3163856 +2111.7,17234.8,20976.4,12913.1,3177654 +2116.4,17236.2,20976.4,12914.6,3197824 +2121.0,17238.7,20976.4,12917.1,3197824 +2125.7,17241.6,20976.4,12919.9,3208531 +2130.5,17248.8,20976.4,12927.4,3223157 +2135.3,17244.9,20976.4,12923.2,3223157 +2139.9,17243.7,20976.4,12921.9,3229709 +2144.5,17240.5,20976.4,12918.9,3245887 +2149.5,17241.8,20976.4,12920.4,3245887 +2154.2,17236.1,20976.4,12914.4,3254095 +2158.9,17236.7,20976.4,12914.7,3274415 +2163.4,17231.8,20976.4,12910.3,3287729 +2168.1,17241.3,20976.4,12919.6,3287729 +2172.7,17241.7,20976.4,12920.0,3295284 +2177.4,17240.7,20976.4,12919.1,3313718 +2181.9,17238.3,20976.4,12916.4,3332350 +2186.3,17239.9,20976.4,12917.7,3332350 +2191.1,17235.5,20976.4,12914.6,3332350 +2195.9,17235.6,20976.4,12913.8,3347949 +2200.5,17234.9,20976.4,12913.0,3347949 +2205.5,17239.4,20976.4,12917.3,3353132 +2210.9,17236.5,20976.4,12914.2,3368328 +2216.8,17237.8,20976.4,12915.4,3377350 +2223.1,17231.6,20976.4,12909.7,3388176 +2229.7,17230.4,20976.4,12908.1,3388176 +2235.7,17233.1,20976.4,12911.6,3388176 +2241.5,17231.7,20976.4,12909.9,3388176 +2247.6,17232.9,20976.4,12910.2,3388176 +2253.7,17227.6,20976.4,12905.2,3393936 +2260.0,17229.7,20976.4,12907.4,3400720 +2265.5,17231.2,20976.4,12909.0,3400720 +2271.2,17231.3,20976.4,12909.4,3400720 +2276.6,17227.9,20976.4,12906.1,3404425 +2282.3,17228.9,20976.4,12906.4,3409875 +2288.0,17228.8,20976.4,12906.7,3416303 +2294.5,17230.2,20976.4,12908.2,3426365 +2300.5,17230.3,20976.4,12908.2,3432419 +2307.2,17231.9,20976.4,12909.7,3440421 +2313.9,17230.3,20976.4,12908.5,3440421 +2320.0,17233.3,20976.4,12911.2,3440421 +2325.2,17235.1,20976.4,12912.7,3440421 +2330.1,17235.1,20976.4,12912.5,3455560 +2335.3,17233.3,20976.4,12911.4,3466790 +2340.3,17236.2,20976.4,12913.4,3466790 +2345.1,17232.7,20976.4,12910.7,3468854 +2350.0,17236.2,20976.4,12914.6,3486628 +2355.1,17234.5,20976.4,12912.3,3498526 +2359.9,17239.6,20976.4,12917.5,3498526 +2365.4,17236.4,20976.4,12913.9,3499763 +2370.6,17235.8,20976.4,12913.8,3509413 +2375.6,17236.6,20976.4,12914.3,3522331 +2381.3,17234.0,20976.4,12911.6,3522331 +2386.8,17232.7,20976.4,12910.0,3522331 +2392.7,17232.0,20976.4,12910.0,3527569 +2398.1,17231.4,20976.4,12909.6,3543495 +2402.7,17235.1,20976.4,12912.9,3543495 +2407.4,17233.2,20976.4,12911.1,3549196 +2412.2,17234.7,20976.4,12912.2,3569050 +2417.1,17238.7,20976.4,12915.9,3585258 +2421.8,17233.2,20976.4,12910.7,3585258 +2426.5,17236.8,20976.4,12914.7,3588195 +2431.2,17234.3,20976.4,12912.2,3606321 +2436.2,17243.6,20976.4,12921.2,3606321 +2440.9,17239.8,20976.4,12917.6,3614677 +2445.6,17245.4,20976.4,12923.0,3627823 +2450.3,17239.9,20976.4,12917.2,3642433 +2455.0,17241.1,20976.4,12918.8,3642433 +2459.6,17241.4,20976.4,12919.1,3650725 +2464.3,17243.6,20976.4,12921.9,3667919 +2469.4,17243.6,20976.4,12921.4,3686507 +2474.1,17243.0,20976.4,12921.7,3686507 +2478.9,17242.2,20976.4,12919.9,3689217 +2483.7,17239.2,20976.4,12918.2,3703491 +2488.3,17236.7,20976.4,12914.8,3723103 +2492.8,17239.3,20976.4,12917.2,3723103 +2497.5,17240.4,20976.4,12918.2,3729799 +2502.2,17235.4,20976.4,12913.2,3737443 +2507.2,17317.7,20976.4,12996.0,3752989 +2512.0,17314.3,20976.4,12992.4,3752989 +2516.9,17316.5,20976.4,12994.6,3762014 +2521.5,17313.3,20976.4,12991.6,3773530 +2526.4,17316.9,20976.4,12995.5,3773530 +2531.1,17317.8,20976.4,12995.3,3777568 +2535.7,17319.7,20976.4,12997.2,3800336 +2540.3,17316.4,20976.4,12994.1,3815610 +2544.9,17319.9,20976.4,12997.8,3815610 +2549.6,17318.1,20976.4,12995.7,3821012 +2554.2,17319.9,20976.4,12997.2,3840098 +2558.8,17316.4,20976.4,12993.8,3852940 +2563.5,17317.1,20976.4,12994.4,3852940 +2568.1,17323.8,20976.4,13002.0,3857745 +2572.7,17320.8,20976.4,12998.8,3876769 +2577.4,17320.8,20976.4,12998.8,3890029 +2582.2,17316.4,20976.4,12993.8,3890029 +2586.7,17318.2,20976.4,12995.8,3890029 +2591.3,17317.1,20976.4,12994.9,3905837 +2595.9,17318.9,20976.4,12996.8,3917935 +2600.7,17318.9,20976.4,12996.9,3934947 +2605.5,17320.6,20976.4,12998.8,3934947 +2609.9,17315.0,20976.4,12993.0,3934947 +2614.6,17319.0,20976.4,12997.0,3947914 +2619.4,17319.0,20976.4,12996.7,3965590 +2624.1,17320.8,20976.4,12998.1,3965590 +2628.7,17318.8,20976.4,12996.4,3969456 +2633.3,17322.0,20976.4,13000.3,3986658 +2637.8,17320.9,20976.4,12998.7,4005370 +2642.2,17318.1,20976.4,12996.2,4005370 +2646.8,17319.1,20976.4,12997.2,4008356 +2651.3,17317.5,20976.4,12995.3,4022090 +2655.9,17317.7,20976.4,12995.2,4038788 +2660.7,17318.2,20976.4,12996.0,4038788 +2665.5,17320.3,20976.4,12998.0,4039562 +2670.4,17316.8,20976.4,12994.4,4052682 +2675.2,17317.5,20976.4,12995.3,4071644 +2679.8,17317.8,20976.4,12995.8,4090562 +2684.6,17319.5,20976.4,12997.7,4090562 +2689.1,17317.8,20976.4,12995.6,4090562 +2693.9,17317.4,20976.4,12995.6,4102916 +2698.5,17318.9,20976.4,12996.3,4121724 +2703.1,17320.3,20976.4,12998.2,4121724 +2708.0,17323.3,20976.4,13001.0,4122824 +2712.5,17318.0,20976.4,12995.6,4142788 +2717.1,17317.4,20976.4,12994.9,4158886 +2721.8,17315.8,20976.4,12993.4,4158886 +2726.5,17317.1,20976.4,12994.6,4158886 +2731.0,17317.6,20976.4,12995.2,4169674 +2735.7,17318.5,20976.4,12996.4,4188604 +2740.2,17316.1,20976.4,12993.9,4204812 +2745.0,17314.9,20976.4,12992.9,4204812 +2749.7,17314.1,20976.4,12992.0,4204812 +2754.4,17313.0,20976.4,12990.8,4213798 +2759.4,17316.1,20976.4,12994.2,4234970 +2764.2,17317.6,20976.4,12995.9,4234970 +2768.8,17319.2,20976.4,12997.1,4238673 +2773.3,17318.5,20976.4,12996.0,4257443 +2778.1,17323.3,20976.4,13000.9,4280451 +2782.8,17320.6,20976.4,12998.2,4280451 +2787.3,17322.7,20976.4,13000.1,4280451 +2791.7,17319.8,20976.4,12998.1,4294168 +2796.3,17318.7,20976.4,12996.8,4309888 +2801.2,17318.1,20976.4,12996.2,4330208 +2805.6,17320.7,20976.4,12998.3,4330208 +2810.1,17317.7,20976.4,12995.0,4330208 +2814.7,17318.7,20976.4,12996.7,4338766 +2819.9,17319.1,20976.4,12996.9,4361164 +2824.6,17317.5,20976.4,12995.6,4361164 +2829.1,17318.9,20976.4,12996.1,4361164 +2833.7,17320.6,20976.4,12998.5,4373581 +2838.2,17319.1,20976.4,12997.0,4391661 +2842.6,17321.0,20976.4,12998.6,4408695 +2847.3,17318.0,20976.4,12995.1,4408695 +2852.0,17315.2,20976.4,12992.6,4408695 +2856.4,17315.2,20976.4,12992.5,4420184 +2861.0,17319.3,20976.4,12996.7,4435078 +2865.7,17319.4,20976.4,12997.1,4435078 +2870.1,17316.7,20976.4,12994.7,4435078 +2874.9,17315.4,20976.4,12993.2,4449391 +2879.5,17314.9,20976.4,12992.4,4463793 +2884.6,17317.3,20976.4,12995.3,4486479 +2889.3,17321.0,20976.4,12998.7,4486479 +2893.8,17318.2,20976.4,12995.8,4486479 +2898.6,17317.8,20976.4,12995.4,4498758 +2903.1,17316.3,20976.4,12994.0,4519726 +2907.9,17315.9,20976.4,12993.9,4519726 +2912.3,17315.4,20976.4,12993.1,4519726 +2916.8,17316.4,20976.4,12993.9,4534352 +2921.5,17312.3,20976.4,12989.7,4549982 +2926.3,17314.8,20976.4,12992.9,4569260 +2931.1,17315.6,20976.4,12993.5,4569260 +2935.6,17319.1,20976.4,12997.5,4569260 +2940.6,17319.4,20976.4,12996.9,4581405 +2945.1,17322.7,20976.4,13000.8,4598319 +2949.9,17317.2,20976.4,12994.8,4616583 +2954.6,17317.6,20976.4,12995.0,4616583 +2959.4,17317.1,20976.4,12994.9,4616583 +2964.1,17319.0,20976.4,12996.7,4629260 +2968.9,17318.3,20976.4,12996.2,4645838 +2973.6,17318.1,20976.4,12996.3,4645838 +2978.4,17317.7,20976.4,12996.2,4645838 +2982.9,17317.7,20976.4,12996.1,4655465 +2987.5,17320.4,20976.4,12998.8,4670515 +2992.0,17317.6,20976.4,12995.4,4689053 +2996.8,17317.4,20976.4,12995.5,4689053 +3001.4,17313.8,20976.4,12993.3,4689053 +3006.1,17333.2,20978.2,13012.1,4707257 +3010.8,17331.9,20980.2,13011.3,4727993 +3015.7,17328.7,20980.2,13007.8,4743851 +3020.5,17317.3,20980.2,12995.5,4743851 +3025.4,17316.2,20980.2,12994.5,4743851 +3030.0,17314.8,20980.2,12993.0,4753488 +3034.7,17310.9,20980.2,12989.8,4766070 +3039.2,17314.1,20980.2,12992.2,4781378 +3043.8,17352.5,20980.2,13031.3,4781378 +3048.6,17420.7,20986.0,13100.0,4789592 +3053.4,17415.3,21000.4,13093.7,4812808 +3058.1,17424.6,21006.1,13103.0,4841758 +3062.8,17416.9,21008.6,13094.8,4874386 +3067.6,17402.9,21008.6,13081.4,4874386 +3072.3,17416.4,21008.6,13094.4,4874386 +3077.0,17406.4,21009.8,13085.1,4892973 +3081.6,17411.7,21009.8,13088.3,4922947 +3086.5,17404.9,21009.8,13082.7,4948099 +3091.3,17387.9,21009.8,13064.1,4973079 +3096.0,17391.6,21009.8,13069.4,4973079 +3100.7,17380.9,21009.8,13058.0,4973079 +3105.3,17385.5,21009.8,13063.3,4973079 +3109.9,17375.2,21009.8,13053.2,4997018 +3114.7,17373.5,21009.8,13051.0,5014658 +3119.3,17370.2,21009.8,13047.5,5041626 +3124.1,17365.3,21009.8,13042.7,5060702 +3128.8,17356.7,21009.8,13034.2,5060702 +3133.4,17372.0,21009.8,13049.5,5060702 +3138.0,17369.8,21009.8,13047.3,5060702 +3142.6,17362.6,21009.8,13040.5,5071522 +3147.4,17355.1,21009.8,13032.8,5090166 +3151.7,17365.8,21009.8,13043.5,5110998 +3156.3,17406.0,21009.8,13083.9,5110998 +3161.0,17402.0,21010.8,13079.1,5110998 +3165.8,17375.2,21010.8,13052.1,5126072 +3170.5,17348.5,21010.8,13025.6,5138556 +3175.1,17342.8,21010.8,13019.6,5153656 +3179.9,17340.4,21010.8,13017.2,5153656 +3184.5,17338.4,21010.8,13015.9,5153656 +3189.1,17339.0,21010.8,13016.4,5153656 +3193.8,17338.8,21010.8,13016.6,5162598 +3198.6,17341.3,21010.8,13019.1,5179994 +3203.3,17339.1,21010.8,13016.8,5197822 +3207.9,17342.7,21010.8,13019.8,5197822 +3212.5,17337.9,21010.8,13015.2,5197822 +3217.2,17339.8,21010.8,13017.7,5203532 +3221.6,17340.3,21010.8,13017.9,5219304 +3226.5,17340.8,21010.8,13017.5,5237446 +3231.3,17339.3,21010.8,13016.4,5237446 +3235.7,17337.2,21010.8,13014.5,5237446 +3240.3,17334.6,21010.8,13012.1,5242166 +3244.8,17334.8,21010.8,13012.3,5256134 +3249.7,17338.2,21010.8,13015.7,5274342 +3254.3,17337.8,21010.8,13015.1,5274342 +3258.9,17341.3,21010.8,13018.5,5274342 +3263.6,17339.3,21010.8,13016.6,5282533 +3268.1,17336.9,21010.8,13014.4,5300147 +3272.7,17334.5,21010.8,13012.2,5317623 +3277.4,17336.9,21010.8,13014.3,5333007 +3282.0,17337.5,21010.8,13014.5,5333007 +3286.5,17337.4,21010.8,13013.8,5333007 +3291.2,17334.7,21010.8,13012.3,5334513 +3295.6,17334.0,21010.8,13011.6,5352151 +3300.4,17332.7,21010.8,13010.1,5365195 +3305.2,17333.3,21010.8,13010.5,5365195 +3309.6,17336.6,21010.8,13013.9,5365195 +3314.1,17336.0,21010.8,13013.5,5372371 +3318.7,17336.1,21010.8,13013.5,5383859 +3323.4,17339.8,21010.8,13017.3,5399793 +3327.8,17338.8,21010.8,13016.1,5416215 +3332.3,17337.1,21010.8,13014.5,5416215 +3337.0,17334.2,21010.8,13011.5,5416215 +3341.8,17335.9,21010.8,13013.2,5422721 +3346.4,17337.9,21010.8,13015.7,5439497 +3351.3,17336.3,21010.8,13014.2,5454843 +3355.7,17336.0,21010.8,13014.0,5454843 +3360.2,17338.3,21010.8,13016.1,5454843 +3365.0,17335.7,21010.8,13012.9,5462378 +3369.6,17336.0,21010.8,13013.0,5479944 +3374.1,17333.1,21010.8,13010.6,5495900 +3378.7,17331.8,21010.8,13009.4,5495900 +3383.4,17331.8,21010.8,13009.3,5495900 +3388.2,17335.1,21010.8,13012.5,5501464 +3392.7,17336.1,21010.8,13013.7,5512680 +3397.5,17333.6,21010.8,13010.8,5529706 +3402.3,17336.3,21010.8,13013.2,5548848 +3406.8,17333.9,21010.8,13011.3,5548848 +3411.4,17333.3,21010.8,13010.6,5548848 +3416.1,17331.4,21010.8,13008.8,5553259 +3420.5,17333.0,21010.8,13010.6,5570871 +3425.3,17332.0,21010.8,13009.7,5584857 +3429.8,17332.0,21010.8,13009.1,5599645 +3434.6,17329.9,21010.8,13007.4,5599645 +3439.1,17332.8,21010.8,13010.3,5599645 +3443.8,17334.0,21010.8,13011.4,5606586 +3448.5,17333.2,21010.8,13010.5,5620836 +3453.4,17337.7,21010.8,13015.5,5635984 +3458.1,17335.8,21010.8,13013.1,5635984 +3462.9,17336.0,21010.8,13013.3,5635984 +3467.5,17335.9,21010.8,13013.2,5639598 +3472.1,17335.8,21010.8,13013.4,5658674 +3476.7,17334.5,21010.8,13012.0,5672660 +3481.5,17336.1,21010.8,13013.7,5689860 +3486.0,17336.1,21010.8,13013.5,5689860 +3490.7,17335.2,21010.8,13012.8,5689860 +3495.4,17332.9,21010.8,13010.3,5689860 +3500.2,17334.7,21010.8,13012.1,5701797 +3504.8,17333.5,21010.8,13010.6,5715565 +3509.6,17336.7,21010.8,13013.9,5715565 +3514.2,17338.8,21010.8,13016.3,5715565 +3519.1,17337.0,21010.8,13014.5,5730011 +3523.8,17332.2,21010.8,13009.9,5742317 +3528.7,17329.5,21010.8,13006.5,5755389 +3533.3,17331.5,21010.8,13008.6,5755389 +3538.0,17332.6,21010.8,13010.0,5755389 +3542.4,17333.6,21010.8,13010.5,5763833 +3547.1,17333.7,21010.8,13010.9,5779991 +3551.8,17334.1,21010.8,13011.4,5796913 +3556.7,17333.9,21010.8,13011.2,5796913 +3561.4,17335.4,21010.8,13012.8,5796913 +3566.2,17336.2,21010.8,13013.6,5808021 +3570.9,17330.7,21010.8,13008.1,5825817 +3575.8,17331.5,21010.8,13009.0,5840701 +3580.4,17330.2,21010.8,13007.4,5854689 +3585.1,17329.9,21010.8,13007.0,5854689 +3589.9,17328.7,21010.8,13005.8,5854689 +3594.4,17330.2,21010.8,13007.8,5859890 +3599.2,17329.2,21010.8,13006.5,5877764 +3603.8,17334.0,21010.8,13011.3,5888230 +3608.8,17335.4,21010.8,13012.4,5903798 +3613.4,17335.6,21010.8,13012.9,5903798 +3618.1,17332.3,21010.8,13010.0,5903798 +3622.8,17332.8,21010.8,13010.2,5903798 +3627.6,17332.6,21010.8,13010.2,5910948 +3632.0,17333.8,21010.8,13011.2,5925096 +3636.8,17333.0,21010.8,13011.0,5942300 +3641.4,17332.2,21010.8,13009.9,5942300 +3646.0,17335.8,21010.8,13013.2,5942300 +3650.7,17331.9,21010.8,13009.1,5944391 +3655.5,17332.7,21010.8,13010.6,5960281 +3660.0,17334.3,21010.8,13011.9,5977303 +3664.6,17337.3,21010.8,13014.5,5991895 +3669.3,17337.1,21010.8,13015.0,5991895 +3674.2,17331.8,21010.8,13009.8,5991895 +3679.0,17332.2,21010.8,13009.6,5999176 +3683.6,17331.0,21010.8,13008.8,6013626 +3688.2,17333.5,21010.8,13011.5,6029928 +3692.9,17332.6,21010.8,13010.2,6045602 +3697.6,17334.0,21010.8,13011.9,6045602 +3702.2,17335.2,21010.8,13012.7,6045602 +3706.9,17332.2,21010.8,13009.3,6045602 +3711.7,17338.4,21010.8,13015.9,6063972 +3716.3,17336.8,21010.8,13014.5,6083170 +3721.2,17334.8,21010.8,13012.3,6098106 +3725.9,17336.4,21010.8,13014.0,6098106 +3730.7,17336.8,21010.8,13014.3,6098106 +3735.4,17333.9,21010.8,13011.7,6098106 +3740.1,17334.3,21010.8,13011.5,6112712 +3745.1,17333.7,21010.8,13011.1,6129550 +3749.6,17338.0,21010.8,13015.1,6129550 +3754.2,17337.1,21010.8,13014.2,6129550 +3759.4,17339.3,21010.8,13016.6,6130165 +3765.0,17338.9,21010.8,13016.4,6143697 +3770.5,17331.2,21010.8,13008.7,6152051 +3776.8,17328.1,21010.8,13005.5,6160859 +3782.7,17329.4,21010.8,13007.4,6170129 +3789.1,17332.3,21010.8,13010.2,6177607 +3795.5,17328.5,21010.8,13006.1,6184495 +3801.9,17326.2,21010.8,13003.7,6191879 +3808.7,17328.3,21010.8,13006.0,6191879 +3815.0,17330.7,21010.8,13008.5,6191879 +3821.2,17334.5,21010.8,13012.1,6191879 +3828.1,17331.7,21010.8,13009.0,6191879 +3835.0,17329.5,21010.8,13007.2,6191879 +3841.1,17325.9,21010.8,13002.9,6191879 +3847.4,17324.4,21010.8,13001.9,6191879 +3853.8,17323.5,21010.8,13001.3,6194029 +3860.4,17327.9,21010.8,13005.4,6201377 +3867.2,17323.1,21010.8,13000.7,6209893 +3872.3,17327.4,21010.8,13004.9,6221925 +3877.2,17331.8,21010.8,13009.3,6221925 +3882.2,17332.1,21010.8,13009.5,6221925 +3887.5,17331.6,21010.8,13009.0,6221925 +3892.3,17338.0,21010.8,13015.5,6234940 +3897.1,17335.0,21010.8,13012.8,6251776 +3902.6,17334.6,21010.8,13011.9,6268512 +3907.4,17333.3,21010.8,13010.4,6268512 +3912.4,17338.0,21010.8,13015.6,6268512 +3918.0,17338.7,21010.8,13016.7,6268512 +3923.3,17332.5,21010.8,13010.8,6272312 +3928.6,17328.9,21010.8,13006.5,6282588 +3934.3,17329.4,21010.8,13007.2,6292184 +3939.6,17330.5,21010.8,13008.3,6302996 +3944.6,17338.2,21010.8,13015.7,6302996 +3949.4,17352.6,21010.8,13030.0,6302996 +3954.0,17365.9,21010.8,13043.1,6318443 +3958.7,17356.8,21010.8,13034.3,6332589 +3963.5,17350.1,21010.8,13027.5,6348829 +3968.3,17346.8,21010.8,13024.0,6362827 +3973.2,17340.7,21010.8,13018.4,6375609 +3977.7,17343.9,21010.8,13021.2,6375609 +3982.2,17342.3,21010.8,13019.8,6375609 +3987.1,17340.4,21010.8,13017.5,6375609 +3991.8,17343.7,21010.8,13020.9,6383778 +3996.4,17340.9,21010.8,13018.7,6396164 +4001.2,17338.7,21010.8,13015.9,6411918 +4005.9,17338.5,21010.8,13015.5,6411918 +4010.6,17338.0,21010.8,13015.2,6411918 +4015.4,17343.0,21010.8,13020.3,6411918 +4020.1,17340.8,21010.8,13018.4,6427949 +4025.0,17339.2,21010.8,13016.5,6446033 +4029.8,17338.5,21010.8,13015.8,6459585 +4034.3,17338.3,21010.8,13016.0,6476545 +4039.0,17338.7,21010.8,13016.2,6476545 +4043.5,17337.3,21010.8,13014.6,6476545 +4048.3,17339.6,21010.8,13016.9,6476545 +4052.9,17333.6,21010.8,13011.0,6482330 +4057.9,17334.7,21010.8,13011.9,6498832 +4062.7,17332.3,21010.8,13009.5,6514040 +4067.4,17334.3,21010.8,13011.6,6514040 +4072.2,17334.8,21010.8,13011.8,6514040 +4076.8,17335.5,21010.8,13012.6,6517449 +4081.6,17336.5,21010.8,13013.6,6534311 +4086.2,17336.9,21010.8,13015.8,6548379 +4090.9,17342.5,21010.8,13019.9,6564681 +4095.3,17339.6,21010.8,13017.2,6564681 +4100.0,17352.3,21010.8,13029.5,6564681 +4104.8,17347.0,21010.8,13023.9,6570247 +4109.5,17339.2,21010.8,13017.0,6577157 +4114.3,17338.7,21010.8,13016.2,6596907 +4118.7,17340.3,21010.8,13017.7,6612515 +4123.2,17340.2,21010.8,13017.4,6612515 +4127.9,17337.9,21010.8,13015.1,6612515 +4132.7,17336.8,21010.8,13013.7,6612515 +4137.1,17335.9,21010.8,13012.9,6622484 +4141.8,17338.4,21010.8,13016.2,6639296 +4146.5,17338.3,21010.8,13015.8,6652952 +4151.1,17339.2,21010.8,13016.8,6652952 +4155.8,17341.4,21010.8,13019.0,6652952 +4160.2,17338.9,21010.8,13016.5,6657936 +4164.8,17336.1,21010.8,13013.0,6666460 +4169.5,17334.9,21010.8,13012.4,6682536 +4174.1,17337.5,21010.8,13014.8,6699144 +4178.8,17338.8,21010.8,13016.4,6699144 +4183.3,17338.7,21010.8,13015.7,6699144 +4187.8,17333.8,21010.8,13011.1,6699144 +4192.5,17357.6,21010.8,13035.3,6711555 +4197.2,17409.5,21010.8,13086.8,6728541 +4202.0,17433.1,21010.8,13109.9,6740279 +4206.5,17435.7,21010.8,13113.0,6740279 +4211.1,17436.1,21010.8,13113.4,6740279 +4215.6,17435.9,21011.2,13113.3,6746095 +4220.3,17436.4,21011.2,13113.8,6757301 +4225.1,17437.3,21012.0,13114.4,6773659 +4229.7,17435.4,21012.0,13112.8,6788755 +4234.3,17438.3,21012.1,13115.3,6803501 +4239.0,17433.9,21012.1,13110.9,6803501 +4243.7,17433.4,21012.1,13110.7,6803501 +4248.3,17435.9,21012.1,13112.9,6804791 +4252.8,17433.3,21012.1,13110.7,6820813 +4257.3,17435.9,21012.1,13113.1,6837025 +4261.9,17432.5,21012.1,13109.3,6852333 +4266.7,17436.3,21012.1,13114.5,6865737 +4271.2,17440.9,21012.1,13118.5,6865737 +4275.8,17447.8,21013.5,13125.0,6865737 +4280.3,17441.0,21013.6,13117.9,6865737 +4285.1,17439.6,21014.2,13117.1,6884091 +4289.8,17435.6,21014.4,13113.0,6900139 +4294.3,17438.4,21014.4,13116.1,6916923 +4299.0,17443.5,21015.7,13121.2,6916923 +4303.6,17443.6,21015.9,13120.8,6916923 +4308.6,17439.3,21016.1,13116.3,6916923 +4313.4,17441.7,21016.1,13118.9,6926963 +4318.1,17439.7,21016.1,13116.8,6942095 +4323.0,17435.9,21016.1,13113.0,6957639 +4327.8,17438.3,21016.1,13115.7,6971773 +4332.5,17434.3,21016.1,13111.4,6971773 +4337.1,17433.9,21016.1,13111.0,6971773 +4341.8,17434.2,21016.1,13112.0,6971773 +4346.6,17434.7,21016.1,13112.4,6978584 +4351.2,17435.2,21016.1,13112.6,6995840 +4355.9,17437.8,21016.1,13115.2,7013468 +4360.7,17436.0,21016.1,13113.6,7028610 +4365.5,17437.6,21016.1,13114.9,7028610 +4370.0,17438.2,21016.1,13115.3,7028610 +4374.4,17437.0,21016.1,13114.1,7028610 +4379.2,17435.9,21016.1,13113.1,7038459 +4383.9,17433.6,21016.1,13110.9,7055277 +4388.9,17432.5,21016.1,13109.8,7071369 +4393.5,17432.1,21016.1,13109.8,7071369 +4397.9,17433.3,21016.1,13110.5,7071369 +4402.6,17431.1,21016.1,13108.7,7071369 +4407.2,17439.3,21016.1,13116.5,7081462 +4411.9,17443.3,21016.1,13121.0,7096890 +4416.6,17440.2,21016.6,13117.8,7114236 +4421.2,17435.4,21016.6,13113.1,7127860 +4425.9,17435.0,21016.6,13112.5,7127860 +4430.7,17438.7,21016.6,13116.1,7127860 +4435.1,17439.4,21016.6,13117.0,7127860 +4439.8,17436.6,21016.6,13114.6,7132190 +4444.5,17438.8,21016.6,13116.8,7149066 +4449.2,17436.3,21016.6,13114.3,7163454 +4454.1,17437.9,21016.6,13115.3,7179648 +4458.9,17433.0,21016.6,13110.2,7190256 +4463.4,17435.7,21016.6,13112.7,7190256 +4468.0,17436.0,21017.6,13113.7,7190256 +4472.7,17434.3,21018.1,13111.5,7190256 +4477.3,17438.6,21018.1,13115.7,7195829 +4481.8,17432.1,21018.1,13109.2,7206759 +4486.3,17432.3,21018.1,13109.4,7218537 +4491.2,17430.9,21018.1,13108.4,7233457 +4496.0,17594.0,21018.1,13271.3,7233457 +4500.7,17593.6,21018.1,13270.6,7233457 +4505.4,17593.5,21018.1,13270.7,7233457 +4510.2,17598.6,21018.1,13276.1,7244019 +4515.0,17602.5,21019.3,13280.5,7261471 +4519.3,17595.1,21019.3,13272.9,7276851 +4523.9,17593.7,21019.3,13270.9,7292991 +4528.5,17596.4,21019.3,13273.7,7292991 +4533.3,17594.9,21019.3,13272.0,7292991 +4537.9,17596.6,21019.3,13273.9,7292991 +4542.6,17592.4,21019.3,13269.7,7302064 +4547.3,17602.2,21021.5,13279.5,7318058 +4552.1,17600.3,21021.5,13277.3,7334586 +4556.5,17600.1,21021.5,13277.3,7349428 +4561.1,17600.2,21021.5,13277.7,7349428 +4565.7,17598.5,21021.5,13276.0,7349428 +4570.5,17595.9,21021.5,13273.1,7349428 +4575.0,17596.2,21021.5,13273.5,7352353 +4579.4,17595.3,21021.5,13272.6,7367741 +4584.1,17595.9,21021.5,13273.5,7384211 +4588.8,17596.2,21021.5,13273.8,7399949 +4593.5,17595.4,21021.5,13272.7,7399949 +4598.3,17595.4,21021.5,13273.0,7399949 +4603.2,17594.6,21021.5,13271.8,7399949 +4607.8,17595.2,21021.5,13272.2,7403066 +4612.6,17594.1,21021.5,13271.0,7418460 +4617.3,17600.7,21021.5,13278.4,7433678 +4622.2,17596.6,21021.5,13274.2,7450852 +4627.0,17596.2,21021.5,13273.4,7462132 +4631.8,17595.0,21021.5,13272.2,7462132 +4636.6,17594.1,21021.5,13271.2,7462132 +4641.3,17590.9,21021.5,13268.0,7462132 +4646.1,17596.0,21021.5,13273.7,7467435 +4650.8,17595.0,21021.5,13272.3,7480695 +4655.7,17593.6,21021.5,13270.7,7498037 +4660.2,17596.3,21021.5,13273.5,7498037 +4665.0,17593.4,21021.5,13270.6,7498037 +4669.8,17592.7,21021.5,13270.2,7498037 +4674.6,17603.4,21021.5,13280.5,7498037 +4679.2,17600.2,21021.5,13277.5,7512639 +4683.9,17595.1,21021.5,13272.4,7521387 +4688.6,17590.4,21021.5,13267.8,7537175 +4693.3,17592.5,21021.5,13269.5,7537175 +4698.1,17592.6,21021.5,13269.5,7537175 +4702.8,17591.1,21021.5,13268.2,7537175 +4707.5,17597.4,21021.5,13274.5,7550093 +4712.2,17597.2,21021.5,13274.4,7564959 +4717.0,17595.7,21021.5,13273.6,7581239 +4721.7,17593.6,21021.5,13271.1,7594985 +4726.6,17595.1,21021.5,13272.5,7594985 +4731.2,17598.5,21021.5,13275.9,7594985 +4735.9,17598.0,21021.5,13275.6,7594985 +4740.8,17598.1,21021.5,13275.6,7596004 +4745.7,17591.0,21021.5,13268.2,7597230 +4750.5,17605.9,21021.5,13283.2,7602038 +4755.2,17614.9,21021.5,13292.1,7628302 +4759.9,17608.6,21021.5,13285.6,7637814 +4764.8,17601.7,21021.5,13278.9,7654414 +4769.3,17595.6,21021.5,13273.0,7667778 +4773.7,17594.9,21021.5,13271.8,7667778 +4778.5,17594.4,21021.5,13271.4,7667778 +4783.2,17592.0,21021.5,13269.3,7667778 +4787.7,17591.0,21021.5,13268.4,7668253 +4792.3,17590.1,21021.5,13267.3,7674131 +4796.9,17587.5,21021.5,13265.4,7683587 +4802.4,17595.1,21021.5,13272.2,7697575 +4808.0,17597.5,21021.5,13274.6,7707945 +4814.2,17593.5,21021.5,13270.9,7717707 +4820.7,17589.0,21021.5,13266.1,7725525 +4827.0,17589.9,21021.5,13266.8,7732979 +4832.8,17588.9,21021.5,13265.3,7732979 +4839.6,17590.5,21021.5,13267.7,7732979 +4846.7,17587.2,21021.5,13265.1,7732979 +4853.8,17585.1,21021.5,13262.4,7732979 +4866.6,17587.1,21021.5,13265.0,7732979 +4874.3,17587.5,21021.5,13264.6,7732979 +4882.2,17586.9,21021.5,13264.2,7732979 +4889.9,17583.0,21021.5,13261.0,7732979 +4898.1,17585.6,21021.5,13263.0,7733677 +4905.9,17586.3,21021.5,13263.6,7740173 +4913.6,17583.4,21021.5,13261.5,7744365 +4921.1,17585.5,21021.5,13262.9,7751381 +4929.5,17582.4,21021.5,13260.0,7756379 +4937.2,17583.7,21021.5,13260.7,7761351 +4943.6,17582.7,21021.5,13260.0,7767061 +4951.7,17594.1,21021.5,13271.7,7767061 +4959.2,17591.1,21021.5,13268.2,7767061 +4966.0,17590.7,21021.5,13268.2,7767061 +4973.5,17586.3,21021.5,13263.9,7767061 +4981.0,17586.8,21021.5,13264.8,7767061 +4988.7,17587.8,21021.5,13265.4,7767061 +4996.1,17587.4,21021.5,13264.6,7767061 +5003.2,17588.1,21021.5,13265.8,7772297 +5011.4,17585.0,21021.5,13261.9,7781507 +5019.2,17584.8,21021.5,13262.0,7787723 +5026.6,17586.3,21021.5,13263.5,7795085 +5035.7,17589.5,21021.5,13267.1,7802997 +5045.2,17587.3,21021.5,13265.0,7809947 +5053.2,17583.5,21021.5,13260.9,7809947 +5060.2,17584.3,21021.5,13261.7,7809947 +5069.4,17586.7,21021.5,13264.6,7809947 +5077.5,17586.9,21021.5,13264.3,7809947 +5085.1,17586.6,21021.5,13264.2,7809947 +5093.5,17586.6,21021.5,13264.1,7812072 +5101.9,17590.2,21021.5,13267.8,7818822 +5110.0,17592.7,21021.5,13270.8,7827116 +5118.4,17591.2,21021.5,13268.8,7834546 +5127.1,17586.7,21021.5,13264.6,7841374 +5136.2,17588.4,21021.5,13265.9,7850478 +5143.2,17590.9,21021.5,13268.6,7850478 +5152.1,17590.4,21021.5,13268.2,7850478 +5159.9,17584.4,21021.5,13262.3,7850478 +5169.6,17587.3,21021.5,13265.2,7850478 +5176.8,17587.3,21021.5,13264.4,7850478 +5184.3,17590.2,21021.5,13267.1,7853682 +5193.7,17585.9,21021.5,13263.5,7861122 +5201.9,17587.3,21021.5,13264.7,7867938 +5209.2,17586.2,21021.5,13264.4,7874614 +5218.3,17591.9,21021.5,13269.3,7883184 +5225.3,17591.6,21021.5,13269.3,7892768 +5230.7,17589.4,21021.5,13267.0,7892768 +5235.6,17588.5,21021.5,13266.1,7892768 +5240.8,17588.6,21021.5,13266.1,7892768 +5245.7,17591.7,21021.5,13269.4,7893931 +5251.1,17596.2,21021.5,13273.2,7905215 +5255.9,17596.3,21021.5,13273.6,7921805 +5260.8,17595.8,21021.5,13272.9,7931791 +5266.6,17593.7,21021.5,13270.8,7942111 +5272.2,17594.6,21021.5,13271.9,7942111 +5277.9,17590.7,21021.5,13267.7,7942111 +5283.2,17591.3,21021.5,13269.0,7942111 +5289.0,17589.1,21021.5,13266.7,7942555 +5294.1,17589.5,21021.5,13267.0,7952545 +5299.6,17604.3,21021.5,13282.0,7962693 +5304.4,17603.5,21021.5,13281.2,7978587 +5309.1,17596.9,21021.5,13274.0,7994409 +5313.7,17602.4,21021.8,13279.6,8009321 +5318.5,17600.5,21022.1,13277.7,8009321 +5322.9,17596.4,21022.1,13273.7,8009321 +5327.4,17596.3,21022.1,13273.9,8009321 +5332.4,17596.1,21022.1,13273.0,8009321 +5337.5,17600.7,21022.1,13277.6,8009321 +5342.2,17613.3,21022.1,13290.7,8025210 +5346.8,17612.6,21022.1,13290.2,8042062 +5351.6,17601.9,21022.1,13279.4,8050130 +5356.6,17596.5,21022.1,13274.9,8061152 +5361.2,17599.5,21022.1,13276.7,8076312 +5365.9,17595.7,21022.1,13272.9,8089512 +5370.7,17593.0,21022.1,13270.6,8089512 +5375.2,17597.7,21022.1,13275.1,8089512 +5379.7,17595.1,21022.1,13272.4,8089512 +5384.5,17595.2,21022.1,13272.2,8089512 +5389.5,17594.7,21022.1,13271.6,8099687 +5394.3,17595.8,21022.1,13273.3,8114103 +5399.1,17595.6,21022.1,13272.8,8130321 +5404.0,17601.9,21022.3,13279.0,8145853 +5408.6,17598.1,21022.3,13275.0,8145853 +5413.5,17594.7,21022.3,13271.8,8145853 +5418.2,17595.0,21022.3,13271.8,8145853 +5422.7,17592.5,21022.3,13269.8,8145853 +5427.2,17599.9,21022.3,13276.9,8155690 +5431.7,17598.2,21022.3,13275.1,8169446 +5436.5,17595.7,21022.3,13272.7,8183264 +5441.3,17596.9,21022.3,13274.0,8196256 +5446.3,17598.1,21022.3,13275.3,8196256 +5451.0,17595.7,21022.3,13272.9,8196256 +5456.0,17596.5,21022.3,13273.5,8196256 +5461.1,17594.0,21022.3,13270.9,8205731 +5465.7,17594.8,21022.3,13271.9,8214731 +5470.3,17592.2,21022.3,13269.7,8230947 +5474.8,17593.3,21022.3,13270.7,8243611 +5479.7,17586.0,21022.3,13263.5,8255937 +5484.5,17596.9,21022.3,13274.7,8255937 +5489.6,17600.0,21022.3,13277.6,8255937 +5494.3,17601.4,21022.3,13278.6,8255937 +5498.8,17605.6,21022.3,13282.8,8263805 +5503.4,17604.8,21022.3,13282.1,8280745 +5508.6,17597.2,21022.3,13274.6,8290135 +5513.7,17598.5,21022.3,13275.7,8309429 +5518.6,17593.3,21022.3,13270.5,8309429 +5523.5,17592.8,21022.3,13270.2,8309429 +5528.2,17593.2,21022.3,13270.5,8309429 +5532.9,17596.4,21022.3,13273.6,8309429 +5537.5,17596.6,21022.3,13274.3,8319971 +5542.1,17593.7,21022.3,13271.1,8335367 +5547.0,17592.4,21022.3,13269.5,8342625 +5552.0,17590.1,21022.3,13267.7,8358569 +5556.8,17592.6,21022.3,13269.9,8371833 +5561.7,17593.0,21022.3,13270.2,8371833 +5567.3,17592.7,21022.3,13270.1,8371833 +5572.2,17595.0,21022.3,13272.1,8371833 +5576.9,17594.3,21022.3,13271.6,8380178 +5581.6,17595.4,21022.3,13272.9,8394660 +5586.1,17596.8,21022.3,13274.2,8409714 +5590.9,17595.2,21022.3,13272.6,8423014 +5595.9,17595.9,21022.3,13273.3,8439776 +5600.7,17597.7,21022.3,13274.8,8439776 +5605.3,17596.3,21022.3,13274.0,8439776 +5610.0,17595.0,21022.3,13272.7,8439776 +5614.6,17591.6,21022.3,13269.1,8439776 +5619.3,17593.0,21022.3,13270.7,8449801 +5623.8,17614.7,21029.9,13292.0,8464755 +5628.5,17614.3,21030.1,13291.5,8478045 +5633.5,17609.1,21030.4,13286.5,8492643 +5638.3,17609.6,21030.4,13287.3,8492643 +5643.1,17607.3,21030.4,13285.0,8492643 +5647.8,17609.5,21030.4,13286.8,8492643 +5652.4,17611.3,21030.4,13288.6,8492643 +5657.0,17613.2,21030.4,13290.2,8504927 +5661.7,17613.2,21030.4,13290.3,8521763 +5666.3,17609.5,21030.4,13286.6,8534619 +5671.2,17608.7,21030.4,13285.6,8550761 +5675.9,17611.0,21030.4,13288.2,8550761 +5680.6,17610.5,21030.4,13287.9,8550761 +5685.4,17610.8,21030.4,13288.0,8550761 +5689.9,17612.3,21030.4,13290.0,8553142 +5694.3,17609.5,21030.4,13286.7,8567478 +5699.1,17609.0,21030.4,13285.9,8587038 +5704.0,17607.6,21030.4,13284.9,8600046 +5708.9,17607.6,21030.4,13284.4,8615458 +5713.8,17608.6,21030.4,13286.1,8615458 +5718.5,17609.5,21030.4,13286.4,8615458 +5723.1,17604.0,21030.4,13281.2,8615458 +5727.8,17608.5,21030.4,13286.4,8615458 +5732.7,17606.2,21030.4,13284.0,8625740 +5737.4,17605.0,21030.4,13282.4,8634352 +5742.2,17599.8,21030.4,13277.5,8639462 +5746.9,17602.9,21030.4,13280.4,8654614 +5751.7,17605.2,21030.4,13282.1,8654614 +5756.7,17606.2,21030.4,13283.2,8654614 +5761.5,17611.0,21030.4,13288.5,8654614 +5766.3,17625.6,21031.8,13303.1,8654614 +5770.9,17622.2,21032.2,13299.5,8657251 +5775.5,17618.4,21032.2,13294.8,8673619 +5780.0,17606.2,21032.4,13283.7,8688555 +5784.9,17603.8,21032.4,13280.9,8706071 +5789.9,17606.3,21032.4,13283.2,8721835 +5794.6,17606.2,21032.4,13284.1,8721835 +5799.2,17608.5,21032.4,13285.6,8721835 +5803.7,17607.3,21032.4,13284.9,8721835 +5808.3,17614.7,21032.4,13291.8,8721835 +5813.1,17612.6,21032.4,13290.1,8732910 +5817.9,17614.6,21032.4,13291.8,8749532 +5822.3,17612.1,21032.4,13289.6,8757570 +5826.8,17613.3,21032.4,13290.6,8768980 +5831.6,17608.7,21032.4,13286.3,8768980 +5836.4,17610.4,21032.4,13287.4,8768980 +5840.9,17605.7,21032.4,13282.6,8768980 +5845.8,17608.1,21032.4,13285.2,8768980 +5850.7,17607.5,21032.4,13284.8,8786330 +5855.4,17611.4,21032.4,13288.7,8800948 +5860.3,17607.6,21032.4,13285.0,8804992 +5865.1,17609.9,21032.4,13287.1,8823108 +5870.0,17609.3,21032.4,13286.8,8838700 +5874.7,17612.0,21032.4,13288.9,8838700 +5879.3,17607.3,21032.4,13284.9,8838700 +5884.5,17606.1,21032.4,13283.4,8838700 +5890.5,17605.8,21032.4,13283.1,8838700 +5897.0,17610.1,21032.4,13287.1,8838700 +5903.4,17610.9,21032.4,13288.0,8848358 +5910.4,17605.7,21032.4,13282.8,8855934 +5917.2,17602.0,21032.4,13279.3,8863932 +5923.9,17599.3,21032.4,13276.7,8871526 +5930.9,17597.3,21032.4,13274.5,8877602 +5938.1,17599.7,21032.4,13276.7,8884070 +5945.8,17596.8,21032.4,13274.2,8890820 +5954.5,17598.6,21032.4,13276.3,8898504 +5961.8,17597.0,21032.4,13274.7,8904932 +5972.9,17596.1,21032.4,13273.1,8913742 +5981.9,17597.5,21032.4,13274.5,8920852 +5991.5,17595.3,21032.4,13272.5,8927392 +6001.6,17598.8,21032.4,13276.2,8934774 +6009.4,17602.6,21032.4,13279.0,8934774 +6016.4,17601.9,21032.4,13279.5,8934774 +6027.1,17597.5,21032.4,13275.1,8934774 +6036.8,17600.5,21032.4,13278.1,8934774 +6045.7,17600.3,21032.4,13278.0,8934774 +6054.3,17599.0,21032.4,13276.5,8934774 +6062.1,17602.3,21032.4,13279.9,8934774 +6069.3,17607.3,21032.4,13285.0,8934774 +6076.2,17601.7,21032.4,13279.1,8934774 +6083.3,17602.9,21032.4,13280.0,8934774 +6091.6,17602.1,21032.4,13278.1,8936308 +6099.8,17602.8,21032.4,13280.4,8943484 +6107.2,17602.7,21032.4,13280.3,8950716 +6114.9,17597.5,21032.4,13274.8,8958012 +6123.0,17599.4,21032.4,13277.0,8964492 +6131.4,17602.4,21032.4,13278.3,8974952 +6139.2,17601.5,21032.4,13278.7,8974952 +6145.9,17598.4,21032.4,13275.8,8974952 +6152.1,17600.9,21032.4,13278.3,8974952 +6158.5,17607.7,21032.4,13285.4,8974952 +6165.6,17617.4,21032.4,13294.7,8974952 +6172.1,17612.6,21032.4,13289.8,8974952 +6181.6,17598.7,21032.4,13275.7,8974952 +6187.6,17621.0,21033.1,13298.7,8978422 +6193.9,17609.1,21033.3,13286.8,8985588 +6200.1,17604.4,21033.3,13281.4,8993330 +6205.6,17604.0,21033.3,13281.2,9003416 +6210.9,17604.7,21033.3,13281.7,9003416 +6215.8,17610.6,21033.3,13287.7,9003416 +6220.6,17605.3,21033.3,13282.4,9003416 +6225.5,17605.6,21033.3,13284.4,9009711 +6230.3,17605.4,21033.3,13282.8,9025339 +6235.1,17605.5,21033.3,13283.0,9040889 +6239.7,17609.3,21033.3,13286.3,9054757 +6244.5,17604.9,21033.3,13282.2,9065629 +6249.4,17604.8,21033.3,13281.9,9081651 +6254.0,17606.0,21033.3,13283.2,9081651 +6258.7,17611.7,21033.3,13289.2,9081651 +6263.3,17610.4,21033.3,13287.8,9081651 +6268.0,17612.5,21033.3,13289.5,9081651 +6272.8,17611.8,21033.3,13288.7,9094690 +6277.4,17608.9,21033.3,13286.2,9105006 +6282.1,17609.0,21033.3,13286.2,9119428 +6287.3,17610.9,21033.3,13288.3,9133546 +6292.1,17609.6,21033.3,13286.6,9133546 +6296.6,17612.3,21033.3,13289.8,9133546 +6301.1,17613.9,21033.3,13290.9,9133546 +6306.0,17610.4,21033.3,13287.5,9133546 +6310.8,17606.7,21033.3,13284.2,9143607 +6315.8,17605.7,21033.3,13282.8,9156509 +6320.8,17605.3,21033.3,13282.6,9170123 +6325.6,17605.2,21033.3,13282.2,9183151 +6331.0,17606.2,21033.3,13283.2,9197223 +6335.7,17617.2,21033.4,13294.2,9197223 +6340.4,17613.0,21033.4,13289.8,9197223 +6345.2,17615.5,21033.4,13292.9,9197223 +6349.9,17611.2,21033.4,13288.4,9197223 +6354.4,17621.8,21033.4,13299.1,9210715 +6359.1,17621.7,21033.4,13298.9,9228645 +6363.7,17615.4,21033.4,13293.1,9242895 +6368.6,17614.6,21033.4,13292.3,9259403 +6373.3,17608.1,21033.4,13285.3,9273839 +6377.9,17609.7,21033.4,13286.9,9273839 +6382.9,17605.4,21033.4,13282.5,9273839 +6387.7,17605.5,21033.4,13282.6,9273839 +6392.6,17604.5,21033.4,13281.9,9273839 +6397.4,17605.1,21033.4,13282.2,9273839 +6402.2,17607.9,21033.4,13285.4,9285089 +6406.8,17609.4,21033.4,13286.3,9298687 +6411.8,17604.5,21033.4,13281.3,9311835 +6416.5,17603.8,21033.4,13280.6,9324459 +6421.1,17601.5,21033.4,13278.8,9324459 +6425.7,17608.8,21033.4,13286.3,9324459 +6430.4,17609.3,21033.4,13287.2,9324459 +6435.4,17607.1,21033.4,13284.5,9324459 +6440.2,17603.7,21033.4,13280.9,9337386 +6445.1,17603.4,21033.4,13280.6,9354360 +6450.0,17606.4,21033.4,13283.7,9367400 +6455.0,17606.0,21033.4,13282.9,9377108 +6460.4,17604.7,21033.4,13282.0,9390678 +6465.6,17607.5,21033.4,13285.1,9390678 +6470.2,17617.7,21034.7,13294.4,9390678 +6474.7,17610.6,21034.7,13287.8,9390678 +6479.2,17610.2,21034.7,13287.4,9390678 +6483.9,17610.8,21034.7,13287.8,9395641 +6488.4,17608.9,21034.7,13285.5,9406389 +6492.9,17609.0,21034.7,13286.4,9423077 +6497.3,17609.4,21034.7,13286.7,9438569 +6502.0,17609.4,21034.7,13286.3,9455937 +6506.9,17609.6,21034.7,13288.6,9471719 +6511.4,17622.4,21036.6,13299.7,9471719 +6516.1,17620.0,21036.6,13297.2,9471719 +6520.9,17611.3,21036.6,13289.6,9471719 +6525.6,17611.2,21036.6,13289.0,9471719 +6530.4,17604.9,21036.6,13282.5,9479225 +6535.0,17610.5,21036.6,13287.8,9497249 +6539.6,17614.6,21036.7,13291.5,9511121 +6544.4,17615.0,21036.7,13292.1,9524437 +6548.9,17609.5,21036.7,13287.0,9538413 +6553.5,17611.9,21036.7,13289.7,9538413 +6558.3,17619.6,21037.5,13296.9,9538413 +6562.9,17616.9,21037.7,13294.7,9538413 +6568.0,17617.9,21037.7,13295.6,9538413 +6572.8,17629.9,21038.0,13307.8,9541274 +6577.4,17633.2,21038.0,13310.9,9555074 +6582.1,17619.0,21038.0,13296.4,9566310 +6586.8,17614.0,21038.0,13290.8,9567724 +6591.6,17616.8,21038.0,13294.2,9580778 +6596.3,17618.6,21038.0,13295.9,9600488 +6601.0,17614.9,21038.0,13292.1,9615640 +6605.7,17611.0,21038.0,13288.4,9615640 +6610.4,17608.3,21038.0,13285.4,9615640 +6614.9,17610.4,21038.0,13288.0,9615640 +6619.6,17609.0,21038.0,13286.3,9615640 +6624.5,17607.9,21038.0,13285.1,9627628 +6629.3,17608.9,21038.0,13286.4,9646006 +6634.0,17608.8,21038.0,13285.8,9660282 +6638.6,17607.3,21038.0,13284.4,9679626 +6643.1,17607.3,21038.0,13284.5,9679626 +6647.7,17609.9,21038.0,13287.0,9679626 +6652.4,17609.9,21038.0,13287.2,9679626 +6657.0,17611.7,21038.0,13288.9,9679626 +6661.7,17608.9,21038.0,13286.0,9693082 +6666.6,17607.4,21038.0,13284.3,9705902 +6671.2,17606.1,21038.0,13283.4,9723292 +6676.0,17607.1,21038.0,13284.4,9742802 +6680.7,17604.6,21038.0,13282.0,9759486 +6685.3,17607.7,21038.0,13284.8,9759486 +6689.9,17606.0,21038.0,13283.2,9759486 +6694.6,17605.5,21038.0,13282.9,9759486 +6699.3,17606.0,21038.0,13283.5,9759486 +6703.9,17606.2,21038.0,13283.0,9759486 +6708.7,17607.6,21038.0,13284.5,9772801 +6713.2,17604.2,21038.0,13281.2,9791193 +6717.9,17604.9,21038.0,13282.4,9801285 +6722.7,17607.3,21038.0,13285.2,9818909 +6727.4,17607.5,21038.0,13285.2,9836249 +6731.9,17609.9,21038.0,13287.8,9836249 +6736.5,17616.5,21038.0,13294.6,9836249 +6741.4,17627.8,21038.0,13305.5,9836249 +6746.2,17642.5,21041.4,13319.4,9836249 +6750.9,17638.7,21043.0,13315.6,9836249 +6755.7,17634.0,21043.0,13311.3,9849090 +6760.4,17613.6,21043.0,13291.2,9867012 +6765.0,17611.6,21043.0,13288.9,9880774 +6769.5,17606.6,21043.0,13283.9,9897164 +6774.3,17609.0,21043.0,13286.3,9914966 +6779.1,17609.9,21043.0,13286.9,9914966 +6783.8,17608.5,21043.0,13285.6,9914966 +6788.4,17607.1,21043.0,13284.2,9914966 +6793.2,17604.3,21043.0,13282.1,9914966 +6797.8,17609.8,21043.0,13287.6,9915803 +6802.4,17606.8,21043.0,13284.3,9932587 +6807.2,17608.5,21043.0,13285.6,9940985 +6811.6,17605.9,21043.0,13283.1,9958541 +6816.5,17606.0,21043.0,13283.1,9975957 +6821.1,17607.3,21043.0,13284.8,9985673 +6826.1,17605.6,21043.0,13282.6,9985673 +6831.4,17609.1,21043.0,13285.9,9985673 +6837.0,17605.9,21043.0,13282.7,9985673 +6843.2,17602.2,21043.0,13280.1,9985673 +6849.1,17612.4,21043.0,13289.6,9985673 +6854.8,17605.3,21043.0,13282.6,9985673 +6860.6,17601.0,21043.0,13278.3,9987898 +6867.5,17604.0,21043.0,13281.6,9995744 +6874.9,17602.7,21043.0,13279.8,10006030 +6881.9,17602.5,21043.0,13279.8,10016422 +6889.2,17600.8,21043.0,13278.1,10026290 +6897.4,17604.7,21043.0,13282.2,10036350 +6905.3,17600.5,21043.0,13277.3,10044462 +6911.7,17596.3,21043.0,13273.5,10050908 +6917.8,17601.4,21043.0,13278.4,10050908 +6925.2,17606.6,21043.0,13284.1,10050908 +6935.0,17602.3,21043.0,13279.1,10050908 +6940.7,17599.4,21043.0,13276.8,10050908 +6945.4,17602.2,21043.0,13279.8,10050908 +6950.8,17603.0,21043.0,13280.4,10050908 +6955.8,17604.8,21043.0,13282.2,10055202 +6960.9,17606.5,21043.0,13284.2,10066026 +6965.8,17608.7,21043.0,13285.6,10076704 +6970.5,17606.7,21043.0,13284.0,10086722 +6975.3,17608.5,21043.0,13285.7,10099404 +6980.3,17606.0,21043.0,13283.1,10099404 +6985.9,17608.3,21043.0,13285.8,10099404 +6991.2,17604.0,21043.0,13281.4,10099404 +6997.0,17603.1,21043.0,13280.2,10099404 +7002.4,17602.4,21043.0,13279.8,10099404 +7010.2,17599.4,21043.0,13276.7,10101237 +7015.9,17600.8,21043.0,13278.4,10111091 +7021.7,17600.4,21043.0,13278.1,10120589 +7027.7,17603.3,21043.0,13280.6,10127349 +7033.9,17599.6,21043.0,13277.0,10134155 +7040.1,17598.1,21043.0,13275.5,10143043 +7046.1,17620.7,21043.0,13298.7,10151071 +7053.3,17643.6,21045.4,13321.2,10163251 +7060.3,17633.8,21046.7,13311.7,10174323 +7067.0,17645.7,21054.3,13323.0,10187389 +7072.5,17652.0,21059.9,13329.8,10201397 +7078.1,17665.1,21066.9,13342.7,10216105 +7085.1,17660.9,21069.8,13338.5,10216105 +7090.8,17652.2,21071.2,13331.1,10216105 +7096.2,17654.0,21072.4,13332.8,10216105 +7102.0,17651.1,21073.5,13329.7,10216105 +7108.2,17648.6,21074.8,13326.6,10216105 +7115.8,17642.8,21074.8,13321.1,10216105 +7123.3,17655.9,21074.8,13333.5,10216105 +7129.6,17668.1,21076.4,13345.8,10225206 +7135.0,17655.5,21077.5,13333.7,10231546 +7140.3,17664.1,21078.5,13341.6,10253124 +7146.0,17672.6,21081.1,13350.1,10267456 +7151.6,17674.4,21082.4,13351.1,10291942 +7156.5,17676.5,21083.9,13353.4,10315520 +7161.5,17670.9,21086.8,13347.5,10315520 +7166.9,17675.6,21088.2,13355.1,10315520 +7171.9,17679.0,21090.9,13356.4,10315520 +7177.8,17654.9,21097.6,13331.7,10315520 +7184.1,17645.4,21097.6,13321.9,10315520 +7190.3,17639.8,21097.6,13316.8,10322829 +7196.0,17645.3,21097.6,13322.5,10335789 +7201.8,17641.7,21097.6,13319.4,10352517 +7207.0,17652.1,21097.6,13329.6,10370729 +7211.9,17664.8,21098.9,13341.9,10395675 +7216.8,17665.0,21100.7,13341.8,10423237 +7221.5,17668.0,21100.7,13345.1,10442615 +7226.3,17664.3,21101.9,13341.6,10442615 +7230.7,17661.5,21101.9,13338.2,10442615 +7235.4,17657.0,21101.9,13335.7,10442615 +7240.3,17669.3,21106.5,13347.0,10442615 +7245.0,17679.1,21106.5,13356.4,10445490 +7249.9,17658.1,21106.5,13335.8,10467688 +7254.7,17667.7,21106.5,13344.2,10489850 +7259.3,17676.9,21106.5,13353.9,10513516 +7264.2,17665.5,21107.6,13342.4,10538688 +7269.0,17661.0,21107.6,13337.6,10559076 +7274.7,17658.3,21107.6,13335.3,10581554 +7279.5,17657.9,21107.6,13334.7,10601434 +7284.5,17669.5,21110.1,13346.2,10601434 +7289.2,17667.9,21111.4,13344.8,10601434 +7293.9,17667.1,21113.5,13344.4,10601434 +7298.7,17663.8,21113.5,13340.7,10601434 +7303.0,17657.4,21113.5,13334.2,10601434 +7307.7,17665.6,21113.5,13342.3,10606493 +7312.3,17661.9,21113.5,13339.3,10627765 +7317.3,17668.5,21113.5,13345.2,10647635 +7322.2,17662.3,21113.5,13339.9,10666997 +7327.1,17668.5,21113.5,13345.4,10686353 +7332.1,17663.9,21113.5,13341.8,10704261 +7336.9,17665.7,21113.5,13342.8,10722819 +7341.7,17658.4,21113.5,13335.7,10722819 +7346.3,17658.1,21113.5,13335.0,10722819 +7351.0,17655.3,21113.5,13332.4,10722819 +7355.8,17661.8,21113.5,13339.3,10722819 +7360.6,17674.6,21113.5,13351.8,10722819 +7365.5,17688.1,21114.1,13365.3,10722819 +7370.2,17700.7,21117.3,13377.7,10723345 +7374.8,17693.8,21117.4,13370.8,10749415 +7379.4,17664.9,21117.4,13342.3,10775051 +7384.0,17662.9,21117.4,13339.9,10793317 +7388.9,17656.2,21117.4,13333.2,10817189 +7393.7,17653.5,21117.4,13330.5,10836099 +7398.8,17654.5,21117.4,13331.6,10855841 +7403.6,17654.8,21117.4,13331.9,10855841 +7408.5,17656.6,21117.4,13333.9,10855841 +7413.4,17662.7,21117.4,13339.5,10855841 +7418.9,17658.3,21117.4,13335.6,10855841 +7423.3,17663.8,21117.4,13340.8,10855841 +7428.2,17661.7,21117.4,13338.2,10855841 +7433.0,17666.0,21117.4,13342.7,10870865 +7437.9,17667.3,21117.4,13344.5,10883305 +7442.8,17667.7,21117.4,13344.5,10908645 +7447.6,17662.9,21117.4,13339.6,10931301 +7452.3,17661.5,21117.4,13338.3,10951377 +7457.2,17652.8,21117.4,13330.2,10968503 +7462.1,17656.4,21117.4,13333.6,10968503 +7466.6,17659.9,21117.4,13337.0,10968503 +7471.0,17658.0,21117.4,13334.8,10968503 +7475.7,17655.7,21117.4,13332.9,10968503 +7480.5,17652.0,21117.4,13329.1,10968503 +7485.2,17657.4,21117.4,13335.2,10981941 +7489.7,17663.0,21117.4,13340.3,10998029 +7494.3,17656.3,21117.4,13333.4,11021109 +7499.2,17650.9,21117.4,13328.2,11041089 +7504.1,17654.1,21117.4,13331.1,11060339 +7508.7,17650.7,21117.4,13327.6,11080949 +7513.3,17650.0,21117.4,13326.9,11080949 +7518.0,17656.2,21117.4,13333.2,11080949 +7522.4,17652.2,21117.4,13329.1,11080949 +7527.2,17651.2,21117.4,13328.7,11080949 +7531.8,17650.6,21117.4,13327.3,11080949 +7536.5,17654.2,21117.4,13331.6,11094274 +7541.2,17653.5,21117.4,13330.3,11109850 +7545.9,17648.8,21117.4,13326.3,11128594 +7550.4,17652.9,21117.4,13330.0,11149012 +7555.1,17654.2,21117.4,13331.3,11169162 +7559.9,17659.5,21117.4,13336.0,11186628 +7564.8,17658.4,21117.4,13334.9,11203732 +7569.6,17652.4,21117.4,13329.1,11223246 +7574.2,17651.0,21117.4,13328.0,11223246 +7579.0,17653.8,21117.4,13330.8,11223246 +7583.8,17653.0,21117.4,13330.2,11223246 +7588.6,17658.0,21117.4,13336.3,11223246 +7593.4,17654.5,21117.4,13332.0,11223246 +7598.0,17651.9,21117.4,13328.7,11223476 +7602.6,17647.8,21117.4,13325.6,11247376 +7607.4,17648.5,21117.4,13326.8,11267092 +7612.8,17648.9,21117.4,13326.3,11281922 +7619.3,17640.7,21117.4,13317.7,11295518 +7625.7,17640.2,21117.4,13317.3,11308346 +7631.8,17636.7,21117.4,13313.7,11315794 +7637.6,17637.6,21117.4,13314.6,11325914 +7643.4,17641.5,21117.4,13318.2,11334454 +7649.8,17638.5,21117.4,13315.5,11342614 +7656.6,17640.9,21117.4,13318.0,11351086 +7662.9,17639.3,21117.4,13316.6,11357514 +7669.5,17632.3,21117.4,13309.2,11370510 +7676.7,17637.6,21117.4,13314.6,11376848 +7685.2,17638.7,21117.4,13316.2,11389064 +7691.8,17636.6,21117.4,13313.6,11397112 +7699.3,17636.0,21117.4,13313.0,11407974 +7706.5,17639.9,21117.4,13316.7,11407974 +7713.9,17639.9,21117.4,13317.1,11407974 +7719.9,17644.5,21117.4,13321.5,11407974 +7724.7,17651.0,21117.4,13328.1,11407974 +7729.7,17641.2,21117.4,13317.9,11407974 +7734.8,17649.0,21117.4,13325.6,11407974 +7741.5,17644.7,21117.4,13321.1,11407974 +7747.6,17641.3,21117.4,13317.5,11407974 +7754.0,17642.4,21117.4,13318.5,11407974 +7760.2,17641.2,21117.4,13316.6,11407974 +7766.8,17639.4,21117.4,13314.6,11407974 +7773.6,17642.5,21117.4,13318.7,11407974 +7780.4,17637.0,21117.4,13313.3,11407974 +7787.3,17635.3,21117.4,13310.6,11407974 +7794.9,17638.5,21117.4,13313.8,11407974 +7802.1,17637.0,21117.4,13312.8,11407974 +7808.4,17637.1,21117.4,13313.1,11408302 +7814.7,17640.9,21117.4,13316.9,11416564 +7821.6,17635.1,21117.4,13310.7,11422932 +7828.6,17637.8,21117.4,13314.0,11429548 +7835.4,17638.3,21117.4,13313.9,11435706 +7841.7,17637.8,21117.4,13314.1,11442242 +7850.1,17635.7,21117.4,13311.4,11451728 +7856.5,17636.4,21117.4,13312.5,11459190 +7863.0,17640.5,21117.4,13316.2,11466676 +7869.2,17636.0,21117.4,13311.9,11473192 +7875.1,17637.1,21117.4,13313.1,11482832 +7881.0,17641.2,21117.4,13317.0,11489528 +7886.6,17639.7,21117.4,13315.6,11502456 +7892.3,17646.7,21117.4,13322.9,11515594 +7898.2,17647.2,21117.4,13323.0,11528856 +7904.0,17647.4,21117.4,13323.3,11528856 +7909.7,17645.4,21117.4,13320.9,11528856 +7915.9,17645.4,21117.4,13321.4,11528856 +7923.5,17645.2,21117.4,13320.5,11528856 +7930.1,17641.5,21117.4,13317.0,11528856 +7938.2,17636.0,21117.4,13311.9,11528856 +7945.1,17640.9,21117.4,13316.8,11528856 +7951.9,17640.0,21117.4,13315.9,11528856 +7961.0,17643.9,21117.4,13319.4,11528856 +7968.3,17641.8,21117.4,13317.6,11528856 +7975.1,17644.5,21117.4,13320.3,11528856 +7982.1,17640.3,21117.4,13315.8,11528856 +7990.0,17641.7,21117.4,13317.6,11528856 +7996.9,17635.2,21117.4,13311.1,11528856 +8003.9,17634.7,21117.4,13311.0,11528856 +8011.2,17635.4,21117.4,13311.5,11532690 +8017.8,17634.3,21117.4,13310.7,11540654 +8025.2,17641.0,21117.4,13316.4,11548126 +8032.0,17642.9,21117.4,13318.3,11560450 +8038.6,17648.0,21117.4,13324.0,11566822 +8044.9,17648.7,21117.4,13324.7,11572632 +8051.4,17639.9,21117.4,13316.3,11580364 +8057.1,17645.9,21117.4,13321.5,11594078 +8063.6,17649.8,21117.4,13325.3,11604798 +8068.7,17653.6,21117.4,13328.7,11618796 +8073.8,17652.3,21117.4,13327.9,11618796 +8078.7,17647.0,21117.4,13322.3,11618796 +8083.7,17653.3,21117.4,13329.0,11618796 +8088.5,17655.6,21117.4,13331.9,11618796 +8093.6,17656.9,21117.4,13333.0,11618796 +8099.0,17649.1,21117.4,13324.9,11618796 +8103.9,17648.3,21117.4,13324.4,11621464 +8109.3,17644.9,21117.4,13320.8,11633260 +8114.8,17646.5,21117.4,13322.8,11640984 +8120.7,17648.0,21117.4,13323.1,11651734 +8126.1,17646.2,21117.4,13322.0,11664778 +8131.4,17649.4,21117.4,13324.8,11680568 +8136.4,17650.6,21117.4,13326.8,11694728 +8141.6,17654.1,21117.4,13329.6,11709108 +8146.4,17658.7,21117.4,13335.0,11709108 +8151.4,17662.0,21117.4,13337.4,11709108 +8156.4,17654.7,21117.4,13330.2,11709108 +8161.3,17652.9,21117.4,13328.7,11709108 +8166.0,17652.3,21117.4,13328.8,11709108 +8170.8,17653.0,21117.4,13328.8,11711226 +8175.7,17654.4,21117.4,13330.2,11728900 +8180.6,17655.5,21117.4,13331.8,11746994 +8185.5,17653.6,21117.4,13329.1,11764170 +8190.5,17652.0,21117.4,13327.6,11785308 +8195.4,17650.1,21117.4,13325.7,11802892 +8199.9,17644.2,21117.4,13319.9,11815780 +8204.8,17646.6,21117.4,13322.3,11815780 +8209.4,17654.1,21117.4,13330.1,11815780 +8214.1,17667.2,21117.4,13343.4,11815780 +8218.9,17663.2,21117.4,13339.1,11815780 +8223.7,17658.2,21117.4,13333.8,11815780 +8228.4,17650.7,21117.4,13326.9,11818597 +8233.0,17649.2,21117.4,13324.6,11836279 +8238.4,17652.5,21117.4,13328.1,11854885 +8243.2,17655.5,21117.4,13331.9,11870975 +8248.1,17652.5,21117.4,13328.4,11888925 +8253.4,17648.7,21117.4,13324.2,11906383 +8258.4,17646.6,21117.4,13322.3,11921637 +8262.9,17645.4,21117.4,13321.9,11921637 +8267.5,17647.9,21117.4,13323.2,11921637 +8272.2,17645.8,21117.4,13321.7,11921637 +8277.1,17643.5,21117.4,13319.9,11921637 +8281.9,17647.3,21117.4,13323.6,11921637 +8286.8,17645.0,21117.4,13320.7,11930320 +8291.6,17653.8,21117.4,13329.6,11947340 +8296.3,17652.0,21117.4,13327.6,11960574 +8301.2,17649.4,21117.4,13325.6,11980118 +8305.9,17654.3,21117.4,13329.8,11998238 +8310.4,17654.4,21117.4,13330.1,12015194 +8315.2,17648.7,21117.4,13324.3,12031686 +8320.1,17646.2,21117.4,13321.5,12031686 +8324.6,17645.4,21117.4,13321.0,12031686 +8329.3,17646.0,21117.4,13321.5,12031686 +8333.9,17643.0,21117.4,13318.5,12031686 +8338.5,17644.2,21117.4,13320.3,12031686 +8343.3,17647.4,21117.4,13323.0,12031686 +8348.0,17652.7,21117.4,13328.6,12035338 +8352.5,17646.9,21117.4,13322.9,12051640 +8357.1,17648.4,21117.4,13324.4,12073312 +8362.1,17648.3,21117.4,13324.3,12089536 +8367.0,17646.6,21117.4,13323.0,12107044 +8372.1,17646.6,21117.4,13322.2,12124034 +8376.9,17646.1,21117.4,13321.7,12140092 +8381.9,17646.2,21117.4,13321.9,12140092 +8386.8,17647.0,21117.4,13322.4,12140092 +8391.7,17650.3,21117.4,13325.7,12140092 +8396.4,17648.7,21117.4,13324.1,12140092 +8401.2,17656.0,21117.4,13331.7,12140092 +8405.9,17653.3,21117.4,13329.4,12140092 +8410.5,17652.1,21117.4,13328.0,12158995 +8415.2,17646.6,21117.4,13323.1,12171825 +8420.1,17649.0,21117.4,13324.7,12189041 +8424.8,17650.6,21117.4,13326.2,12209845 +8429.5,17651.9,21117.4,13325.9,12227957 +8434.2,17648.5,21117.4,13324.6,12244197 +8439.0,17649.9,21117.4,13326.2,12244197 +8443.5,17654.0,21117.4,13330.0,12244197 +8448.1,17649.7,21117.4,13325.4,12244197 +8452.5,17645.2,21117.4,13321.2,12244197 +8457.2,17647.2,21117.4,13322.9,12244197 +8461.9,17649.4,21117.4,13325.8,12244197 +8466.4,17652.5,21117.4,13328.7,12254459 +8471.0,17656.0,21117.4,13332.0,12272801 +8475.8,17654.2,21117.4,13330.2,12294147 +8480.5,17654.0,21117.4,13329.5,12315533 +8485.1,17655.6,21117.4,13330.7,12326955 +8489.8,17652.3,21117.4,13328.4,12345613 +8494.8,17647.6,21117.4,13322.7,12364317 +8499.8,17646.9,21117.4,13322.8,12382419 +8504.5,17646.2,21117.4,13322.3,12382419 +8509.1,17648.6,21117.4,13324.2,12382419 +8513.9,17644.8,21117.4,13320.3,12382419 +8518.3,17645.2,21117.4,13321.4,12382419 +8523.2,17641.0,21117.4,13317.1,12382419 +8527.6,17644.1,21117.4,13319.7,12382419 +8532.4,17643.1,21117.4,13318.8,12388793 +8537.2,17642.9,21117.4,13318.6,12404565 +8541.9,17641.3,21117.4,13317.3,12424285 +8546.5,17641.9,21117.4,13318.0,12442363 +8551.0,17642.0,21117.4,13317.7,12460925 +8555.8,17643.2,21117.4,13319.4,12480141 +8560.6,17643.3,21117.4,13319.2,12480141 +8565.2,17648.0,21117.4,13324.5,12480141 +8569.8,17648.2,21117.4,13324.3,12480141 +8574.5,17658.6,21117.4,13334.4,12480141 +8579.3,17654.3,21117.4,13330.0,12480141 +8584.1,17647.6,21117.4,13323.5,12480141 +8588.5,17645.0,21117.4,13321.1,12489981 +8593.3,17686.8,21119.4,13362.9,12519109 +8597.7,17716.4,21134.5,13392.2,12548379 +8602.1,17723.1,21145.8,13399.0,12573713 +8606.7,17763.1,21174.2,13440.3,12620317 +8611.8,17739.6,21175.3,13414.4,12644361 +8616.3,17737.7,21179.4,13415.4,12687049 +8621.2,17731.3,21179.4,13406.5,12687049 +8625.7,17727.0,21180.1,13402.3,12687049 +8630.5,17726.2,21180.1,13401.2,12687049 +8635.3,17716.1,21180.1,13391.6,12687049 +8640.3,17718.0,21180.1,13393.3,12693359 +8645.1,17702.6,21180.1,13378.3,12709985 +8650.0,17700.5,21180.1,13375.7,12730875 +8654.6,17716.1,21180.1,13391.2,12752777 +8659.3,17712.9,21180.1,13388.2,12790687 +8663.8,17701.0,21180.1,13376.9,12806875 +8668.6,17708.1,21180.1,13383.8,12839747 +8673.5,17708.1,21180.1,13383.2,12868817 +8678.3,17707.1,21180.1,13382.8,12899267 +8683.2,17707.4,21180.1,13383.2,12920329 +8688.2,17715.5,21180.1,13390.8,12947643 +8692.7,17707.2,21180.1,13382.2,12977239 +8697.4,17704.9,21180.1,13380.3,12977239 +8702.3,17696.7,21180.1,13372.2,12977239 +8707.1,17703.2,21180.1,13379.3,12977239 +8711.6,17699.6,21180.1,13375.2,12977239 +8716.4,17703.9,21180.1,13380.1,12977239 +8721.2,17711.4,21180.1,13386.6,12977239 +8725.8,17715.3,21180.1,13391.2,12977239 +8730.6,17713.4,21180.1,13388.4,12977239 +8735.3,17703.6,21180.1,13378.7,12998291 +8740.0,17714.6,21180.1,13389.2,13034389 +8744.7,17716.7,21180.1,13391.9,13058567 +8749.8,17702.4,21180.1,13377.8,13088525 +8755.3,17687.1,21180.1,13362.2,13101401 +8761.4,17685.8,21180.1,13361.4,13122687 +8766.9,17677.2,21180.1,13352.5,13132191 +8772.8,17676.0,21180.1,13351.6,13142733 +8778.6,17672.5,21180.1,13347.5,13153435 +8784.5,17664.2,21180.1,13339.4,13169965 +8790.5,17672.2,21180.1,13347.4,13180743 +8797.5,17673.9,21180.1,13349.4,13194627 +8803.5,17669.4,21180.1,13344.9,13204293 +8808.9,17673.4,21180.1,13348.9,13214381 +8814.3,17669.4,21180.1,13345.1,13223633 +8820.2,17676.7,21180.1,13351.9,13233595 +8825.7,17671.7,21180.1,13347.4,13244611 +8831.4,17670.0,21180.1,13345.8,13256195 +8837.2,17667.3,21180.1,13343.1,13269575 +8843.7,17670.8,21180.1,13346.0,13282663 +8852.2,17676.9,21180.1,13351.9,13293001 +8859.4,17663.9,21180.1,13338.3,13302835 +8865.5,17671.5,21180.1,13346.8,13317807 +8870.4,17681.9,21180.1,13357.1,13340739 +8876.0,17672.8,21180.1,13348.2,13340739 +8881.4,17675.9,21180.1,13351.2,13340739 +8886.4,17680.0,21180.1,13355.5,13340739 +8891.3,17686.7,21180.1,13362.4,13340739 +8896.1,17690.2,21180.1,13366.0,13340739 +8900.7,17681.0,21180.1,13356.0,13340739 +8905.9,17678.6,21180.1,13354.5,13340739 +8911.0,17674.2,21180.1,13349.6,13340739 +8916.5,17678.6,21180.1,13354.2,13340739 +8922.0,17669.8,21180.1,13345.2,13340739 +8926.9,17672.6,21180.1,13347.9,13340739 +8932.2,17671.0,21180.1,13346.6,13340739 +8937.4,17680.8,21180.1,13356.6,13340739 +8941.8,17687.9,21180.1,13363.9,13340739 +8946.6,17690.5,21180.1,13366.2,13348260 +8951.6,17686.8,21180.1,13362.5,13365376 +8956.2,17685.4,21180.1,13360.7,13386614 +8961.0,17682.5,21180.1,13357.4,13409474 +8965.9,17676.4,21180.1,13351.3,13428408 +8970.3,17686.2,21180.1,13361.7,13455876 +8975.0,17686.4,21180.1,13361.8,13473924 +8980.0,17678.1,21180.1,13353.9,13473924 +8984.6,17687.0,21180.1,13363.0,13473924 +8989.0,17707.4,21180.1,13382.5,13473924 +8993.5,17703.1,21180.1,13378.1,13473924 +8998.3,17683.7,21180.1,13359.0,13473924 +9003.1,17677.3,21180.1,13352.4,13473924 +9007.9,17677.5,21180.1,13353.0,13473924 +9012.7,17675.9,21180.1,13351.2,13473924 +9017.4,17675.2,21180.1,13350.2,13473924 +9022.2,17679.6,21180.1,13355.4,13489685 +9026.9,17683.6,21180.1,13359.6,13511439 +9031.7,17691.0,21180.1,13367.0,13534979 +9036.5,17688.0,21180.1,13364.1,13562493 +9041.3,17687.1,21180.1,13362.8,13585375 +9046.2,17680.4,21180.1,13355.8,13608105 +9050.9,17680.8,21180.1,13356.4,13608105 +9055.7,17687.0,21180.1,13362.1,13608105 +9060.4,17683.3,21180.1,13358.3,13608105 +9065.1,17686.4,21180.1,13362.7,13608105 +9069.9,17681.5,21180.1,13356.8,13608105 +9074.7,17684.8,21180.1,13359.9,13608105 +9079.3,17679.1,21180.1,13354.2,13608105 +9083.8,17676.9,21180.1,13351.7,13615375 +9088.6,17677.5,21180.1,13352.7,13630491 +9093.4,17673.8,21180.1,13348.7,13654413 +9098.0,17674.7,21180.1,13349.7,13675651 +9102.9,17670.6,21180.1,13345.8,13691655 +9107.5,17669.5,21180.1,13344.5,13712343 +9112.2,17676.9,21180.1,13352.2,13734873 +9117.2,17680.9,21180.1,13356.2,13754957 +9122.1,17679.2,21180.1,13354.3,13776997 +9127.2,17676.6,21180.1,13351.6,13798629 +9132.0,17671.5,21180.1,13347.8,13798629 +9136.8,17675.0,21180.1,13349.9,13798629 +9141.5,17680.6,21180.1,13355.8,13798629 +9146.4,17678.5,21180.1,13353.4,13798629 +9151.0,17682.4,21180.1,13357.8,13798629 +9155.8,17683.3,21180.1,13358.3,13798629 +9160.4,17670.9,21180.1,13346.2,13798629 +9165.5,17675.4,21180.1,13350.7,13798629 +9170.4,17675.9,21180.1,13351.1,13798737 +9175.0,17678.3,21180.1,13353.2,13818431 +9179.9,17670.9,21180.1,13345.9,13838059 +9184.7,17677.1,21180.1,13352.5,13852571 +9189.8,17678.8,21180.1,13354.2,13873317 +9194.9,17674.4,21180.1,13349.6,13890521 +9199.4,17671.1,21180.1,13346.3,13907911 +9204.2,17676.6,21180.1,13351.6,13907911 +9209.0,17672.2,21180.1,13347.2,13907911 +9213.5,17668.1,21180.1,13343.8,13907911 +9218.2,17671.1,21180.1,13346.7,13907911 +9222.9,17674.7,21180.1,13350.0,13907911 +9227.6,17666.6,21180.1,13342.2,13907911 +9232.0,17671.4,21180.1,13346.9,13907911 +9236.8,17673.5,21180.1,13349.5,13927762 +9241.5,17676.0,21180.1,13351.3,13949698 +9246.3,17673.6,21180.1,13349.3,13969748 +9250.8,17670.6,21180.1,13345.8,13989974 +9255.6,17672.2,21180.1,13348.9,14011582 +9260.4,17673.6,21180.1,13349.8,14032788 +9265.0,17673.6,21180.1,13349.1,14052986 +9269.7,17674.9,21180.1,13349.5,14052986 +9274.2,17676.4,21180.1,13351.7,14052986 +9279.0,17672.0,21180.1,13347.7,14052986 +9283.7,17673.4,21180.1,13348.5,14052986 +9288.5,17674.2,21180.1,13349.7,14052986 +9293.1,17675.4,21180.1,13351.0,14052986 +9297.6,17669.5,21180.1,13344.6,14052986 +9302.2,17670.5,21180.1,13345.7,14072231 +9306.8,17671.4,21180.1,13346.9,14091949 +9311.4,17700.5,21181.1,13376.4,14110945 +9316.1,17696.7,21181.1,13372.1,14135157 +9320.7,17713.3,21188.2,13389.0,14158439 +9325.2,17731.3,21189.0,13406.8,14183461 +9329.9,17730.9,21189.2,13406.9,14205683 +9334.7,17732.8,21189.2,13407.6,14232851 +9339.2,17726.9,21189.2,13402.1,14232851 +9344.0,17733.0,21189.2,13408.7,14232851 +9348.8,17729.1,21190.4,13404.1,14232851 +9353.4,17727.6,21190.4,13402.4,14232851 +9358.1,17731.9,21191.0,13407.0,14232851 +9363.0,17729.1,21191.8,13404.2,14232851 +9367.3,17725.6,21191.8,13401.7,14232851 +9372.0,17728.3,21191.9,13403.0,14232851 +9376.5,17725.4,21191.9,13400.9,14248106 +9380.9,17721.8,21191.9,13397.1,14268144 +9385.6,17725.8,21191.9,13401.3,14281542 +9390.3,17725.5,21191.9,13401.1,14299536 +9395.1,17724.1,21191.9,13399.2,14321272 +9400.1,17726.0,21192.6,13401.5,14340220 +9405.1,17726.4,21192.6,13401.6,14357134 +9410.0,17723.2,21192.6,13399.1,14373644 +9414.6,17726.5,21192.6,13401.4,14390694 +9419.2,17731.2,21192.6,13406.8,14390694 +9424.2,17730.7,21193.8,13405.9,14390694 +9428.8,17727.6,21193.8,13402.8,14390694 +9433.4,17726.4,21193.8,13402.0,14390694 +9438.2,17723.6,21193.8,13398.8,14390694 +9442.9,17725.9,21193.8,13401.3,14390694 +9447.5,17722.9,21193.8,13398.2,14390694 +9451.9,17724.6,21193.8,13400.3,14395154 +9456.7,17727.3,21193.8,13402.6,14419460 +9461.4,17718.6,21193.8,13394.0,14432324 +9466.2,17721.5,21193.8,13397.0,14449928 +9470.9,17725.1,21193.8,13400.4,14466388 +9475.6,17720.6,21193.8,13395.9,14487814 +9480.3,17727.8,21193.8,13403.3,14508204 +9484.8,17728.4,21193.8,13403.8,14525202 +9489.3,17723.5,21193.8,13398.5,14547620 +9494.2,17726.6,21193.8,13401.6,14547620 +9498.9,17726.1,21193.8,13400.9,14547620 +9503.6,17723.2,21193.8,13398.6,14547620 +9508.4,17722.1,21193.8,13398.1,14547620 +9513.0,17718.7,21193.8,13393.8,14547620 +9517.7,17712.0,21193.8,13387.1,14547620 +9522.2,17711.2,21193.8,13386.2,14547620 +9527.0,17713.1,21193.8,13388.5,14547620 +9531.8,17718.1,21193.8,13393.3,14568600 +9536.5,17718.0,21193.8,13393.2,14585142 +9541.3,17719.4,21194.0,13394.8,14603800 +9546.0,17722.8,21194.3,13397.7,14625504 +9550.8,17723.3,21194.3,13398.8,14644548 +9555.4,17724.2,21194.3,13400.1,14662618 +9560.0,17719.3,21194.3,13395.6,14680776 +9564.4,17725.3,21194.3,13400.7,14697442 +9568.9,17719.5,21194.3,13394.6,14719398 +9573.8,17720.3,21194.3,13395.4,14719398 +9578.3,17719.2,21194.3,13394.5,14719398 +9582.9,17718.2,21194.3,13393.6,14719398 +9587.7,17717.7,21194.3,13393.3,14719398 +9592.2,17721.2,21194.3,13396.1,14719398 +9596.8,17718.1,21194.3,13393.1,14719398 +9601.5,17720.1,21194.3,13395.2,14719398 +9606.3,17721.2,21194.3,13396.0,14722633 +9611.1,17722.1,21194.3,13397.5,14741101 +9616.0,17722.0,21194.3,13396.9,14762939 +9620.6,17717.9,21194.3,13393.2,14786481 +9625.3,17717.9,21194.3,13392.7,14808893 +9630.0,17717.6,21194.3,13393.1,14829183 +9634.5,17718.5,21194.3,13394.0,14846125 +9639.2,17720.8,21194.3,13396.4,14867991 +9644.1,17721.4,21194.3,13396.7,14888113 +9649.1,17719.1,21194.3,13394.3,14910073 +9653.8,17719.9,21194.3,13395.5,14910073 +9658.5,17718.7,21194.3,13393.4,14910073 +9663.2,17729.2,21195.5,13404.8,14910073 +9667.7,17723.3,21195.5,13398.5,14910073 +9672.5,17724.6,21195.5,13399.6,14910073 +9677.4,17719.2,21195.5,13394.6,14910073 +9682.1,17723.5,21195.5,13398.8,14910073 +9686.7,17718.2,21195.5,13393.3,14910073 +9691.4,17719.2,21195.5,13394.6,14925005 +9696.1,17719.2,21195.5,13394.6,14948591 +9700.9,17717.1,21195.5,13392.3,14971577 +9705.7,17721.6,21195.5,13396.6,14993355 +9710.7,17715.7,21195.5,13390.9,15015705 +9715.4,17721.1,21195.5,13396.2,15037533 +9720.0,17717.0,21195.5,13392.6,15062117 +9724.5,17720.3,21195.5,13395.3,15082967 +9729.4,17714.7,21195.5,13389.9,15082967 +9734.1,17717.8,21195.5,13393.4,15082967 +9738.8,17714.2,21195.5,13389.3,15082967 +9743.3,17716.7,21195.5,13392.0,15082967 +9748.1,17717.4,21195.5,13392.7,15082967 +9752.8,17718.5,21195.5,13393.9,15082967 +9757.5,17716.5,21195.5,13392.1,15082967 +9762.2,17720.2,21195.5,13395.9,15082967 +9766.7,17716.7,21195.5,13392.2,15086965 +9771.2,17718.8,21195.5,13394.0,15107511 +9775.8,17722.8,21195.7,13398.3,15130693 +9780.5,17721.1,21195.7,13396.2,15145525 +9785.0,17724.8,21195.7,13399.5,15166597 +9789.5,17719.9,21195.7,13395.4,15186903 +9794.0,17722.1,21195.9,13397.0,15207311 +9798.4,17718.5,21196.1,13393.9,15227039 +9802.8,17718.4,21196.1,13394.1,15247109 +9807.6,17718.4,21196.1,13393.2,15247109 +9812.0,17719.7,21196.1,13395.1,15247109 +9816.4,17717.8,21196.1,13393.1,15247109 +9821.0,17712.9,21196.1,13388.2,15247109 +9825.7,17716.7,21196.1,13391.8,15247109 +9830.6,17717.8,21196.1,13393.2,15247109 +9835.0,17718.6,21196.1,13393.7,15247109 +9839.7,17712.6,21196.1,13387.9,15247109 +9844.2,17712.3,21196.1,13387.7,15247811 +9848.9,17712.5,21196.1,13388.2,15272009 +9853.6,17714.4,21196.1,13389.5,15287329 +9858.3,17709.5,21196.1,13384.3,15307139 +9862.9,17715.5,21196.1,13390.8,15328779 +9867.6,17712.6,21196.1,13388.0,15349285 +9872.2,17716.5,21196.1,13392.2,15369107 +9876.7,17719.7,21196.1,13394.6,15389485 +9881.1,17718.9,21196.1,13394.4,15413951 +9885.7,17720.1,21196.1,13395.8,15413951 +9890.3,17714.9,21196.1,13390.5,15413951 +9895.0,17717.4,21196.1,13393.0,15413951 +9899.8,17715.9,21196.1,13391.2,15413951 +9904.5,17715.5,21196.1,13391.1,15413951 +9909.0,17714.1,21196.1,13390.2,15413951 +9913.6,17714.5,21196.1,13389.6,15413951 +9918.3,17713.3,21196.1,13388.7,15413951 +9923.0,17714.8,21196.1,13390.1,15428702 +9927.7,17711.3,21196.1,13386.3,15448682 +9932.3,17716.1,21196.1,13391.6,15470032 +9937.0,17718.5,21196.1,13394.0,15492942 +9941.6,17716.8,21196.1,13391.7,15512364 +9946.3,17712.7,21196.1,13387.7,15530602 +9951.0,17711.2,21196.1,13386.5,15551636 +9955.7,17717.4,21196.1,13392.8,15573468 +9960.5,17721.7,21196.1,13396.7,15598924 +9965.2,17720.1,21196.1,13394.9,15598924 +9969.9,17714.2,21196.1,13389.5,15598924 +9974.6,17716.5,21196.1,13391.5,15598924 +9979.4,17714.9,21196.1,13390.3,15598924 +9984.1,17715.9,21196.1,13391.1,15598924 +9988.6,17711.4,21196.1,13386.9,15598924 +9993.3,17715.7,21196.1,13390.6,15598924 +9998.0,17712.1,21196.1,13387.5,15598924 +10002.5,17713.5,21196.1,13389.0,15598924 +10007.2,17715.9,21196.1,13391.2,15613391 +10011.8,17716.5,21196.1,13391.5,15636765 +10016.5,17715.4,21196.1,13390.4,15652401 +10021.3,17720.0,21196.1,13395.2,15675963 +10025.8,17716.9,21196.1,13392.4,15694815 +10030.5,17716.2,21196.1,13391.7,15718225 +10035.3,17716.0,21196.1,13391.3,15737711 +10039.9,17711.9,21196.1,13386.9,15758341 +10044.5,17713.5,21196.1,13388.7,15758341 +10049.3,17710.1,21196.1,13385.4,15758341 +10054.0,17711.4,21196.1,13387.1,15758341 +10058.8,17707.0,21196.1,13382.4,15758341 +10063.3,17708.5,21196.1,13384.4,15758341 +10068.0,17706.2,21196.1,13382.0,15758341 +10072.8,17708.9,21196.1,13384.1,15758341 +10077.3,17708.1,21196.1,13383.1,15765652 +10081.9,17707.6,21196.1,13382.9,15783626 +10086.5,17705.8,21196.1,13381.1,15799152 +10091.0,17705.0,21196.1,13380.3,15817594 +10095.4,17708.1,21196.1,13383.9,15836806 +10100.2,17709.4,21196.1,13384.6,15855794 +10104.9,17711.1,21196.1,13386.4,15875174 +10109.6,17710.4,21196.1,13385.6,15895682 +10114.2,17710.6,21196.1,13385.6,15914690 +10119.0,17709.1,21196.1,13384.2,15938650 +10123.7,17708.2,21196.1,13384.2,15938650 +10128.5,17706.5,21196.1,13381.7,15938650 +10133.2,17709.6,21196.1,13385.0,15938650 +10137.9,17706.5,21196.1,13382.0,15938650 +10142.7,17707.4,21196.1,13382.7,15938650 +10147.2,17710.5,21196.1,13386.5,15938650 +10152.1,17714.1,21196.1,13389.6,15938650 +10156.8,17712.4,21196.1,13387.5,15938650 +10161.4,17712.7,21196.1,13387.2,15938650 +10166.1,17710.1,21196.1,13385.5,15955023 +10170.6,17708.3,21196.1,13383.6,15977481 +10175.2,17709.9,21196.1,13384.8,15995337 +10179.9,17707.0,21196.1,13383.0,16013551 +10184.9,17709.5,21196.1,13385.1,16035697 +10189.5,17707.1,21196.1,13382.1,16052153 +10194.5,17706.4,21196.1,13381.8,16070183 +10199.0,17709.5,21196.1,13384.6,16088759 +10203.6,17709.1,21196.1,13384.6,16088759 +10208.4,17712.0,21196.1,13387.2,16088759 +10213.0,17711.4,21196.1,13386.6,16088759 +10217.6,17705.9,21196.1,13381.1,16088759 +10222.2,17713.9,21196.1,13389.2,16088759 +10226.9,17711.3,21196.1,13386.4,16088759 +10231.7,17710.8,21196.1,13385.7,16088759 +10236.4,17710.6,21196.1,13385.9,16088759 +10241.1,17713.5,21196.1,13389.1,16101934 +10246.0,17712.4,21196.1,13386.6,16124204 +10250.7,17710.5,21196.1,13385.6,16142782 +10255.3,17713.0,21196.1,13388.1,16162814 +10259.7,17709.5,21196.1,13384.6,16180908 +10264.5,17709.7,21196.1,13385.0,16199104 +10269.5,17710.2,21196.1,13385.8,16220752 +10274.0,17713.7,21196.1,13389.2,16236464 +10278.5,17710.9,21196.1,13386.4,16257876 +10283.1,17713.2,21196.1,13388.9,16257876 +10287.9,17710.7,21196.1,13386.5,16257876 +10292.5,17708.1,21196.1,13383.6,16257876 +10297.1,17706.4,21196.1,13381.3,16257876 +10301.5,17707.1,21196.1,13381.9,16257876 +10306.3,17710.3,21196.1,13385.1,16257876 +10311.1,17710.3,21196.1,13385.5,16257876 +10315.7,17710.4,21196.1,13385.4,16257876 +10320.4,17710.8,21196.1,13385.9,16257876 +10325.1,17710.5,21196.1,13385.7,16271697 +10329.9,17711.1,21196.1,13386.4,16292649 +10334.6,17710.7,21196.1,13386.0,16312577 +10339.3,17707.6,21196.1,13383.1,16330535 +10343.8,18325.5,21196.1,14004.6,16341621 +10348.4,18030.2,21196.1,13705.4,16359353 +10353.2,18032.4,21196.1,13707.8,16378483 +10357.6,18031.6,21196.1,13707.5,16395693 +10362.1,18031.7,21196.1,13706.8,16417509 +10366.4,18029.8,21196.1,13705.0,16436455 +10371.0,18030.6,21196.1,13706.3,16436455 +10375.6,18030.7,21196.1,13705.9,16436455 +10379.9,18030.5,21196.1,13706.0,16436455 +10384.6,18032.8,21196.1,13708.4,16436455 +10389.2,18031.6,21196.1,13707.3,16436455 +10393.9,18031.4,21196.1,13706.7,16436455 +10398.7,18031.1,21196.1,13705.6,16436455 +10403.5,18030.1,21196.1,13705.6,16436455 +10408.2,18033.2,21196.1,13708.3,16446835 +10412.8,18030.9,21196.1,13706.2,16464989 +10417.6,18028.4,21196.1,13703.9,16486259 +10422.4,18031.9,21196.1,13706.9,16507071 +10427.2,18031.7,21196.1,13706.7,16526249 +10431.9,18031.4,21196.1,13706.6,16547857 +10436.5,18031.4,21196.1,13706.9,16564995 +10441.6,18033.1,21196.1,13709.0,16586727 +10446.3,18034.8,21196.1,13710.6,16604419 +10451.0,18035.0,21196.1,13710.3,16624807 +10455.6,18033.8,21196.1,13709.1,16645027 +10460.2,18014.2,21196.1,13689.4,16645027 +10465.0,18015.5,21196.1,13691.1,16645027 +10469.8,18011.1,21196.1,13686.8,16645027 +10474.6,17996.9,21196.1,13672.4,16645027 +10479.4,17981.8,21196.1,13656.9,16645027 +10484.3,17983.1,21196.1,13658.3,16645027 +10488.9,17985.6,21196.1,13660.7,16645027 +10493.7,17985.4,21196.1,13660.6,16645027 +10498.4,17989.5,21196.1,13664.5,16645027 +10503.2,17988.9,21196.1,13664.5,16661402 +10507.8,17988.1,21196.1,13663.9,16678580 +10512.5,17988.9,21196.1,13664.7,16700364 +10517.2,17984.1,21196.1,13659.6,16717452 +10521.9,17985.2,21196.1,13660.7,16737606 +10526.7,17984.5,21196.1,13659.4,16751944 +10531.3,17984.2,21196.1,13659.3,16772258 +10536.0,17986.0,21196.1,13661.7,16787344 +10540.6,17985.0,21196.1,13660.1,16805654 +10545.2,17987.9,21196.1,13663.3,16824658 +10549.9,17988.2,21196.1,13663.2,16824658 +10554.6,17988.2,21196.1,13663.8,16824658 +10559.5,17998.1,21196.1,13674.2,16824658 +10564.2,18012.0,21196.1,13687.0,16824658 +10568.9,18002.9,21196.1,13678.0,16824658 +10573.4,17995.6,21196.1,13671.0,16824658 +10578.0,17987.6,21196.1,13663.0,16824658 +10582.5,17982.9,21196.1,13658.2,16824658 +10587.2,17982.8,21196.1,13658.2,16824658 +10591.8,17984.1,21196.1,13659.9,16824765 +10596.3,17985.3,21196.1,13660.7,16842793 +10600.9,17987.4,21196.1,13662.5,16861287 +10605.7,17988.4,21196.1,13663.7,16883897 +10610.4,17986.4,21196.1,13661.5,16902505 +10615.0,17986.6,21196.1,13662.2,16920771 +10619.6,17986.6,21196.1,13662.2,16938689 +10624.5,17986.3,21196.1,13661.8,16960541 +10629.2,17984.4,21196.1,13659.5,16979719 +10634.0,17985.0,21196.1,13660.2,16999717 +10638.8,17992.3,21196.1,13667.9,16999717 +10643.5,17989.8,21196.1,13665.4,16999717 +10648.2,17987.7,21196.1,13663.3,16999717 +10653.0,18009.6,21197.5,13684.8,16999717 +10657.8,18005.0,21197.8,13680.1,16999717 +10662.3,18000.6,21197.8,13675.8,16999717 +10667.1,17996.4,21197.8,13671.4,16999717 +10671.8,17992.5,21197.8,13667.9,16999717 +10676.5,17993.5,21197.8,13669.3,16999717 +10680.9,17989.7,21197.8,13665.2,17008837 +10685.5,17988.1,21197.8,13663.2,17025359 +10690.2,17983.5,21197.8,13659.0,17046401 +10694.8,17986.5,21197.8,13661.6,17063861 +10699.6,17984.2,21197.8,13659.3,17083481 +10704.5,17982.2,21197.8,13657.5,17098795 +10709.3,17982.1,21197.8,13657.6,17117591 +10713.7,17986.0,21197.8,13661.6,17135505 +10718.4,17986.7,21197.8,13661.8,17157679 +10723.1,17989.7,21197.8,13664.8,17157679 +10727.6,17989.8,21197.8,13665.6,17157679 +10732.3,17990.8,21197.8,13666.3,17157679 +10736.8,17984.5,21197.8,13660.4,17157679 +10741.3,17989.1,21197.8,13664.3,17157679 +10746.1,17988.1,21197.8,13663.3,17157679 +10750.8,17987.4,21197.8,13662.5,17157679 +10755.5,17986.5,21197.8,13661.7,17157679 +10760.2,17987.9,21197.8,13663.3,17157679 +10765.0,17988.6,21197.8,13663.6,17173094 +10769.6,17987.6,21197.8,13663.1,17194226 +10774.2,17989.7,21197.8,13664.8,17211624 +10778.9,17989.7,21197.8,13665.0,17231898 +10783.8,17992.3,21197.8,13668.1,17255026 +10788.4,17988.1,21197.8,13663.4,17269640 +10793.0,17986.2,21197.8,13661.3,17286538 +10797.5,17984.6,21197.8,13659.9,17307774 +10802.0,17989.2,21197.8,13664.3,17326728 +10806.6,17990.1,21197.8,13665.4,17345114 +10811.2,17987.2,21197.8,13662.7,17345114 +10816.0,17986.7,21197.8,13661.9,17345114 +10820.7,17990.0,21197.8,13665.1,17345114 +10825.4,17987.9,21197.8,13663.5,17345114 +10830.1,17985.4,21197.8,13660.7,17345114 +10834.8,17986.5,21197.8,13661.6,17345114 +10839.4,17985.9,21197.8,13661.4,17345114 +10844.2,17985.1,21197.8,13660.1,17345114 +10848.9,17984.1,21197.8,13659.6,17345114 +10853.5,17984.1,21197.8,13659.3,17354860 +10858.2,17985.3,21197.8,13661.0,17375832 +10863.0,17986.4,21197.8,13661.8,17394888 +10867.8,17986.0,21197.8,13661.5,17413426 +10872.4,17984.7,21197.8,13660.4,17433968 +10877.1,17982.6,21197.8,13658.1,17450126 +10881.5,17984.6,21197.8,13660.5,17467256 +10886.5,17985.6,21197.8,13660.9,17489294 +10891.3,17987.9,21197.8,13663.1,17511230 +10895.7,17987.4,21197.8,13662.8,17511230 +10900.3,17986.1,21197.8,13661.6,17511230 +10905.1,17986.7,21197.8,13662.5,17511230 +10909.7,17996.7,21197.8,13672.3,17511230 +10914.4,17997.9,21197.8,13672.7,17511230 +10919.1,17987.7,21197.8,13662.7,17511230 +10923.9,17984.7,21197.8,13659.8,17511230 +10928.4,17983.1,21197.8,13658.0,17511230 +10933.0,17986.2,21197.8,13661.9,17511230 +10937.8,17984.9,21197.8,13660.4,17516728 +10942.5,17983.9,21197.8,13659.5,17537822 +10947.3,17986.1,21197.8,13661.4,17559132 +10952.2,17992.6,21197.8,13668.0,17579942 +10957.0,17985.3,21197.8,13660.2,17600112 +10961.7,17984.2,21197.8,13659.8,17615988 +10966.2,17982.8,21197.8,13658.3,17627894 +10970.8,17983.6,21197.8,13658.9,17645296 +10975.3,17987.2,21197.8,13662.5,17662674 +10980.1,17980.9,21197.8,13656.1,17662674 +10985.0,17984.3,21197.8,13659.9,17662674 +10989.8,17982.5,21197.8,13657.9,17662674 +10994.4,17983.9,21197.8,13659.9,17662674 +10999.0,17984.0,21197.8,13659.0,17662674 +11003.7,17984.5,21197.8,13659.3,17662674 +11008.4,17985.2,21197.8,13660.4,17662674 +11013.2,17986.1,21197.8,13661.3,17662674 +11017.8,17981.4,21197.8,13656.6,17662674 +11022.2,17986.7,21197.8,13662.0,17672228 +11026.8,17987.0,21197.8,13662.5,17693536 +11031.4,17985.0,21197.8,13660.4,17708126 +11036.1,17980.9,21197.8,13655.9,17727126 +11040.8,17982.1,21197.8,13657.3,17745116 +11045.5,17981.2,21197.8,13656.5,17764420 +11050.1,17978.3,21197.8,13654.6,17779242 +11054.7,17979.0,21197.8,13654.4,17798444 +11059.4,17983.1,21197.8,13658.6,17818804 +11064.1,17985.7,21197.8,13660.7,17839304 +11068.7,17980.6,21197.8,13655.9,17839304 +11073.4,17985.3,21197.8,13661.0,17839304 +11078.2,17987.2,21197.8,13662.7,17839304 +11083.0,17986.0,21197.8,13661.4,17839304 +11087.9,17983.5,21197.8,13659.2,17839304 +11092.6,17985.5,21197.8,13660.8,17839304 +11097.4,17983.4,21197.8,13658.9,17839304 +11102.3,17985.9,21197.8,13661.3,17839304 +11107.8,17983.9,21197.8,13659.2,17839304 +11113.7,17979.9,21197.8,13655.3,17840869 +11118.9,17977.9,21197.8,13653.0,17852497 +11124.3,17980.9,21197.8,13656.1,17867255 +11129.8,17982.6,21197.8,13657.3,17885353 +11135.7,17980.4,21197.8,13655.6,17893565 +11141.8,17981.3,21197.8,13656.1,17904781 +11147.4,17975.3,21197.8,13651.0,17916821 +11152.2,17980.1,21197.8,13655.4,17932121 +11157.3,17978.3,21197.8,13653.9,17947535 +11162.7,17980.5,21197.8,13656.2,17960761 +11167.4,17974.9,21197.8,13650.4,17977093 +11172.5,17976.8,21197.8,13652.2,17977093 +11177.8,17978.0,21197.8,13653.5,17977093 +11183.4,17977.4,21197.8,13652.6,17977093 +11188.5,17972.0,21197.8,13647.4,17977093 +11194.2,17972.4,21197.8,13647.3,17977093 +11199.6,17971.2,21197.8,13646.2,17977093 +11204.0,17973.8,21197.8,13649.0,17977093 +11208.7,17977.2,21197.8,13652.4,17977093 +11213.3,17979.1,21197.8,13654.2,17977093 +11218.4,17976.9,21197.8,13652.2,17977093 +11223.1,17975.9,21197.8,13651.1,17981972 +11227.9,17974.9,21197.8,13650.1,18001634 +11233.0,17977.0,21197.8,13652.3,18020002 +11237.9,17975.4,21197.8,13650.8,18034620 +11242.8,17980.9,21197.8,13656.3,18053182 +11247.4,17979.2,21197.8,13654.6,18070802 +11251.8,17980.6,21197.8,13656.2,18091222 +11256.6,17981.0,21197.8,13656.3,18108970 +11261.2,17982.9,21197.8,13658.5,18125888 +11265.7,17980.7,21197.8,13656.5,18125888 +11270.3,17980.3,21197.8,13655.7,18125888 +11274.8,17980.2,21197.8,13656.0,18125888 +11279.7,17982.3,21197.8,13657.4,18125888 +11284.2,17986.4,21197.8,13661.9,18125888 +11288.9,17981.7,21197.8,13657.2,18125888 +11293.5,17978.4,21197.8,13653.8,18125888 +11298.3,17977.1,21197.8,13652.6,18125888 +11303.0,17977.7,21197.8,13652.9,18142915 +11307.7,17974.9,21197.8,13650.3,18161947 +11312.5,17975.4,21197.8,13650.6,18181203 +11317.3,17978.1,21197.8,13653.2,18198047 +11322.2,17975.7,21197.8,13651.0,18220055 +11326.9,17977.7,21197.8,13653.0,18237115 +11331.6,17977.2,21197.8,13651.9,18253919 +11336.2,17978.4,21197.8,13653.4,18270839 +11340.8,17977.8,21197.8,13653.1,18290919 +11345.5,17977.9,21197.8,13652.8,18307371 +11350.2,17978.7,21197.8,13654.1,18327087 +11355.0,17980.2,21197.8,13656.0,18327087 +11359.8,17977.1,21197.8,13652.4,18327087 +11364.6,17974.9,21197.8,13650.3,18327087 +11369.2,17973.5,21197.8,13649.1,18327087 +11373.6,17974.4,21197.8,13650.0,18327087 +11378.3,17979.3,21197.8,13654.8,18327087 +11383.0,17977.9,21197.8,13653.1,18327087 +11387.8,17980.8,21197.8,13656.1,18327087 +11392.6,17982.4,21197.8,13657.6,18327087 +11397.0,17989.8,21197.8,13665.1,18338289 +11401.9,17987.9,21197.8,13663.2,18358625 +11406.6,17982.7,21197.8,13658.3,18377431 +11411.3,17981.9,21197.8,13657.5,18396461 +11416.1,17980.6,21197.8,13655.7,18414025 +11421.0,17980.2,21197.8,13655.6,18430901 +11425.7,17979.5,21197.8,13654.4,18445715 +11430.2,17976.8,21197.8,13652.0,18462449 +11435.3,17978.0,21197.8,13652.7,18478603 +11440.1,17977.7,21197.8,13653.2,18494725 +11445.1,17982.7,21197.8,13658.0,18494725 +11449.9,17980.0,21197.8,13655.3,18494725 +11454.6,17978.6,21197.8,13654.2,18494725 +11459.5,17976.3,21197.8,13651.5,18494725 +11465.5,17975.2,21197.8,13650.4,18494725 +11471.1,17973.8,21197.8,13648.8,18494725 +11475.8,17975.4,21197.8,13650.6,18494725 +11480.8,17975.0,21197.8,13649.9,18494725 +11485.7,17977.8,21197.8,13653.3,18494725 +11490.4,17980.6,21197.8,13655.4,18494725 +11495.0,17979.5,21197.8,13654.9,18507819 +11499.7,17982.6,21197.8,13659.3,18526539 +11504.4,17978.4,21197.8,13653.5,18541679 +11509.1,17975.3,21197.8,13650.6,18562887 +11513.7,17977.6,21197.8,13653.3,18579095 +11518.4,17977.9,21197.8,13653.3,18596337 +11523.0,17977.0,21197.8,13652.2,18611059 +11527.7,17974.9,21197.8,13650.0,18629253 +11532.4,17980.8,21197.8,13655.9,18645483 +11537.3,17977.5,21197.8,13652.2,18661703 +11542.2,17977.9,21197.8,13652.8,18680115 +11547.1,17977.1,21197.8,13651.8,18680115 +11551.8,17977.8,21197.8,13653.0,18680115 +11556.4,17976.5,21197.8,13651.9,18680115 +11560.9,17976.8,21197.8,13652.6,18680115 +11565.5,17982.1,21197.8,13657.7,18680115 +11570.0,18000.9,21197.8,13676.4,18680115 +11574.6,18015.7,21197.8,13691.0,18680115 +11579.0,17998.6,21197.8,13674.1,18680115 +11583.5,17984.1,21197.8,13659.8,18680115 +11588.2,17975.7,21197.8,13651.1,18680115 +11592.9,17978.6,21197.8,13653.9,18680115 +11597.5,17979.8,21197.8,13655.7,18680115 +11602.0,17984.6,21197.8,13660.0,18699045 +11606.5,17984.0,21197.8,13659.6,18717095 +11610.9,17980.7,21197.8,13655.8,18739013 +11615.5,17980.9,21197.8,13655.5,18753201 +11620.3,17977.9,21197.8,13653.1,18772591 +11624.8,17979.4,21197.8,13654.6,18787437 +11629.7,17982.2,21197.8,13657.3,18809957 +11634.2,17996.5,21197.8,13671.7,18827537 +11639.0,17993.8,21197.8,13669.9,18847689 +11643.5,17995.6,21197.8,13670.9,18865157 +11648.1,17997.3,21197.8,13672.9,18865157 +11652.6,17997.0,21197.8,13671.9,18865157 +11657.4,17997.3,21197.8,13671.9,18865157 +11662.0,17997.0,21197.8,13672.0,18865157 +11666.7,17993.6,21197.8,13668.5,18865157 +11671.3,17993.4,21197.8,13668.3,18865157 +11676.2,17993.7,21197.8,13669.2,18865157 +11680.9,17993.1,21197.8,13668.6,18865157 +11685.5,17993.7,21197.8,13669.4,18865157 +11690.1,17994.8,21197.8,13670.5,18865157 +11695.0,17996.7,21197.8,13671.2,18871521 +11699.7,17993.0,21197.8,13669.9,18890463 +11704.2,17995.7,21197.8,13671.1,18906743 +11708.9,17997.5,21197.8,13672.7,18913747 +11713.5,17996.9,21197.8,13672.2,18934733 +11718.2,17995.5,21197.8,13670.3,18950669 +11723.0,17996.2,21197.8,13671.6,18968341 +11727.6,17994.8,21197.8,13670.6,18985199 +11732.7,17994.1,21197.8,13669.5,19005145 +11737.6,17991.1,21197.8,13665.9,19021527 +11742.5,17992.5,21197.8,13667.5,19039107 +11747.0,17992.7,21197.8,13667.7,19057605 +11751.8,17994.8,21197.8,13669.8,19057605 +11756.3,17994.4,21197.8,13669.8,19057605 +11761.0,17994.0,21197.8,13669.1,19057605 +11765.7,17991.8,21197.8,13667.5,19057605 +11770.4,17991.8,21197.8,13667.3,19057605 +11775.2,17993.7,21197.8,13668.7,19057605 +11779.8,17991.1,21197.8,13666.8,19057605 +11784.6,17993.4,21197.8,13669.3,19057605 +11789.2,17993.0,21197.8,13668.4,19057605 +11793.6,17992.3,21197.8,13667.6,19057605 +11798.3,17994.4,21197.8,13669.7,19076111 +11803.0,17994.2,21197.8,13669.6,19091937 +11807.8,17994.1,21197.8,13669.5,19112341 +11812.5,17992.5,21197.8,13667.9,19128535 +11817.0,17994.3,21197.8,13669.4,19145099 +11821.9,17991.6,21197.8,13666.7,19162421 +11826.5,17991.3,21197.8,13666.5,19180257 +11831.2,17990.5,21197.8,13666.1,19200227 +11835.7,17990.6,21197.8,13666.0,19215827 +11840.2,17991.8,21197.8,13667.3,19232099 +11845.4,17992.0,21197.8,13667.2,19252697 +11850.1,17995.0,21197.8,13670.8,19252697 +11854.6,17996.3,21197.8,13671.4,19252697 +11859.2,17998.8,21197.8,13674.1,19252697 +11863.9,18005.0,21197.8,13680.2,19252697 +11868.4,18008.3,21197.8,13683.3,19252697 +11873.2,18000.2,21197.8,13675.5,19252697 +11877.8,17994.5,21197.8,13669.7,19252697 +11882.5,17996.7,21197.8,13672.7,19252697 +11887.0,17996.1,21197.8,13671.5,19252697 +11891.8,17996.7,21197.8,13672.9,19252697 +11896.9,17997.3,21197.8,13672.8,19252697 +11902.8,17994.5,21197.8,13670.0,19252697 +11908.2,17991.2,21197.8,13667.0,19253721 +11913.1,17988.0,21197.8,13663.2,19270083 +11917.7,17990.9,21197.8,13666.4,19286377 +11922.5,17991.5,21197.8,13667.0,19301475 +11926.9,17990.1,21197.8,13665.7,19320423 +11931.9,17993.1,21197.8,13668.4,19334951 +11936.8,17990.3,21197.8,13665.3,19351167 +11941.4,17990.5,21197.8,13665.9,19368293 +11945.9,17993.7,21197.8,13669.3,19386139 +11950.5,17993.1,21197.8,13668.0,19403527 +11955.4,17993.6,21197.8,13668.6,19403527 +11960.1,17995.1,21197.8,13670.5,19403527 +11964.8,17995.4,21197.8,13670.7,19403527 +11969.6,17996.3,21197.8,13671.9,19403527 +11974.2,17996.6,21197.8,13671.6,19403527 +11978.6,17995.6,21197.8,13670.3,19403527 +11983.0,17996.5,21197.8,13671.3,19403527 +11987.7,17993.1,21197.8,13668.3,19403527 +11992.5,17996.6,21197.8,13671.1,19403527 +11996.9,17990.8,21197.8,13665.8,19403527 +12001.6,17993.4,21197.8,13668.6,19421342 +12006.2,17992.1,21197.8,13667.1,19438598 +12010.9,17988.8,21197.8,13664.1,19450728 +12015.7,17990.7,21197.8,13665.8,19465134 +12020.4,17992.2,21197.8,13667.7,19483812 +12025.1,18040.6,21201.9,13716.4,19502228 +12030.1,18081.6,21215.5,13756.1,19532530 +12035.2,18105.7,21227.8,13781.2,19581384 +12039.7,18076.2,21228.3,13751.5,19606972 +12044.6,18071.7,21228.3,13747.6,19630348 +12049.4,18065.5,21228.3,13741.2,19665992 +12054.2,18057.5,21228.3,13732.8,19699552 +12058.8,18067.7,21228.3,13743.2,19699552 +12063.7,18061.3,21228.3,13736.3,19699552 +12068.5,18050.8,21228.3,13726.2,19699552 +12073.0,18058.1,21228.3,13733.6,19699552 +12077.7,18056.2,21228.3,13731.4,19699552 +12082.3,18052.5,21228.3,13728.2,19699552 +12087.1,18056.4,21228.3,13732.2,19699552 +12091.7,18055.8,21228.3,13731.1,19712411 +12096.5,18051.1,21228.3,13725.9,19746567 +12101.2,18047.5,21228.3,13722.6,19776973 +12105.8,18041.8,21228.3,13717.3,19805545 +12110.6,18048.6,21228.3,13723.4,19843173 +12116.2,18043.9,21228.3,13718.8,19865935 +12121.6,18040.9,21228.3,13716.4,19881821 +12126.6,18032.7,21228.3,13708.4,19909457 +12131.3,18045.4,21228.3,13721.2,19935561 +12135.7,18050.0,21228.3,13725.2,19962465 +12140.4,18046.6,21228.3,13722.2,19986185 +12145.0,18043.0,21228.3,13718.1,20010437 +12149.6,18040.6,21228.3,13716.3,20035631 +12154.0,18032.5,21228.3,13707.7,20058645 +12158.9,18033.4,21228.3,13708.5,20092267 +12163.5,18040.1,21228.3,13715.3,20113551 +12168.2,18041.6,21228.3,13717.3,20113551 +12172.9,18045.8,21228.3,13721.1,20113551 +12177.7,18044.6,21228.3,13719.8,20113551 +12182.4,18032.5,21228.3,13708.4,20113551 +12187.0,18037.4,21228.3,13713.2,20113551 +12191.7,18040.3,21228.3,13715.5,20113551 +12196.3,18047.6,21228.3,13723.1,20113551 +12200.6,18034.7,21228.3,13709.9,20113551 +12205.3,18031.7,21228.3,13707.5,20113551 +12209.8,18030.8,21228.3,13706.0,20113551 +12214.5,18025.3,21228.3,13700.7,20113551 +12219.1,18020.5,21228.3,13696.1,20113551 +12223.9,18036.5,21228.3,13711.4,20118553 +12228.8,18046.0,21228.3,13721.4,20146689 +12233.3,18044.6,21228.3,13720.0,20169321 +12238.0,18027.3,21228.3,13702.4,20193865 +12242.7,18025.8,21228.3,13701.3,20221041 +12247.4,18027.0,21228.3,13702.6,20240691 +12252.1,18026.1,21228.3,13701.1,20270155 +12257.1,18016.8,21228.3,13692.1,20294003 +12261.5,18010.6,21228.3,13686.2,20313075 +12266.5,18018.0,21228.3,13693.5,20338215 +12271.3,18015.2,21228.3,13690.5,20365595 +12276.0,18019.3,21228.3,13695.0,20390329 +12280.9,18025.9,21228.3,13701.2,20418341 +12285.7,18027.6,21228.3,13703.0,20442989 +12290.6,18018.4,21228.3,13693.4,20468369 +12295.2,18019.4,21228.3,13694.6,20491115 +12299.7,18015.7,21228.3,13690.9,20491115 +12304.3,18023.4,21228.3,13698.2,20491115 +12309.2,18023.0,21228.3,13698.2,20491115 +12313.8,18020.2,21228.3,13696.1,20491115 +12318.4,18017.0,21228.3,13692.5,20491115 +12323.2,18016.8,21228.3,13692.1,20491115 +12327.7,18029.4,21228.3,13704.8,20491115 +12332.5,18027.1,21228.3,13702.7,20491115 +12338.3,18020.1,21228.3,13695.5,20491115 +12344.2,18016.9,21228.3,13692.1,20491115 +12349.1,18019.6,21228.3,13695.2,20491115 +12353.9,18013.5,21228.3,13688.4,20491115 +12358.6,18010.2,21228.3,13685.0,20491115 +12363.5,18028.1,21228.3,13703.4,20491115 +12368.3,18023.0,21228.3,13698.5,20491115 +12372.9,18014.6,21228.3,13690.0,20491115 +12377.6,18020.8,21228.3,13696.4,20506337 +12382.4,18012.4,21228.3,13687.6,20528035 +12386.9,18002.3,21228.3,13677.1,20547657 +12391.8,18004.8,21228.3,13679.9,20571785 +12396.4,18004.7,21228.3,13680.2,20586037 +12401.1,18017.8,21228.3,13692.8,20613393 +12405.8,18018.1,21228.3,13692.9,20638129 +12410.5,18015.3,21228.3,13690.2,20662903 +12415.3,18013.5,21228.3,13688.8,20684093 +12419.8,18006.6,21228.3,13682.0,20702833 +12424.6,18007.2,21228.3,13682.8,20722787 +12429.2,18011.7,21228.3,13687.1,20747669 +12433.9,18007.3,21228.3,13682.4,20747669 +12438.6,18001.8,21228.3,13676.5,20747669 +12443.2,18008.3,21228.3,13683.5,20747669 +12447.8,18007.9,21228.3,13683.9,20747669 +12452.3,18009.3,21228.3,13685.5,20747669 +12456.7,18004.8,21228.3,13679.7,20747669 +12461.4,18011.3,21228.3,13686.9,20747669 +12466.3,18010.0,21228.3,13685.5,20747669 +12471.0,18010.0,21228.3,13685.3,20747669 +12475.5,18003.7,21228.3,13679.3,20747669 +12480.2,17998.7,21228.3,13673.9,20747669 +12484.9,18005.1,21228.3,13680.1,20747669 +12489.7,18002.2,21228.3,13677.5,20747669 +12494.4,18002.5,21228.3,13677.8,20747669 +12499.1,18006.7,21228.3,13682.1,20760252 +12503.9,18007.0,21228.3,13682.5,20780852 +12508.7,18003.6,21228.3,13678.7,20806278 +12513.2,18004.5,21228.3,13679.4,20825194 +12517.9,17999.5,21228.3,13675.0,20849338 +12522.9,18000.4,21228.3,13675.4,20870758 +12528.0,17998.5,21228.3,13674.1,20890724 +12532.5,17996.6,21228.3,13672.0,20909314 +12537.3,17992.7,21228.3,13667.7,20924298 +12542.4,17992.5,21228.3,13668.0,20941168 +12547.2,17991.3,21228.3,13666.2,20958228 +12552.1,17995.0,21228.3,13670.5,20976314 +12556.7,17999.2,21228.3,13674.9,20976314 +12561.3,17996.9,21228.3,13672.4,20976314 +12566.0,18000.9,21228.3,13676.7,20976314 +12570.5,18006.4,21228.3,13681.8,20976314 +12575.1,18006.0,21228.3,13681.4,20976314 +12579.7,18006.5,21228.3,13682.4,20976314 +12584.4,18000.9,21228.3,13676.5,20976314 +12589.2,18000.4,21228.3,13675.8,20976314 +12593.7,18002.1,21228.3,13676.9,20976314 +12598.3,17997.6,21228.3,13673.0,20976314 +12603.1,17998.7,21228.3,13673.7,20976314 +12607.6,17995.6,21228.3,13670.8,20976314 +12612.4,17996.1,21228.3,13671.3,20996918 +12617.0,17996.1,21228.3,13671.2,21013662 +12621.5,18000.7,21228.3,13675.8,21033974 +12626.2,17997.2,21228.3,13673.0,21044850 +12630.7,18001.4,21228.3,13677.1,21067558 +12635.4,17998.6,21228.3,13673.9,21088642 +12640.3,17994.7,21228.3,13670.0,21109796 +12645.0,17996.1,21228.3,13671.3,21125258 +12649.9,17999.8,21228.3,13675.0,21146710 +12654.6,18000.2,21228.3,13675.9,21163636 +12659.1,18000.2,21228.3,13675.5,21185690 +12663.9,18000.0,21228.3,13675.8,21202784 +12668.8,17997.4,21228.3,13672.9,21225728 +12673.6,18002.5,21228.3,13677.8,21243686 +12678.0,17998.0,21228.3,13673.2,21243686 +12682.7,17998.9,21228.3,13673.8,21243686 +12687.5,17991.9,21228.3,13667.1,21243686 +12692.0,17993.5,21228.3,13669.3,21243686 +12696.8,17994.4,21228.3,13669.8,21243686 +12701.5,17994.1,21228.3,13669.3,21243686 +12706.0,17998.6,21228.3,13674.1,21243686 +12710.7,17997.4,21228.3,13672.8,21243686 +12715.1,17993.3,21228.3,13668.5,21243686 +12719.9,17997.9,21228.3,13673.1,21243686 +12724.5,17997.8,21228.3,13673.0,21243686 +12729.2,17994.8,21228.3,13670.9,21243686 +12734.1,17992.8,21228.3,13668.5,21244592 +12738.7,17990.2,21228.3,13666.0,21264116 +12743.4,17992.1,21228.3,13667.2,21286138 +12748.2,17990.9,21228.3,13665.9,21303476 +12752.9,17991.3,21228.3,13666.2,21324786 +12757.5,17991.1,21228.3,13665.9,21339250 +12762.2,17992.4,21228.3,13667.5,21358484 +12766.6,17992.3,21228.3,13667.6,21373598 +12771.2,17993.8,21228.3,13669.0,21389890 +12776.1,17992.3,21228.3,13667.5,21406112 +12781.0,17989.3,21228.3,13664.4,21422642 +12785.9,17991.4,21228.3,13666.2,21442992 +12790.6,17992.1,21228.3,13667.3,21442992 +12795.3,17992.4,21228.3,13668.0,21442992 +12800.0,17993.2,21228.3,13668.4,21442992 +12804.7,17990.0,21228.3,13665.2,21442992 +12809.5,17988.8,21228.3,13663.7,21442992 +12814.0,17993.2,21228.3,13668.5,21442992 +12818.7,18001.1,21228.3,13677.3,21442992 +12823.4,18007.6,21228.3,13682.8,21442992 +12828.1,18000.0,21228.3,13675.2,21442992 +12832.7,17996.1,21228.3,13671.4,21442992 +12837.3,17989.9,21228.3,13665.3,21442992 +12842.0,17988.3,21228.3,13663.9,21442992 +12846.7,17988.8,21228.3,13664.8,21444684 +12851.5,17990.8,21228.3,13665.9,21465944 +12856.2,17989.8,21228.3,13665.0,21486512 +12860.7,17990.4,21228.3,13665.6,21507260 +12865.5,17992.4,21228.3,13667.5,21528052 +12870.0,17993.6,21228.3,13669.2,21547914 +12874.7,17997.2,21228.3,13672.7,21567628 +12879.5,17996.0,21228.3,13671.5,21583146 +12884.3,17993.6,21228.3,13669.0,21603678 +12888.8,17995.9,21228.3,13671.0,21623772 +12893.4,17994.7,21228.3,13669.6,21644244 +12898.2,17992.1,21228.3,13667.2,21644244 +12903.0,17992.1,21228.3,13667.3,21644244 +12907.5,17988.1,21228.3,13662.9,21644244 +12912.2,17987.7,21228.3,13662.8,21644244 +12916.9,17986.6,21228.3,13662.4,21644244 +12921.5,17987.9,21228.3,13663.4,21644244 +12926.2,17987.5,21228.3,13663.4,21644244 +12931.0,17989.8,21228.3,13665.5,21644244 +12935.4,17993.1,21228.3,13668.7,21644244 +12940.2,17990.4,21228.3,13665.6,21644244 +12944.9,17991.3,21228.3,13666.6,21644244 +12949.6,17991.8,21228.3,13667.6,21660255 +12954.5,17989.9,21228.3,13665.4,21678399 +12959.1,17990.1,21228.3,13665.0,21697913 +12963.7,17987.5,21228.3,13662.5,21717523 +12968.5,17987.4,21228.3,13663.2,21741275 +12973.1,17989.5,21228.3,13664.7,21757961 +12977.9,17986.8,21228.3,13662.7,21776641 +12982.6,17985.5,21228.3,13661.4,21792203 +12987.3,17984.6,21228.3,13660.4,21809475 +12992.0,17985.0,21228.3,13660.5,21829759 +12996.6,17987.9,21228.3,13663.2,21848943 +13001.1,17987.9,21228.3,13663.2,21864559 +13005.6,17988.1,21228.3,13664.1,21884135 +13010.5,17990.0,21228.3,13665.6,21884135 +13015.0,17993.1,21228.3,13668.6,21884135 +13019.7,17993.6,21228.3,13668.5,21884135 +13024.4,17989.3,21228.3,13664.4,21884135 +13029.1,17978.1,21228.3,13653.0,21884135 +13033.9,17975.6,21228.3,13650.7,21884135 +13038.6,17973.4,21228.3,13648.9,21884135 +13043.3,17978.6,21228.3,13654.2,21884135 +13047.8,17975.2,21228.3,13650.2,21884135 +13052.7,17975.2,21228.3,13650.4,21884135 +13057.5,17979.7,21228.3,13655.3,21884135 +13062.2,17976.5,21228.3,13651.8,21884135 +13066.6,17976.2,21228.3,13651.2,21884135 +13071.1,17973.9,21228.3,13648.5,21903240 +13075.9,17973.6,21228.3,13648.5,21924092 +13080.4,17973.6,21228.3,13649.1,21944722 +13084.9,17975.1,21228.3,13650.6,21957456 +13089.7,17975.6,21228.3,13650.8,21976546 +13094.5,17983.5,21228.3,13659.3,21997146 +13099.1,17978.0,21228.3,13653.5,22015182 +13103.6,17976.2,21228.3,13651.7,22033240 +13108.5,17973.0,21228.3,13648.2,22048264 +13113.3,17972.5,21228.3,13647.9,22066948 +13118.1,17969.9,21228.3,13645.2,22081522 +13123.0,17967.5,21228.3,13642.9,22099112 +13127.7,17968.4,21228.3,13644.2,22099112 +13132.4,17975.4,21228.3,13650.6,22099112 +13137.0,17979.3,21228.3,13654.6,22099112 +13141.4,17977.7,21228.3,13653.5,22099112 +13146.3,17981.7,21228.3,13657.5,22099112 +13151.0,17978.7,21228.3,13654.2,22099112 +13155.7,17980.3,21228.3,13656.1,22099112 +13160.3,17973.7,21228.3,13649.6,22099112 +13165.0,17980.9,21228.3,13656.5,22099112 +13169.8,17979.6,21228.3,13654.6,22099112 +13174.6,17977.7,21228.3,13653.1,22099112 +13179.3,17977.5,21228.3,13652.7,22099112 +13184.0,17972.6,21228.3,13647.7,22118620 +13188.7,17972.4,21228.3,13647.3,22136454 +13193.5,17972.7,21228.3,13648.2,22156304 +13198.1,17972.1,21228.3,13647.6,22170890 +13202.9,17973.2,21228.3,13648.2,22189210 +13207.6,17970.8,21228.3,13645.8,22205486 +13212.3,17969.9,21228.3,13645.2,22226876 +13217.2,17969.6,21228.3,13644.7,22239830 +13222.4,17969.0,21228.3,13645.0,22259632 +13226.8,17971.7,21228.3,13647.5,22274020 +13231.4,17974.3,21228.3,13649.4,22291642 +13236.3,17970.8,21228.3,13646.1,22309956 +13240.9,17970.5,21228.3,13646.0,22324878 +13245.5,17972.9,21228.3,13648.5,22324878 +13250.1,17972.5,21228.3,13647.8,22324878 +13254.8,17970.3,21228.3,13645.7,22324878 +13259.2,17969.6,21228.3,13644.9,22324878 +13263.7,17966.7,21228.3,13641.8,22324878 +13268.4,17969.4,21228.3,13644.4,22324878 +13273.1,17971.2,21228.3,13646.8,22324878 +13277.9,17969.6,21228.3,13645.0,22324878 +13282.6,17975.3,21228.3,13650.6,22324878 +13287.4,17971.8,21228.3,13647.0,22324878 +13291.9,17973.0,21228.3,13647.9,22324878 +13296.5,17971.6,21228.3,13646.8,22324878 +13301.3,17967.4,21228.3,13643.3,22343634 +13305.8,17969.3,21228.3,13644.8,22360434 +13310.5,17968.3,21228.3,13643.7,22381620 +13315.3,17968.4,21228.3,13644.1,22397234 +13320.0,17967.6,21228.3,13643.7,22417396 +13324.8,17971.1,21228.3,13646.6,22435078 +13329.6,17968.9,21228.3,13643.9,22454604 +13334.1,17969.7,21228.3,13644.9,22469286 +13338.6,17966.0,21228.3,13640.9,22482166 +13343.2,17963.7,21228.3,13639.1,22497506 +13347.7,17966.2,21228.3,13642.0,22514590 +13352.5,17967.1,21228.3,13642.3,22530308 +13357.1,17965.5,21228.3,13641.2,22545418 +13361.7,17966.6,21228.3,13641.9,22545418 +13366.2,17964.7,21228.3,13640.3,22545418 +13370.8,17966.5,21228.3,13641.4,22545418 +13375.4,17968.1,21228.3,13644.1,22545418 +13380.1,17967.2,21228.3,13642.6,22545418 +13384.7,17969.9,21228.3,13644.8,22545418 +13389.4,17968.7,21228.3,13643.7,22545418 +13394.1,17969.9,21228.3,13645.0,22545418 +13398.9,17968.7,21228.3,13643.5,22545418 +13403.6,17969.0,21228.3,13643.9,22545418 +13408.4,17964.7,21228.3,13639.7,22545418 +13413.1,17967.3,21228.3,13642.7,22545418 +13417.7,17967.6,21228.3,13642.9,22545418 +13422.3,17968.6,21228.3,13644.2,22561009 +13427.1,17964.6,21228.3,13640.0,22575595 +13431.7,17963.4,21228.3,13638.9,22593791 +13436.4,17969.5,21228.3,13644.8,22610343 +13441.0,17970.5,21228.3,13645.9,22627871 +13445.8,17969.9,21228.3,13645.1,22644581 +13450.3,17975.8,21228.3,13651.3,22663321 +13454.9,17975.2,21228.3,13650.5,22679785 +13459.8,17974.0,21228.3,13649.5,22696259 +13464.4,17972.8,21228.3,13648.4,22712663 +13469.3,17967.2,21228.3,13642.3,22733049 +13473.9,17969.2,21228.3,13644.4,22749377 +13478.6,17973.8,21228.3,13649.1,22749377 +13483.4,17970.2,21228.3,13645.6,22749377 +13488.1,17969.0,21228.3,13644.5,22749377 +13492.8,17965.5,21228.3,13641.1,22749377 +13497.4,17966.9,21228.3,13642.5,22749377 +13502.2,17966.5,21228.3,13641.6,22749377 +13506.9,17967.0,21228.3,13642.4,22749377 +13511.5,17966.5,21228.3,13642.2,22749377 +13516.0,17966.2,21228.3,13641.7,22749377 +13520.7,17964.0,21228.3,13639.3,22749377 +13525.5,17963.7,21228.3,13638.7,22749377 +13530.3,17960.0,21228.3,13635.2,22754884 +13535.0,17963.0,21228.3,13638.2,22770256 +13539.7,17961.3,21228.3,13636.1,22787134 +13544.4,17963.5,21228.3,13638.2,22802754 +13548.9,17965.0,21228.3,13640.2,22819150 +13553.6,17960.9,21228.3,13636.1,22838198 +13558.4,17968.9,21228.3,13644.1,22856008 +13563.3,17968.0,21228.3,13643.6,22870784 +13567.9,17967.2,21228.3,13642.9,22886196 +13572.7,17966.1,21228.3,13641.6,22901580 +13577.2,17967.8,21228.3,13643.0,22917674 +13581.9,17966.2,21228.3,13641.6,22932218 +13586.6,17963.7,21228.3,13639.2,22948722 +13591.1,17967.5,21228.3,13643.3,22948722 +13595.6,17964.0,21228.3,13639.6,22948722 +13600.2,17965.8,21228.3,13642.0,22948722 +13604.9,17968.6,21228.3,13645.6,22948722 +13609.7,17989.2,21228.3,13665.5,22948722 +13614.5,17983.0,21228.3,13658.2,22948722 +13619.3,17977.0,21228.3,13651.9,22948722 +13624.0,17968.9,21228.3,13644.1,22948722 +13628.5,17963.8,21228.3,13639.5,22948722 +13633.0,17963.0,21228.3,13638.2,22948722 +13637.6,17964.4,21228.3,13639.6,22948722 +13642.2,17965.5,21228.3,13641.4,22948722 +13646.9,17965.1,21228.3,13640.6,22948722 +13651.6,17967.0,21228.3,13642.7,22948722 +13656.2,17968.4,21228.3,13644.0,22960153 +13660.9,17966.8,21228.3,13641.9,22977225 +13665.5,17970.3,21228.3,13645.5,22996097 +13670.0,17963.3,21228.3,13638.2,23011185 +13674.8,17969.0,21228.3,13644.7,23030159 +13679.4,17965.1,21228.3,13640.6,23044649 +13684.1,17965.4,21228.3,13640.6,23062709 +13688.9,17965.5,21228.3,13640.4,23079151 +13693.7,17965.5,21228.3,13641.5,23095381 +13698.4,17969.7,21228.3,13645.2,23109939 +13703.1,17970.1,21228.3,13645.2,23123819 +13707.7,17968.4,21228.3,13643.5,23123819 +13712.3,17967.7,21228.3,13643.4,23123819 +13716.8,17968.3,21228.3,13643.6,23123819 +13721.6,17968.4,21228.3,13644.1,23123819 +13726.4,17967.3,21228.3,13642.4,23123819 +13730.9,17961.5,21228.3,13636.7,23123819 +13735.5,17966.9,21228.3,13642.7,23123819 +13740.3,17964.1,21228.3,13639.6,23123819 +13745.0,17966.0,21228.3,13641.4,23123819 +13749.8,17965.7,21228.3,13641.5,23123819 +13754.6,17966.3,21228.3,13641.6,23123819 +13759.2,17972.1,21228.3,13647.4,23123819 +13763.6,17965.4,21228.3,13641.2,23127713 +13768.2,17964.5,21228.3,13639.8,23144497 +13773.0,17965.5,21228.3,13640.6,23162409 +13777.7,17961.5,21228.3,13636.5,23178511 +13782.2,17963.1,21228.3,13638.5,23192197 +13786.8,17964.6,21228.3,13639.3,23207853 +13791.3,17964.2,21228.3,13639.2,23223851 +13795.7,17964.1,21228.3,13640.0,23237341 +13800.6,17963.6,21228.3,13639.2,23253509 +13805.3,17962.1,21228.3,13637.3,23268765 +13810.2,17960.4,21228.3,13635.8,23286773 +13815.3,17963.1,21228.3,13638.4,23304067 +13820.1,17959.7,21228.3,13635.3,23316035 +13824.8,17962.0,21228.3,13637.5,23316035 +13829.7,17966.6,21228.3,13642.4,23316035 +13834.3,17964.5,21228.3,13640.9,23316035 +13839.1,17961.0,21228.3,13636.7,23316035 +13843.6,17964.2,21228.3,13640.2,23316035 +13848.4,17973.4,21228.3,13648.6,23316035 +13852.9,17969.1,21228.3,13644.3,23316035 +13857.6,17959.8,21228.3,13634.6,23316035 +13862.1,17961.2,21228.3,13636.1,23316035 +13866.8,17961.9,21228.3,13636.9,23316035 +13871.6,17960.6,21228.3,13636.0,23316035 +13876.1,17959.8,21228.3,13635.2,23316035 +13880.4,17960.7,21228.3,13636.1,23316035 +13885.2,17960.9,21228.3,13636.5,23317113 +13889.9,17960.7,21228.3,13635.8,23332715 +13894.4,17959.6,21228.3,13634.7,23347895 +13899.1,17961.5,21228.3,13636.8,23366691 +13903.8,17964.3,21228.3,13639.4,23386295 +13908.4,17964.0,21228.3,13639.1,23400571 +13912.8,17963.5,21228.3,13638.4,23416957 +13917.7,17961.8,21228.3,13637.2,23432709 +13922.5,17961.3,21228.3,13637.9,23448687 +13927.1,17963.1,21228.3,13638.3,23464563 +13932.1,17963.5,21228.3,13638.7,23479645 +13936.8,17962.9,21228.3,13637.9,23495659 +13941.5,17962.4,21228.3,13638.0,23495659 +13946.0,17963.1,21228.3,13638.8,23495659 +13950.4,17960.7,21228.3,13635.7,23495659 +13955.2,17961.6,21228.3,13637.0,23495659 +13959.8,17962.6,21228.3,13637.9,23495659 +13964.6,17965.2,21228.3,13640.7,23495659 +13969.4,17966.9,21228.3,13641.8,23495659 +13974.2,17963.8,21228.3,13638.9,23495659 +13978.9,17964.0,21228.3,13639.0,23495659 +13983.5,17961.5,21228.3,13636.9,23495659 +13988.0,17963.5,21228.3,13639.0,23495659 +13992.7,17965.0,21228.3,13640.4,23503388 +13997.5,17959.8,21228.3,13635.2,23524952 +14002.2,17959.8,21228.3,13635.3,23536638 +14006.8,17959.9,21228.3,13635.4,23554776 +14011.6,17958.3,21228.3,13634.2,23572682 +14016.0,17959.9,21228.3,13635.1,23587892 +14020.7,17959.9,21228.3,13635.8,23603100 +14025.9,17960.7,21228.3,13635.7,23621854 +14030.3,17961.7,21228.3,13637.1,23638028 +14034.8,17959.4,21228.3,13634.4,23652192 +14039.4,17958.4,21228.3,13633.9,23666924 +14044.2,17959.5,21228.3,13635.1,23683612 +14049.0,17959.4,21228.3,13634.9,23700196 +14053.8,17960.4,21228.3,13635.5,23717858 +14058.6,17960.3,21228.3,13636.1,23717858 +14063.1,17959.4,21228.3,13634.6,23717858 +14067.7,17958.9,21228.3,13634.7,23717858 +14072.4,17964.2,21228.3,13640.0,23717858 +14077.0,17960.2,21228.3,13636.8,23717858 +14081.5,17960.7,21228.3,13636.3,23717858 +14085.9,17958.5,21228.3,13634.3,23717858 +14090.6,17962.6,21228.3,13638.3,23717858 +14094.9,17962.0,21228.3,13637.7,23717858 +14099.3,17960.9,21228.3,13636.3,23717858 +14104.0,17959.9,21228.3,13635.5,23717858 +14108.7,17960.6,21228.3,13636.1,23717858 +14113.5,17959.4,21228.3,13634.6,23723960 +14118.1,17962.1,21228.3,13637.4,23739794 +14122.8,17962.1,21228.3,13637.2,23755454 +14127.2,17960.4,21228.3,13635.2,23773048 +14131.7,17959.6,21228.3,13634.6,23787452 +14136.4,17960.9,21228.3,13636.7,23805710 +14140.8,17960.9,21228.3,13636.8,23820692 +14145.6,17961.9,21228.3,13637.2,23838450 +14150.4,17956.1,21228.3,13631.5,23853572 +14155.0,17960.3,21228.3,13635.2,23869180 +14159.9,17958.3,21228.3,13634.0,23883164 +14164.7,17956.4,21228.3,13631.5,23899004 +14169.3,17957.2,21228.3,13632.5,23899004 +14174.1,17956.6,21228.3,13632.1,23899004 +14178.7,17959.6,21228.3,13634.9,23899004 +14183.4,17952.8,21228.3,13628.4,23899004 +14187.9,17953.8,21228.3,13629.0,23899004 +14192.5,17965.4,21228.3,13640.6,23899004 +14197.0,17962.0,21228.3,13637.3,23899004 +14201.7,17960.2,21228.3,13634.9,23899004 +14206.3,17967.2,21228.3,13642.2,23899004 +14211.1,17968.3,21228.3,13643.4,23899004 +14215.8,17973.2,21228.3,13648.6,23899004 +14220.5,17975.9,21228.3,13651.2,23899004 +14225.2,17973.7,21228.3,13649.1,23899004 +14229.6,17972.2,21228.3,13647.6,23899004 +14234.2,17973.2,21228.3,13648.4,23905438 +14238.9,17972.0,21228.3,13647.0,23922434 +14243.6,17973.8,21228.3,13649.2,23941072 +14248.4,17971.6,21228.3,13646.8,23952980 +14253.0,17970.5,21228.3,13645.7,23970830 +14257.8,17967.5,21228.3,13642.9,23987090 +14262.7,17968.8,21228.3,13644.0,24000328 +14267.6,17967.7,21228.3,13643.3,24017582 +14272.5,17967.5,21228.3,13642.8,24033020 +14277.3,17970.6,21228.3,13645.8,24048550 +14282.2,17969.5,21228.3,13644.9,24063936 +14286.9,17970.5,21228.3,13645.4,24063936 +14291.6,17968.5,21228.3,13643.7,24063936 +14296.4,17968.6,21228.3,13643.9,24063936 +14300.9,17967.5,21228.3,13642.7,24063936 +14305.6,17967.2,21228.3,13642.4,24063936 +14310.2,17969.7,21228.3,13644.8,24063936 +14314.8,17970.3,21228.3,13645.7,24063936 +14319.5,17974.7,21228.3,13649.6,24063936 +14324.0,17974.9,21228.3,13650.4,24063936 +14328.6,17976.7,21228.3,13652.0,24063936 +14333.2,17978.3,21228.3,13653.6,24071246 +14338.0,17978.4,21228.3,13653.4,24090652 +14342.7,17978.9,21228.3,13654.0,24108220 +14347.3,17974.4,21228.3,13649.7,24126384 +14352.0,17972.8,21228.3,13648.3,24143088 +14356.6,17974.4,21228.3,13650.6,24160300 +14361.4,17974.3,21228.3,13650.0,24176224 +14365.8,17979.3,21228.3,13654.8,24194354 +14370.4,17975.8,21228.3,13651.4,24209690 +14375.3,17976.0,21228.3,13651.0,24223458 +14380.1,17974.5,21228.3,13650.0,24239920 +14384.9,17971.3,21228.3,13646.7,24257892 +14389.8,17968.8,21228.3,13643.8,24273740 +14394.2,17972.0,21228.3,13647.0,24273740 +14398.8,17970.6,21228.3,13646.1,24273740 +14403.6,17969.6,21228.3,13645.6,24273740 +14408.3,17968.8,21228.3,13644.6,24273740 +14413.0,17969.9,21228.3,13645.3,24273740 +14417.4,17973.4,21228.3,13648.9,24273740 +14422.1,17973.8,21228.3,13648.4,24273740 +14426.6,17973.0,21228.3,13649.2,24273740 +14431.1,17973.9,21228.3,13650.0,24273740 +14435.7,17977.3,21228.3,13652.7,24273740 +14440.5,17973.4,21228.3,13648.3,24273740 +14445.3,17971.8,21228.3,13647.0,24284170 +14450.1,17970.5,21228.3,13645.7,24301944 +14454.7,17972.5,21228.3,13647.9,24318804 +14459.0,17971.7,21228.3,13646.9,24328828 +14463.8,17972.6,21228.3,13647.6,24344358 +14468.5,17971.1,21228.3,13646.8,24362820 +14473.2,17970.3,21228.3,13646.1,24381344 +14477.9,17972.3,21228.3,13647.3,24398618 +14482.5,17970.6,21228.3,13645.8,24416414 +14487.0,17970.4,21228.3,13645.4,24429318 +14491.5,17970.0,21228.3,13645.5,24444160 +14496.3,17967.0,21228.3,13642.5,24459830 +14501.1,17968.9,21228.3,13644.4,24475214 +14506.0,17969.0,21228.3,13644.1,24492380 +14510.5,17970.8,21228.3,13646.3,24492380 +14515.2,17966.3,21228.3,13641.5,24492380 +14519.7,17968.9,21228.3,13644.4,24492380 +14524.3,17974.4,21228.3,13650.6,24492380 +14529.0,17978.2,21228.3,13653.7,24492380 +14533.6,17979.2,21228.3,13654.3,24492380 +14538.3,17978.6,21228.3,13653.7,24492380 +14543.1,17976.1,21228.3,13651.3,24492380 +14547.8,17973.9,21228.3,13649.3,24492380 +14552.3,17973.2,21228.3,13648.3,24492380 +14557.0,17969.6,21228.3,13644.1,24492380 +14561.7,17974.7,21228.3,13650.2,24492380 +14566.6,17971.9,21228.3,13646.9,24492380 +14571.3,17974.4,21228.3,13649.4,24500188 +14576.0,17971.8,21228.3,13647.2,24516306 +14580.6,17970.9,21228.3,13646.4,24533244 +14585.0,17970.7,21228.3,13646.0,24550038 +14589.6,17970.7,21228.3,13645.5,24566382 +14594.3,17972.6,21228.3,13647.6,24583392 +14598.8,17970.6,21228.3,13645.8,24600110 +14603.8,17970.6,21228.3,13647.7,24616216 +14608.5,17971.9,21228.3,13647.6,24631118 +14613.2,17969.4,21228.3,13644.8,24648376 +14618.0,17968.8,21228.3,13644.1,24664558 +14622.9,17969.9,21228.3,13645.4,24679510 +14627.7,17968.8,21228.3,13644.4,24679510 +14632.5,17970.8,21228.3,13646.2,24679510 +14637.1,17974.2,21228.3,13649.4,24679510 +14641.7,17979.9,21228.3,13655.0,24679510 +14646.2,17975.8,21228.3,13650.5,24679510 +14650.6,17967.5,21228.3,13643.0,24679510 +14655.1,17974.8,21228.3,13650.4,24679510 +14659.9,17976.4,21228.3,13651.9,24679510 +14664.6,17975.9,21228.3,13651.0,24679510 +14669.1,17974.0,21228.3,13649.3,24679510 +14673.7,17973.8,21228.3,13649.0,24679510 +14678.4,17973.2,21228.3,13647.4,24679510 +14683.0,17970.0,21228.3,13645.0,24679510 +14687.5,17967.9,21228.3,13643.1,24680263 +14692.3,17971.6,21228.3,13646.6,24700477 +14697.3,17973.5,21228.3,13648.6,24718883 +14702.0,17971.7,21228.3,13647.2,24736117 +14706.7,17970.6,21228.3,13646.0,24751589 +14711.1,17968.6,21228.3,13644.1,24767311 +14715.6,17970.8,21228.3,13646.1,24783037 +14720.2,17970.8,21228.3,13645.8,24803087 +14725.2,17973.9,21228.3,13648.6,24823013 +14729.9,17973.0,21228.3,13648.2,24841657 +14734.8,17971.4,21228.3,13646.6,24860729 +14739.7,17973.1,21228.3,13648.2,24879799 +14744.7,17972.2,21228.3,13647.3,24898565 +14749.3,17970.9,21228.3,13646.1,24914673 +14753.9,17972.1,21228.3,13646.9,24914673 +14758.7,17971.9,21228.3,13646.9,24914673 +14763.4,17970.1,21228.3,13645.3,24914673 +14768.2,17969.2,21228.3,13644.3,24914673 +14772.7,17975.9,21228.3,13650.8,24914673 +14777.4,17974.5,21228.3,13649.6,24914673 +14782.1,17973.3,21228.3,13648.4,24914673 +14786.8,17969.2,21228.3,13644.1,24914673 +14791.4,17970.5,21228.3,13645.5,24914673 +14795.8,17966.0,21228.3,13641.3,24914673 +14800.5,17972.6,21228.3,13647.7,24914673 +14805.3,17966.1,21228.3,13641.1,24914673 +14809.9,17968.0,21228.3,13643.1,24921472 +14814.7,17968.7,21228.3,13643.9,24940924 +14819.4,17971.0,21228.3,13646.5,24959226 +14824.0,17969.0,21228.3,13644.2,24976496 +14828.8,17969.3,21228.3,13644.5,24994492 +14833.5,17968.8,21228.3,13644.4,25014052 +14838.3,17966.1,21228.3,13641.2,25028204 +14842.9,17963.7,21228.3,13638.5,25042010 +14847.8,17959.8,21228.3,13634.7,25057026 +14852.5,17964.7,21228.3,13640.5,25074040 +14857.3,17965.3,21228.3,13640.6,25090792 +14862.3,17964.8,21228.3,13640.1,25109820 +14867.0,17968.2,21228.3,13643.8,25127782 +14871.7,17968.0,21228.3,13643.2,25143692 +14876.4,17967.2,21228.3,13643.2,25143692 +14881.1,17966.9,21228.3,13642.1,25143692 +14885.6,17968.1,21228.3,13643.5,25143692 +14890.3,17965.3,21228.3,13640.7,25143692 +14894.9,17966.3,21228.3,13641.8,25143692 +14899.8,17967.9,21228.3,13643.3,25143692 +14904.3,17968.0,21228.3,13643.1,25143692 +14909.1,17966.0,21228.3,13640.7,25143692 +14913.7,17965.6,21228.3,13640.9,25143692 +14918.4,17968.4,21228.3,13643.6,25143692 +14922.9,17975.0,21228.3,13650.6,25143692 +14927.6,17970.7,21228.3,13645.6,25143692 +14932.4,17969.5,21228.3,13644.6,25146076 +14937.0,17969.8,21228.3,13645.3,25162652 +14941.8,17966.7,21228.3,13642.2,25180030 +14946.3,17968.5,21228.3,13643.9,25195324 +14950.7,17965.1,21228.3,13640.4,25215210 +14955.5,17967.1,21228.3,13642.8,25233608 +14960.3,17966.6,21228.3,13641.9,25252566 +14965.1,17967.6,21228.3,13642.8,25269448 +14969.9,17968.5,21228.3,13643.7,25289516 +14974.7,17965.3,21228.3,13640.2,25310400 +14979.3,17985.3,21228.3,13660.7,25328334 +14984.1,18028.1,21228.3,13702.6,25358666 +14988.8,18066.1,21231.5,13741.7,25402084 +14993.5,18073.0,21236.4,13748.0,25439352 +14998.4,18049.3,21236.4,13724.2,25475224 +15003.1,18058.0,21236.4,13726.6,25502282 +15007.7,18035.1,21236.4,13708.0,25502282 +15012.3,18046.1,21236.4,13721.3,25502282 +15017.1,18044.6,21238.6,13720.6,25502282 +15022.0,18044.4,21238.6,13720.6,25502282 +15026.7,18038.7,21238.6,13714.2,25502282 +15031.4,18035.1,21238.6,13710.6,25502282 +15036.1,18031.8,21238.6,13707.3,25502282 +15040.8,18028.2,21238.6,13704.5,25502282 +15045.5,18031.1,21238.6,13706.2,25502282 +15050.3,18031.6,21238.6,13707.0,25529257 +15055.0,18027.5,21238.6,13703.0,25564057 +15059.5,18025.2,21238.6,13700.7,25592807 +15064.3,18028.3,21238.6,13704.1,25628419 +15069.0,18023.8,21238.6,13698.8,25661785 +15073.6,18020.1,21238.6,13695.1,25694537 +15078.1,18014.5,21238.6,13690.2,25719423 +15082.8,18021.4,21238.6,13696.8,25757503 +15087.5,18020.0,21238.6,13695.3,25777007 +15092.0,18026.8,21238.6,13702.3,25804623 +15096.6,18019.3,21238.6,13695.4,25838905 +15101.2,18018.7,21238.6,13693.9,25865319 +15105.9,18017.2,21238.6,13692.3,25894459 +15110.4,18004.2,21238.6,13679.6,25918591 +15115.2,18009.7,21238.6,13685.0,25943527 +15120.1,18006.9,21238.6,13682.8,25967703 +15124.7,17998.9,21238.6,13674.6,25990851 +15129.5,18001.0,21238.6,13676.2,26014347 +15134.2,18008.3,21238.6,13684.0,26039581 +15139.1,18005.7,21238.6,13680.6,26063855 +15144.1,18008.4,21238.6,13683.4,26091625 +15148.8,18004.3,21238.6,13679.4,26114849 +15153.6,17999.7,21238.6,13674.7,26114849 +15158.4,17997.7,21238.6,13673.1,26114849 +15163.1,17995.7,21238.6,13671.6,26114849 +15167.7,17999.1,21238.6,13674.3,26114849 +15172.2,18005.6,21238.6,13681.2,26114849 +15177.2,18007.2,21238.6,13683.1,26114849 +15182.0,17998.1,21238.6,13673.6,26114849 +15186.7,17997.7,21238.6,13672.8,26114849 +15191.4,17994.3,21238.6,13669.4,26114849 +15196.1,17995.6,21238.6,13670.8,26114849 +15200.9,17992.3,21238.6,13667.9,26114849 +15205.6,17993.4,21238.6,13668.8,26114849 +15210.4,17994.9,21238.6,13669.7,26114849 +15215.2,17993.4,21238.6,13668.6,26114849 +15220.0,17993.2,21238.6,13668.1,26114849 +15224.7,17992.7,21238.6,13667.2,26114849 +15229.2,17994.5,21238.6,13669.9,26114849 +15233.8,17997.5,21238.6,13672.2,26115253 +15238.4,17995.3,21238.6,13670.6,26135791 +15243.2,17995.2,21238.6,13670.3,26156551 +15248.1,17993.6,21238.6,13668.9,26183599 +15253.0,17990.9,21238.6,13666.8,26209379 +15257.8,17990.6,21238.6,13666.2,26229297 +15262.7,17988.6,21238.6,13663.5,26253987 +15267.6,17983.6,21238.6,13658.9,26272607 +15272.5,17987.6,21238.6,13662.7,26295197 +15277.2,17983.6,21238.6,13659.7,26316221 +15282.2,17987.1,21238.6,13663.0,26335433 +15287.1,17981.9,21238.6,13657.5,26355247 +15291.8,17979.7,21238.6,13655.4,26378459 +15296.4,17980.3,21238.6,13656.4,26397089 +15301.3,17984.0,21238.6,13658.9,26419761 +15305.9,17980.9,21238.6,13655.7,26441819 +15310.5,17979.3,21238.6,13654.4,26461625 +15315.1,17977.5,21238.6,13652.6,26484963 +15319.9,17982.7,21238.6,13657.9,26506959 +15324.6,17981.5,21238.6,13657.0,26506959 +15329.2,17980.7,21238.6,13656.0,26506959 +15334.1,17978.6,21238.6,13653.9,26506959 +15338.7,17976.6,21238.6,13652.3,26506959 +15343.6,17976.3,21238.6,13652.7,26506959 +15348.2,17980.8,21238.6,13656.3,26506959 +15353.0,17983.0,21238.6,13658.4,26506959 +15357.8,17979.7,21238.6,13655.1,26506959 +15362.3,17979.3,21238.6,13655.3,26506959 +15367.1,17978.1,21238.6,13653.6,26506959 +15371.9,17977.0,21238.6,13652.5,26506959 +15376.7,17980.2,21238.6,13656.3,26506959 +15381.4,17976.5,21238.6,13651.9,26506959 +15386.1,17975.5,21238.6,13650.5,26506959 +15390.8,17981.5,21238.6,13656.7,26506959 +15395.7,17976.6,21238.6,13652.1,26506959 +15400.3,17976.1,21238.6,13651.2,26506959 +15404.9,17974.8,21238.6,13650.0,26506959 +15409.7,17974.3,21238.6,13649.4,26506959 +15414.3,17970.5,21238.6,13645.6,26506959 +15419.1,17956.7,21238.6,13631.6,26520613 +15423.9,17957.1,21238.6,13633.0,26540919 +15428.9,17954.6,21238.6,13630.0,26558803 +15433.8,17954.9,21238.6,13630.5,26573843 +15438.5,17957.8,21238.6,13633.3,26593289 +15443.6,17956.5,21238.6,13632.6,26614615 +15448.5,17958.7,21238.6,13634.2,26629603 +15453.5,17958.6,21238.6,13634.5,26649611 +15458.3,17955.2,21238.6,13630.8,26671603 +15463.2,17958.4,21238.6,13633.7,26688957 +15468.0,17955.2,21238.6,13630.3,26703927 +15472.5,17955.5,21238.6,13630.7,26724241 +15477.3,17959.8,21238.6,13635.5,26724241 +15481.9,17957.0,21238.6,13632.9,26724241 +15486.5,17956.9,21238.6,13632.5,26724241 +15491.0,17960.5,21238.6,13636.4,26724241 +15495.7,17957.0,21238.6,13632.5,26724241 +15500.6,17957.6,21238.6,13633.0,26724241 +15505.3,17957.2,21238.6,13632.2,26724241 +15509.8,17956.5,21238.6,13631.7,26724241 +15514.4,17959.9,21238.6,13635.3,26724241 +15519.0,17958.2,21238.6,13633.1,26724241 +15523.8,17958.1,21238.6,13633.3,26724241 +15528.5,17959.1,21238.6,13634.2,26724241 +15533.3,17959.3,21238.6,13634.5,26724241 +15537.9,17957.7,21238.6,13632.9,26724241 +15542.7,17958.3,21238.6,13633.2,26724241 +15547.4,17955.4,21238.6,13630.7,26724241 +15551.8,17956.9,21238.6,13632.5,26724241 +15556.7,17954.0,21238.6,13629.4,26729156 +15561.1,17954.1,21238.6,13629.5,26747114 +15565.6,17952.2,21238.6,13627.3,26766896 +15570.3,17957.0,21238.6,13632.1,26786422 +15575.0,17954.4,21238.6,13629.3,26806732 +15579.8,17956.8,21238.6,13632.3,26825166 +15584.6,17956.0,21238.6,13631.5,26845080 +15589.3,17959.2,21238.6,13634.2,26860350 +15594.0,17957.9,21238.6,13632.9,26879294 +15598.9,17954.1,21238.6,13629.5,26900472 +15603.8,17953.3,21238.6,13628.7,26919744 +15608.7,17952.7,21238.6,13628.3,26937456 +15613.6,17951.6,21238.6,13627.2,26951800 +15618.2,17952.5,21238.6,13627.9,26951800 +15622.8,17952.5,21238.6,13628.0,26951800 +15627.4,17951.3,21238.6,13627.1,26951800 +15632.1,17954.6,21238.6,13630.2,26951800 +15636.8,17955.0,21238.6,13630.0,26951800 +15641.7,17955.0,21238.6,13630.2,26951800 +15646.3,17954.5,21238.6,13629.8,26951800 +15651.1,17963.3,21238.6,13638.6,26951800 +15655.9,17956.0,21238.6,13631.2,26951800 +15660.5,17955.5,21238.6,13631.5,26951800 +15665.1,17951.4,21238.6,13626.2,26951800 +15669.9,17951.9,21238.6,13626.7,26951800 +15674.6,17947.5,21238.6,13623.2,26951800 +15679.3,17948.3,21238.6,13623.9,26951800 +15683.9,17947.7,21238.6,13622.6,26951800 +15688.6,17950.5,21238.6,13626.0,26951800 +15693.2,17953.2,21238.6,13628.6,26967361 +15697.9,17953.4,21238.6,13628.2,26987581 +15702.4,17952.2,21238.6,13627.2,27004299 +15707.1,17952.5,21238.6,13627.3,27020887 +15711.8,17952.5,21238.6,13627.4,27038331 +15716.6,17947.9,21238.6,13622.8,27057433 +15721.5,17954.7,21238.6,13629.9,27074967 +15726.4,17949.4,21238.6,13624.8,27090637 +15731.3,17952.1,21238.6,13627.3,27107269 +15736.0,17946.9,21238.6,13622.4,27123331 +15740.7,17947.9,21238.6,13623.6,27138853 +15745.5,17949.0,21238.6,13624.4,27154893 +15749.9,17950.4,21238.6,13626.2,27154893 +15754.5,17946.1,21238.6,13621.3,27154893 +15759.0,17946.5,21238.6,13621.8,27154893 +15763.6,17945.9,21238.6,13621.7,27154893 +15768.1,17941.8,21238.6,13617.7,27154893 +15773.3,17940.9,21238.6,13616.4,27154893 +15778.0,17938.3,21238.6,13614.3,27154893 +15782.8,17941.3,21238.6,13617.0,27154893 +15787.2,17942.6,21238.6,13618.0,27154893 +15791.7,17944.6,21238.6,13620.0,27154893 +15796.3,17946.0,21238.6,13621.6,27154893 +15801.2,17947.0,21238.6,13622.5,27154893 +15805.9,17943.9,21238.6,13619.2,27154893 +15810.3,17945.1,21238.6,13620.8,27154893 +15815.0,17944.3,21238.6,13619.5,27154893 +15819.5,17944.4,21238.6,13620.1,27154893 +15824.1,17946.2,21238.6,13621.4,27169871 +15828.9,17944.7,21238.6,13620.1,27188417 +15833.6,17947.6,21238.6,13623.0,27205293 +15838.5,17947.1,21238.6,13622.6,27220963 +15843.4,17950.0,21238.6,13626.1,27236525 +15848.1,17947.9,21238.6,13623.5,27254575 +15852.8,17945.6,21238.6,13621.1,27267853 +15857.7,17943.5,21238.6,13618.8,27283031 +15862.6,17943.9,21238.6,13618.8,27299321 +15867.6,17942.1,21238.6,13617.2,27315089 +15872.3,17943.9,21238.6,13618.8,27332131 +15877.1,17946.1,21238.6,13621.4,27332131 +15881.8,17947.8,21238.6,13622.9,27332131 +15886.3,17945.4,21238.6,13620.6,27332131 +15890.9,17948.6,21238.6,13623.7,27332131 +15895.5,17940.8,21238.6,13616.6,27332131 +15900.1,17944.9,21238.6,13620.4,27332131 +15904.8,17944.2,21238.6,13619.2,27332131 +15909.5,17946.6,21238.6,13621.6,27332131 +15914.1,17946.4,21238.6,13621.6,27332131 +15918.8,17944.6,21238.6,13620.7,27332131 +15923.5,17941.6,21238.6,13617.7,27332131 +15928.0,17937.5,21238.6,13612.8,27332131 +15932.7,17936.0,21238.6,13611.0,27332131 +15937.2,17937.0,21238.6,13612.5,27341615 +15941.9,17936.5,21238.6,13612.0,27354673 +15946.4,17944.4,21238.6,13619.8,27370771 +15950.9,17940.4,21238.6,13616.1,27385557 +15955.5,17944.0,21238.6,13619.2,27402099 +15960.0,17940.6,21238.6,13615.9,27417951 +15964.6,17942.9,21238.6,13618.5,27434927 +15969.2,17945.2,21238.6,13620.8,27452443 +15973.9,17947.8,21238.6,13623.2,27468557 +15978.4,17947.5,21238.6,13623.0,27485125 +15983.2,17950.9,21238.6,13626.8,27503217 +15988.3,17951.4,21238.6,13626.1,27520361 +15993.0,17950.9,21238.6,13626.0,27536991 +15997.5,17949.1,21238.6,13624.1,27553641 +16002.4,17948.2,21238.6,13623.6,27570731 +16006.9,17947.6,21238.6,13623.1,27570731 +16011.6,17948.5,21238.6,13623.7,27570731 +16016.2,17946.4,21238.6,13621.7,27570731 +16020.9,17946.6,21238.6,13621.9,27570731 +16025.5,17945.2,21238.6,13620.9,27570731 +16030.3,17947.3,21238.6,13622.7,27570731 +16035.2,17946.1,21238.6,13621.5,27570731 +16039.7,17948.1,21238.6,13623.0,27570731 +16044.5,17947.4,21238.6,13623.0,27570731 +16049.0,17946.2,21238.6,13621.7,27570731 +16053.5,17950.4,21238.6,13625.8,27570731 +16058.2,17949.6,21238.6,13625.0,27570731 +16063.0,17951.7,21238.6,13626.9,27570731 +16067.7,17953.6,21238.6,13628.9,27570731 +16072.6,17954.4,21238.6,13629.6,27588665 +16077.1,17951.3,21238.6,13626.5,27605059 +16081.8,17949.1,21238.6,13625.3,27624869 +16086.3,17948.0,21238.6,13623.7,27633845 +16091.0,17949.5,21238.6,13625.1,27655401 +16095.5,17947.4,21238.6,13622.8,27674847 +16100.2,17949.1,21238.6,13624.2,27691379 +16105.1,17945.6,21238.6,13620.9,27705685 +16109.7,17945.2,21238.6,13620.5,27719983 +16114.3,17945.0,21238.6,13620.1,27737509 +16118.7,17944.2,21238.6,13619.3,27754013 +16123.5,17946.1,21238.6,13621.5,27771057 +16128.4,17943.3,21238.6,13618.7,27784439 +16133.2,17943.6,21238.6,13618.4,27801371 +16138.1,17944.7,21238.6,13620.2,27818309 +16143.0,17945.0,21238.6,13620.2,27832835 +16147.8,17947.1,21238.6,13622.6,27851553 +16152.4,17949.2,21238.6,13624.6,27851553 +16157.1,17948.3,21238.6,13623.5,27851553 +16161.7,17948.0,21238.6,13623.5,27851553 +16166.3,17947.9,21238.6,13623.2,27851553 +16171.1,17951.4,21238.6,13627.5,27851553 +16175.8,17947.0,21238.6,13622.3,27851553 +16180.6,17949.3,21238.6,13624.9,27851553 +16185.2,17950.8,21238.6,13626.0,27851553 +16190.0,17949.3,21238.6,13624.5,27851553 +16194.5,17947.1,21238.6,13622.6,27851553 +16198.9,17946.2,21238.6,13619.3,27851553 +16203.7,17944.0,21238.6,13619.2,27851553 +16208.2,17944.9,21238.6,13620.2,27851553 +16212.8,17946.6,21238.6,13621.2,27855661 +16217.5,17946.7,21238.6,13621.9,27874391 +16222.1,17945.8,21238.6,13621.0,27892475 +16226.8,17948.2,21238.6,13623.5,27910737 +16231.7,17950.8,21238.6,13625.9,27933351 +16236.1,17951.2,21238.6,13627.0,27947807 +16240.7,17949.3,21238.6,13624.4,27965777 +16245.4,17946.8,21238.6,13622.4,27983775 +16250.0,17942.2,21238.6,13617.8,28002363 +16254.6,17941.8,21238.6,13617.3,28020415 +16259.4,17944.6,21238.6,13619.7,28038759 +16264.1,17945.5,21238.6,13621.1,28055531 +16268.5,17949.0,21238.6,13624.6,28070349 +16273.1,17948.5,21238.6,13623.8,28084985 +16277.8,17943.2,21238.6,13618.7,28097181 +16282.4,17946.1,21238.6,13621.1,28110155 +16287.1,17947.1,21238.6,13622.8,28127467 +16291.9,17949.6,21238.6,13625.0,28127467 +16296.3,17949.8,21238.6,13624.9,28127467 +16301.2,17949.8,21238.6,13625.0,28127467 +16305.9,17945.0,21238.6,13620.4,28127467 +16310.6,17948.4,21238.6,13623.5,28127467 +16315.5,17942.9,21238.6,13617.8,28127467 +16320.1,17943.1,21238.6,13618.7,28127467 +16324.9,17946.1,21238.6,13621.3,28127467 +16329.8,17942.9,21238.6,13618.2,28127467 +16334.4,17942.8,21238.6,13618.8,28127467 +16338.9,17944.0,21238.6,13619.5,28127467 +16343.4,17942.5,21238.6,13617.9,28127467 +16348.3,17943.4,21238.6,13618.9,28127467 +16352.9,17944.5,21238.6,13619.4,28127467 +16357.5,17945.0,21238.6,13620.5,28127467 +16362.1,17941.9,21238.6,13617.2,28127467 +16367.0,17941.4,21238.6,13617.0,28127467 +16371.4,17941.4,21238.6,13617.1,28127467 +16376.0,17941.0,21238.6,13616.7,28127467 +16380.8,17945.8,21238.6,13620.8,28127467 +16385.5,17944.6,21238.6,13619.8,28127467 +16390.1,17946.4,21238.6,13622.0,28127467 +16394.8,17944.0,21238.6,13618.8,28127467 +16399.5,17943.1,21238.6,13618.2,28127467 +16404.3,17944.0,21238.6,13619.1,28127467 +16408.8,17942.7,21238.6,13618.3,28127467 +16413.5,17944.8,21238.6,13620.6,28127467 +16418.2,17947.3,21238.6,13622.7,28127467 +16422.8,17945.4,21238.6,13620.9,28140232 +16427.4,17941.6,21238.6,13616.5,28140232 +16432.2,17938.7,21238.6,13613.8,28148151 +16437.0,17941.1,21238.6,13616.4,28160897 +16441.5,17941.7,21238.6,13617.0,28160897 +16446.3,17939.8,21238.6,13614.8,28167712 +16451.1,17941.2,21238.6,13616.4,28183124 +16456.0,17941.4,21238.6,13616.4,28197752 +16460.9,17942.2,21238.6,13617.3,28197752 +16465.5,17941.1,21238.6,13616.7,28204135 +16469.9,17938.7,21238.6,13614.0,28218435 +16474.7,17938.8,21238.6,13614.3,28234939 +16479.5,17937.8,21238.6,13612.9,28234939 +16484.2,17937.6,21238.6,13612.6,28237256 +16488.9,17936.7,21238.6,13612.4,28248220 +16493.7,17939.0,21238.6,13614.5,28265682 +16498.5,17939.4,21238.6,13614.5,28265682 +16503.1,17940.0,21238.6,13615.1,28270336 +16508.0,17939.3,21238.6,13614.6,28282248 +16512.5,17942.0,21238.6,13617.2,28282248 +16517.2,17937.8,21238.6,13613.3,28286085 +16521.9,17940.3,21238.6,13616.3,28299565 +16526.6,17937.2,21238.6,13612.1,28317005 +16531.4,17940.2,21238.6,13616.5,28317005 +16536.2,17939.1,21238.6,13614.7,28322218 +16541.0,17940.0,21238.6,13615.1,28337748 +16545.7,17938.5,21238.6,13613.6,28352798 +16550.1,17939.9,21238.6,13615.3,28368280 +16554.8,17938.9,21238.6,13614.6,28368280 +16559.3,17939.7,21238.6,13615.1,28368280 +16564.1,17940.1,21238.6,13615.4,28380085 +16568.8,17944.4,21238.6,13619.7,28397923 +16573.3,17942.2,21238.6,13617.2,28397923 +16578.1,17941.4,21238.6,13616.2,28397923 +16582.6,17941.2,21238.6,13616.4,28409062 +16587.6,17940.2,21238.6,13615.9,28426312 +16592.3,17938.0,21238.6,13613.3,28441740 +16597.1,17937.3,21238.6,13612.9,28441740 +16601.8,17937.5,21238.6,13612.9,28441740 +16606.6,17941.5,21238.6,13617.1,28454405 +16611.2,17939.6,21238.6,13614.9,28470869 +16615.9,17944.6,21238.6,13619.7,28487433 +16620.6,17945.2,21238.6,13620.5,28504055 +16625.3,17939.1,21238.6,13614.5,28504055 +16629.8,17940.6,21238.6,13615.8,28504055 +16634.6,17942.6,21238.6,13618.0,28512395 +16639.4,17946.3,21238.6,13621.6,28529375 +16644.3,17960.1,21238.6,13635.4,28546165 +16649.2,17962.6,21238.6,13638.1,28546165 +16653.9,17955.7,21238.6,13630.8,28546165 +16658.3,17951.3,21238.6,13626.5,28548350 +16662.8,17951.0,21238.6,13626.1,28563886 +16667.6,17948.4,21238.6,13623.6,28577744 +16672.3,17950.8,21238.6,13626.4,28592474 +16676.8,17950.5,21238.6,13625.8,28592474 +16681.5,17951.1,21238.6,13626.2,28592474 +16686.2,17949.7,21238.6,13624.6,28596565 +16691.0,17951.6,21238.6,13626.6,28613049 +16695.6,17949.9,21238.6,13625.4,28629279 +16700.7,17950.6,21238.6,13626.0,28648867 +16705.2,17951.0,21238.6,13626.3,28648867 +16709.8,17948.0,21238.6,13623.3,28648867 +16714.5,17947.5,21238.6,13622.9,28648867 +16719.2,17949.1,21238.6,13624.4,28662108 +16723.8,17947.0,21238.6,13623.5,28673926 +16728.5,17946.7,21238.6,13622.0,28684462 +16733.4,17943.6,21238.6,13618.9,28684462 +16738.1,17944.2,21238.6,13619.1,28684462 +16742.7,17941.9,21238.6,13616.8,28684462 +16747.2,17940.7,21238.6,13615.5,28696906 +16752.0,17942.7,21238.6,13617.0,28710792 +16756.7,17943.2,21238.6,13618.2,28723902 +16761.5,17945.0,21238.6,13620.9,28723902 +16766.3,17950.2,21238.6,13625.6,28723902 +16771.1,17954.7,21238.6,13630.3,28731863 +16775.9,17957.1,21238.6,13632.3,28746705 +16780.3,17954.5,21238.6,13629.6,28764583 +16784.9,17952.1,21238.6,13627.5,28782507 +16789.5,17952.0,21238.6,13627.3,28797925 +16794.3,17951.7,21238.6,13627.0,28797925 +16799.2,17954.5,21238.6,13629.6,28797925 +16804.1,17951.3,21238.6,13626.5,28797925 +16809.0,17951.8,21238.6,13627.1,28811589 +16813.6,17954.1,21238.6,13629.2,28830661 +16818.3,17953.5,21238.6,13628.9,28848099 +16823.1,17951.0,21238.6,13626.2,28863685 +16827.8,17954.8,21238.6,13631.0,28863685 +16832.6,17953.7,21238.6,13629.5,28863685 +16837.4,17957.0,21238.6,13632.3,28863685 +16842.0,17954.4,21238.6,13629.7,28864266 +16846.8,17951.8,21238.6,13627.2,28878786 +16851.5,17954.9,21238.6,13629.5,28896190 +16856.2,17950.8,21238.6,13626.3,28913070 +16861.1,17953.4,21238.6,13630.0,28926050 +16865.8,17949.6,21238.6,13625.3,28943550 +16870.5,17950.5,21238.6,13626.0,28943550 +16875.2,17951.2,21238.6,13626.7,28943550 +16880.0,17949.7,21238.6,13625.7,28943550 +16884.7,17949.7,21238.6,13625.2,28943550 +16889.4,17952.4,21238.6,13628.1,28959335 +16894.2,17954.7,21238.6,13630.2,28977783 +16898.9,17952.5,21238.6,13628.0,28995561 +16903.4,17954.3,21238.6,13629.5,29011921 +16908.3,17956.6,21238.6,13631.7,29011921 +16913.0,17956.5,21238.6,13631.8,29011921 +16917.6,17954.5,21238.6,13629.7,29011921 +16922.2,17956.7,21238.6,13632.1,29011921 +16926.9,17953.4,21238.6,13629.4,29026170 +16931.6,17950.3,21238.6,13626.1,29042024 +16936.4,17955.1,21238.6,13630.3,29057610 +16941.0,17951.6,21238.6,13627.0,29070726 +16945.6,17948.3,21238.6,13624.0,29084122 +16950.2,17951.8,21238.6,13627.1,29084122 +16954.8,17952.6,21238.6,13628.2,29084122 +16959.6,17950.6,21238.6,13626.0,29084122 +16964.2,17953.2,21238.6,13628.4,29084122 +16968.9,17955.7,21238.6,13630.8,29099281 +16973.4,17953.4,21238.6,13628.5,29115747 +16977.8,17951.3,21238.6,13626.5,29134213 +16982.7,17951.0,21238.6,13626.7,29150867 +16987.2,17957.9,21238.6,13632.7,29171175 +16991.8,17959.4,21238.6,13634.8,29185337 +16996.5,17958.0,21238.6,13633.2,29185337 +17001.1,17958.9,21238.6,13634.1,29185337 +17006.0,17953.0,21238.6,13628.8,29185337 +17010.6,17951.3,21238.6,13626.4,29185337 +17015.2,17947.9,21238.6,13623.3,29185337 +17019.9,17952.2,21238.6,13628.1,29202333 +17024.8,17946.3,21238.6,13621.8,29212885 +17029.5,17951.8,21238.6,13627.1,29232555 +17034.1,17949.8,21238.6,13624.9,29248615 +17038.9,17952.0,21238.6,13627.5,29265901 +17043.5,17951.7,21238.6,13627.4,29265901 +17048.3,17948.2,21238.6,13623.3,29265901 +17053.1,17952.0,21238.6,13627.5,29265901 +17058.0,17949.9,21238.6,13625.3,29265901 +17062.7,17953.0,21238.6,13628.9,29272276 +17067.4,17951.6,21238.6,13627.0,29290874 +17072.0,17951.4,21238.6,13626.4,29311556 +17076.9,17946.8,21238.6,13622.3,29324692 +17081.7,17948.2,21238.6,13623.2,29342900 +17086.2,17951.5,21238.6,13627.2,29360178 +17090.9,17953.4,21238.6,13628.8,29360178 +17095.6,17948.4,21238.6,13623.8,29360178 +17100.4,17948.3,21238.6,13624.2,29360178 +17105.1,17947.3,21238.6,13623.3,29360178 +17109.7,17950.0,21238.6,13625.1,29360178 +17114.4,17949.4,21238.6,13624.8,29369285 +17119.2,17952.1,21238.6,13627.2,29389635 +17124.0,17950.6,21238.6,13625.5,29407853 +17128.7,17986.8,21238.6,13662.4,29426789 +17133.6,18017.0,21238.6,13692.8,29456481 +17138.0,18017.3,21238.6,13693.1,29478901 +17142.7,18012.4,21238.6,13687.9,29478901 +17147.6,18013.7,21238.6,13689.5,29478901 +17152.1,17998.4,21238.6,13673.7,29478901 +17156.6,17989.1,21238.6,13664.5,29478901 +17161.4,18014.4,21238.6,13687.4,29484754 +17166.1,18010.2,21238.6,13685.3,29510648 +17170.9,17999.1,21238.6,13674.4,29539144 +17175.6,17997.6,21238.6,13673.5,29568154 +17180.4,18001.2,21238.6,13676.3,29598078 +17185.5,17989.5,21238.6,13665.4,29620670 +17190.2,17985.9,21238.6,13661.4,29649092 +17195.1,17989.0,21238.6,13664.3,29680678 +17199.7,17992.7,21238.6,13667.6,29702708 +17204.5,17994.4,21238.6,13670.5,29724888 +17209.2,17998.7,21238.6,13674.5,29724888 +17213.7,17986.9,21238.6,13662.9,29724888 +17218.4,18006.6,21238.6,13681.7,29724888 +17223.2,17999.6,21238.6,13674.9,29724888 +17228.0,17973.9,21238.6,13649.5,29724888 +17232.8,17961.8,21238.6,13637.3,29724888 +17237.5,17946.3,21238.6,13621.8,29724888 +17242.1,17944.4,21238.6,13619.6,29724888 +17246.8,17936.8,21238.6,13612.8,29724888 +17251.4,17943.9,21238.6,13619.2,29740359 +17256.2,17967.8,21238.6,13644.2,29761113 +17261.0,17977.6,21238.6,13653.5,29786261 +17266.2,17975.2,21238.6,13651.1,29809731 +17271.6,17970.6,21238.6,13646.2,29831581 +17277.5,17968.1,21238.6,13644.5,29831581 +17283.9,17960.0,21238.6,13635.1,29831581 +17290.0,17962.6,21238.6,13638.6,29831581 +17296.3,17966.6,21238.6,13642.7,29831581 +17302.4,17958.0,21238.6,13633.4,29831581 +17308.0,17960.8,21238.6,13636.5,29831581 +17314.4,17952.1,21238.6,13627.4,29831581 +17320.2,17964.0,21238.6,13639.2,29831581 +17325.6,17966.9,21238.6,13642.2,29831581 +17332.0,17958.4,21238.6,13633.7,29831581 +17337.6,17964.3,21238.6,13639.8,29831581 +17343.2,17967.5,21238.6,13643.0,29831581 +17349.4,17968.2,21238.6,13644.1,29839631 +17355.8,17959.4,21238.6,13635.0,29848271 +17361.8,17953.9,21238.6,13629.7,29858887 +17368.2,17957.2,21238.6,13633.0,29872213 +17374.8,17952.4,21238.6,13627.8,29880699 +17380.4,17954.5,21238.6,13629.9,29896785 +17386.3,17962.4,21238.6,13638.0,29914301 +17393.4,17959.2,21238.6,13634.5,29927245 +17399.9,17955.8,21238.6,13631.4,29935291 +17406.8,17955.7,21238.6,13631.5,29945149 +17413.1,17961.0,21238.6,13636.7,29957947 +17419.9,17957.3,21238.6,13632.6,29964935 +17427.1,17961.4,21238.6,13637.0,29980569 +17435.4,17954.5,21238.6,13628.7,29988427 +17443.3,17961.5,21238.6,13636.6,30001967 +17449.9,17950.2,21238.6,13626.1,30008103 +17457.8,17956.6,21238.6,13632.0,30018325 +17464.3,17954.4,21238.6,13629.5,30025913 +17471.0,17955.0,21238.6,13630.4,30034283 +17478.3,17954.1,21238.6,13630.1,30040873 +17484.8,17955.9,21238.6,13631.4,30050175 +17492.6,17959.1,21238.6,13634.5,30050175 +17500.0,17958.8,21238.6,13634.2,30050175 +17507.5,17960.2,21238.6,13635.5,30050175 +17514.4,17949.4,21238.6,13624.7,30050175 +17521.4,17949.7,21238.6,13625.7,30050175 +17528.8,17955.5,21238.6,13630.9,30050175 +17535.5,17954.0,21238.6,13629.5,30050175 +17540.6,17959.1,21238.6,13634.6,30050175 +17546.0,17958.4,21238.6,13633.6,30050175 +17551.2,17962.4,21238.6,13638.0,30050175 +17556.1,17963.9,21238.6,13639.4,30050175 +17561.0,17964.3,21238.6,13639.5,30050175 +17565.6,17963.9,21238.6,13639.8,30050175 +17570.4,17960.9,21238.6,13636.2,30058299 +17575.4,17959.5,21238.6,13635.3,30078067 +17581.1,17956.6,21238.6,13631.8,30092835 +17586.8,17962.2,21238.6,13637.6,30106435 +17592.7,17953.9,21238.6,13629.1,30119583 +17598.3,17951.9,21238.6,13627.8,30133751 +17604.1,17951.9,21238.6,13628.0,30147097 +17609.6,17957.1,21238.6,13632.5,30162225 +17615.1,17963.2,21238.6,13638.0,30182565 +17619.8,17967.7,21238.6,13643.0,30182565 +17624.6,17969.8,21238.6,13644.8,30182565 +17629.2,17967.7,21238.6,13642.8,30182565 +17633.9,17955.2,21238.6,13630.3,30182565 +17638.5,17960.5,21238.6,13635.3,30182565 +17643.4,17964.9,21238.6,13640.0,30182565 +17648.2,17961.6,21238.6,13636.9,30182565 +17652.9,17962.2,21238.6,13637.2,30182565 +17657.8,17957.8,21238.6,13633.0,30182565 +17662.6,17949.6,21238.6,13625.0,30190682 +17667.4,17947.2,21238.6,13623.0,30200518 +17672.0,17955.7,21238.6,13631.9,30221676 +17676.5,17960.6,21238.6,13636.0,30240798 +17681.3,17960.0,21238.6,13635.3,30257180 +17686.2,17958.4,21238.6,13633.7,30271010 +17691.2,17959.9,21238.6,13634.9,30290452 +17696.1,17962.7,21238.6,13637.8,30309440 +17701.0,17962.9,21238.6,13638.1,30328536 +17705.9,17963.3,21238.6,13638.8,30347302 +17710.6,17961.3,21238.6,13636.6,30347302 +17715.5,17961.4,21238.6,13636.7,30347302 +17720.1,17965.3,21238.6,13641.2,30347302 +17725.1,17960.9,21238.6,13636.3,30347302 +17730.1,17961.2,21238.6,13636.6,30347302 +17734.9,17956.5,21238.6,13632.0,30347302 +17739.4,17956.5,21238.6,13631.8,30347302 +17744.2,17957.1,21238.6,13632.6,30347302 +17748.8,17961.8,21238.6,13637.5,30357600 +17753.6,17957.3,21238.6,13632.8,30379326 +17758.3,17959.5,21238.6,13635.0,30394494 +17763.1,17960.2,21238.6,13635.4,30410026 +17767.9,17955.7,21238.6,13630.9,30432080 +17772.6,17958.2,21238.6,13633.1,30453786 +17777.5,17960.8,21238.6,13636.2,30470644 +17782.5,17958.3,21238.6,13633.7,30486208 +17787.6,17953.1,21238.6,13629.0,30501068 +17792.5,17953.2,21238.6,13628.4,30518028 +17797.4,17951.9,21238.6,13627.0,30535290 +17802.3,17952.3,21238.6,13627.6,30535290 +17806.9,17952.8,21238.6,13627.9,30535290 +17811.4,17955.6,21238.6,13631.3,30535290 +17816.0,17953.9,21238.6,13629.8,30535290 +17820.6,17953.7,21238.6,13629.4,30535290 +17825.2,17954.1,21238.6,13629.2,30535290 +17830.0,17952.6,21238.6,13627.6,30535290 +17834.7,17955.7,21238.6,13630.8,30535290 +17839.5,17959.9,21238.6,13635.3,30546493 +17844.0,17958.0,21238.6,13633.5,30566871 +17848.6,17958.1,21238.6,13633.7,30583195 +17853.2,17958.0,21238.6,13633.0,30603899 +17857.7,17956.5,21238.6,13632.1,30623571 +17862.4,17955.7,21238.6,13631.5,30642327 +17867.2,17953.6,21238.6,13628.9,30661579 +17872.0,17951.9,21238.6,13627.1,30681767 +17876.7,17955.5,21238.6,13630.6,30701215 +17882.1,17952.5,21238.6,13627.4,30720509 +17886.9,17950.8,21238.6,13625.9,30739551 +17891.9,17955.2,21238.6,13630.5,30761229 +17896.5,17959.5,21238.6,13634.3,30781137 +17900.9,17961.5,21238.6,13636.3,30781137 +17905.5,17956.2,21238.6,13631.4,30781137 +17910.3,17955.9,21238.6,13631.2,30781137 +17915.0,17954.8,21238.6,13630.5,30781137 +17919.5,17948.7,21238.6,13624.4,30781137 +17924.4,17954.1,21238.6,13629.5,30781137 +17929.1,17956.5,21238.6,13631.7,30781137 +17933.9,17959.3,21238.6,13635.3,30781137 +17938.6,17958.5,21238.6,13633.6,30781137 +17943.3,17954.6,21238.6,13630.0,30781137 +17947.9,17959.5,21238.6,13634.7,30800941 +17952.6,17957.7,21238.6,13633.2,30818655 +17957.4,17953.7,21238.6,13629.3,30844845 +17962.2,17953.4,21238.6,13628.7,30864855 +17966.8,17953.3,21238.6,13628.4,30885593 +17971.7,17958.7,21238.6,13633.9,30910731 +17976.2,17955.4,21238.6,13630.6,30929711 +17980.8,17954.3,21238.6,13630.1,30951105 +17985.7,17953.6,21238.6,13629.1,30969009 +17990.6,17958.0,21238.6,13633.6,30989569 +17995.4,17960.1,21238.6,13635.7,31011687 +17999.9,17956.0,21238.6,13630.8,31033171 +18004.7,17952.8,21238.6,13627.9,31053163 +18009.3,17951.5,21238.6,13626.6,31053163 +18013.9,17960.4,21238.6,13635.8,31053163 +18018.4,17957.5,21238.6,13632.7,31053163 +18023.1,17953.2,21238.6,13628.8,31053163 +18027.8,17955.2,21238.6,13630.3,31053163 +18032.6,17953.0,21238.6,13628.1,31053163 +18037.2,17952.6,21238.6,13628.1,31053163 +18041.7,17952.2,21238.6,13627.2,31053163 +18046.3,17956.8,21238.6,13632.4,31053163 +18051.0,17957.3,21238.6,13632.8,31053163 +18055.7,17956.0,21238.6,13631.5,31053163 +18060.5,17959.9,21238.6,13635.0,31069719 +18065.2,17955.8,21238.6,13630.7,31093483 +18069.8,17954.5,21238.6,13629.5,31111863 +18074.7,17953.5,21238.6,13629.0,31133405 +18079.5,17956.8,21238.6,13632.4,31155563 +18084.3,17948.8,21238.6,13624.2,31170945 +18089.1,17947.1,21238.6,13622.4,31189977 +18094.2,17946.7,21238.6,13621.9,31212483 +18098.7,17949.3,21238.6,13624.4,31231531 +18103.3,17951.6,21238.6,13627.1,31248935 +18108.0,17951.4,21238.6,13626.6,31268487 +18112.6,17952.4,21238.6,13627.8,31286469 +18117.5,17950.2,21238.6,13625.8,31306899 +18122.6,17949.9,21238.6,13625.8,31328245 +18127.3,17950.2,21238.6,13625.5,31328245 +18132.0,17939.1,21238.6,13614.4,31328245 +18136.7,17934.6,21238.6,13609.8,31328245 +18141.4,17929.0,21238.6,13604.0,31328245 +18146.1,17927.1,21238.6,13602.6,31328245 +18150.8,17928.7,21238.6,13604.0,31328245 +18155.3,17929.9,21238.6,13606.1,31328245 +18160.1,17934.7,21238.6,13610.0,31328245 +18164.9,17933.5,21238.6,13608.8,31328245 +18169.4,17930.2,21238.6,13605.9,31328245 +18173.9,17932.2,21238.6,13605.8,31328245 +18178.4,17935.3,21238.6,13611.0,31328245 +18183.1,17933.4,21238.6,13608.6,31328245 +18187.8,17933.9,21238.6,13608.6,31329370 +18192.5,17933.4,21238.6,13608.8,31345496 +18197.3,17930.6,21238.6,13606.1,31363302 +18201.8,17930.7,21238.6,13606.1,31382820 +18206.3,17930.0,21238.6,13605.3,31399740 +18211.0,17932.0,21238.6,13607.6,31417576 +18215.9,17930.9,21238.6,13606.4,31436792 +18220.8,17930.1,21238.6,13605.6,31453192 +18225.8,17928.2,21238.6,13603.4,31469544 +18230.6,17927.9,21238.6,13603.1,31485738 +18235.6,17928.2,21238.6,13603.2,31503882 +18240.2,17929.6,21238.6,13604.5,31522258 +18245.1,17930.3,21238.6,13605.6,31538142 +18249.7,17931.0,21238.6,13606.0,31554500 +18254.4,17933.1,21238.6,13608.3,31554500 +18259.3,17932.2,21238.6,13607.4,31554500 +18264.0,17933.3,21238.6,13608.8,31554500 +18268.7,17941.6,21238.6,13616.8,31554500 +18273.4,17939.5,21238.6,13615.1,31554500 +18278.2,17931.5,21238.6,13606.8,31554500 +18282.9,17928.7,21238.6,13603.9,31554500 +18287.8,17929.1,21238.6,13604.6,31554500 +18292.6,17928.5,21238.6,13604.0,31554500 +18297.3,17928.4,21238.6,13604.1,31554500 +18302.1,17927.8,21238.6,13603.3,31554500 +18306.8,17925.6,21238.6,13601.2,31554500 +18311.6,17935.2,21238.6,13611.3,31554500 +18316.2,17933.9,21238.6,13609.0,31554500 +18321.0,17931.6,21238.6,13607.5,31569090 +18325.7,17930.8,21238.6,13605.8,31582178 +18330.3,17933.1,21238.6,13608.2,31600844 +18335.1,17932.2,21238.6,13607.5,31620702 +18339.8,17932.9,21238.6,13608.4,31636260 +18344.5,17931.0,21238.6,13606.2,31655156 +18349.4,17930.5,21238.6,13605.7,31675944 +18354.2,17929.9,21238.6,13605.1,31691012 +18359.2,17932.2,21238.6,13607.2,31707222 +18364.2,17929.9,21238.6,13605.1,31724048 +18369.2,17927.4,21238.6,13602.4,31740138 +18374.0,17928.9,21238.6,13604.3,31757020 +18378.9,17930.8,21238.6,13606.1,31757020 +18383.5,17929.2,21238.6,13604.4,31757020 +18388.3,17950.9,21238.6,13626.3,31757020 +18393.0,17957.5,21238.6,13632.6,31757020 +18397.5,17946.3,21238.6,13621.7,31757020 +18402.3,17935.8,21238.6,13611.1,31757020 +18406.9,17925.4,21238.6,13600.6,31757020 +18411.7,17924.9,21238.6,13600.5,31757020 +18416.2,17925.4,21238.6,13601.0,31757020 +18421.1,17927.3,21238.6,13602.5,31757020 +18425.6,17926.1,21238.6,13601.5,31757020 +18430.2,17927.9,21238.6,13602.9,31757020 +18434.9,17924.4,21238.6,13600.2,31765690 +18439.6,17924.9,21238.6,13600.2,31785320 +18443.9,17924.5,21238.6,13600.1,31803148 +18448.7,17925.4,21238.6,13600.9,31819634 +18453.4,17928.3,21238.6,13603.5,31841302 +18458.1,17926.8,21238.6,13602.2,31859376 +18462.6,17928.2,21238.6,13603.5,31879058 +18467.4,17926.7,21238.6,13602.0,31894996 +18472.3,17932.7,21238.6,13608.1,31915186 +18477.2,17930.1,21238.6,13604.9,31932716 +18482.2,17928.0,21238.6,13602.8,31946944 +18487.1,17933.1,21238.6,13608.6,31965446 +18492.1,17929.9,21238.6,13605.1,31983788 +18497.0,17931.3,21238.6,13606.4,31999400 +18501.9,17924.7,21238.6,13600.1,32016122 +18506.6,17924.5,21238.6,13599.6,32016122 +18511.2,17925.7,21238.6,13600.8,32016122 +18516.0,17925.0,21238.6,13600.3,32016122 +18520.8,17961.7,21238.6,13637.0,32016122 +18525.7,17946.9,21238.6,13622.3,32016122 +18530.6,17939.4,21238.6,13615.4,32016122 +18535.2,17933.0,21238.6,13608.6,32016122 +18539.9,17931.3,21238.6,13607.4,32016122 +18544.8,17925.9,21238.6,13601.0,32016122 +18549.4,17930.4,21238.6,13605.3,32016122 +18554.3,17926.0,21238.6,13601.4,32016122 +18559.0,17925.4,21238.6,13601.1,32016122 +18564.0,17927.6,21238.6,13603.0,32016122 +18568.9,17927.6,21238.6,13603.3,32016122 +18573.9,17932.3,21238.6,13607.7,32016122 +18578.7,17937.8,21238.6,13613.1,32016122 +18583.5,17933.1,21238.6,13608.2,32016122 +18588.3,17928.7,21238.6,13603.6,32016122 +18593.0,17926.2,21238.6,13601.7,32016122 +18597.7,17930.6,21238.6,13605.5,32016122 +18602.6,17930.1,21238.6,13605.2,32016122 +18607.1,17930.6,21238.6,13605.9,32016122 +18611.9,17933.3,21238.6,13608.4,32016122 +18616.5,17932.1,21238.6,13607.0,32016122 +18621.6,17932.8,21238.6,13608.2,32027137 +18626.2,17929.8,21238.6,13605.4,32027137 +18630.9,17925.6,21238.6,13601.3,32036126 +18635.8,17922.7,21238.6,13598.2,32051048 +18640.3,17924.1,21238.6,13599.9,32051048 +18645.0,17925.2,21238.6,13600.9,32063090 +18649.8,17927.7,21238.6,13603.5,32079768 +18654.4,17929.3,21238.6,13604.7,32079768 +18658.9,17927.1,21238.6,13602.6,32087703 +18663.5,17927.2,21238.6,13602.5,32104009 +18668.4,17928.9,21238.6,13603.7,32104009 +18673.0,17928.0,21238.6,13603.2,32117078 +18677.7,17925.5,21238.6,13600.5,32132590 +18682.2,17925.9,21238.6,13602.1,32132590 +18686.8,17926.7,21238.6,13602.6,32142660 +18691.4,17925.7,21238.6,13600.7,32157936 +18696.1,17928.2,21238.6,13603.7,32157936 +18700.9,17923.6,21238.6,13599.0,32160334 +18705.4,17927.8,21238.6,13603.4,32177888 +18709.9,17927.6,21238.6,13602.7,32193566 +18715.2,17925.5,21238.6,13600.5,32193566 +18719.8,17930.5,21238.6,13605.5,32193566 +18724.7,17925.9,21238.6,13601.6,32211902 +18729.3,17927.5,21238.6,13602.2,32224822 +18734.0,17924.7,21238.6,13599.8,32244556 +18738.8,17928.0,21238.6,13603.2,32244556 +18743.3,17925.2,21238.6,13600.1,32244556 +18748.0,17925.9,21238.6,13601.7,32258926 +18752.8,17927.2,21238.6,13602.2,32274974 +18757.4,17926.3,21238.6,13601.8,32274974 +18762.1,17930.1,21238.6,13605.8,32275084 +18766.9,17929.0,21238.6,13604.7,32292036 +18771.6,17925.1,21238.6,13600.1,32308058 +18776.1,17934.4,21238.6,13610.1,32308058 +18781.0,17928.3,21238.6,13604.1,32308058 +18785.6,17927.4,21238.6,13602.9,32322681 +18790.3,17929.8,21238.6,13605.5,32341761 +18794.8,17927.5,21238.6,13603.3,32358553 +18799.6,17930.1,21238.6,13605.2,32358553 +18804.1,17927.5,21238.6,13602.8,32358553 +18808.6,17927.7,21238.6,13603.2,32371522 +18813.3,17927.3,21238.6,13602.3,32386430 +18818.1,17925.7,21238.6,13601.0,32399020 +18822.7,17925.8,21238.6,13601.2,32399020 +18827.5,17927.3,21238.6,13602.7,32399020 +18832.1,17926.2,21238.6,13601.3,32405439 +18836.8,17923.5,21238.6,13599.1,32419577 +18841.3,17922.7,21238.6,13598.3,32431687 +18846.0,17922.8,21238.6,13598.1,32431687 +18850.7,17925.2,21238.6,13600.8,32431687 +18855.3,17924.1,21238.6,13599.9,32431743 +18860.2,17925.3,21238.6,13600.7,32450443 +18864.7,17925.0,21238.6,13599.9,32468373 +18869.2,17926.4,21238.6,13601.4,32483949 +18873.6,17924.6,21238.6,13600.2,32483949 +18878.2,17926.6,21238.6,13602.3,32483949 +18883.0,17927.9,21238.6,13603.2,32486611 +18887.8,17933.1,21238.6,13608.1,32503389 +18892.5,17923.2,21238.6,13598.6,32519105 +18897.2,17925.1,21238.6,13600.3,32536019 +18901.9,17921.9,21238.6,13596.6,32536019 +18906.7,17925.5,21238.6,13600.8,32536019 +18911.5,17925.2,21238.6,13600.6,32536019 +18916.3,17923.6,21238.6,13598.6,32550309 +18921.1,17926.8,21238.6,13601.7,32565903 +18925.8,17926.2,21238.6,13601.5,32583161 +18930.7,17929.8,21238.6,13604.4,32583161 +18935.4,17929.1,21238.6,13604.3,32583161 +18940.0,17930.0,21238.6,13605.1,32585456 +18944.6,17925.8,21238.6,13601.0,32599052 +18949.4,17927.9,21238.6,13603.4,32617446 +18954.1,17926.4,21238.6,13601.9,32635106 +18958.9,17928.9,21238.6,13604.2,32635106 +18963.4,17928.3,21238.6,13603.9,32635106 +18968.1,17925.6,21238.6,13600.7,32635230 +18972.8,17925.0,21238.6,13600.4,32652888 +18977.6,17927.4,21238.6,13602.4,32668262 +18982.4,17926.1,21238.6,13600.9,32683752 +18987.4,17923.6,21238.6,13598.7,32698384 +18992.4,17926.9,21238.6,13602.1,32698384 +18997.2,17924.9,21238.6,13600.2,32698384 +19001.9,17928.7,21238.6,13603.8,32698384 +19006.5,17928.4,21238.6,13603.9,32712179 +19011.3,17925.8,21238.6,13601.1,32727527 +19016.2,17929.0,21238.6,13604.3,32745143 +19020.9,17930.3,21238.6,13605.8,32763407 +19025.4,17928.7,21238.6,13604.2,32763407 +19030.0,17926.9,21238.6,13602.3,32763407 +19034.8,17924.3,21238.6,13599.5,32763407 +19039.5,17923.0,21238.6,13598.8,32772255 +19044.1,17923.3,21238.6,13599.2,32787667 +19048.9,17923.7,21238.6,13599.3,32803669 +19053.8,17926.0,21238.6,13601.7,32820155 +19058.4,17924.2,21238.6,13600.0,32820155 +19063.1,17925.0,21238.6,13600.3,32820155 +19067.9,17923.5,21238.6,13599.1,32820155 +19072.6,17925.2,21238.6,13600.8,32829221 +19077.1,17922.3,21238.6,13598.0,32846389 +19081.9,17926.2,21238.6,13601.8,32864287 +19086.7,17927.9,21238.6,13603.3,32884341 +19091.2,17926.8,21238.6,13602.4,32898745 +19095.9,17925.3,21238.6,13600.8,32898745 +19100.6,17924.6,21238.6,13599.9,32898745 +19105.1,17925.1,21238.6,13600.5,32898745 +19109.7,17927.0,21238.6,13602.0,32898745 +19114.4,17925.4,21238.6,13601.1,32909756 +19119.2,17931.7,21238.6,13606.6,32931086 +19124.0,17924.5,21238.6,13599.5,32948146 +19129.0,17926.8,21238.6,13602.0,32965260 +19133.8,17927.0,21238.6,13602.3,32982240 +19138.5,17927.2,21238.6,13602.3,32982240 +19143.3,17926.3,21238.6,13601.4,32982240 +19148.0,17926.9,21238.6,13602.8,32982240 +19152.6,17925.9,21238.6,13601.2,32982259 +19157.4,17926.6,21238.6,13601.7,33003207 +19162.1,17928.5,21238.6,13603.7,33023535 +19166.8,17923.8,21238.6,13599.4,33039717 +19171.8,17952.6,21238.6,13628.3,33061331 +19176.5,17946.7,21238.6,13622.1,33076487 +19181.4,17934.7,21238.6,13609.8,33076487 +19186.1,17925.2,21238.6,13600.2,33076487 +19190.9,17920.4,21238.6,13595.5,33076487 +19195.7,17920.7,21238.6,13596.2,33076487 +19200.1,17921.2,21238.6,13596.8,33080493 +19204.9,17927.1,21238.6,13602.6,33096507 +19209.7,17927.5,21238.6,13602.4,33111887 +19214.3,17925.5,21238.6,13601.0,33130783 +19219.1,17932.4,21238.6,13607.5,33147399 +19223.9,17928.6,21238.6,13603.4,33147399 +19228.5,17945.8,21238.6,13620.4,33147399 +19233.0,18001.0,21238.6,13676.4,33147399 +19237.6,17976.2,21238.6,13651.2,33164586 +19242.6,17976.1,21238.6,13651.5,33191416 +19247.3,17971.8,21238.6,13647.1,33217598 +19251.7,17981.1,21238.6,13656.2,33246698 +19256.3,17969.8,21238.6,13645.2,33268820 +19261.0,17963.7,21238.6,13639.5,33292066 +19265.9,17951.1,21238.6,13625.9,33309502 +19270.6,17958.6,21238.6,13634.4,33309502 +19275.4,17965.6,21238.6,13641.0,33309502 +19280.1,17970.6,21238.6,13645.8,33309502 +19284.8,17963.7,21238.6,13639.4,33309502 +19289.4,18617.1,21238.6,14292.3,33309502 +19294.1,18615.7,21238.6,14291.5,33325959 +19298.9,18624.5,21238.6,14299.7,33344729 +19303.7,18613.0,21238.6,14287.9,33359555 +19308.5,18618.3,21238.6,14294.0,33380901 +19313.0,18618.3,21238.6,14294.3,33400889 +19317.8,18637.7,21238.6,14313.3,33431187 +19322.7,18640.3,21238.6,14316.1,33450431 +19327.4,18631.0,21238.6,14306.9,33467861 +19332.2,18621.8,21238.6,14296.6,33467861 +19336.7,18612.0,21238.6,14287.1,33467861 +19341.5,18614.7,21238.6,14290.3,33467861 +19346.3,18613.7,21238.6,14289.2,33467861 +19350.9,18612.2,21238.6,14287.9,33467861 +19355.6,18610.8,21238.6,14286.2,33467861 +19360.4,18610.8,21238.6,14286.5,33472413 +19364.9,18614.8,21238.6,14290.4,33495103 +19369.7,18614.0,21238.6,14289.2,33515341 +19374.2,18613.2,21238.6,14288.5,33532755 +19379.0,18613.4,21238.6,14288.4,33551973 +19383.7,18610.3,21238.6,14285.4,33564689 +19388.7,18613.3,21238.6,14288.8,33585295 +19393.5,18612.7,21238.6,14287.8,33585295 +19398.1,18611.9,21238.6,14287.2,33585295 +19402.6,18609.4,21238.6,14284.6,33585295 +19407.3,18609.0,21238.6,14284.2,33585295 +19411.9,18610.1,21238.6,14285.2,33585295 +19416.7,18612.1,21238.6,14287.0,33585295 +19421.4,18607.0,21238.6,14282.1,33587830 +19426.3,18606.0,21238.6,14281.4,33605094 +19431.0,18605.0,21238.6,14280.4,33627090 +19435.9,18606.9,21238.6,14282.4,33645004 +19441.0,18607.6,21238.6,14282.7,33665854 +19445.9,18605.0,21238.6,14280.4,33683568 +19450.7,18606.6,21238.6,14282.0,33702506 +19455.5,18606.2,21238.6,14281.5,33719148 +19460.3,18600.7,21238.6,14277.0,33719148 +19464.9,18601.5,21238.6,14277.2,33719148 +19469.4,18602.9,21238.6,14279.7,33719148 +19473.9,18616.0,21238.6,14291.3,33719148 +19478.4,18615.1,21238.6,14290.4,33719148 +19483.0,18609.8,21238.6,14284.6,33719148 +19487.7,18602.4,21238.6,14278.4,33719148 +19492.3,18599.9,21238.6,14275.1,33732920 +19497.1,18598.9,21238.6,14274.1,33750820 +19501.8,18600.6,21238.6,14276.4,33764906 +19506.6,18599.5,21238.6,14274.4,33782926 +19511.4,18595.2,21238.6,14270.2,33796056 +19516.0,18599.9,21238.6,14275.5,33816404 +19520.8,18598.2,21238.6,14273.8,33834662 +19525.4,18596.3,21238.6,14271.5,33834662 +19530.2,18592.0,21238.6,14267.5,33834662 +19534.8,18585.4,21238.6,14260.7,33834662 +19539.3,18579.4,21238.6,14255.0,33834662 +19543.9,18587.2,21238.6,14263.1,33834662 +19548.6,18589.3,21238.6,14264.3,33834662 +19553.2,18586.6,21238.6,14261.4,33834662 +19557.9,18584.2,21238.6,14259.7,33834662 +19562.6,18594.0,21238.6,14269.0,33834662 +19567.4,18591.3,21238.6,14266.2,33849319 +19572.1,18590.8,21238.6,14266.0,33861483 +19576.9,18594.8,21238.6,14270.2,33883767 +19581.5,18595.0,21238.6,14270.4,33898423 +19586.3,18597.0,21238.6,14272.5,33917263 +19590.8,18593.2,21238.6,14268.6,33932097 +19595.3,18593.9,21238.6,14269.0,33950827 +19599.9,18593.8,21238.6,14269.4,33950827 +19604.6,18593.4,21238.6,14268.9,33950827 +19609.3,18592.7,21238.6,14268.0,33950827 +19614.1,18594.0,21238.6,14269.5,33950827 +19618.8,18594.1,21238.6,14269.9,33950827 +19623.4,18592.1,21238.6,14267.8,33950827 +19628.2,18595.0,21238.6,14270.6,33962947 +19632.6,18593.6,21238.6,14268.9,33980705 +19637.3,18594.7,21238.6,14269.9,33997655 +19642.2,18595.1,21238.6,14270.4,34014771 +19646.9,18596.9,21238.6,14272.1,34030093 +19651.4,18593.9,21238.6,14269.8,34049441 +19656.2,18591.6,21238.6,14266.8,34060151 +19660.9,18591.5,21238.6,14267.3,34071775 +19665.7,18590.4,21238.6,14265.9,34071775 +19670.5,18587.5,21238.6,14262.9,34071775 +19675.3,18589.7,21238.6,14265.2,34071775 +19680.0,18591.6,21238.6,14267.0,34071775 +19684.7,18591.3,21238.6,14266.8,34071775 +19689.6,18593.1,21238.6,14267.9,34071775 +19694.3,18590.0,21238.6,14265.3,34071775 +19698.8,18591.0,21238.6,14267.0,34071775 +19703.6,18594.4,21238.6,14269.9,34086329 +19708.2,18593.5,21238.6,14268.9,34103625 +19713.0,18588.0,21238.6,14263.6,34113883 +19717.7,18586.4,21238.6,14261.3,34131265 +19722.4,18590.1,21238.6,14265.0,34146529 +19727.1,18593.2,21238.6,14268.3,34163419 +19731.6,18592.4,21238.6,14267.5,34180489 +19736.7,18591.7,21238.6,14266.8,34196787 +19741.5,18588.1,21238.6,14263.9,34213187 +19746.1,18589.7,21238.6,14265.6,34228017 +19750.9,18587.3,21238.6,14263.2,34228017 +19755.6,18590.1,21238.6,14265.4,34228017 +19760.2,18591.2,21238.6,14266.5,34228017 +19764.7,18588.4,21238.6,14264.4,34228017 +19769.2,18587.1,21238.6,14262.5,34228017 +19773.9,18582.6,21238.6,14258.2,34228017 +19778.4,18587.4,21238.6,14263.0,34228017 +19783.1,18591.9,21238.6,14267.5,34228017 +19787.8,18589.2,21238.6,14264.3,34230775 +19792.5,18586.9,21238.6,14262.4,34246685 +19797.4,18583.1,21238.6,14258.4,34262497 +19802.2,18585.1,21238.6,14260.3,34277591 +19806.9,18584.1,21238.6,14259.3,34295981 +19811.7,18585.8,21238.6,14261.4,34309887 +19816.5,18584.3,21238.6,14259.6,34323933 +19821.2,18587.1,21238.6,14262.3,34342793 +19825.8,18589.4,21238.6,14264.8,34342793 +19830.7,18589.7,21238.6,14265.3,34342793 +19835.4,18591.6,21238.6,14267.3,34342793 +19840.1,18592.1,21238.6,14267.8,34342793 +19844.5,18593.3,21238.6,14268.7,34342793 +19849.2,18589.3,21238.6,14265.0,34342793 +19853.9,18592.1,21238.6,14267.7,34342793 +19858.3,18593.5,21238.6,14268.6,34345554 +19862.8,18590.5,21238.6,14265.2,34355420 +19867.6,18588.7,21238.6,14263.5,34370156 +19872.1,18588.9,21238.6,14264.6,34385344 +19877.0,18587.9,21238.6,14263.1,34401236 +19881.7,18586.5,21238.6,14261.9,34412954 +19886.5,18588.4,21238.6,14263.7,34430700 +19891.3,18587.6,21238.6,14262.3,34448468 +19895.7,18588.1,21238.6,14263.2,34462028 +19900.4,18588.8,21238.6,14264.3,34476738 +19905.1,18589.2,21238.6,14264.5,34495922 +19909.8,18591.2,21238.6,14266.2,34495922 +19914.4,18588.9,21238.6,14264.6,34495922 +19918.9,18588.6,21238.6,14264.0,34495922 +19923.5,18585.9,21238.6,14261.3,34495922 +19928.3,18590.9,21238.6,14266.3,34495922 +19932.9,18591.8,21238.6,14267.2,34495922 +19937.7,18589.6,21238.6,14264.6,34495922 +19942.4,18584.8,21238.6,14259.9,34495922 +19947.0,18589.1,21238.6,14264.3,34503822 +19951.7,18588.9,21238.6,14263.9,34522858 +19956.2,18589.7,21238.6,14264.2,34539716 +19960.8,18587.3,21238.6,14262.8,34557102 +19965.4,18586.0,21238.6,14262.0,34575172 +19970.2,18583.7,21238.6,14259.4,34586844 +19975.0,18588.0,21238.6,14263.3,34605712 +19979.8,18585.6,21238.6,14261.1,34619832 +19984.6,18584.1,21238.6,14259.9,34633960 +19989.5,18586.1,21238.6,14261.5,34649962 +19994.1,18585.9,21238.6,14261.7,34665338 +19998.7,18587.6,21238.6,14262.8,34665338 +20003.4,18587.9,21238.6,14263.4,34665338 +20008.1,18587.5,21238.6,14262.7,34665338 +20012.9,18583.3,21238.6,14258.3,34665338 +20017.5,18584.7,21238.6,14259.6,34665338 +20022.3,18584.8,21238.6,14259.9,34665338 +20027.0,18585.5,21238.6,14260.8,34665338 +20031.7,18582.3,21238.6,14258.0,34665338 +20036.5,18585.4,21238.6,14260.6,34665338 +20041.2,18585.2,21238.6,14260.3,34665338 +20045.9,18583.6,21238.6,14258.4,34675188 +20050.4,18587.2,21238.6,14262.6,34692636 +20055.0,18584.8,21238.6,14259.7,34709942 +20059.8,18582.7,21238.6,14258.0,34723410 +20064.2,18585.2,21238.6,14260.4,34738204 +20068.8,18583.7,21238.6,14259.1,34755002 +20073.6,18584.3,21238.6,14259.7,34773148 +20078.2,18582.4,21238.6,14257.6,34784520 +20083.0,18583.0,21238.6,14258.3,34798206 +20087.7,18582.6,21238.6,14258.0,34813874 +20092.1,18583.0,21238.6,14258.5,34813874 +20096.8,18585.6,21238.6,14260.9,34813874 +20101.3,18598.7,21238.6,14275.2,34813874 +20106.1,18605.7,21238.6,14281.0,34813874 +20111.0,18594.1,21238.6,14269.4,34813874 +20115.7,18591.3,21238.6,14267.0,34813874 +20120.5,18586.6,21238.6,14262.2,34813874 +20125.0,18585.2,21238.6,14260.5,34813874 +20129.7,18582.8,21238.6,14258.7,34813874 +20134.4,18588.1,21238.6,14263.3,34831162 +20139.3,18592.8,21238.6,14268.2,34849256 +20143.9,18587.6,21238.6,14262.9,34865268 +20148.6,18587.4,21238.6,14262.9,34881842 +20153.3,18585.4,21238.6,14260.4,34899002 +20158.2,18586.1,21238.6,14261.6,34915858 +20163.1,18587.6,21238.6,14263.8,34932180 +20167.6,18591.5,21238.6,14267.6,34948864 +20172.2,18646.3,21238.6,14321.6,34948864 +20177.1,18743.4,21263.4,14419.2,35016500 +20182.1,18642.4,21264.2,14317.9,35034522 +20186.8,18602.4,21264.2,14278.2,35034522 +20191.3,18596.1,21264.2,14271.4,35034522 +20195.9,18590.9,21264.2,14266.6,35034522 +20200.6,18596.0,21264.2,14271.6,35034522 +20205.3,18589.7,21264.2,14265.3,35034522 +20210.2,18594.2,21264.2,14269.9,35034522 +20214.8,18591.0,21264.2,14266.1,35034522 +20219.6,18592.2,21264.2,14268.0,35034522 +20224.2,18596.2,21264.2,14271.1,35034522 +20228.9,18597.7,21264.2,14273.3,35034522 +20233.4,18589.7,21264.2,14264.7,35034522 +20238.3,18594.7,21264.2,14269.7,35046529 +20242.9,18591.7,21264.2,14267.2,35066851 +20247.7,18591.1,21264.2,14266.9,35082883 +20252.4,18591.0,21264.2,14266.6,35100909 +20257.2,18586.2,21264.2,14261.8,35118029 +20262.0,18587.4,21264.2,14263.2,35134473 +20266.8,18586.3,21264.2,14261.5,35149325 +20271.5,18590.9,21264.2,14266.1,35166151 +20276.3,18589.1,21264.2,14264.7,35182185 +20281.2,18587.1,21264.2,14262.3,35197289 +20286.0,18588.4,21264.2,14263.2,35213627 +20290.8,18590.0,21264.2,14265.5,35231615 +20295.6,18593.4,21264.2,14268.3,35231615 +20300.0,18592.5,21264.2,14267.2,35231615 +20304.7,18591.7,21264.2,14267.2,35231615 +20309.0,18593.1,21264.2,14268.3,35231615 +20313.6,18592.6,21264.2,14268.3,35231615 +20318.4,18594.6,21264.2,14269.8,35231615 +20323.2,18594.5,21264.2,14269.8,35231615 +20327.8,18599.0,21264.2,14274.5,35231615 +20332.7,18599.1,21264.2,14274.0,35231615 +20337.4,18598.4,21264.2,14274.2,35231615 +20342.0,18596.0,21264.2,14271.4,35247846 +20346.6,18591.2,21264.2,14266.8,35267498 +20351.3,18595.4,21264.2,14270.5,35286392 +20356.1,18593.2,21264.2,14268.1,35309584 +20360.7,18644.9,21264.2,14320.6,35341898 +20365.4,18654.8,21264.2,14330.9,35379322 +20370.0,18646.0,21264.2,14321.3,35409138 +20374.4,18639.0,21264.2,14314.4,35430600 +20379.2,18630.0,21264.2,14305.5,35461568 +20384.3,18626.2,21264.2,14301.1,35488372 +20389.1,18619.8,21264.2,14295.0,35511782 +20393.6,18619.9,21264.2,14295.1,35537910 +20398.1,18621.4,21264.2,14296.5,35563524 +20402.7,18608.6,21264.2,14283.6,35580418 +20407.4,18609.8,21264.2,14286.7,35580418 +20412.2,18610.8,21264.2,14286.0,35580418 +20416.8,18608.3,21264.2,14283.7,35580418 +20421.5,18607.9,21264.2,14283.6,35580418 +20426.3,18599.8,21264.2,14275.5,35580418 +20431.1,18595.6,21264.2,14273.3,35580418 +20435.7,18595.8,21264.2,14270.7,35580418 +20440.3,18597.7,21264.2,14273.2,35580418 +20445.0,18594.3,21264.2,14269.6,35580418 +20450.0,18598.8,21264.2,14274.9,35580418 +20454.6,18596.9,21264.2,14272.3,35580418 +20459.3,18584.3,21264.2,14259.9,35580418 +20464.3,18587.3,21264.2,14262.3,35589379 +20469.0,18588.1,21264.2,14263.3,35613863 +20473.7,18584.7,21264.2,14260.6,35632441 +20478.4,18579.9,21264.2,14255.2,35653781 +20483.1,18580.7,21264.2,14256.0,35675905 +20487.6,18581.6,21264.2,14256.8,35695811 +20492.1,18579.3,21264.2,14254.7,35715727 +20496.8,18580.9,21264.2,14256.3,35735849 +20501.7,18581.9,21264.2,14257.3,35757017 +20506.3,18579.2,21264.2,14254.2,35772495 +20511.1,18579.3,21264.2,14254.2,35791945 +20515.6,18571.0,21264.2,14246.3,35804401 +20520.3,18569.6,21264.2,14245.1,35820289 +20524.8,18571.7,21264.2,14246.5,35835923 +20529.4,18569.8,21264.2,14244.9,35835923 +20534.1,18570.1,21264.2,14245.5,35835923 +20538.9,18564.0,21264.2,14239.3,35835923 +20543.3,18568.0,21264.2,14243.5,35835923 +20548.1,18566.4,21264.2,14241.8,35835923 +20552.7,18568.6,21264.2,14244.0,35835923 +20557.5,18569.9,21264.2,14245.1,35835923 +20562.5,18566.7,21264.2,14242.1,35835923 +20567.2,18571.2,21264.2,14246.1,35835923 +20571.8,18571.6,21264.2,14247.6,35835923 +20576.5,18573.3,21264.2,14249.4,35835923 +20581.4,18574.4,21264.2,14249.8,35835923 +20586.0,18579.9,21264.2,14255.6,35835923 +20590.6,18578.5,21264.2,14253.7,35835923 +20595.4,18576.2,21264.2,14251.7,35835923 +20599.9,18572.4,21264.2,14248.0,35835923 +20604.7,18575.1,21264.2,14250.6,35835923 +20609.5,18576.2,21264.2,14251.6,35852778 +20614.1,18574.1,21264.2,14249.8,35871592 +20618.6,18570.0,21264.2,14245.1,35886240 +20623.3,18567.9,21264.2,14244.7,35897284 +20627.8,18572.1,21264.2,14247.1,35912664 +20632.3,18569.2,21264.2,14244.7,35923114 +20637.0,18572.6,21264.2,14247.9,35939738 +20641.6,18570.9,21264.2,14246.5,35953672 +20646.1,18568.4,21264.2,14243.5,35966648 +20650.5,18573.6,21264.2,14249.0,35978450 +20655.2,18568.5,21264.2,14244.2,35992942 +20659.9,18571.3,21264.2,14246.0,35992942 +20664.5,18570.2,21264.2,14245.9,35992942 +20669.3,18571.1,21264.2,14246.5,35992942 +20674.2,18569.3,21264.2,14244.4,35992942 +20678.8,18569.0,21264.2,14244.2,35992942 +20683.4,18570.4,21264.2,14245.3,35992942 +20688.1,18571.8,21264.2,14246.7,35992942 +20692.6,18571.7,21264.2,14246.8,35992942 +20697.2,18571.2,21264.2,14246.8,35992942 +20701.9,18567.5,21264.2,14243.0,35992942 +20706.8,18564.4,21264.2,14239.7,35992942 +20711.4,18565.4,21264.2,14240.8,35992942 +20716.2,18561.3,21264.2,14236.4,35992942 +20721.0,18565.9,21264.2,14241.3,35992942 +20725.8,18568.6,21264.2,14244.0,35992942 +20730.3,18571.4,21264.2,14247.0,35992942 +20735.1,18573.3,21264.2,14248.3,35992942 +20740.0,18567.0,21264.2,14242.2,35992942 +20744.7,18566.9,21264.2,14241.7,35992942 +20749.5,18570.0,21264.2,14245.2,35992942 +20754.4,18565.7,21264.2,14241.0,35992942 +20758.8,18568.5,21264.2,14244.0,35992942 +20763.3,18571.2,21264.2,14246.5,35992942 +20767.9,18572.4,21264.2,14247.4,35992942 +20772.7,18571.7,21264.2,14246.5,35999787 +20777.2,18567.2,21264.2,14242.3,35999787 +20782.0,18567.9,21264.2,14242.8,36014129 +20786.8,18568.7,21264.2,14244.1,36014129 +20791.3,18566.0,21264.2,14241.3,36031711 +20796.0,18571.7,21264.2,14246.7,36031711 +20800.5,18568.8,21264.2,14244.4,36044519 +20805.0,18569.1,21264.2,14244.3,36056289 +20809.8,18568.4,21264.2,14243.6,36056289 +20814.4,18567.3,21264.2,14242.0,36071270 +20819.3,18567.8,21264.2,14242.7,36071270 +20823.9,18565.9,21264.2,14241.3,36086358 +20828.6,18575.8,21264.2,14251.8,36099998 +20833.4,18571.0,21264.2,14246.3,36099998 +20838.2,18570.6,21264.2,14245.6,36113271 +20843.1,18566.4,21264.2,14241.3,36130115 +20847.8,18566.5,21264.2,14241.5,36130115 +20852.6,18565.3,21264.2,14240.7,36141612 +20857.2,18566.5,21264.2,14242.2,36141612 +20861.8,18567.3,21264.2,14242.4,36151214 +20866.5,18566.2,21264.2,14241.4,36170412 +20871.0,18565.7,21264.2,14242.0,36170412 +20875.7,18568.9,21264.2,14244.0,36179070 +20880.5,18565.3,21264.2,14240.6,36197012 +20885.0,18565.0,21264.2,14240.6,36197012 +20889.7,18566.7,21264.2,14241.9,36202273 +20894.4,18564.0,21264.2,14239.7,36216451 +20899.2,18569.5,21264.2,14244.9,36235917 +20904.0,18570.4,21264.2,14245.6,36235917 +20908.7,18567.1,21264.2,14242.3,36241730 +20913.4,18565.1,21264.2,14240.7,36253814 +20918.2,18563.8,21264.2,14239.4,36253814 +20922.7,18562.2,21264.2,14237.0,36258850 +20927.4,18562.5,21264.2,14237.9,36275110 +20931.9,18562.0,21264.2,14237.8,36288112 +20936.5,18562.2,21264.2,14237.5,36288112 +20941.2,18565.2,21264.2,14240.2,36291447 +20945.7,18565.4,21264.2,14240.5,36308769 +20950.1,18563.7,21264.2,14238.7,36308769 +20954.8,18563.4,21264.2,14239.0,36311478 +20959.4,18563.1,21264.2,14238.7,36328384 +20964.0,18563.5,21264.2,14238.7,36342134 +20968.7,18560.8,21264.2,14235.8,36342134 +20973.5,18562.9,21264.2,14238.0,36342134 +20978.1,18563.8,21264.2,14238.8,36354050 +20982.6,18561.4,21264.2,14236.7,36368634 +20987.5,18560.8,21264.2,14236.0,36385368 +20992.3,18562.4,21264.2,14237.7,36385368 +20996.9,18559.0,21264.2,14234.6,36385368 +21001.2,18560.6,21264.2,14236.4,36393579 +21005.7,18554.6,21264.2,14230.0,36405983 +21010.3,18554.9,21264.2,14230.0,36405983 +21014.8,18551.8,21264.2,14227.2,36405983 +21019.4,18551.8,21264.2,14227.6,36421055 +21023.9,18555.5,21264.2,14230.9,36431929 +21028.7,18554.3,21264.2,14229.8,36431929 +21033.3,18556.0,21264.2,14231.4,36431929 +21038.1,18555.5,21264.2,14230.8,36442822 +21042.8,18555.6,21264.2,14230.9,36454896 +21047.4,18556.9,21264.2,14232.6,36468924 +21052.2,18556.8,21264.2,14232.2,36468924 +21056.9,18553.4,21264.2,14228.4,36470707 +21061.5,18557.5,21264.2,14232.4,36483157 +21066.0,18551.8,21264.2,14227.0,36496225 +21070.7,18555.7,21264.2,14230.8,36496225 +21075.1,18553.6,21264.2,14228.7,36496225 +21079.7,18552.0,21264.2,14227.5,36508222 +21084.4,18554.1,21264.2,14229.4,36518266 +21089.1,18551.6,21264.2,14226.6,36518266 +21093.6,18551.6,21264.2,14226.7,36518266 +21098.3,18554.5,21264.2,14230.0,36531907 +21103.1,18559.1,21264.2,14234.2,36545145 +21107.7,18560.4,21264.2,14235.9,36545145 +21112.4,18556.4,21264.2,14231.9,36545145 +21116.9,18555.8,21264.2,14231.3,36559700 +21121.6,18555.9,21264.2,14231.6,36567092 +21126.0,18565.9,21264.2,14241.3,36585582 +21130.8,18564.1,21264.2,14239.6,36585582 +21135.6,18561.2,21264.2,14236.3,36585582 +21140.2,18557.6,21264.2,14232.9,36598422 +21144.9,18556.7,21264.2,14232.2,36613186 +21149.6,18554.2,21264.2,14229.8,36628114 +21154.3,18554.4,21264.2,14230.0,36628114 +21158.9,18552.4,21264.2,14227.5,36628114 +21163.7,18550.8,21264.2,14225.9,36631142 +21168.4,18553.8,21264.2,14228.6,36646350 +21173.1,18552.2,21264.2,14227.1,36660758 +21177.9,18551.9,21264.2,14227.3,36660758 +21182.4,18565.5,21264.2,14240.8,36660758 +21187.0,18559.1,21264.2,14234.5,36673798 +21191.5,18558.5,21264.2,14233.4,36690144 +21196.2,18553.2,21264.2,14229.0,36703722 +21200.8,18553.1,21264.2,14228.7,36703722 +21205.4,18554.2,21264.2,14230.0,36703722 +21210.4,18551.6,21264.2,14227.0,36704308 +21215.2,18550.0,21264.2,14225.2,36713952 +21219.9,18552.8,21264.2,14227.9,36728984 +21224.5,18551.4,21264.2,14226.6,36741900 +21229.4,18550.3,21264.2,14225.1,36741900 +21234.2,18553.5,21264.2,14228.6,36741900 +21238.7,18550.2,21264.2,14225.4,36748195 +21243.3,18551.5,21264.2,14226.4,36762721 +21247.8,18549.8,21264.2,14224.7,36776749 +21252.4,18548.1,21264.2,14223.2,36788661 +21257.2,18546.9,21264.2,14222.5,36788661 +21261.9,18549.4,21264.2,14225.0,36788661 +21266.6,18546.6,21264.2,14222.0,36793843 +21271.1,18548.5,21264.2,14224.2,36806313 +21275.8,18548.2,21264.2,14223.7,36817377 +21280.6,18545.7,21264.2,14220.9,36817377 +21285.3,18547.2,21264.2,14222.5,36817377 +21290.1,18545.8,21264.2,14221.6,36820980 +21294.8,18545.6,21264.2,14221.2,36833776 +21299.5,18546.1,21264.2,14221.6,36847494 +21304.4,18548.9,21264.2,14223.5,36861630 +21308.9,18546.1,21264.2,14221.4,36861630 +21313.6,18546.2,21264.2,14221.5,36861630 +21318.1,18553.9,21264.2,14229.7,36864614 +21322.9,18552.0,21264.2,14227.5,36878942 +21327.6,18555.2,21264.2,14230.5,36893658 +21332.1,18555.3,21264.2,14230.7,36905266 +21336.9,18549.6,21264.2,14224.8,36905266 +21341.7,18550.4,21264.2,14226.0,36905266 +21346.4,18552.2,21264.2,14227.7,36905266 +21350.9,18558.4,21264.2,14233.6,36920893 +21355.7,18556.5,21264.2,14231.2,36934675 +21360.4,18572.8,21264.2,14247.8,36951459 +21365.3,18575.6,21264.2,14250.9,36951459 +21370.1,18573.9,21264.2,14249.3,36951459 +21374.6,18568.4,21264.2,14244.0,36951459 +21379.0,18567.9,21264.2,14243.5,36961850 +21383.7,18566.3,21264.2,14241.9,36974026 +21388.3,18565.4,21264.2,14241.2,36989472 +21393.0,18564.2,21264.2,14240.0,37002386 +21397.6,18566.6,21264.2,14242.2,37002386 +21402.2,18567.4,21264.2,14242.9,37002386 +21406.7,18563.8,21264.2,14239.0,37002386 +21411.6,18562.1,21264.2,14237.0,37012674 +21416.2,18564.5,21264.2,14240.3,37026088 +21420.8,18567.5,21264.2,14242.6,37041148 +21425.4,18568.4,21264.2,14243.4,37056912 +21430.2,18570.9,21264.2,14245.8,37056912 +21434.9,18575.5,21264.2,14249.7,37056912 +21439.5,18574.1,21264.2,14249.3,37056980 +21443.9,18574.9,21264.2,14250.2,37072686 +21448.4,18573.1,21264.2,14248.8,37089734 +21453.3,18572.4,21264.2,14247.8,37106238 +21458.1,18571.3,21264.2,14246.3,37118726 +21462.6,18570.4,21264.2,14245.7,37118726 +21467.4,18570.5,21264.2,14246.2,37118726 +21472.2,18570.0,21264.2,14245.2,37118726 +21477.0,18567.6,21264.2,14243.2,37130986 +21481.8,18565.0,21264.2,14239.9,37147366 +21486.5,18569.9,21264.2,14245.1,37163766 +21491.2,18566.1,21264.2,14241.2,37176410 +21496.1,18566.5,21264.2,14241.5,37176410 +21500.6,18569.1,21264.2,14243.9,37176410 +21505.1,18570.9,21264.2,14246.7,37176410 +21509.7,18570.2,21264.2,14245.3,37181308 +21514.4,18567.2,21264.2,14242.3,37199784 +21518.8,18569.1,21264.2,14244.1,37212496 +21523.5,18567.3,21264.2,14242.0,37226792 +21528.4,18570.3,21264.2,14245.1,37240348 +21533.0,18565.0,21264.2,14240.9,37240348 +21537.6,18569.3,21264.2,14245.1,37240348 +21542.3,18569.4,21264.2,14244.9,37240348 +21546.9,18567.8,21264.2,14243.4,37240348 +21551.8,18569.7,21264.2,14244.8,37256975 +21556.5,18567.0,21264.2,14242.4,37271775 +21561.3,18569.1,21264.2,14244.9,37289315 +21566.1,18566.4,21264.2,14242.0,37307475 +21570.8,18571.4,21264.2,14246.7,37323323 +21575.5,18566.3,21264.2,14241.7,37323323 +21580.5,18570.0,21264.2,14245.7,37323323 +21585.2,18572.2,21264.2,14247.1,37323323 +21589.7,18571.1,21264.2,14247.0,37323323 +21594.1,18570.3,21264.2,14245.9,37333606 +21598.9,18568.5,21264.2,14244.0,37350360 +21603.6,18567.4,21264.2,14243.1,37364476 +21608.2,18567.4,21264.2,14242.8,37381218 +21612.7,18567.2,21264.2,14242.3,37396464 +21617.2,18565.2,21264.2,14240.4,37396464 +21621.7,18565.4,21264.2,14240.7,37396464 +21626.5,18563.8,21264.2,14238.8,37396464 +21631.2,18563.7,21264.2,14239.0,37396464 +21635.9,18565.0,21264.2,14240.3,37403352 +21640.5,18567.0,21264.2,14242.1,37419582 +21645.4,18567.4,21264.2,14242.6,37436538 +21650.1,18563.6,21264.2,14238.9,37448990 +21654.7,18564.6,21264.2,14239.9,37465114 +21659.7,18563.7,21264.2,14239.0,37465114 +21664.3,18563.6,21264.2,14238.5,37465114 +21669.0,18566.7,21264.2,14241.8,37465114 +21673.8,18567.0,21264.2,14242.0,37465114 +21678.2,18567.5,21264.2,14242.7,37478800 +21683.0,18565.4,21264.2,14240.8,37494760 +21687.7,18566.1,21264.2,14242.0,37510414 +21692.2,18566.0,21264.2,14241.4,37525260 +21696.7,18565.7,21264.2,14241.1,37538908 +21701.2,18563.0,21264.2,14238.6,37538908 +21705.5,18558.2,21264.2,14233.9,37538908 +21710.2,18551.6,21264.2,14227.3,37538908 +21714.9,18548.0,21264.2,14223.7,37538908 +21719.2,18546.7,21264.2,14222.4,37538908 +21724.0,18546.5,21264.2,14222.2,37538908 +21728.5,18546.5,21264.2,14222.2,37538908 +21732.8,18546.3,21264.2,14222.0,37538908 +21737.5,18546.3,21264.2,14222.0,37538908 +21741.8,18545.9,21264.2,14221.6,37538908 +21746.4,18545.9,21264.2,14221.7,37538908 +21751.0,18545.9,21264.2,14221.6,37538908 +21755.5,18545.9,21264.2,14221.6,37538908 +21759.9,18545.8,21264.2,14221.5,37538908 +21764.4,18545.8,21264.2,14221.5,37538908 +21768.7,18545.8,21264.2,14221.5,37538908 +21773.1,18545.7,21264.2,14221.4,37538908 +21777.5,18558.5,21264.2,14234.3,37538908 +21782.3,18586.8,21264.2,14262.2,37538908 +21786.8,18616.1,21264.2,14291.9,37538908 +21791.5,18648.1,21264.2,14323.8,37538908 +21795.9,18676.1,21264.2,14351.8,37538908 +21800.7,18705.7,21264.2,14381.2,37538908 +21805.3,18737.5,21264.2,14413.2,37538908 +21810.1,18767.4,21264.2,14443.1,37538908 +21814.8,18811.2,21264.2,14486.9,37538908 +21819.4,18923.7,21264.2,14599.9,37538908 +21823.9,19038.2,21264.2,14714.0,37538908 +21828.6,19143.4,21264.2,14821.2,37538908 +21832.9,19255.6,21264.2,14931.4,37538908 +21837.6,19329.3,21264.2,15005.0,37538908 +21842.1,19284.5,21264.2,14960.2,37538908 +21846.5,18690.1,21264.2,14365.8,37538908 +21851.1,18554.1,21264.2,14229.8,37538908 +21855.8,18554.0,21264.2,14229.7,37538908 +21860.1,18554.0,21264.2,14229.7,37538908 +21864.5,18553.8,21264.2,14229.5,37538908 +21869.2,18554.0,21264.2,14229.4,37538908 +21873.8,18553.8,21264.2,14229.3,37538908 +21878.5,18553.7,21264.2,14229.2,37538908 diff --git a/benchmarks/realtime-data-indexing/results/50gb-postrun-qdrant.json b/benchmarks/realtime-data-indexing/results/50gb-postrun-qdrant.json new file mode 100644 index 0000000..d259be5 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/50gb-postrun-qdrant.json @@ -0,0 +1,62 @@ +{ + "exact_count": { + "result": { + "count": 37472786 + }, + "status": "ok", + "time": 6.670093051 + }, + "collection": { + "result": { + "status": "green", + "optimizer_status": "ok", + "indexed_vectors_count": 37472786, + "points_count": 37472786, + "segments_count": 11, + "config": { + "params": { + "vectors": { + "embedding": { + "size": 256, + "distance": "Cosine" + } + }, + "shard_number": 1, + "replication_factor": 1, + "write_consistency_factor": 1, + "on_disk_payload": true + }, + "hnsw_config": { + "m": 16, + "ef_construct": 100, + "full_scan_threshold": 10000, + "max_indexing_threads": 0, + "on_disk": false + }, + "optimizer_config": { + "deleted_threshold": 0.2, + "vacuum_min_vector_number": 1000, + "default_segment_number": 0, + "max_segment_size": null, + "memmap_threshold": null, + "indexing_threshold": 10000, + "flush_interval_sec": 5, + "max_optimization_threads": null + }, + "wal_config": { + "wal_capacity_mb": 32, + "wal_segments_ahead": 0, + "wal_retain_closed": 1 + }, + "quantization_config": null, + "strict_mode_config": { + "enabled": false + } + }, + "payload_schema": {} + }, + "status": "ok", + "time": 0.373571209 + }, + "note": "Independent post-run check; published chunk count retains the original approximate-count methodology" +} diff --git a/benchmarks/realtime-data-indexing/results/50gb-preparation.json b/benchmarks/realtime-data-indexing/results/50gb-preparation.json new file mode 100644 index 0000000..cc007c4 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/50gb-preparation.json @@ -0,0 +1,17 @@ +{ + "target_bytes": 52428800000, + "written_bytes": 52428765709, + "documents": 17083603, + "language_configs": [ + "20231101.en", + "20231101.de", + "20231101.fr", + "20231101.es", + "20231101.it", + "20231101.pt", + "20231101.nl" + ], + "trimmed_trailing_documents": 20, + "trimmed_bytes": 35697, + "cutoff_reason": "Reconstruct the previous published corpus boundary: identical file count and approximate page count; full nominal target differs by only 20 trailing Dutch articles" +} diff --git a/benchmarks/realtime-data-indexing/results/50gb-processes.jsonl.gz b/benchmarks/realtime-data-indexing/results/50gb-processes.jsonl.gz new file mode 100644 index 0000000..235ad24 Binary files /dev/null and b/benchmarks/realtime-data-indexing/results/50gb-processes.jsonl.gz differ diff --git a/benchmarks/realtime-data-indexing/results/50gb-summary.json b/benchmarks/realtime-data-indexing/results/50gb-summary.json new file mode 100644 index 0000000..3dec179 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/50gb-summary.json @@ -0,0 +1,297 @@ +{ + "dataset": "50gb", + "corpus_mb": 50000.0, + "documents": 17083603, + "parsed_documents": 17083603, + "expected_chunks": 37472786, + "verified_exact_chunks": 37472786, + "chunks": 37472786, + "max_observed_chunks": 37538908, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count", + "indexing_seconds": 21696.7, + "throughput_mb_per_min": 138.27, + "peak_rss_mb": 19329.3, + "peak_pss_mb": 15005.0, + "peak_processes": [ + { + "pid": 1, + "worker_rank": null, + "VmHWM": 1026128, + "VmRSS": 1026128, + "Pss": 661645, + "Private_Clean": 192, + "Private_Dirty": 619144, + "mapping_groups": { + "embedding": { + "Pss": 49862, + "private_kb": 19860 + }, + "other": { + "Pss": 611273, + "private_kb": 599476 + } + } + }, + { + "pid": 131, + "worker_rank": null, + "VmHWM": 203192, + "VmRSS": 203192, + "Pss": 127389, + "Private_Clean": 212, + "Private_Dirty": 118460, + "mapping_groups": { + "embedding": { + "Pss": 0, + "private_kb": 0 + }, + "other": { + "Pss": 127180, + "private_kb": 118672 + } + } + }, + { + "pid": 146, + "worker_rank": 0, + "VmHWM": 10140564, + "VmRSS": 8319676, + "Pss": 7821336, + "Private_Clean": 756, + "Private_Dirty": 7757932, + "mapping_groups": { + "embedding": { + "Pss": 64443, + "private_kb": 19900 + }, + "other": { + "Pss": 7756377, + "private_kb": 7738788 + } + } + }, + { + "pid": 147, + "worker_rank": 1, + "VmHWM": 1482860, + "VmRSS": 1455256, + "Pss": 960982, + "Private_Clean": 1212, + "Private_Dirty": 894580, + "mapping_groups": { + "embedding": { + "Pss": 68050, + "private_kb": 20632 + }, + "other": { + "Pss": 892414, + "private_kb": 875160 + } + } + }, + { + "pid": 148, + "worker_rank": 2, + "VmHWM": 1505668, + "VmRSS": 1484532, + "Pss": 985338, + "Private_Clean": 340, + "Private_Dirty": 922124, + "mapping_groups": { + "embedding": { + "Pss": 64921, + "private_kb": 19944 + }, + "other": { + "Pss": 919900, + "private_kb": 902520 + } + } + }, + { + "pid": 149, + "worker_rank": 3, + "VmHWM": 1481272, + "VmRSS": 1463444, + "Pss": 962441, + "Private_Clean": 88, + "Private_Dirty": 900628, + "mapping_groups": { + "embedding": { + "Pss": 64511, + "private_kb": 19868 + }, + "other": { + "Pss": 897418, + "private_kb": 880848 + } + } + }, + { + "pid": 150, + "worker_rank": 4, + "VmHWM": 1486244, + "VmRSS": 1465708, + "Pss": 965231, + "Private_Clean": 216, + "Private_Dirty": 903224, + "mapping_groups": { + "embedding": { + "Pss": 64582, + "private_kb": 19900 + }, + "other": { + "Pss": 900136, + "private_kb": 883540 + } + } + }, + { + "pid": 151, + "worker_rank": 5, + "VmHWM": 1479416, + "VmRSS": 1460408, + "Pss": 963530, + "Private_Clean": 1676, + "Private_Dirty": 896860, + "mapping_groups": { + "embedding": { + "Pss": 67509, + "private_kb": 20060 + }, + "other": { + "Pss": 895499, + "private_kb": 878476 + } + } + }, + { + "pid": 152, + "worker_rank": 6, + "VmHWM": 1492516, + "VmRSS": 1472604, + "Pss": 975371, + "Private_Clean": 1864, + "Private_Dirty": 911452, + "mapping_groups": { + "embedding": { + "Pss": 64378, + "private_kb": 19868 + }, + "other": { + "Pss": 910482, + "private_kb": 893448 + } + } + }, + { + "pid": 153, + "worker_rank": 7, + "VmHWM": 1476656, + "VmRSS": 1442236, + "Pss": 941850, + "Private_Clean": 208, + "Private_Dirty": 879580, + "mapping_groups": { + "embedding": { + "Pss": 64450, + "private_kb": 19844 + }, + "other": { + "Pss": 876887, + "private_kb": 859944 + } + } + } + ], + "peak_elapsed_s": 21837.5939020887, + "jemalloc_conf": "background_thread:true", + "omp_threads": 12, + "indexer_cpu_seconds": 194611.7, + "avg_parallelism": 9.0, + "vector_db_disk_mb": 86481.0, + "qdrant_image_id": "sha256:0ad2e23181e5646b286253c04cea23f07040ca313cc377d9fa7b21db184b6406", + "vector_db_mb_per_corpus_mb": 1.73, + "retrieval_accuracy": "13/20", + "embedder": "sentence-transformers/static-retrieval-mrl-en-v1 (local, 256 dims)", + "approx_text_pages": 26214382, + "accuracy_misses": [ + { + "expected": "0000/6000-Tom-Tango.txt", + "query": "Tom Tango: Tom Tango and \"TangoTiger\" are aliases used online by a baseball sabermetrics and ice hockey statistics analy", + "got": [ + "0546/9191-Tango-music.txt", + "0543/5737-Tango.txt", + "0598/4990-History-of-the-tango.txt", + "0546/9191-Tango-music.txt", + "0295/8960-Histoire-du-Tango.txt" + ] + }, + { + "expected": "0001/6000-Ricky-Fuji.txt", + "query": "Ricky Fuji: (born September 27, 1965) better known under his ring name is a Japanese professional wrestler currently si", + "got": [ + "0592/7826-Kazuyuki-Fujita.txt", + "0369/6383-Sho-wrestler.txt", + "0592/1210-Tatsumi-Fujinami.txt", + "0454/6595-Daiki-Inaba.txt", + "0321/0463-Jeff-Cobb.txt" + ] + }, + { + "expected": "0003/2000-Theroigne-de-Mericourt.txt", + "query": "Theroigne de Mericourt: Anne-Jos\u00e8phe Th\u00e9roigne de M\u00e9ricourt (born Anne-Jos\u00e8phe Terwagne; 13 August 1762 \u2013 8 June 1817) w", + "got": [ + "0966/4911-Coalition-d-Auvergne.txt", + "0023/8794-Jean-Baptiste-Jacques-Augustin.txt", + "0541/4392-List-of-French-language-authors.txt", + "1439/0373-Parlamento-di-Provenza.txt", + "0918/6788-Liste-des-membres-de-l-exp-dition-de-La-P-rouse.txt" + ] + }, + { + "expected": "0003/6000-Black-capped-fruit-bat.txt", + "query": "Black-capped fruit bat: The black-capped fruit bat (Chironax melanocephalus) is a species of megabat in the monotypic ge", + "got": [ + "0551/7828-Leaf-nosed-bat.txt", + "0033/2843-Black-winged-little-yellow-bat.txt", + "0046/2629-Antillean-fruit-eating-bat.txt", + "0100/5548-List-of-bats.txt", + "0585/7845-Jamaican-fruit-bat.txt" + ] + }, + { + "expected": "0005/0000-Clockers-film.txt", + "query": "Clockers (film): Clockers is a 1995 American crime drama film directed by Spike Lee.", + "got": [ + "0143/2934-Clocker.txt", + "0734/5709-Clock.txt", + "0577/8055-The-Big-Clock-film.txt", + "0012/9555-The-Big-Clock.txt", + "0233/9531-Sidewalk-clock-on-Jamaica-Avenue.txt" + ] + }, + { + "expected": "0005/6000-Jamie-Mueller.txt", + "query": "Jamie Mueller: Jamie F.", + "got": [ + "0285/7552-Jamie-disambiguation.txt", + "0000/6975-Jamie.txt", + "0000/6975-Jamie.txt", + "0623/5680-Jamie-Oliver.txt", + "0000/6975-Jamie.txt" + ] + }, + { + "expected": "0005/8000-Ruhleben-internment-camp.txt", + "query": "Ruhleben internment camp: Ruhleben internment camp was a civilian detention camp in Germany during World War I.", + "got": [ + "1390/4284-Ngamiland-Delta.txt", + "0700/8955-Ruhleben-Begriffskl-rung.txt", + "0793/4665-D-hlener-Wetterschacht.txt", + "0624/8269-Ruhleben.txt", + "0674/3640-Friedrich-G-nther-Schwarzburg-Rudolstadt.txt" + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/realtime-data-indexing/results/README.md b/benchmarks/realtime-data-indexing/results/README.md new file mode 100644 index 0000000..df0f7da --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/README.md @@ -0,0 +1,114 @@ +# Indexing benchmark results + +Measured from 2026-10-12T19:00:22.529304+00:00, on 96 logical CPUs. +Exact CPU description, RAM, kernel, image ID and installed package versions: +`run-environment.json`. Resolved compose configurations redact the license key. +The workload, 180-second quiet window, 30-core activity threshold and +approximate-counter timing formula are retained. The current completion +detector also requires all input files to be parsed and, when an expected +count is supplied, verifies the exact output count. Independent splitter +counts match the saved final counts for all six sizes. The saved 50 GB run +additionally records complete parsing progress and an exact Qdrant count; +the five smaller runs record approximate backend counts. +The separate 50 GB audit scanned 17,083,603 frozen input files (52,428,765,709 +bytes) in the original immutable benchmark image and produced exactly +37,472,786 chunks, matching the saved run and exact Qdrant count. +It ran outside the measured indexing workload; memory and timing results +are unchanged. `50gb-expected-output.json` records this independent audit. + +| corpus target | ≈ pages | files | chunks (approx.) | indexing time | peak memory (PSS, GiB) | in Qdrant (GiB) | +|---|---|---|---|---|---|---| +| 100 MB | 52 431 | 12 969 | 53 651 | 44 s | 6.86 | 0.61 | +| 1 GB | 524 288 | 240 516 | 604 299 | 4.9 min | 6.82 | 1.85 | +| 3 GB | 1 572 864 | 841 890 | 1 888 846 | 15.4 min | 7.18 | 4.89 | +| 10 GB | 5 242 884 | 3 423 359 | 6 761 214 | 1.0 h | 8.16 | 16.07 | +| 30 GB | 15 728 646 | 9 202 620 | 20 933 962 | 3.1 h | 10.40 | 48.57 | +| 50 GB | 26 214 382 | 17 083 603 | 37 472 786 | 6.0 h | 14.65 | 84.45 | + +Input labels retain the published nominal sizes: 1 GB means a target of +1,000 MiB of text; 100 MB means 100 MiB. Memory and disk columns use GiB; +CSV columns ending in `_mb` contain MiB. Pages use 2,000 bytes per page. +The 50 GB input restores the historical Portuguese/Dutch continuation and +matches the archived file count and approximate page count. Strictly filling +the nominal byte target adds 20 trailing Dutch articles (35,697 bytes); +these are preserved outside the input to reproduce the published cutoff. +`50gb-preparation.json` records the final boundary and the trimmed bytes +and document count. + +Peaks are the maximum sampled total PSS, not kernel high-water marks. +Sampling continues through the existing completion window. +Only indexer processes are included: Qdrant and API-server memory, unmapped +filesystem cache and kernel memory are outside this measurement. It is not +a RAM requirement for the whole stack. Sampling starts after server readiness; +each iteration includes a three-second sleep plus collection and polling +overhead, so brief peaks may be missed. +The chunk-count methodology is unchanged: the summary uses Qdrant's +approximate count at completion, while CSV `chunks` retains the maximum +observed count used by the completion detector. Segment optimization can +reduce approximate counts, so these two fields need not be identical; +`max_observed_chunks` records the CSV's value in the summary explicitly. + +Published indexing time is an estimate: elapsed time from before launching +the indexer and server to the last increase in the running maximum of the +approximate chunk count. Preparation, model downloads and Qdrant startup are +outside this interval. A temporary overcount can make the last new maximum +precede the final write. Checking the exact output count at completion does +not revise that timestamp or measure when the backend's search index is ready. +The saved timing values retain this definition. + +`*-memory.csv` and `*-processes.jsonl.gz` record aggregate and per-process +measurements at the same times; `*-summary.json` includes the process snapshot +at peak PSS. `*-indexer.txt.gz` contains the local container log and is +excluded from Git. Measurements, derived values, this report and the two +published figures in `docs/assets/` are committed. Per-size `*-memory.png` +plots, the standalone `table.md` and `consistency-check.json` stay local and +are ignored by Git; they can be regenerated from the saved measurements. + +One final run per corpus size is published, without an estimate of run-to-run +variability. File count and text volume both change across sizes, so these +measurements do not establish constant memory at a fixed file count. They +describe this local static-embedding model, plain-text workload and configuration. +The product splitter merges trailing fragments shorter than 100 characters +into the preceding chunk; older published measurements predate that behavior. +Differences in chunk counts, disk usage and timing therefore cannot be +attributed to allocator tuning alone. + +The saved runs do not establish a controlled cold-cache result. Fresh database +and persistence state do not imply a cold operating-system cache. The current +rerun driver reads the entire corpus to count expected chunks before indexing, +which can warm that cache. Saved image IDs and key package versions identify +the measured environment; the build recipe still uses floating image tags +and dependency versions, and model and dataset downloads have no pinned +revision. A future rebuild can use different software or inputs. + +The right panel estimates file-watch memory as rank 0's private-page excess +over the median of the seven other workers at the final sample. It includes +any other unequal worker allocations; it is not an allocation-level measurement. +The line is a least-squares fit with an intercept (~361 bytes per file; +fixed overhead estimate 45.9 MiB). +The pie attributes local-embedding private memory using an eight-process +controlled diagnostic in the same image: graph imports before the embedder, +UTF-8 parser and configured splitter initialized and warmed, embedding +runtime imported, model loaded and warmed on three 2,048-text batches, then +model released. `embedder-calibration.json` records all stages. The private +increment after warming estimates the local stack's private cost across eight +workers. This is capped at the observed non-file-watch worker private memory +minus the calibrated pre-embedder baseline, since live and diagnostic buffers +have different lifetimes. Supervisor private pages are left in the residual; +any embedding allocations there are not subtracted speculatively. +Shared embedding-library PSS starts from mapping samples at the 10 GB peak +and is capped by the diagnostic's shared-library increment, so libraries +already needed by the parser are retained in the residual. Other shared +pages are separated. Anonymous attribution +and the indexer-without-local-embedder figure are estimates, not an actual +external-service benchmark; client overhead and changes in working buffers +are not measured. The embedding service's own memory is outside that figure. +Derived values are saved in `derived.json`. + +Regenerate with `python plot_results.py --update-readme`. +The script checks CSV peaks, chunk counts and process snapshots against summaries +and requires all six sizes before updating the publication. +Run `python verify_results.py` directly from a checkout to audit every process +sample, both published tables and the pie sum. It does not require a benchmark +rerun or regenerated per-size plots. It prints the result and writes a local +`consistency-check.json` with hashes of both published figures. diff --git a/benchmarks/realtime-data-indexing/results/derived.json b/benchmarks/realtime-data-indexing/results/derived.json new file mode 100644 index 0000000..45ea66a --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/derived.json @@ -0,0 +1,105 @@ +{ + "connector_bytes_per_file_estimate": 360.973937266504, + "connector_fixed_overhead_bytes_estimate": 48147350.7307731, + "peak_memory_ratio": 2.134788299567494, + "runs": [ + { + "size": "100mb", + "peak_components": { + "workers_private_mb": 5722.9921875, + "connector_extra_mb_estimate": 98.18359375, + "supervisors_private_mb": 666.4375, + "shared_pss_mb": 541.181640625 + }, + "final_components": { + "workers_private_mb": 5656.12890625, + "connector_extra_mb_estimate": 98.06640625, + "supervisors_private_mb": 666.4375, + "shared_pss_mb": 534.6943359375 + } + }, + { + "size": "1gb", + "peak_components": { + "workers_private_mb": 5574.11328125, + "connector_extra_mb_estimate": 197.2109375, + "supervisors_private_mb": 666.3984375, + "shared_pss_mb": 543.037109375 + }, + "final_components": { + "workers_private_mb": 5541.84765625, + "connector_extra_mb_estimate": 193.98828125, + "supervisors_private_mb": 666.3984375, + "shared_pss_mb": 543.044921875 + } + }, + { + "size": "3gb", + "peak_components": { + "workers_private_mb": 5761.92578125, + "connector_extra_mb_estimate": 330.859375, + "supervisors_private_mb": 715.20703125, + "shared_pss_mb": 543.8564453125 + }, + "final_components": { + "workers_private_mb": 5758.62890625, + "connector_extra_mb_estimate": 291.51171875, + "supervisors_private_mb": 715.20703125, + "shared_pss_mb": 543.8369140625 + } + }, + { + "size": "10gb", + "peak_components": { + "workers_private_mb": 5969.25, + "connector_extra_mb_estimate": 1184.8359375, + "supervisors_private_mb": 666.98828125, + "shared_pss_mb": 537.5595703125 + }, + "final_components": { + "workers_private_mb": 5968.08984375, + "connector_extra_mb_estimate": 1104.76953125, + "supervisors_private_mb": 666.9921875, + "shared_pss_mb": 537.5859375 + } + }, + { + "size": "30gb", + "peak_components": { + "workers_private_mb": 6179.609375, + "connector_extra_mb_estimate": 3275.96484375, + "supervisors_private_mb": 666.48828125, + "shared_pss_mb": 531.7294921875 + }, + "final_components": { + "workers_private_mb": 6090.828125, + "connector_extra_mb_estimate": 3269.671875, + "supervisors_private_mb": 666.48828125, + "shared_pss_mb": 531.7255859375 + } + }, + { + "size": "50gb", + "peak_components": { + "workers_private_mb": 7045.671875, + "connector_extra_mb_estimate": 6697.23828125, + "supervisors_private_mb": 720.7109375, + "shared_pss_mb": 541.3720703125 + }, + "final_components": { + "workers_private_mb": 7044.953125, + "connector_extra_mb_estimate": 5922.1484375, + "supervisors_private_mb": 720.7109375, + "shared_pss_mb": 541.3662109375 + } + } + ], + "10gb_embedder_breakdown": { + "local_embedder_private_mb_estimate": 4713.703125, + "local_embedder_shared_libraries_mb": 291.4873046875, + "file_watch_mb_estimate": 1184.8359375, + "remaining_indexer_private_mb_estimate": 1922.53515625, + "other_shared_mb": 246.072265625 + }, + "10gb_indexer_without_local_embedder_mb_estimate": 3353.4095703125004 +} diff --git a/benchmarks/realtime-data-indexing/results/embedder-calibration.json b/benchmarks/realtime-data-indexing/results/embedder-calibration.json new file mode 100644 index 0000000..9a5ab45 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/embedder-calibration.json @@ -0,0 +1,211 @@ +{ + "description": "Diagnostic: eight workers, before/after local embedder initialization.\n\nRun between indexing sizes, in the same image and environment. This does\nnot replace the indexing benchmark. Anonymous local-embedding memory is\nestimated by the private-page increment; shared code comes from main-run\nmapping samples. Keep workers alive at each barrier to count shared pages\nproportionally across all eight rather than multiplying a single RSS value.\n", + "workers": 8, + "baseline_includes": "graph imports, UTF-8 parser, initialized and warmed configured splitter", + "jemalloc_conf": "background_thread:true", + "omp_threads": "12", + "stages": { + "baseline": [ + { + "Pss": 168251, + "Private_Clean": 116, + "Private_Dirty": 155880, + "embedding_shared_kb": 0 + }, + { + "Pss": 168292, + "Private_Clean": 124, + "Private_Dirty": 155852, + "embedding_shared_kb": 0 + }, + { + "Pss": 175644, + "Private_Clean": 28, + "Private_Dirty": 163648, + "embedding_shared_kb": 0 + }, + { + "Pss": 168493, + "Private_Clean": 600, + "Private_Dirty": 155820, + "embedding_shared_kb": 0 + }, + { + "Pss": 176026, + "Private_Clean": 432, + "Private_Dirty": 163684, + "embedding_shared_kb": 0 + }, + { + "Pss": 179083, + "Private_Clean": 84, + "Private_Dirty": 166604, + "embedding_shared_kb": 0 + }, + { + "Pss": 178026, + "Private_Clean": 208, + "Private_Dirty": 165684, + "embedding_shared_kb": 0 + }, + { + "Pss": 169489, + "Private_Clean": 192, + "Private_Dirty": 156724, + "embedding_shared_kb": 0 + } + ], + "runtime": [ + { + "Pss": 634314, + "Private_Clean": 152, + "Private_Dirty": 582752, + "embedding_shared_kb": 35649 + }, + { + "Pss": 635180, + "Private_Clean": 244, + "Private_Dirty": 582880, + "embedding_shared_kb": 36014 + }, + { + "Pss": 642793, + "Private_Clean": 364, + "Private_Dirty": 590808, + "embedding_shared_kb": 35983 + }, + { + "Pss": 635196, + "Private_Clean": 1248, + "Private_Dirty": 582796, + "embedding_shared_kb": 35603 + }, + { + "Pss": 644433, + "Private_Clean": 1680, + "Private_Dirty": 590836, + "embedding_shared_kb": 36432 + }, + { + "Pss": 648148, + "Private_Clean": 524, + "Private_Dirty": 594764, + "embedding_shared_kb": 36816 + }, + { + "Pss": 644526, + "Private_Clean": 280, + "Private_Dirty": 592572, + "embedding_shared_kb": 35966 + }, + { + "Pss": 636337, + "Private_Clean": 372, + "Private_Dirty": 584656, + "embedding_shared_kb": 35315 + } + ], + "model_warm": [ + { + "Pss": 887400, + "Private_Clean": 144, + "Private_Dirty": 834484, + "embedding_shared_kb": 36982 + }, + { + "Pss": 896661, + "Private_Clean": 244, + "Private_Dirty": 843052, + "embedding_shared_kb": 37323 + }, + { + "Pss": 883749, + "Private_Clean": 372, + "Private_Dirty": 830436, + "embedding_shared_kb": 37296 + }, + { + "Pss": 886815, + "Private_Clean": 1272, + "Private_Dirty": 833000, + "embedding_shared_kb": 36972 + }, + { + "Pss": 901423, + "Private_Clean": 1688, + "Private_Dirty": 846448, + "embedding_shared_kb": 37781 + }, + { + "Pss": 903614, + "Private_Clean": 536, + "Private_Dirty": 848828, + "embedding_shared_kb": 38201 + }, + { + "Pss": 884094, + "Private_Clean": 280, + "Private_Dirty": 830808, + "embedding_shared_kb": 37275 + }, + { + "Pss": 889771, + "Private_Clean": 412, + "Private_Dirty": 836676, + "embedding_shared_kb": 36653 + } + ], + "model_released": [ + { + "Pss": 887737, + "Private_Clean": 144, + "Private_Dirty": 834908, + "embedding_shared_kb": 36895 + }, + { + "Pss": 895914, + "Private_Clean": 244, + "Private_Dirty": 842392, + "embedding_shared_kb": 37236 + }, + { + "Pss": 883947, + "Private_Clean": 364, + "Private_Dirty": 830748, + "embedding_shared_kb": 37191 + }, + { + "Pss": 887083, + "Private_Clean": 1248, + "Private_Dirty": 833428, + "embedding_shared_kb": 36836 + }, + { + "Pss": 901629, + "Private_Clean": 1624, + "Private_Dirty": 846836, + "embedding_shared_kb": 37663 + }, + { + "Pss": 902465, + "Private_Clean": 528, + "Private_Dirty": 847832, + "embedding_shared_kb": 38057 + }, + { + "Pss": 884231, + "Private_Clean": 280, + "Private_Dirty": 831032, + "embedding_shared_kb": 37188 + }, + { + "Pss": 888996, + "Private_Clean": 408, + "Private_Dirty": 836000, + "embedding_shared_kb": 36559 + } + ] + }, + "image": "sha256:84b80c81426ebd36d7a1b9635d536c1ed8d05df4079e199f9924d3dbe4839c81", + "completed_utc": "2026-10-13T00:20:28.105584+00:00" +} diff --git a/benchmarks/realtime-data-indexing/results/run-environment.json b/benchmarks/realtime-data-indexing/results/run-environment.json new file mode 100644 index 0000000..2bc1265 --- /dev/null +++ b/benchmarks/realtime-data-indexing/results/run-environment.json @@ -0,0 +1,24 @@ +{ + "started_utc": "2026-10-12T19:00:22.529304+00:00", + "git_commit": "6e219870e381c73c98dd15995caf88bbc5cfa6f7", + "cpu": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 43 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7402 24-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 2\nCore(s) per socket: 24\nSocket(s): 2\nStepping: 0\nFrequency boost: enabled\nCPU(s) scaling MHz: 99%\nCPU max MHz: 2800.0000\nCPU min MHz: 1500.0000\nBogoMIPS: 5589.53\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es\nVirtualization: AMD-V\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 256 MiB (16 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23,48-71\nNUMA node1 CPU(s): 24-47,72-95\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected", + "logical_cpus": 96, + "memory": "MemTotal: 2100643220 kB\nMemFree: 677079912 kB\nMemAvailable: 1182867364 kB\nBuffers: 1240 kB\nCached: 376573736 kB\nSwapCached: 1183132 kB\nActive: 140373948 kB\nInactive: 1031714416 kB\nActive(anon): 3448152 kB\nInactive(anon): 810799672 kB\nActive(file): 136925796 kB\nInactive(file): 220914744 kB\nUnevictable: 272780 kB\nMlocked: 272780 kB\nSwapTotal: 4194300 kB\nSwapFree: 0 kB\nDirty: 904 kB\nWriteback: 0 kB\nAnonPages: 785781864 kB\nMapped: 19469476 kB\nShmem: 18579620 kB\nKReclaimable: 161692716 kB\nSlab: 214152920 kB\nSReclaimable: 161692716 kB\nSUnreclaim: 52460204 kB\nKernelStack: 428944 kB\nPageTables: 2708320 kB\nNFS_Unstable: 0 kB\nBounce: 0 kB\nWritebackTmp: 0 kB\nCommitLimit: 1054515908 kB\nCommitted_AS: 1347338332 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed: 1733276 kB\nVmallocChunk: 0 kB\nPercpu: 17190912 kB\nHardwareCorrupted: 0 kB\nAnonHugePages: 183640064 kB\nShmemHugePages: 0 kB\nShmemPmdMapped: 0 kB\nFileHugePages: 0 kB\nFilePmdMapped: 0 kB\nCmaTotal: 0 kB\nCmaFree: 0 kB\nHugePages_Total: 0\nHugePages_Free: 0\nHugePages_Rsvd: 0\nHugePages_Surp: 0\nHugepagesize: 2048 kB\nHugetlb: 0 kB\nDirectMap4k: 770803372 kB\nDirectMap2M: 1289414656 kB\nDirectMap1G: 75497472 kB\n", + "kernel": "5.14.0-160.el9.x86_64", + "docker": "Docker version 29.2.1, build a5c7197", + "versions": { + "serviette": "0.1.7", + "pathway": "0.33.0", + "torch": "2.14.1", + "sentence-transformers": "6.1.0", + "qdrant-client": "1.19.1" + }, + "image": "sha256:84b80c81426ebd36d7a1b9635d536c1ed8d05df4079e199f9924d3dbe4839c81", + "environment": { + "BENCH_DATA_ROOT": "/mnt/scratch/sergey/serviette-consistency-20261012/datasets", + "BENCH_JEMALLOC_CONF": "background_thread:true", + "BENCH_OMP_THREADS": "12", + "COMPOSE_PROJECT_NAME": "serviette-consistency-bench" + }, + "qdrant_image": "sha256:0ad2e23181e5646b286253c04cea23f07040ca313cc377d9fa7b21db184b6406" +} diff --git a/benchmarks/realtime-data-indexing/run_bench.py b/benchmarks/realtime-data-indexing/run_bench.py old mode 100644 new mode 100755 index e56478d..b3e50d3 --- a/benchmarks/realtime-data-indexing/run_bench.py +++ b/benchmarks/realtime-data-indexing/run_bench.py @@ -1,10 +1,9 @@ #!/usr/bin/env python3 """Run the real-time indexing benchmark and produce the memory report. -Flow: start Qdrant → start the serviette indexer (static run over the dataset) -and the serviette server → sample the indexer's memory (VmRSS + kernel-tracked -peak VmHWM from /proc/1/status inside the container) every few seconds until -it finishes → replay the accuracy questions through the server's /api/v1 +Flow: start Qdrant → start the streaming serviette indexer +and server → sample total RSS/PSS and per-process memory every few seconds +until indexing is complete → replay accuracy questions through /api/v1 retrieve endpoint → write a memory-over-time plot, a CSV, and a JSON summary into ./results. @@ -15,14 +14,109 @@ from __future__ import annotations import argparse +import gzip import json import subprocess +import threading import time from pathlib import Path import httpx ROOT = Path(__file__).parent + + +class _ParsingProgress: + """Observe cold-run input progress without entering the measured container.""" + + def __init__(self, container): + self.container = container + self.parsed_documents = 0 + self.failed_documents = 0 + self.last_change = 0.0 + self.reader_error = None + + def _observe(self, line): + if b'INFO:serviette.indexer.graph:parsed ' in line: + self.parsed_documents += 1 + self.last_change = time.monotonic() + elif b'document is indexed as EMPTY' in line: + self.failed_documents += 1 + + def _read(self): + try: + for line in self.process.stdout: + self._observe(line) + except (OSError, ValueError) as exc: + self.reader_error = str(exc) + + def __enter__(self): + self.process = subprocess.Popen( + ['docker', 'logs', '--follow', self.container], + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + self.reader = threading.Thread(target=self._read, daemon=True) + self.reader.start() + return self + + def __exit__(self, *_): + self.process.terminate() + try: + self.process.wait(timeout=10) + except subprocess.TimeoutExpired: + self.process.kill() + self.process.wait() + self.reader.join(timeout=10) + self.process.stdout.close() + + +def _ready_to_finish(*, documents, parsed_documents, chunks, last_change, + last_busy, now, stats_available): + # A chunk-rich subset can exceed the number of input files. Require every + # file to have reached parsing before applying the existing quiet window. + return (stats_available and parsed_documents >= documents and chunks >= documents + and now - last_change >= STABLE_WINDOW + and now - last_busy >= STABLE_WINDOW) + +# Observe the same processes as the aggregate sampler, with worker ranks so +# chart components can be traced to measurements. Never record full environs +# or command lines: they may contain credentials. +PROC_SAMPLE = r''' +import json, os +from pathlib import Path +rows = [] +for proc in Path('/proc').iterdir(): + if not proc.name.isdigit() or int(proc.name) == os.getpid(): + continue + try: + env = dict(item.split(b'=', 1) for item in (proc / 'environ').read_bytes().split(b'\0') if b'=' in item) + row = {'pid': int(proc.name), 'worker_rank': int(env[b'PATHWAY_PROCESS_ID']) if b'PATHWAY_PROCESS_ID' in env else None} + for filename in ('status', 'smaps_rollup'): + for line in (proc / filename).read_text().splitlines(): + if line.startswith(('VmRSS:', 'VmHWM:', 'Pss:', 'Private_Clean:', 'Private_Dirty:')): + key, value = line.split(':', 1) + row[key] = int(value.split()[0]) + # File mappings identify shared local-embedding libraries. Anonymous + # allocations require a separate controlled initialization measurement. + groups = {'embedding': {'Pss': 0, 'private_kb': 0}, + 'other': {'Pss': 0, 'private_kb': 0}} + group = 'other' + for line in (proc / 'smaps').read_text().splitlines(): + parts = line.split() + if parts and '-' in parts[0] and ':' not in parts[0]: + path = parts[-1] if len(parts) >= 6 else '' + group = 'embedding' if any(part in path for part in + ('/torch/', '/nvidia/', '/triton/', '/sentence_transformers/', '/tokenizers/', '/safetensors/', + '/models--sentence-transformers--static-retrieval-mrl-en-v1/')) else 'other' + elif line.startswith('Pss:'): + groups[group]['Pss'] += int(parts[1]) + elif line.startswith(('Private_Clean:', 'Private_Dirty:')): + groups[group]['private_kb'] += int(parts[1]) + row['mapping_groups'] = groups + rows.append(row) + except (OSError, ValueError): + continue +print(json.dumps(rows)) +''' SAMPLE_INTERVAL = 3.0 # Streaming never exits; indexing is "done" when BOTH hold for this long: # the chunk counter stopped growing AND the indexer's CPU dropped to idle. @@ -59,7 +153,7 @@ def _container_id(size: str, service: str) -> str: return out -def _proc_status(container: str) -> dict[str, int] | None: +def _proc_status(container: str) -> dict | None: """VmRSS/VmHWM (kB) summed over ALL processes in the container. With ``indexer.workers`` > 1 the container holds the spawn parent plus N @@ -70,10 +164,10 @@ def _proc_status(container: str) -> dict[str, int] | None: """ proc = subprocess.run( - ["docker", "exec", container, "sh", "-c", - "cat /proc/[0-9]*/status /proc/[0-9]*/smaps_rollup 2>/dev/null"], + ["docker", "exec", container, "python", "-c", PROC_SAMPLE], capture_output=True, text=True, + check=False, ) if proc.returncode != 0: return None @@ -81,10 +175,11 @@ def _proc_status(container: str) -> dict[str, int] | None: # worker) proportionally, so its sum is the honest physical footprint; # the VmRSS sum counts each shared page once per process. values = {"VmRSS": 0, "VmHWM": 0, "Pss": 0} - for line in proc.stdout.splitlines(): - if line.startswith(("VmRSS:", "VmHWM:", "Pss:")): - key, rest = line.split(":", 1) - values[key] += int(rest.strip().split()[0]) + rows = json.loads(proc.stdout) + for row in rows: + for key in values: + values[key] += row.get(key, 0) + values["processes"] = rows return values @@ -95,6 +190,7 @@ def _proc_cpu_seconds(container: str) -> float | None: ["docker", "exec", container, "sh", "-c", "cat /proc/[0-9]*/stat"], capture_output=True, text=True, + check=False, ) if proc.returncode != 0: return None @@ -196,6 +292,8 @@ def main() -> None: parser.add_argument("--server-port", type=int, default=8300) parser.add_argument("--qdrant-port", type=int, default=8301) parser.add_argument("--keep-up", action="store_true", help="don't docker compose down") + parser.add_argument('--expected-chunks', type=int, + help='Optional exact output count for a frozen corpus; validates completeness') args = parser.parse_args() import os as _os @@ -207,6 +305,16 @@ def main() -> None: questions = json.loads((dataset / "questions.json").read_text()) results_dir = ROOT / "results" results_dir.mkdir(exist_ok=True) + # Save the resolved setup without the license secret. + compose_config = _compose("config", "--format", "json", size=args.size, capture=True) + if compose_config.returncode: + raise SystemExit(compose_config.stderr) + resolved = json.loads(compose_config.stdout) + for service in resolved["services"].values(): + env = service.get("environment", {}) + if "PATHWAY_LICENSE_KEY" in env: + env["PATHWAY_LICENSE_KEY"] = "" + (results_dir / f"{args.size}-compose.json").write_text(json.dumps(resolved, indent=2)) import os @@ -222,7 +330,8 @@ def main() -> None: # Pre-warm the HF model cache on the host: containers run HF_HUB_OFFLINE. import yaml as _yaml - model = _yaml.safe_load((ROOT / "config.yaml").read_text())["embedder"].get( + bench_config = _yaml.safe_load((ROOT / "config.yaml").read_text()) + model = bench_config["embedder"].get( "model" ) if model: @@ -252,12 +361,14 @@ def main() -> None: base = f"http://127.0.0.1:{args.server_port}" _wait_http(f"{base}/api/v1/health", 180, "server not ready") - def _chunks() -> int: + def _chunks() -> int | None: try: - value = httpx.get(f"{base}/api/v1/stats", timeout=10).json().get("chunks") - return int(value) if value is not None else 0 + response = httpx.get(f"{base}/api/v1/stats", timeout=10) + response.raise_for_status() + value = response.json().get("chunks") + return int(value) if value is not None else None except (httpx.HTTPError, ValueError): - return 0 + return None # The indexer runs in its product mode — STREAMING — and never exits, so # "indexing finished" is detected by the chunk counter going stable: it @@ -266,61 +377,92 @@ def _chunks() -> int: # The CSV is appended live, so an interrupted run keeps its memory curve. csv_path = results_dir / f"{args.size}-memory.csv" csv_path.write_text("elapsed_s,rss_mb,hwm_mb,pss_mb,chunks\n") - - samples: list[tuple[float, int, int, int, int]] = [] # (s, rss, hwm, pss, chunks) - last_count = 0 - last_change = started - last_busy = started - prev_cpu: float | None = None - prev_time = started - cpu_seconds: float | None = None - while True: - status = _proc_status(indexer) - if status is None: - _compose("logs", "--tail", "50", "indexer", size=args.size) - raise SystemExit("indexer container exited unexpectedly") - now = time.monotonic() - count = _chunks() - if count > last_count: - last_count = count - last_change = now - cpu = _proc_cpu_seconds(indexer) - if cpu is not None: - if prev_cpu is not None and now > prev_time: - if (cpu - prev_cpu) / (now - prev_time) >= IDLE_CORES: + process_path = results_dir / f"{args.size}-processes.jsonl.gz" + peak_processes = [] + peak_pss_kb = -1 + verified_exact_count = None + with _ParsingProgress(indexer) as parsing, gzip.open(process_path, "wt") as process_log: + + samples: list[tuple[float, int, int, int, int]] = [] # (s, rss, hwm, pss, chunks) + last_count = 0 + last_change = started + last_busy = started + prev_cpu: float | None = None + prev_time = started + cpu_seconds: float | None = None + while True: + status = _proc_status(indexer) + if status is None: + _compose("logs", "--tail", "50", "indexer", size=args.size) + raise SystemExit("indexer container exited unexpectedly") + now = time.monotonic() + count = _chunks() + if parsing.failed_documents or parsing.reader_error: + raise RuntimeError('Input parsing failed or its progress could not be observed') + if parsing.process.poll() is not None: + raise RuntimeError('Input progress log stream ended unexpectedly') + if count is None: + # An unavailable stats endpoint is not evidence of inactivity. + last_busy = now + elif count > last_count: + last_count = count + last_change = now + cpu = _proc_cpu_seconds(indexer) + if cpu is not None: + if (prev_cpu is not None and now > prev_time + and (cpu - prev_cpu) / (now - prev_time) >= IDLE_CORES): last_busy = now - prev_cpu, prev_time, cpu_seconds = cpu, now, cpu - elapsed = now - started - sample = ( - elapsed, - status.get("VmRSS", 0), - status.get("VmHWM", 0), - status.get("Pss", 0), - last_count, - ) - samples.append(sample) - with csv_path.open("a") as fh: - fh.write( - f"{sample[0]:.1f},{sample[1] / 1024:.1f},{sample[2] / 1024:.1f}," - f"{sample[3] / 1024:.1f},{sample[4]}\n" + prev_cpu, prev_time, cpu_seconds = cpu, now, cpu + last_busy = max(last_busy, parsing.last_change) + elapsed = now - started + sample = ( + elapsed, + status.get("VmRSS", 0), + status.get("VmHWM", 0), + status.get("Pss", 0), + last_count, ) - print( - f"\r t={elapsed:7.0f}s rss={status.get('VmRSS', 0) / 1024:7.1f} MB " - f"peak={status.get('VmHWM', 0) / 1024:7.1f} MB chunks={last_count}", - end="", - flush=True, - ) - if ( - # Every document yields at least one chunk, so completion is - # impossible before the counter reaches the document count — - # this cleanly excludes the low-CPU initial-scan phase. - last_count >= docs_count - and now - last_change >= STABLE_WINDOW - and now - last_busy >= STABLE_WINDOW - ): - break - time.sleep(SAMPLE_INTERVAL) - print() + samples.append(sample) + process_log.write(json.dumps({"elapsed_s": elapsed, "processes": status["processes"], + "parsed_documents": parsing.parsed_documents, + "observed_chunks": count}) + "\n") + process_log.flush() + if sample[3] > peak_pss_kb: + peak_pss_kb = sample[3] + peak_processes = status["processes"] + with csv_path.open("a") as fh: + fh.write( + f"{sample[0]:.1f},{sample[1] / 1024:.1f},{sample[2] / 1024:.1f}," + f"{sample[3] / 1024:.1f},{sample[4]}\n" + ) + print( + f"\r t={elapsed:7.0f}s rss={status.get('VmRSS', 0) / 1024:7.1f} MB " + f"peak={status.get('VmHWM', 0) / 1024:7.1f} MB chunks={last_count} " + f"parsed={parsing.parsed_documents}/{docs_count}", + end="", + flush=True, + ) + if _ready_to_finish(documents=docs_count, parsed_documents=parsing.parsed_documents, + chunks=last_count, last_change=last_change, last_busy=last_busy, + now=now, stats_available=count is not None): + if args.expected_chunks is not None: + collection = bench_config['vector_db']['collection'] + response = httpx.post( + f'http://127.0.0.1:{args.qdrant_port}/collections/{collection}/points/count', + json={'exact': True}, timeout=120) + response.raise_for_status() + verified_exact_count = response.json()['result']['count'] + if verified_exact_count > args.expected_chunks: + raise RuntimeError('Output exceeds the expected frozen-corpus point count') + if verified_exact_count < args.expected_chunks: + last_busy = time.monotonic() + time.sleep(SAMPLE_INTERVAL) + continue + (results_dir / f"{args.size}-postrun-qdrant.json").write_text( + json.dumps({"exact_count": response.json()}, indent=2) + "\n") + break + time.sleep(SAMPLE_INTERVAL) + print() duration = last_change - started # Vector-DB footprint: on-disk storage inside the qdrant container. @@ -329,6 +471,7 @@ def _chunks() -> int: ["docker", "exec", qdrant, "du", "-sb", "/qdrant/storage"], capture_output=True, text=True, + check=False, ) db_bytes = int(du.stdout.split()[0]) if du.returncode == 0 and du.stdout else None @@ -346,15 +489,27 @@ def _chunks() -> int: "dataset": args.size, "corpus_mb": round(size_mb, 1), "documents": docs_count, + "parsed_documents": parsing.parsed_documents, + "expected_chunks": args.expected_chunks, + "verified_exact_chunks": verified_exact_count, "chunks": stats.get("chunks"), + "max_observed_chunks": last_count, + "chunk_count_metric": "Qdrant approximate count at completion; CSV chunks is the maximum observed count", "indexing_seconds": round(duration, 1), "throughput_mb_per_min": round(size_mb / (duration / 60), 2), "peak_rss_mb": round(peak_mb, 1), # Honest physical peak: shared pages divided among processes. "peak_pss_mb": round(peak_pss_mb, 1), + "peak_processes": peak_processes, + "peak_elapsed_s": max(samples, key=lambda s: s[3])[0], + "jemalloc_conf": os.environ.get("BENCH_JEMALLOC_CONF", ""), + "omp_threads": int(os.environ.get("BENCH_OMP_THREADS", "12")), "indexer_cpu_seconds": round(cpu_seconds, 1) if cpu_seconds else None, "avg_parallelism": round(cpu_seconds / duration, 1) if cpu_seconds else None, "vector_db_disk_mb": round(db_bytes / 1024 / 1024, 1) if db_bytes else None, + "qdrant_image_id": subprocess.check_output( + ["docker", "inspect", "--format", "{{.Image}}", qdrant], text=True + ).strip(), "vector_db_mb_per_corpus_mb": ( round(db_bytes / 1024 / 1024 / size_mb, 2) if db_bytes else None ), @@ -374,14 +529,14 @@ def _chunks() -> int: fig, ax = plt.subplots(figsize=(9, 4.5)) minutes = [s[0] / 60 for s in samples] - ax.plot(minutes, [s[1] / 1024 for s in samples], label="indexer RSS", linewidth=1.8) - ax.axhline(peak_mb, linestyle="--", linewidth=1, alpha=0.6, label=f"peak {peak_mb:.0f} MB") + ax.plot(minutes, [s[3] / 1024**2 for s in samples], label="indexer PSS", linewidth=1.8) + ax.axhline(peak_pss_mb / 1024, linestyle="--", linewidth=1, alpha=0.6, label=f"peak {peak_pss_mb / 1024:.2f} GiB") ax.set_xlabel("minutes since start") - ax.set_ylabel("memory, MB") + ax.set_ylabel("memory (PSS), GiB") ax.set_ylim(bottom=0) ax2 = ax.twinx() - ax2.plot(minutes, [s[4] for s in samples], color="tab:green", alpha=0.6, linewidth=1.4, label="chunks in DB") - ax2.set_ylabel("chunks indexed") + ax2.plot(minutes, [s[4] for s in samples], color="tab:green", alpha=0.6, linewidth=1.4, label="maximum observed chunks") + ax2.set_ylabel("maximum observed chunks") ax2.set_ylim(bottom=0) ax.set_title(f"serviette indexer — {args.size} corpus ({docs_count} docs, streaming)") lines, labels = ax.get_legend_handles_labels() @@ -395,6 +550,11 @@ def _chunks() -> int: print(json.dumps(summary, indent=2)) print(f"plot: {plot_path}\ncsv: {csv_path}") + logs = _compose("logs", "--no-color", "indexer", size=args.size, capture=True) + with gzip.open(results_dir / f"{args.size}-indexer.txt.gz", "wt") as fh: + fh.write(logs.stdout) + fh.write(logs.stderr) + if not args.keep_up: _compose("down", "-v", size=args.size) diff --git a/benchmarks/realtime-data-indexing/test_completion_guard.py b/benchmarks/realtime-data-indexing/test_completion_guard.py new file mode 100644 index 0000000..3bb9aa0 --- /dev/null +++ b/benchmarks/realtime-data-indexing/test_completion_guard.py @@ -0,0 +1,38 @@ +"""Regression checks for premature completion on chunk-rich input subsets.""" +import unittest + +from run_bench import _ParsingProgress, _ready_to_finish + + +class CompletionGuardTests(unittest.TestCase): + def ready(self, **changes): + values = {"documents": 17083603, "parsed_documents": 17083603, + "chunks": 37472786, "last_change": 100, "last_busy": 100, + "now": 281, "stats_available": True} + values.update(changes) + return _ready_to_finish(**values) + + def test_chunk_rich_subset_is_incomplete(self): + self.assertFalse(self.ready(parsed_documents=13889356, chunks=31535316)) + + def test_complete_quiet_output(self): + self.assertTrue(self.ready()) + + def test_unavailable_stats_and_recent_activity(self): + self.assertFalse(self.ready(stats_available=False)) + self.assertFalse(self.ready(last_busy=200)) + self.assertFalse(self.ready(last_change=200)) + + def test_parsing_progress(self): + progress = _ParsingProgress("unused") + progress._observe(b"INFO:serviette.indexer.graph:parsing doc (1 KB)") + self.assertEqual(progress.parsed_documents, 0) + progress._observe(b"INFO:serviette.indexer.graph:parsed doc: 1000 chars in 0.0s") + self.assertEqual(progress.parsed_documents, 1) + self.assertGreater(progress.last_change, 0) + progress._observe(b"ERROR:document is indexed as EMPTY") + self.assertEqual(progress.failed_documents, 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/realtime-data-indexing/test_rerun_all.py b/benchmarks/realtime-data-indexing/test_rerun_all.py new file mode 100644 index 0000000..32f587f --- /dev/null +++ b/benchmarks/realtime-data-indexing/test_rerun_all.py @@ -0,0 +1,59 @@ +"""Exercise publication orchestration without running the multi-hour workload.""" +import contextlib +import io +import json +import os +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +import rerun_all + + +class RerunTests(unittest.TestCase): + def test_reused_corpora_are_counted_before_indexing(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + datasets = root / "datasets" + for size, _ in rerun_all.SIZES: + corpus = datasets / size + corpus.mkdir(parents=True) + (corpus / "questions.json").write_text("[]") + calls = [] + + def output(command, **kwargs): + if "import importlib.metadata" in command[-1]: + return '{}' + return "test-image" + + def run(command, **kwargs): + calls.append(command) + if "/bench/count_expected_chunks.py" in command: + size = command[-1] + (root / "results" / f"{size}-expected-output.json").write_text( + json.dumps({"documents": 2, "exact_chunks": 7})) + elif str(root / "run_bench.py") in command: + size = command[command.index("--size") + 1] + self.assertEqual(command[-2:], ["--expected-chunks", "7"]) + self.assertIn("image", json.loads( + (root / "results" / f"{size}-expected-output.json").read_text())) + (root / "results" / f"{size}-summary.json").write_text( + json.dumps({"chunks": 7, "documents": 2, "peak_pss_mb": 10})) + return SimpleNamespace(stdout='{}', stderr='') + + with patch.object(rerun_all, "ROOT", root), patch.dict(os.environ, { + "PATHWAY_LICENSE_KEY": "test", "BENCH_JEMALLOC_CONF": "background_thread:true", + "BENCH_DATA_ROOT": str(datasets), + }), patch.object(rerun_all.subprocess, "check_output", side_effect=output), \ + patch.object(rerun_all.subprocess, "run", side_effect=run), \ + contextlib.redirect_stdout(io.StringIO()): + rerun_all.main() + self.assertFalse(any(str(root / "prepare_dataset.py") in c for c in calls)) + self.assertEqual(sum('/bench/count_expected_chunks.py' in c for c in calls), 6) + self.assertIn(str(root / "plot_results.py"), calls[-1]) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/realtime-data-indexing/verify_results.py b/benchmarks/realtime-data-indexing/verify_results.py new file mode 100644 index 0000000..1768b66 --- /dev/null +++ b/benchmarks/realtime-data-indexing/verify_results.py @@ -0,0 +1,77 @@ +"""Audit published tables against every saved memory/process sample.""" +import csv +import gzip +import hashlib +import json + +from plot_results import REPO, ROOT, SIZES, load, table + + +def verify(): + results = ROOT / "results" + runs = load(results) + published = table(runs) + for path in [REPO / "README.md", results / "README.md"]: + assert published in path.read_text(), f"Different table in {path}" + audited = [] + for run in runs: + size, summary = run["size"], run["summary"] + with (results / f"{size}-memory.csv").open() as stream: + rows = list(csv.DictReader(stream)) + previous_time = previous_chunks = -1 + last_increase = peak = 0 + peak_time = 0 + with gzip.open(results / f"{size}-processes.jsonl.gz", "rt") as stream: + count = 0 + for row in rows: + line = stream.readline() + assert line, f"{size}: missing process sample" + sample = json.loads(line) + elapsed, chunks = float(row["elapsed_s"]), int(row["chunks"]) + assert elapsed > previous_time, f"{size}: nonmonotonic sample time" + assert chunks >= previous_chunks, f"{size}: decreasing high-water count" + assert abs(elapsed - sample["elapsed_s"]) <= 0.11 + for column, metric in [("pss_mb", "Pss"), ("rss_mb", "VmRSS")]: + measured = sum(p.get(metric, 0) for p in sample["processes"]) / 1024 + assert abs(float(row[column]) - measured) <= 0.11, (size, column, elapsed) + pss = sum(p.get("Pss", 0) for p in sample["processes"]) / 1024 + if pss > peak: + peak, peak_time = pss, elapsed + if chunks > previous_chunks and chunks > 0: + last_increase = elapsed + previous_time, previous_chunks = elapsed, chunks + count += 1 + assert not stream.readline(), f"{size}: extra process samples" + assert abs(peak - summary["peak_pss_mb"]) <= 0.11 + assert abs(peak_time - summary["peak_elapsed_s"]) <= 0.11 + assert abs(last_increase - summary["indexing_seconds"]) <= 0.11 + assert abs(sum(run["peak_components"].values()) - peak) <= 0.11 + audited.append({"size": size, "samples_checked": count, + "peak_pss_gib": summary["peak_pss_mb"] / 1024, + "last_chunk_increase_seconds": last_increase}) + expected = json.loads((results / "50gb-expected-output.json").read_text()) + backend = json.loads((results / "50gb-postrun-qdrant.json").read_text()) + assert backend["exact_count"]["result"]["count"] == expected["exact_chunks"] + derived = json.loads((results / "derived.json").read_text()) + ten = next(r for r in runs if r["size"] == "10gb")["summary"] + components = derived["10gb_embedder_breakdown"] + assert abs(sum(components.values()) - ten["peak_pss_mb"]) <= 0.11 + remaining = (ten["peak_pss_mb"] - components["local_embedder_private_mb_estimate"] + - components["local_embedder_shared_libraries_mb"]) + assert remaining == derived["10gb_indexer_without_local_embedder_mb_estimate"] + assert derived["peak_memory_ratio"] == runs[-1]["summary"]["peak_pss_mb"] / runs[0]["summary"]["peak_pss_mb"] + assets = [REPO / "docs/assets" / name for name in ["bench-memory.png", "bench-memory-breakdown.png"]] + readme = (REPO / "README.md").read_text() + for path in assets: + assert f'src="docs/assets/{path.name}"' in readme + report = {"status": "passed", "sizes": len(SIZES), "runs": audited, + "tables_identical": True, "all_process_samples_match_csv": True, + "pie_components_match_measured_peak": True, + "external_embedder_indexer_gib_estimate": remaining / 1024, + "figure_sha256": {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in assets}} + (results / "consistency-check.json").write_text(json.dumps(report, indent=2) + "\n") + print(json.dumps(report, indent=2)) + + +if __name__ == "__main__": + verify() diff --git a/docs/README.md b/docs/README.md index f0d1d74..f6d479b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -248,7 +248,7 @@ string, which would only surface later as a provider 401). | `splitter.chunk_size` | int | `512` | indexer | Max chunk size (tokens). | | `splitter.chunk_overlap` | int | `50` | indexer | Overlap (used by `recursive`). | | `indexer.fetch_retries` | int | `3` | indexer | Retries (exponential backoff from 1s) of a document's byte fetch before it is given up on: indexed as empty and ERROR-logged. Remote sources fail transiently under bulk backfills. | -| `indexer.workers` | int | `1` | indexer | Worker **processes** (sharded via `pathway spawn`). Raise for large backfills — each worker carries its own embedding stack (~1 GB with local embeddings); the benchmarks run with `8`. | +| `indexer.workers` | int | `1` | indexer | Worker **processes** (sharded via `pathway spawn`). Raise for large backfills — each worker carries its own local embedding stack; memory depends on the model, and library pages may be shared. The benchmarks run with `8`. | | `workdir_path` | str | `./serviette-workdir` | indexer | serviette's working directory — everything it keeps between runs: the configuration fingerprint at its top level, the Pathway persistence state (and the parse cache, `runtime_calls/`) under `persistence/`. Relative to the process's working directory. Silent default — the wizard does not ask. | | `persistence.enabled` | bool | `true` | indexer | See [Persistence](#5-persistence). | | `persistence.backend` | `filesystem` | `filesystem` | indexer | Persistence backend. | diff --git a/docs/assets/bench-memory-breakdown.png b/docs/assets/bench-memory-breakdown.png index a684418..9944020 100644 Binary files a/docs/assets/bench-memory-breakdown.png and b/docs/assets/bench-memory-breakdown.png differ diff --git a/docs/assets/bench-memory.png b/docs/assets/bench-memory.png index 158301d..14a402a 100644 Binary files a/docs/assets/bench-memory.png and b/docs/assets/bench-memory.png differ