Rename from simple_triton->triteia

.github/workflows/black.yml

@@ -9,4 +9,4 @@ jobs:
       - uses: actions/checkout@v2
       - uses: psf/black@stable
         with:
-          src: "./simple_triton"
+          src: "./triteia"

.gitignore

@@ -1,3 +1,4 @@
+out
 plot_out/
 *.csv
 *_results/

README.md

@@ -1,8 +1,8 @@
-# simple-triton
+# triteia
 
-simple-triton is a Python client for performing inference on the NVIDIA Triton Inference Server. It provides model deployment, configuration, and optimization capabilities for the TensorFlow, ONNX, and Python Triton backends directly from Python. This was developed to address limitations in the [PyTriton](https://github.com/triton-inference-server/pytriton) package that only suppports deployments with the Python backend where TensorRT, XLA, and mixed precision are not available.
+triteia is a Python client for performing inference on the NVIDIA Triton Inference Server. It provides model deployment, configuration, and optimization capabilities for the TensorFlow, ONNX, and Python Triton backends directly from Python. This was developed to address limitations in the [PyTriton](https://github.com/triton-inference-server/pytriton) package that only suppports deployments with the Python backend where TensorRT, XLA, and mixed precision are not available.
 
-![triton_overview.png](triton_overview.png)
+![doc/overview_figure.png](doc/overview_figure.png)
 
 # User guide <a name="user-guide"></a>
 
@@ -22,25 +22,25 @@ simple-triton is a Python client for performing inference on the NVIDIA Triton I
 
 ## Quick start <a name="quick-start"></a>
 
-simple-triton requires `histomcs_stream` and `large_image` packages with the tiff reader
+triteia requires `histomcs_stream` and `large_image` packages with the tiff reader
 ```
-git clone https://github.com/PathologyDataScience/simple_triton.git
-pip install --editable ./simple_triton
+git clone https://github.com/PathologyDataScience/triteia.git
+pip install --editable ./triteia
 ```
-> `--editable` ensures that updates to the `simple_triton` package (after `git pull`) immediately takes effect.
+> `--editable` ensures that updates to the `triteia` package (after `git pull`) immediately takes effect.
 
-Or, you can try the Docker image. First, run `git clone` (as above) or make sure to do a git pull inside the "simple_triton" directory. Then:
+Or, you can try the Docker image. First, run `git clone` (as above) or make sure to do a git pull inside the "triteia" directory. Then:
 ```bash
 # optional: download test data
 python download_test_data.py
 
-# simple_triton_client will be the name of the Docker image
-docker build -f client.Dockerfile . -t simple_triton_client:latest --build-arg DOCKER_GROUP_ID=$(getent group docker | cut -d: -f3)
+# triteia will be the name of the Docker image
+docker build -f client.Dockerfile . -t triteia:latest --build-arg DOCKER_GROUP_ID=$(getent group docker | cut -d: -f3)
 docker run \
   --security-opt seccomp:unconfined --network=host \
   --rm -it --shm-size=1g \
   -v ${PWD}/test_data:/data:ro \
-  --name tritonclient simple_triton_client:latest
+  --name tritonclient triteia:latest
 ```
 > **_NOTE:_**  `--network=` option allows the Docker image to access ports from other containers or the host. The default shared memory size for Docker containers is 64MB; use `--shm-size=` to increase it if you need to process large whole-slide images. The `--security-opt seccomp:unconfined` option may be needed on larger machines to enable [OpenBLAS](https://www.openblas.net/) threading support. The `--rm` option removes the container after it stops, so be cautious if you need persistent data.
 
@@ -64,13 +64,13 @@ docker run \
   -v ${PWD}/examples:/examples:rw \
   --user $UID --rm -it \
   --shm-size=1g \
-  --name tritonclient simple_triton_client:latest \
+  --name tritonclient triteia:latest \
   bash -c "jupyter-lab --notebook-dir /examples/ --no-browser"
 ```
 
 ### Running the Triton container <a name="container"></a>
 
-simple-triton is tested with [Triton version 25.02](https://github.com/triton-inference-server/server/releases/tag/v2.55.0).
+triteia is tested with [Triton version 25.02](https://github.com/triton-inference-server/server/releases/tag/v2.55.0).
 Support for Tensorflow is deprecated in later versions, but other model backends (like PyTorch) should still work.
 
 We recommend starting from the repository’s root directory (the same as the directory containing this README.md file). You can run using the `./launch_server.sh` script (which also has some command-line options) or the command below:
@@ -153,7 +153,7 @@ python feature_extraction.py ~/inputs.tsv ~/ EfficientNetV2S.tensorflow -s
 ```
 
 ## Model wrappers <a name="wrappers"></a>
-simple-triton contains wrappers for serving popular digital pathology models, including CONCH, UNI, Prov-GigaPath, hibou-L, Phikon, Virchow, and Virchow2 on the [Python backend](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/python_backend/README.html). All models are served using mixed precision.
+triteia contains wrappers for serving popular digital pathology models, including CONCH, UNI, Prov-GigaPath, hibou-L, Phikon, Virchow, and Virchow2 on the [Python backend](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/python_backend/README.html). All models are served using mixed precision.
 
 | Model | Input | Output | Size |
 |---|---|---|---|
@@ -195,14 +195,14 @@ docker run \
 ```
 
 ## Model configuration <a name="config"></a>
-`simple_triton.config` includes model configuration classes that implement backend-specific configuration options. These classes enable configuration of batching behavior, specification of model input/output shapes and types, and backend optimizations. Refer to the Triton documentation on [model configuration](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html#model-configuration) and [optimization](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/optimization.html) for further details.
+`triteia.config` includes model configuration classes that implement backend-specific configuration options. These classes enable configuration of batching behavior, specification of model input/output shapes and types, and backend optimizations. Refer to the Triton documentation on [model configuration](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html#model-configuration) and [optimization](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/optimization.html) for further details.
 
 Configuration classes like `PythonConfiguration` and `TensorflowConfiguration` take additional data classes as inputs that configure batching and caching behavior, hardware resources, and model input/output signatures.
 
 When Triton is launched with `--strict-model-config=false`, the server automatically configures basic information such as input/output signatures, and the configuration can omit them.
 ```python
-from simple_triton.config import TensorflowConfig
-from simple_triton.model import TritonModel
+from triteia.config import TensorflowConfig
+from triteia.model import TritonModel
 name = "mymodel.tensorflow"
 config = TensorflowConfig(name, max_batch_size=64)
 model = TritonModel(name, "localhost:8001")
@@ -211,7 +211,7 @@ model.load(config=config.json())
 
 Alternatively, model inputs and output signatures can be defined using the `ModelInput` and `ModelOutput` classes
 ```python
-from simple_triton.config import ModelInput
+from triteia.config import ModelInput
 input = [ModelInput(name="input_0", shape=[224, 224, 3], dtype=np.float32, optional=False)]
 config = TensorflowConfig(name, max_batch_size=64, input=input)
 ```
@@ -220,14 +220,14 @@ Variable-sized input dimensions can be indicated using a value of -1.
 
 The `InstanceGroup` class configures the use of CPU or GPU resources and the number of model instances hosted on each GPU.
 ```python
-from simple_triton.config import InstanceGroup
+from triteia.config import InstanceGroup
 instances = InstanceGroup(count=2, kind="gpu", gpus=[0,1,2,3])
 config = TensorflowConfig(name=name, instance_group=instances)
 ```
 
 `TensorflowOptimization` can be used with `TensorflowMixedPrecision`, `TensorflowXla`, and `TensorRt` to activate automatic mixed precision, XLA compilation, or TensorRT optimization.
 ```python
-from simple_triton.config import TensorflowMixedPrecision, TensorflowXla, TensorflowOptimization
+from triteia.config import TensorflowMixedPrecision, TensorflowXla, TensorflowOptimization
 amp = TensorflowMixedPrecision()
 xla = TensorflowXla(level=2)
 optimizer = TensorflowOptimization(amp=amp, xla=xla)
@@ -253,11 +253,11 @@ File-based configuration is useful for distributing models. When configuring and
 The `TritonModel` class can be used to load/unload models, retrieve model configurations or metadata, or check whether a model is idle or loaded. A model is defined by a model name and a server URL.
 
 ```python
-from simple_triton.model import TritonModel
+from triteia.model import TritonModel
 model = TritonModel("EfficientNetV2S.tensorflow", "localhost:8001")
 ```
 
-When unloading a model, simple-triton will check that the model is idle.
+When unloading a model, triteia will check that the model is idle.
 ```python
 # load model with auto-generated configuration
 # block and timeout after 1 second
@@ -306,7 +306,7 @@ pooch.retrieve(
 ### Using standalone docker container
 To test using the Docker container, launch and build the client as follows:
 ```
-docker build -f client.Dockerfile . -t simple_triton_client:latest --build-arg DOCKER_GROUP_ID=$(getent group docker | cut -d: -f3)
+docker build -f client.Dockerfile . -t triteia:latest --build-arg DOCKER_GROUP_ID=$(getent group docker | cut -d: -f3)
 ./launch_test_container.sh
 ```
 You can now run tests inside the container using `pytest tests` inside the container.
@@ -320,8 +320,11 @@ Be aware that extra performance is not guaranteed.
 PyTorch offers many optimizations that may not be available in the ONNX or TRT backends.
 
 ## Paper results
-To reproduce the (TBD) paper: `OUTPUT_DIR="./results" ./benchmarking/paper_benchmarks.sh $OUTPUT_DIR`
-Results can be inspected either as tensorboards: `tensorboard --logdir=...`, or as figures:
+To reproduce the (TBD) paper
+1. start the NVIDIA triton server with all GPUs and models available: `./launch_server.sh --num-gpus <num_gpu> --http-port 7984 --grpc-port 7985 --metrics-port 7986`
+2. `OUTPUT_DIR="./results" ./benchmarking/paper_benchmarks.sh $OUTPUT_DIR`
+3. Inspect results: `tensorboard --logdir=...`.
+To view results as figures::
 ```bash
 # convert TensorBoard to CSV
 ./benchmarking/tensorboard_to_csv.py
@@ -330,5 +333,5 @@ Results can be inspected either as tensorboards: `tensorboard --logdir=...`, or
 ```
 
 To do the benchmarks using Docker, use the `benchmark_client.Dockerfile` in the benchmarking directory.
-It is identical to the `client.Dockerfile` in this directory, except it has access to CUDA so it can automatically start and stop Triton with GPUs.
-To build it from the git root directory: `docker build -f benchmarking/benchmark_client.Dockerfile . -t simple_triton_client:benchmark`
+It is identical to the `client.Dockerfile` in this directory, except it has access to CUDA so it can export GPU metrics and automatically start and stop Triton with GPUs.
+To build it from the git root directory: `docker build -f benchmarking/benchmark_client.Dockerfile . -t triteia:benchmark`

benchmarking/ConvNeXtXLarge_amp_Batch64_GPU8_iter5_BatchTest.sh

@@ -22,7 +22,7 @@ while [ $gpu_num -ne 8 ]
                                         do
                                                 maxbatchsize=$(($maxbatchsize+32))
                                                 echo "gpu_num:$gpu_num  Workers: $limit maxbatchsize: $maxbatchsize"
-                                                python /tf/notebooks/simple_triton/benchmarking/benchmark_interface.py  --limit 1 --gpu-num $gpu_num  --fileoutput $filename   --iterations 5 --use-amp --precision "FP16"  --maxbatchsize $maxbatchsize  --model-name "ConvNeXtXLarge"
+                                                python /tf/notebooks/triteia/benchmarking/benchmark_interface.py  --limit 1 --gpu-num $gpu_num  --fileoutput $filename   --iterations 5 --use-amp --precision "FP16"  --maxbatchsize $maxbatchsize  --model-name "ConvNeXtXLarge"
                                                 echo "==============================================================="
                                         done
 

benchmarking/benchmark_client.Dockerfile

@@ -1,5 +1,6 @@
 # for the paper, this docker image is built with:
-# docker build -f client.Dockerfile . -t simple_triton_client:benchmark --build-arg DOCKER_GROUP_ID=$(getent group docker | cut -d: -f3) --build-arg UID=$(id -u) --build-arg GID=$(id -g)  --build-arg USERNAME=$USER
+# this file is intended to be similar to the client.Dockerfile, except with GPUs available to the client (to read performance metrics, etc).
+# docker build -f client.Dockerfile . -t triteia:benchmark --build-arg DOCKER_GROUP_ID=$(getent group docker | cut -d: -f3) --build-arg UID=$(id -u) --build-arg GID=$(id -g)  --build-arg USERNAME=$USER
 FROM python:3.10-slim AS build-image
 ARG USERNAME=myuser
 
@@ -14,9 +15,9 @@ RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/00-docker && \
     apt clean && \
     rm -rf /var/lib/apt/lists/*
 
-# install simple-triton
-WORKDIR /home/$USERNAME/code/simple_triton
-COPY simple_triton/ simple_triton
+# install triteia
+WORKDIR /home/$USERNAME/code/triteia
+COPY triteia/ triteia
 COPY pyproject.toml .
 # comment out scm (i.e. git) line in pyproject.toml
 RUN sed -i 's/.*\[tool.setuptools_scm\]/#&/g' pyproject.toml
@@ -44,8 +45,8 @@ RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/00-docker && \
     rm -rf /var/lib/apt/lists/*
 
 USER $USERNAME
-WORKDIR /home/$USERNAME/simple_triton
-COPY --chown=$USERNAME:$USERNAME simple_triton/ simple_triton
+WORKDIR /home/$USERNAME/triteia
+COPY --chown=$USERNAME:$USERNAME triteia/ triteia
 COPY --chown=$USERNAME:$USERNAME pyproject.toml .
 
 
@@ -75,8 +76,8 @@ RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/00-docker && \
     rm -rf /var/lib/apt/lists/*
 RUN curl -fsSL https://get.docker.com | sh
 # for jupyter notebooks as non-root
-RUN mkdir --mode a+rxw /.local /.jupyter /.cache /models/ /.config
-RUN chown $USERNAME:$USERNAME /home/$USERNAME/simple_triton/
+RUN mkdir --mode a+rxw /.local /.jupyter /.cache /.config
+RUN chown $USERNAME:$USERNAME /home/$USERNAME/triteia/
 USER $USERNAME
 
 COPY --chown=$USERNAME:$USERNAME README.md pyproject.toml ./

benchmarking/benchmark_interface.py

@@ -9,11 +9,11 @@
 import tensorflow as tf
 from large_image.cache_util import cachesClear
 
-from simple_triton.config import *
-from simple_triton.feature_extraction import study, inference
-from simple_triton.model import TritonModel
-from simple_triton.tile_iterators import TiffPrefetch
-from simple_triton.utils import analyze
+from triteia.config import *
+from triteia.feature_extraction import study, inference
+from triteia.model import TritonModel
+from triteia.tile_iterators import TiffPrefetch
+from triteia.utils import analyze
 
 
 class Benchmark:
@@ -23,6 +23,9 @@ class Benchmark:
     """
 
     def __init__(self, args_dict):
+        self.times = None
+        self.tile_info = None
+        self.features = None
         self.args_dict = args_dict
 
     def create_hs_study(self, wsi_path, mask_path):
@@ -126,7 +129,7 @@ def callback(user_data, result, error):
         )
         # warm up Model
         print("Warmup Model")
-        (self.features, self.tile_info, self.times, self.failed,) = inference(
+        self.features, self.tile_info, self.times = inference(
             iterator,
             model_name,
             url=self.args_dict["url"],
@@ -152,7 +155,7 @@ def callback(user_data, result, error):
             )
             # start timer
             start = time.time()
-            (self.features, self.tile_info, self.times, self.failed,) = inference(
+            self.features, self.tile_info, self.times = inference(
                 iterator,
                 model_name,
                 url=self.args_dict["url"],
@@ -212,8 +215,8 @@ def gpu_mem_clear(self):
 def install():
     """Install dependencies for running benchmarking interface tool"""
     # install large_image with tile sources as prereq, check feature_extraction.ipynb in examples directory.
-    # install simple_triton
-    subprocess.check_call([sys.executable, "-m", "pip", "install", f"../simple_triton"])
+    # install triteia
+    subprocess.check_call([sys.executable, "-m", "pip", "install", f"../triteia"])
     subprocess.check_call([sys.executable, "-m", "pip", "install", "ray"])
     subprocess.check_call([sys.executable, "-m", "pip", "install", "pyarrow"])