The steps of this setup must be followed in order. Steps labled as OPTIONAL may be skipped without impacting inference.

Basic setup

Clone this repository:

git clone <this repo>

And initialize its submodules:

git submodule update --init --recursive

Download XMem2 checkpoints

cd tracking_models/XMem2/ && ./scripts/download_models.sh

Sam2 checkpoints

Sam2 checkpoints must be downloaded from https://github.com/facebookresearch/sam2, and then placed into tracking_models/sam2.

Currently the inference script points to this path SAM2_CHECKPOINT="/workspace/tracking_models/sam2/sam2.1_hiera_large.pt" for inference, but by modifying this line you may use any of the other checkpoint files that you may have placed under tracking_models/sam2, such as tracking_models/sam2/sam2.1_hiera_small.pt or tracking_models/sam2/sam2.1_hiera_tiny.pt. These checkpoints are all available in the sam2 repo through download links.

Docker

Several dockers are needed, let's build them. From the project root .../SpiceSeg, run the following:

docker-compose build

You should see this output at the end:

[+] Building 4/4 ✔ sam2 Built 0.0s ✔ xmem2 Built 0.0s ✔ foodseg103 Built 0.0s ✔ yolo Built 0.0s

Prepare FoodSeg103 data for YOLO training (OPTIONAL)

First download the zipped data from Kaggle and place the compressed file inside data/. You can download the dataset from the FoodSeg103 Kaggle page:

https://www.kaggle.com/datasets/fontainenathan/foodseg103

Next, we need to prepare the FoodSeg103 data for YOLO training. This includes decompressing the data, splitting into train, test and validation sets and converting the labels into a format that YOLO understands. The following script takes care of that; assuming that the zip file you have downloaded from Kaggle is called "archive.zip", run this from the project root:

docker-compose run --rm --user "$(id -u):$(id -g)" yolo "/workspace/utility_scripts/reorganize_foodseg103.sh /workspace/data/archive.zip"

The splits generated by this script are deterministic, as the random seed is specified.

Fine-tune YOLO on FoodSeg103 (OPTIONAL)

If you've completed the step above, the data for YOLO fine-tuning is ready. Download the base checkpoint from https://docs.ultralytics.com/models/yolo11/#supported-tasks-and-modes , and place it in assets/ckpts/YOLO (make sure the checkpoint that you have downloaded is for the segmentation task, your checkpoint's filename should match "<model_name>-seg.pt").

The current training script ( utility_scripts/train_yolo_fs103.sh ) is made to work with yolo11s-seg , but you can easily use a different one by replacing the string for YOLO_PT:

...
# Change to project root
cd "$PROJECT_ROOT"

# Model configuration
YOLO_PT="<your_checkpoint_name>" # CHANGE THIS

# Run training inside Docker container with GPU support
...

After that, simply call the the script from the project root:

./utility_scripts/train_yolo_fs103.sh

The script will launch a YOLO fine-tune run for the specified checkpoint. Your YOLO run's checkpoints and metrics should appear in your src/yolo/runs/segment folder.

Inference

The recommended entrypoint script for inference is unified_inference.sh. The general pattern for the script is the following:

Simple inference (a single segmentor model):

source ./unified_inference.sh && INPUT_DIR="your/input/dir/path" OUTPUT_DIR="your/output/dir/path" inference [--log_gpu] <segmentor>

Mixed inference (a segmentor model does inference on a number of initial images, and a tracking model takes those predicted masks and propagetes them through time):

source ./unified_inference.sh && INPUT_DIR="your/input/dir/path" OUTPUT_DIR="your/output/dir/path" inference --mixed [--log_gpu] <segmentor> <tracking_model> <n_images>

the --log_gpu option makes it so a csv file is generated in your OUTPUT_DIR, containing the memory usage over time of your device during inference. This uses nvidia-smi, so this must be available to your system if you want to use the option.

Bellow you will find some examples on how to run several of the inference pipelines available in the repo:

YOLO:

source ./unified_inference.sh && INPUT_DIR="data/test_dataset/images" OUTPUT_DIR="data/test_dataset/results/YOLO" inference --log_gpu yolo

YOLO (3 images) + SAM2:

source ./unified_inference.sh && INPUT_DIR="data/test_dataset/images" OUTPUT_DIR="data/test_dataset/results/YOLO_sam2" inference --mixed --log_gpu yolo sam2 3

CCNET_RELEM:

source ./unified_inference.sh && INPUT_DIR="data/test_dataset/images" OUTPUT_DIR="data/test_dataset/results/CCNET_RELEM" inference --log_gpu foodseg_ccnet_relem

SWIN (1 image) + XMEM2:

source ./unified_inference.sh && INPUT_DIR="data/test_dataset/images" OUTPUT_DIR="data/test_dataset/results/SWIN_XMEM2" inference --mixed --log_gpu foodseg_swin_small xmem2 1

Obtain performance metrics

utility_scripts/run_eval_map.sh --submit_dir data/test_dataset/results/YOLO --truth_dir data/test_dataset/ground_truth --output_dir data/test_dataset/metrics/YOLO

IMPORTANT: naming convention

The expected naming convention to be followed by all images to take part in the inference process is the following:

<video_id>_<frame_number>.<extension>

• 'video_id' may be any string
• 'frame_number' must be a zero-padded string with at least 3 digits, but can be longer than three digits (i.e, both '1_002.png' and '1_1132.png' are valid)
• extension must be either png or jpg