From 18fe2180b4c92102746fbf49a4fb4ac47063aba6 Mon Sep 17 00:00:00 2001 From: Veronica Valeros <2458879+verovaleros@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:53:48 +0200 Subject: [PATCH 1/2] Update README files to enhance setup instructions and clarify Docker usage --- README.md | 97 ++++++++++++++++++++++++++++---------------- etc/docker/README.md | 47 +++++++++++++-------- 2 files changed, 91 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index d00bf2d..65889b8 100644 --- a/README.md +++ b/README.md @@ -38,50 +38,75 @@ Each AIP model generates its own blocklist based on a specific criteria. The mai - A more experimental approach to increase blocklist efficiency. -## Installation +## Setup -AIP can be run through Docker (recommended) or natively. In any case, make sure the data is correctly accessible in the `data/raw/` folder and that the files `data/external/` are correctly configured. +AIP can run with Docker (recommended) or directly with Python. Either way, it needs two things: -### AIP Docker +1. **Zeek logs** in `data/raw/`, one folder per day (for example `data/raw/2024-10-22/conn.*`). With Docker you can point AIP to your Zeek logs folder instead of copying them. +2. **Your honeypot IPs** in `data/external/`. Copy the example files and edit them: -The best way to run AIP right now is using [Docker](etc/docker/README.md). + ```bash + cp data/external/honeypots_public_ips_example.csv data/external/honeypots_public_ips.csv + cp data/external/do_not_block_these_ips_example.csv data/external/do_not_block_these_ips.csv + ``` -AIP will automatically attempt to run all the models using the available data. Assuming the Zeek data is located in its usual location: + - `honeypots_public_ips.csv` (required): the public IPs of your honeypots. AIP only looks at traffic sent to these IPs. + - `do_not_block_these_ips.csv` (optional): IPs that must never appear in a blocklist. + +### Run with Docker + +From the AIP folder, assuming your Zeek logs are in `/opt/zeek/logs/`: + +```bash +docker run --rm \ + -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro \ + -v ${PWD}/data/:/home/aip/AIP/data/:rw \ + --name aip stratosphereips/aip:latest bin/aip +``` + +This runs all models for today. To use any of the [commands](#commands) below, write them after `bin/aip`. For example, to run all models for one day: ```bash -:~$ cd AIP -:~$ docker run --rm -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro -v ${PWD}/data/:/home/aip/AIP/data/:rw --name aip stratosphereips/aip:latest bin/aip +docker run --rm \ + -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro \ + -v ${PWD}/data/:/home/aip/AIP/data/:rw \ + --name aip stratosphereips/aip:latest bin/aip run-models --start-date 2026-10-12 --end-date 2026-10-13 ``` -### Local Installation - -To run AIP locally, follow the next steps: - -* Clone the AIP repository: - * `git clone https://github.com/stratosphereips/AIP.git ~/AIP` -* Access the AIP folder: - * `cd ~/AIP` -* Create a Python Virtual Environment to install the requirements - * `python3 -m venv aip-venv` -* Activate the virtual environment - * `source aip-venv/bin/activate` -* Install the Python requirements: - * `pip install -r requirements.txt` -* Set the PYTHONPATH environment variable so Python knows where to find all AIP components: - * `export PYTHONPATH=$(pwd)/lib` -* Let's test that AIP works with a simple test: - * `python3 bin/aip --help` - -### AIP Commands - -Get all the full list of options with `bin/aip --help`: - -- Run AIP for a specific day: - - `bin/aip -v run-models --start-date YYYY-MM-DD` -- Run AIP for a specific day and model: - - `bin/aip -v run-models --start-date YYYY-MM-DD --model Alpha` -- Rebuild AIP knowledge base from a specific date: - - `bin/aip -d rebuild-kb --start-date YYYY-MM-DD` +To build the image yourself, see the [Docker guide](etc/docker/README.md). + +### Run with Python + +```bash +git clone https://github.com/stratosphereips/AIP.git ~/AIP +cd ~/AIP +python3 -m venv aip-venv +source aip-venv/bin/activate +pip install -r requirements.txt +export PYTHONPATH=$(pwd)/lib +bin/aip --help +``` + +Run `source aip-venv/bin/activate` and the `export PYTHONPATH` line again each time you open a new terminal. + +## Commands + +| What you want | Command | +|---|---| +| Run all models for today | `bin/aip` | +| Run all models for one day | `bin/aip run-models --start-date 2026-10-02 --end-date 2026-10-02` | +| Run all models for several days | `bin/aip run-models --start-date 2026-10-01 --end-date 2026-10-12` | +| Run only one model | Add `--model` with `Alpha`, `Alpha7`, `Prioritize_New`, `Prioritize_Consistent` or `Random_Forest` | +| Rebuild the attacker history from a date until today | `bin/aip rebuild-kb --start-date 2026-10-01` | + +Things to know: + +- If you leave out `--end-date`, it is today. So `--start-date` alone runs every day from that date until today. +- The blocklist for a day is built from the attacks seen on the days before it. +- Blocklists are saved in `data/output//`. +- The attacker history is what the Prioritize and Random Forest models learn from. AIP builds it on its own as it runs. +- To see what AIP is doing, add `-v` (progress) or `-d` (full detail) right after `bin/aip`, for example `bin/aip -v run-models`. +- See all options with `bin/aip --help`. ## License diff --git a/etc/docker/README.md b/etc/docker/README.md index 88818af..5e4fccf 100644 --- a/etc/docker/README.md +++ b/etc/docker/README.md @@ -1,40 +1,51 @@ # Docker image for AIP -AIP docker aims to help in development and deployment of AIP algorithms to newcommers. The code of the repository is mounted and available from inside the docker image. The source of the data and the output folder of AIP is also mounted as a data volume inside the data/ folder to decouple from where this data really is in the host machine, easying the AIP deployment. +The Docker image includes AIP and everything it needs to run. Your Zeek logs and the AIP `data/` folder stay on your machine and are shared with the container when you run it. ## Build the image -To build the image, you can run the following command. - ```bash -:~$ git clone https://github.com/stratosphereips/AIP.git -:~$ cd AIP/ -:~/AIP$ docker build --build-arg uid=1000 --file etc/docker/Dockerfile --tag aip:latest . +git clone https://github.com/stratosphereips/AIP.git +cd AIP/ +docker build --build-arg uid=1000 --file etc/docker/Dockerfile --tag aip:latest . ``` ## Prepare the data -AIP needs raw network flow data to run. In this case, we assume you have Zeek logs in `/opt/zeek/logs`. +AIP reads Zeek logs. These steps assume your Zeek logs are in `/opt/zeek/logs/`. -Additionally, the following two files need to be edited and populated: -- `data/external/do_not_block_these_ips_example.csv`: you want to add here IPs that should not appear on the AIP blocklists -- `data/external/honeypots_public_ips_example.csv`: you want to add here the public IP of the honeypot or machine running Zeek +Copy the two example files and edit the copies: -First copy the files and then edit them: ```bash -:~/AIP$ cp data/external/do_not_block_these_ips_example.csv data/external/do_not_block_these_ips.csv -:~/AIP$ cp data/external/honeypots_public_ips_example.csv data/external/honeypots_public_ips.csv +cp data/external/honeypots_public_ips_example.csv data/external/honeypots_public_ips.csv +cp data/external/do_not_block_these_ips_example.csv data/external/do_not_block_these_ips.csv ``` +- `honeypots_public_ips.csv` (required): the public IPs of your honeypots. AIP only looks at traffic sent to these IPs. +- `do_not_block_these_ips.csv` (optional): IPs that must never appear in a blocklist. + ## Run the container -To run the container of that image you can run the following command: +Run all models for today: ```bash -:~/AIP$ docker run --rm -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro -v ${PWD}/data/:/home/aip/AIP/data/:rw --name aip aip:latest bin/aip +docker run --rm \ + -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro \ + -v ${PWD}/data/:/home/aip/AIP/data/:rw \ + --name aip aip:latest bin/aip ``` -An example output is shown below: +Run all models for one day, showing full detail with `-d`: + +```bash +docker run --rm \ + -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro \ + -v ${PWD}/data/:/home/aip/AIP/data/:rw \ + --name aip aip:latest bin/aip -d run-models --start-date 2024-10-22 --end-date 2024-10-22 +``` + +The output starts like this: + ``` 2024-10-23 13:02:36,513 - aip.data.access - DEBUG - Creating attacks for dates ['2024-10-22'] 2024-10-23 13:02:36,513 - aip.data.access - DEBUG - Making dataset from raw data for dates ['2024-10-22'] @@ -42,4 +53,6 @@ An example output is shown below: 2024-10-23 13:02:37,539 - aip.data.access - DEBUG - Writting file: /home/aip/AIP/data/processed/attacks.2024-10-22.csv.gz 2024-10-23 13:02:37,648 - aip.data.access - DEBUG - Creating attacks for dates ['2024-10-20', '2024-10-21'] ... -``` \ No newline at end of file +``` + +Without `-v` or `-d`, AIP only prints errors. Blocklists are saved in `data/output/`. See the [main README](../../README.md#commands) for all commands. From 6330b6c2bba67c3d2e676e394f6c2d2d76a437c9 Mon Sep 17 00:00:00 2001 From: Veronica Valeros <2458879+verovaleros@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:57:25 +0200 Subject: [PATCH 2/2] Fix end date in Docker run command for model execution in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 65889b8..8fddfe9 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ This runs all models for today. To use any of the [commands](#commands) below, w docker run --rm \ -v /opt/zeek/logs/:/home/aip/AIP/data/raw:ro \ -v ${PWD}/data/:/home/aip/AIP/data/:rw \ - --name aip stratosphereips/aip:latest bin/aip run-models --start-date 2026-10-12 --end-date 2026-10-13 + --name aip stratosphereips/aip:latest bin/aip run-models --start-date 2026-10-12 --end-date 2026-10-12 ``` To build the image yourself, see the [Docker guide](etc/docker/README.md).