Usage and installation overview of the core features. Additional helpful information can be found in the README.
Before installing and starting the application, ensure Docker and Docker-compose are installed. Both can be installed according to the official Docker documentation. If you’re a student in the University of Helsinki and using a fuksi laptop, there are some other steps you need to take to install Docker correctly (Docker uses the root partition by default and if it gets filled your computer can become unusable). Information about installing Docker on cubbli-linux here. After installing Docker on cubbli-linux, you can install the Docker-compose plugin according to the official documentation.
To run the application, execute the following commands:
git clone https://github.com/EvoTestOps/VisualLogAnalyzer.git
cd VisualLogAnalyzer
mkdir analysis_results # Optional but recommended to avoid permission issues
docker compose --env-file .env.sample up
To omit the --env-file flag, rename .env.sample to .env. For example with command: mv .env.sample .env
Building the application for the first time may take 1-2 minutes. Once the applitcation is running navigate to http://localhost:5000/dash/ to access the homepage.
The commands below assume you renamed .env.sample to .env. If you did not, add --env-file .env.sample to every docker compose command, not just up — compose reads the env file each time to resolve ${LOG_DATA_DIRECTORY} and the other variables. Likewise, if you are using the development compose file, repeat -f docker-compose-dev.yml on every command.
If you started the application in the foreground with docker compose up, press Ctrl+C in that terminal. To stop it from another terminal, or when it was started in the background, use one of:
docker compose stop # stop the containers but keep them, for a fast restart
docker compose down # stop and remove the containers
Both are safe to run: your projects and analyses are stored in the db_data Docker volume and the results in analysis_results/, and neither command touches them.
docker compose start # after `docker compose stop`
docker compose up -d # after `docker compose down`, or any time; -d runs it in the background
docker compose restart # stop and start in one go, e.g. to clear a stuck analysis worker
Database migrations are applied automatically every time the app container starts, so no extra step is needed after an update.
With -d the application runs in the background and your terminal stays free. Use docker compose ps to see what is running and docker compose logs -f to follow the output (add a service name, e.g. docker compose logs -f celery, to follow just one).
Changes to the .env file take effect when the containers are recreated: docker compose up -d. Changes to the application code or to requirements.txt require the images to be rebuilt first:
docker compose up -d --build
(With the development compose file the code is mounted into the containers, so only requirements.txt changes need a rebuild there.)
docker compose down -v
The -v flag additionally deletes the db_data volume, which removes all projects and analysis records. The result files in analysis_results/ are not deleted, but they are left orphaned — delete its contents as well if you want a completely clean state. The database schema is recreated automatically on the next start.
The repository includes an example light-oauth-2 dataset. It contains a Labeled directory which has known cases (either correct or some type of error), and a Hidden_Group_1 which contains unknown cases. Reviewing the dataset structure is recommended.
To use the dataset:
- Navigate to the homepage.
- Click "Create a new project", give it a name, and set the base path to
./log_data/LO2
With setting the base path we are able to access the directories inside the LO2 directory.
After the project has been created click it on the list which should take you to the project view. Here we have four main components:
- List of analyses run in the project
- "Create a new analysis"-options. When a option is clicked e.g. "Anomaly Detection" it will give you options on which level you want to run it
- Settings box with relevant settings (hover for explanations)
- "Recent analyses" table for monitoring the status of analyses, inspecting logs and error messages.
Lets start by running an simple example analysis:
- Click "High Level Visualisations" and then "Directory Level".
- Select "Labeled" from the "Directory" dropdown and click "Analyze".
This redirects you back to the project page, where the results will appear once the analysis is complete.
Clicking the result will show a plot and analysis details. Hover over data points for more information.
Next, lets try running anomaly detection:
- Go back to the project page and select "Anomaly Detection" and then "Line Level".
- Select
Labeledfrom both "Train data directory" and "Test data directory".
- Compared to high level visualisations and log distance analysis, you will need to input train data directory and test data directory separately. This either means having two directories where one contains only train data and the other one only test data, or by filtering the data in the "Directories to include in train data" and "Directories to include in test data".
- From the "Directories to include in train data" select only the correct cases.
- For the "Directories to include in test data" select the rest which were not selected to the train data.
- Select a "Regex mask type" (e.g. "Myllari" or "Myllari Extended")**.
- In general it is a good idea to use a mask unless there is some really good reason not to.
- Click "Analyze".
The results will be displayed on the list similarly as in the previous analysis. After clicking the results you can select plot (file) to display from the drop down. This will generate a plot and a table containing the analysis results of that specific file in line level. Clicking a data point on the plot takes you to the correct line on the table.
To view moving averages in the plots, navigate back to the project page and select either "Moving averages only" or "Show all" from the settings, and click "Apply". Now when navigating back to the results the moving averages are displayed.
In line level anomaly detection, you can create a multi-plot image to visualize multiple plots side-by-side. In a anomaly detection line level results page, click "Create multi-plot image" and from the form select the files and columns to include. In general using moving averages as the columns is the most useful.
By default, the application looks for datasets in the log_data-directory. To add your own dataset, simply copy/move them in to that directory (you might need to refresh the page to see the directory). To use directories in the root of log_data, leave the base path of the project empty or use the default value. You can organize your datasets similarly as the example dataset was organized and specify the path in the base path input. The base path can also be used to navigate further down in the directory structure what the directory input options normally allows, which might be useful in some log data structures.