Malware Detection Using Machine Learning

A machine learning project for detecting malicious processes from behavioral data using process-level feature engineering and ensemble classification.

The project takes raw process activity data, aggregates it at the process level, generates statistical and temporal features, trains multiple machine learning models, and deploys the final model through an interactive Streamlit dashboard.

Live Demo

https://malware-detection-using-machine-learninggit-ul5eeutzwcqfpyttct.streamlit.app/

GitHub Repository

https://github.com/nikh0011/Malware-Detection-Using-Machine-Learning

Project Overview

Traditional malware detection can rely heavily on signatures, which may not work well for previously unseen or modified malware.

This project explores a behavior-based machine learning approach. Instead of directly classifying individual records, the raw activity data is grouped by process and transformed into meaningful statistical and temporal features.

The final model uses an Extra Trees Classifier and achieved the following results on the evaluated process-level data:

Metric	Score
Accuracy	98.00%
Precision	100.00%
Recall	96.00%
F1 Score	97.96%
ROC-AUC	0.9920

The trained model is integrated into a Streamlit application where users can upload a process-level CSV file and receive malware predictions along with probability scores and model insights.

Dataset

The project uses malwaredataset.csv.

Dataset characteristics

100,000 total records
100 unique processes
33 numerical behavioral features
hash used as the process identifier
millisecond used for temporal information
classification used as the ground-truth label
No missing values in the original dataset

The raw data contains multiple observations for each process over time. Therefore, the modeling pipeline converts these observations into a single process-level representation.

Process-Level Feature Engineering

A major part of the project is converting time-series process activity into fixed-length features suitable for machine learning.

The data is first sorted using:

hash
millisecond

Numerical behavioral features are then aggregated for each process.

For each numerical feature, the following statistics are generated:

Mean
Standard deviation
Minimum
Maximum
Median
25th percentile
75th percentile
Range
First value
Last value
Delta between first and last value
Mean absolute consecutive change

Additional temporal features are generated:

time_start
time_end
time_span

After feature engineering, the final model uses 268 training features.

Avoiding Data Leakage

Since multiple rows belong to the same process, randomly splitting individual rows could cause information from the same process to appear in both training and testing data.

To avoid this issue, the raw observations are first aggregated to the process level.

This ensures that the model works with one feature representation per process instead of treating individual observations as independent samples.

The train-test split is then performed on the resulting process-level dataset using stratification.

Machine Learning Models

Several classification algorithms were evaluated during model development:

Logistic Regression
K-Nearest Neighbors (KNN)
Support Vector Machine (SVM)
Random Forest
Extra Trees
HistGradientBoosting
XGBoost was also considered, but was not used in the final pipeline

The final deployed model is:

Extra Trees Classifier

The production pipeline is:

Input Data
    ↓
Process-Level Feature Engineering
    ↓
Median Imputation
    ↓
Extra Trees Classifier
    ↓
Malware / Benign Prediction
    ↓
Malware Probability

Final Model Configuration

Parameter	Value
Algorithm	Extra Trees Classifier
Number of Trees	500
Class Weight	Balanced
Minimum Samples per Leaf	2
Random State	42
Parallel Processing	-1
Training Features	268
Missing Value Handling	Median Imputation

Model Evaluation

The deployed model was evaluated using ground-truth labels available in the uploaded dataset.

Results

Metric	Score
Accuracy	98.00%
Precision	100.00%
Recall	96.00%
F1 Score	97.96%
ROC-AUC	0.9920

Why these metrics matter

Accuracy — 98.00%

Measures the overall percentage of correctly classified processes.

Precision — 100.00%

Measures how many processes predicted as malware were actually malware. The evaluation produced no false positives among the predicted malware processes.

Recall — 96.00%

Measures how many of the actual malware processes were successfully detected.

F1 Score — 97.96%

Provides a combined measure of Precision and Recall and is useful when both types of classification errors matter.

ROC-AUC — 0.9920

Measures the model's ability to distinguish between malware and benign processes across different classification thresholds.

Streamlit Dashboard

The trained model has been deployed using Streamlit.

The application allows users to upload a process-level CSV file and automatically performs the same feature engineering required by the trained model.

Dashboard includes

Dataset overview
Total records
Unique processes
Numerical feature count
Ground-truth availability
Malware vs benign distribution
Malware detection summary
Malware probability distribution
Top feature importance
Ground-truth evaluation
Confusion matrix
Prediction results
Downloadable prediction CSV
Model configuration

Example deployment result

For the uploaded malwaredataset.csv:

Processes analyzed: 100
Malware detected: 48
Benign detected: 52
Malware rate: 48.00%

The application also calculates evaluation metrics when the uploaded dataset contains the classification ground-truth column.

Feature Importance

The Extra Trees model provides feature importance scores that help identify which engineered behavioral features contribute most to the classification decision.

The Streamlit dashboard displays the Top 15 features based on the trained model's feature importance values.

This provides some interpretability into how process behavior contributes to the malware prediction.

Project Structure

Malware-Detection-Using-Machine-Learning/
│
├── app.py
├── malware_detection.ipynb
│
├── malwaredataset.csv
├── malware_process_features.csv
│
├── malware_model.pkl
├── malware_feature_columns.pkl
│
├── malware_model_cv_results.csv
├── malware_model_test_results.csv
│
├── requirements.txt
├── .gitignore
└── README.md

Important Files

`malware_detection.ipynb`

Contains the complete machine learning workflow:

Data loading
Data exploration
Preprocessing
Process-level aggregation
Feature engineering
Model training
Model comparison
Evaluation
Feature importance
Model serialization

`app.py`

Streamlit application used for deployment.

It handles:

CSV upload
Process-level feature engineering
Model loading
Prediction
Probability calculation
Evaluation against ground truth
Visualization
Prediction download

`malware_model.pkl`

Serialized trained Extra Trees pipeline used by the Streamlit application.

`malware_feature_columns.pkl`

Stores the feature columns expected by the trained model and helps maintain consistency between training and prediction.

`malware_model_test_results.csv`

Contains model evaluation results used for comparing the tested algorithms.

`malware_model_cv_results.csv`

Contains cross-validation results from model development.

Tech Stack

Programming

Python

Data Processing

Pandas
NumPy

Machine Learning

Scikit-learn

Visualization

Matplotlib
Streamlit

Model Persistence

Joblib

Deployment

Streamlit Cloud

Algorithms

Logistic Regression
KNN
SVM
Random Forest
Extra Trees
HistGradientBoosting

Running the Project Locally

1. Clone the repository

git clone https://github.com/nikh0011/Malware-Detection-Using-Machine-Learning.git
cd Malware-Detection-Using-Machine-Learning

2. Create a virtual environment

python -m venv venv

Activate it on Windows:

venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Start the Streamlit application

streamlit run app.py

The application will open in your browser.

Example Workflow

Raw Malware Dataset
        ↓
Data Cleaning
        ↓
Sort by Process + Time
        ↓
Process-Level Aggregation
        ↓
Statistical Features
        ↓
Temporal Features
        ↓
Train/Test Split
        ↓
Model Training
        ↓
Model Evaluation
        ↓
Extra Trees Selection
        ↓
Model Serialization
        ↓
Streamlit Deployment

What I Worked On

This project involved the complete machine learning workflow rather than only training a classifier.

Key areas I worked on include:

Exploratory data analysis
Process-level data aggregation
Behavioral feature engineering
Statistical and temporal feature extraction
Handling class imbalance
Training and comparing multiple ML algorithms
Model evaluation using Accuracy, Precision, Recall, F1 and ROC-AUC
Feature importance analysis
Saving the trained model using Joblib
Building a Streamlit prediction dashboard
Integrating the trained pipeline with the deployed application
Generating downloadable prediction results

Key Takeaways

The project demonstrates how process behavior can be transformed into machine-learning features for malware classification.

The final Extra Trees pipeline achieved:

98.00% Accuracy | 100.00% Precision | 96.00% Recall | 97.96% F1 | 0.9920 ROC-AUC

The main focus was not only model accuracy, but also building a complete pipeline from raw behavioral data → feature engineering → model training → evaluation → deployment.

Project: https://github.com/nikh0011/Malware-Detection-Using-Machine-Learning

Live Demo: https://malware-detection-using-machine-learninggit-ul5eeutzwcqfpyttct.streamlit.app/

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
README.md		README.md
app.py		app.py
malware_detection.ipynb		malware_detection.ipynb
malware_feature_columns.pkl		malware_feature_columns.pkl
malware_model.pkl		malware_model.pkl
malware_model_cv_results.csv		malware_model_cv_results.csv
malware_model_test_results.csv		malware_model_test_results.csv
malware_process_features.csv		malware_process_features.csv
malwaredataset.csv		malwaredataset.csv
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation