SPX Prediction Project

A comprehensive quantitative finance project for S&P 500 (SPX) prediction using advanced statistical methods, machine learning models, and regime change detection techniques.

Project Overview

This repository contains multiple approaches to SPX prediction and analysis:

• Statistical Analysis: EWMA variance calculations, rolling covariance analysis, and correlation matrix validation
• Machine Learning Models: Gradient boosting, LSTM, Random Forest, and ensemble methods
• Regime Change Detection: CUSUM-based and cluster-based regime identification
• Extended Methods: Gaussian Mixture Models and Graph-based approaches

Repository Structure

spx_prediction/
├── stats_calc/              # Statistical calculations and analysis
│   ├── main.py              # Main execution script
│   ├── preprocess_data.py   # Data preprocessing
│   ├── calculate_stats.py   # EWMA, covariance calculations
│   ├── evaluate_stats.py    # Matrix validation
│   ├── config.py            # Configuration management
│   ├── utils.py             # Utility functions
│   ├── requirements.txt     # Python dependencies
│   └── README.md            # Detailed documentation
│
├── spx_prediciton/          # Main prediction models
│   ├── main.py              # Main execution
│   ├── preprocess_data.py   # Data preprocessing
│   ├── build_model.py       # Model building
│   ├── eveluate_model.py    # Model evaluation
│   ├── dataset/             # Data files
│   ├── results/             # Model results and visualizations
│   │   ├── Updated Results.ipynb  # Comprehensive results analysis
│   │   ├── plot_1.png       # Confusion matrix
│   │   ├── plot_2.png       # Cumulative returns comparison
│   │   ├── plot_3.png       # Trading signals
│   │   ├── description.txt   # Results description
│   │   └── *.csv            # Trading signal files
│   └── *.ipynb              # Jupyter notebooks for analysis
│
└── extend/                  # Extended methods and experiments
    ├── Gaussian_Mixture_Model.ipynb
    ├── Graph-based_Method.ipynb
    ├── LSTM_RandomForest.ipynb
    └── Supplyment.pdf       # Supplementary documentation

Features

Statistical Analysis (stats_calc/)

• EWMA Variance: Exponential Weighted Moving Average variance with custom spans
• Blended Variance: Weighted combination of multiple EWMA variances
• Rolling Covariance: Window-based covariance calculations
• Matrix Validation: Positive Semi-Definite (PSD) property checking

Prediction Models (spx_prediciton/)

• Gradient Boosting (Classification & Regression)
• Model evaluation and performance metrics
• Data preprocessing and feature engineering
• Exploratory Data Analysis (EDA)

Extended Methods (extend/)

• Gaussian Mixture Models for regime identification
• Graph-based methods for market structure analysis
• LSTM and Random Forest ensemble approaches

Getting Started

Prerequisites

• Python 3.8+
• Required packages (see stats_calc/requirements.txt)

Installation

1. Clone the repository:

git clone <repository-url>
cd spx_prediction

2. Install dependencies:

cd stats_calc
pip install -r requirements.txt

3. For automated setup (Unix/Mac):

cd stats_calc
chmod +x install.sh
./install.sh

Usage

Statistical Analysis

cd stats_calc
python main.py

Prediction Models

cd spx_prediciton
python main.py

Jupyter Notebooks

Open and run the relevant notebooks in spx_prediciton/ or extend/ directories.

Documentation

• stats_calc/README.md: Detailed documentation for statistical calculations
• stats_calc/PROJECT_OVERVIEW.md: Complete project setup and overview
• extend/Supplyment.pdf: Supplementary documentation for extended methods

Data

The project uses financial time series data including:

• S&P 500 index data
• Commodity prices
• Currency exchange rates
• Energy consumption data

Data files are located in:

• spx_prediciton/dataset/
• stats_calc/data.csv

Results

The project includes comprehensive model evaluation and trading strategy results. Detailed analysis can be found in spx_prediciton/results/Updated Results.ipynb.

Model Performance

The gradient boosting classification model was evaluated on test data with the following performance metrics:

• Accuracy: 54%
• Precision (Class 0): 0.50
• Recall (Class 0): 0.48
• F1-Score (Class 0): 0.49
• Precision (Class 1): 0.57
• Recall (Class 1): 0.58
• F1-Score (Class 1): 0.58

Confusion matrix showing the classification performance on test data

Trading Strategy Performance

The trading strategy based on the model predictions shows promising results when compared to the S&P 500 benchmark:

Cumulative return comparison between the trading strategy and S&P 500 index

Trading signals generated by the model over time

Signal Files

The results include three signal files for different trading strategies:

• long_only.csv: Long-only signals with 1 for long positions and 0 for neutral
• long_short.csv: Long-short signals replacing 0 with -1 for short positions
• long_short_2.csv: Long-short signals excluding the period where the model did not perform well (2015-2016)

Note: The long-short strategy may exhibit higher volatility. While overall returns may be larger, the Sharpe ratio might be smaller. Future improvements could include a three-class classification system (long, neutral, short) for better results.