Netflix-Stream: Video Ingestion & HLS Streaming Microservices

Netflix-Stream is a microservices-based video cataloging, ingestion, encoding, and streaming system. It is designed to handle video uploads, perform distributed HLS transcoding into multiple resolutions, and serve dynamic streaming links.

---

🏗️ Architecture & Component Overview

flowchart TD
    subgraph Client
        Browser[Client Browser / App]
    end

    subgraph Microservices
        CS[Content Service :8081]
        VS[Video Service :8082]
        ES[Encoding Service]
        SS[Streaming Service]
    end

    subgraph Infrastructure
        MySQL[(MySQL DB)]
        Redis[(Redis Cache)]
        Kafka{Kafka Topics}
        S3[[AWS S3 Bucket]]
    end

    %% Flow connections
    Browser -->|1. Add Metadata| CS
    CS <-->|Store/Retrieve| MySQL
    
    Browser -->|2. Upload Video| VS
    VS -->|Save Raw Video| S3
    VS -->|3. Publish video.uploaded| Kafka
    
    Kafka -->|4. Consume Upload Event| ES
    ES -->|Download Raw Video| S3
    ES -->|5. FFmpeg Transcode to HLS| ES
    ES -->|6. Upload Playlist & Segments| S3
    ES -->|7. Publish video.encoded| Kafka
    
    SS <-->|Cache Stream URLs| Redis

Loading

1. Content Service (content-service) — Port 8081

• Purpose: Manages movie catalog metadata.
• Database: MySQL (content_db).
• Lifecycle: Manages the videoStatus transitions (PENDING ➔ UPLOADED ➔ READY).
• Responsibilities:
  • Exposes APIs for cataloging and searching movies.
  • Updates movie records with video keys and HLS stream URLs based on system event signals.

2. Video Service (video-service) — Port 8082

• Purpose: Manages file ingest.
• Storage: Amazon S3 (raw file storage).
• Responsibilities:
  • Accepts raw video file uploads via multipart HTTP requests.
  • Generates unique S3 object keys.
  • Publishes video.uploaded event to Kafka topic to trigger down-stream processing.

3. Encoding Service (encoding-service)

• Purpose: Performs video transcoding and adaptive bitrate segmentation.
• Engine: FFmpeg.
• Responsibilities:
  • Consumes raw video events from Kafka.
  • Downloads raw source files from AWS S3.
  • Encodes videos into 4 target qualities:
    • 1080p (5000k bitrate)
    • 720p (2800k bitrate)
    • 480p (1200k bitrate)
    • 360p (800k bitrate)
  • Generates HLS playlists (.m3u8 playlist files) and segment TS files (.ts).
  • Uploads all artifacts back to AWS S3 and broadcasts completion events.

4. Streaming Service (streaming-service)

• Purpose: Serves stream manifests and caches active stream links.
• Cache: Redis.

---

🛠️ Infrastructure Stack (Docker Compose)

The project includes a docker-compose.yml to spin up the required local backing systems:

• MySQL: Persists catalog tables. Runs on port 3306.
• Redis: Fast caching layers. Runs on port 6379.
• Apache Kafka & Zookeeper: Event distribution stream. Runs on port 9092.

---

🔄 Video Upload & Transcoding Flow

[User App]                           [Content DB]       [S3 Bucket]         [Kafka]
    │                                     │                  │                 │
    │ 1. POST /api/movies                 │                  │                 │
    ├────────────────────────────────────>│                  │                 │
    │    (Creates movie: PENDING)         │                  │                 │
    │                                     │                  │                 │
    │ 2. POST /api/v1/upload/{movie_id}   │                  │                 │
    ├───────────────────────────────────────────────────────>│                 │
    │    (Uploads raw video payload)      │                  │                 │
    │                                     │                  │                 │
    │ 3. Dispatch upload event            │                  │                 │
    │    ─────────────────────────────────────────────────────────────────────>│
    │                                     │                  │  [video.uploaded]
    │                                     │                  │                 │
    │                                     │                  │ 4. Listen event │
    │                                     │                  │   ──────────────┤
    │                                     │                  │   (Starts FFmpeg)
    │                                     │                  │                 │
    │                                     │                  │ 5. Upload HLS   │
    │                                     │                  │   ─────────────>│
    │                                     │                  │   (.m3u8 files) │
    │                                     │                  │                 │

---

🚀 Getting Started

Prerequisites

• Java 17 or higher
• Maven 3.x
• Docker & Docker Compose
• AWS Account (with access credentials for S3)
• FFmpeg (locally installed for the encoding service)

Step 1: Start Infrastructure Containers

From the root directory, run:

docker-compose up -d

Step 2: Configure Environment Variables

Ensure the following variables are set in your environment (or configure them in application.yaml profiles):

• AWS_ACCESS_KEY
• AWS_SECRET_KEY
• AWS_REGION
• S3_BUCKET_NAME

Step 3: Run the Services

Run each service using the Maven Wrapper:

# Start Content Service
cd content-service && ./mvnw spring-boot:run

# Start Video Service (in a new terminal)
cd video-service && ./mvnw spring-boot:run

# Start Encoding Service (in a new terminal)
cd encoding-service && ./mvnw spring-boot:run

---

📞 API Reference

Content Service (:8081)

1. Add Movie Metadata

• Endpoint: POST /api/movies
• Body:

{
  "title": "Inception",
  "description": "A thief who steals corporate secrets through the use of dream-sharing technology.",
  "genre": "SCIENCE_FICTION",
  "director": "Christopher Nolan",
  "cast": "Leonardo DiCaprio, Joseph Gordon-Levitt",
  "releasedate": 2010,
  "rating": 8.8,
  "thumbnailUrlString": "https://example.com/inception.jpg",
  "durationInMinutes": 148
}

2. Get All Movies

• Endpoint: GET /api/movies

3. Search Movies by Title

• Endpoint: GET /api/movies/search?title=Inception

4. Find Movies by Genre

• Endpoint: GET /api/movies/genre/{GENRE_NAME} (e.g. SCIENCE_FICTION)

---

Video Service (:8082)

1. Upload Video File

• Endpoint: POST /api/v1/upload/{movie_id}
• Content-Type: multipart/form-data
• Form Parameters:
  • file: (Raw Video File binary)