Turn raw shopper behavior into trusted, hourly business insight - on a private, cost-first AWS pipeline.

Batch ETL • AWS Serverless • PostgreSQL Data Warehouse • R Shiny Analytics

Storefront • Data pipeline • Dashboard • Local environment • Workshop

On this page: The problem · How we solve it · What it costs · How the pipeline runs · AWS services and the problem each solves · Technical deep dive · Roadmap · Repositories

Every click tells a story. Most online shops cannot read it.

E-commerce teams rarely lack data. What they lack is a reliable way to turn scattered shopper behavior into decisions - at the right time, in the right context, and on infrastructure they can actually run. Once a business takes behavior-driven marketing seriously, the same questions keep coming up:

"Where exactly do shoppers give up?"

Product views, clicks, add-to-cart and checkout are recorded in different places and in different shapes. Nobody can draw the full journey, so the drop-off points stay a guess.

"Why did last week's numbers change?"

Every feature and every release sends events its own way. The same report gives different answers over time, and the business slowly stops trusting the dashboard.

"Can we fix this report without breaking history?"

With no original copy of the data underneath, every change to the processing logic is a gamble. Numbers cannot be audited, errors cannot be traced, and past data cannot be replayed.

"Can analytics run safely inside a locked-down network?"

Real environments keep databases in private networks, with no SSH and strict isolation. Setups that work in a demo often break once the doors are closed.

"Do we really need a streaming platform for hourly decisions?"

Always-on streaming stacks, NAT gateways and managed warehouses cost far more than a team that decides hour by hour can justify.

Scope. The storefront - sign-in, browsing, cart and checkout - is the foundation of a working e-commerce site. The heart of this project is what we built on top of it: the clickstream pipeline, batch processing, data warehouse and analytics dashboard.

_{The SHOPCART storefront - every browse, click and add-to-cart here becomes a clickstream event.}

We built a batch-based clickstream analytics platform that puts data discipline, day-to-day operability and cost ahead of novelty. Each answer below mirrors one of the questions above.

One event language, from browser to dashboard.

A single publisher built into the storefront sends every interaction in the same shape, tied to the same shopper, browser and session. For the first time, the shopper journey can be drawn end to end.

The same numbers, every release.

A fixed event contract is enforced twice - once in the browser and once in the warehouse schema - and every event is stamped with one trusted server clock. Last week's report means the same thing today.

Every raw click is kept.

Events land untouched in Amazon S3, filed by hour, before anything transforms them. Any report can be rebuilt and any hour replayed, without ever touching the original data.

Raw event in S3 - the original payload plus server metadata	Curated row in the warehouse - one fixed schema for every event (IDs blurred)

Private by default, and still easy to run.

The warehouse, the processing and the dashboard have no public address. The team manages them through AWS Systems Manager without SSH, and an email alert goes out as soon as a pipeline run fails.

Hourly batches, pay-per-use.

Processing runs once an hour - the pace at which marketing and product teams actually make decisions - on serverless compute that costs nothing while idle. No NAT gateway, no streaming cluster.

The R Shiny dashboard reads straight from the data warehouse and refreshes every 10 seconds:

• KPI cards - Total events, Unique clients, Checkout rate (checkout_complete ÷ add_to_cart_click).
• Overview - Events over time, Top 10 event types, Events by login state (logged-in vs anonymous shoppers).
• Products - Top categories and Top brands by engagement.
• Raw sample - the latest product events, paged and newest first, for spot checks.
• Filters - date range and user login state.

_{Overview tab - KPI cards, events over time, top event types and events by login state}

_{Products tab - top categories and top brands by shopper engagement}

Cost-first is not a slogan - these are the real AWS bills from our development and demo period in December 2025.

The serverless pieces - API Gateway, both Lambda functions, EventBridge and S3 - barely register on the bill, because they cost nothing while the store is quiet.

_{Daily AWS cost by service during development (2-7 December 2025)}

The numbers below match the numbered circles in the diagram.

sequenceDiagram
    actor Shopper
    participant Pub as ClickStream publisher
    participant API as API Gateway
    participant Ing as Lambda Ingest
    participant Raw as S3 raw bucket
    participant EB as EventBridge
    participant ETL as Lambda ETL
    participant DW as PostgreSQL DW
    participant Dash as R Shiny
    actor Analyst
    Shopper->>Pub: Clicks Add to cart at 14:20 UTC
    Pub->>API: POST /clickstream (add_to_cart_click)
    API->>Ing: Invoke
    Ing->>Raw: Write events/2026/09/28/14/event-uuid.json
    Note over Raw: Hour 14 keeps filling until 15:00 UTC
    EB->>ETL: 15:05 UTC - process hour 14
    ETL->>Raw: List and read hour 14 via S3 Gateway Endpoint
    ETL->>DW: Insert rows, skip duplicate event_id
    Dash->>DW: Query every 10 seconds
    Dash->>Analyst: Updated KPIs and charts

Every service in the platform earns its place by removing a concrete problem for the business.

📘 Step-by-step workshop: AWS First Cloud Journey - Clickstream workshop