RetailFlow

Dashboard

Pipeline

Olist CSVs → Airflow → Python → PostgreSQL → dbt → Streamlit

Airflow orchestrates ingestion and then triggers dbt to build and test the analytics models. Docker Compose runs the local services.

What I Built

RetailFlow is an end-to-end ELT pipeline built around the Olist e-commerce dataset. The pipeline ingests raw CSV data into PostgreSQL, transforms it with dbt, validates the resulting models with automated tests, and exposes a daily sales mart through an interactive Streamlit dashboard.

Apache Airflow orchestrates the workflow, while Docker Compose provides a reproducible local environment for the warehouse and orchestration services.

Key Engineering Decisions

• Used an ELT architecture: raw source data is loaded into PostgreSQL first, while transformations happen inside the warehouse with dbt.
• Separated raw, staging, and analytics layers so ingestion, cleaning, dimensional modeling, and business reporting remain clearly defined.
• Modeled order items at a precise grain using order_id + order_item_id, which prevents accidental duplication in the fact table.
• Handled Olist customer identity carefully by using customer_unique_id to represent the underlying customer across multiple orders.
• Added automated dbt tests for uniqueness, null values, relationships, and fact-table grain.
• Preserved downstream dependencies during reloads by refreshing raw tables without dropping and recreating them.
• Kept the dashboard dependent only on the final analytics mart, rather than embedding business logic or complex joins inside the frontend.
• Used a lightweight exported mart snapshot for the public dashboard, which allows the live demo to remain self-contained without exposing database credentials or requiring a continuously running cloud warehouse.