This project analyzes the Expedia hotel recommendations dataset (803 K booking events) to surface six key travel-intelligence signals:
| # | Job | Insight |
|---|---|---|
| 1 | Hotel Cluster Rankings | Which of the 100 hotel clusters is booked most? |
| 2 | Top 10 Clusters | Quick-glance leaderboard for the highest-demand clusters |
| 3 | Family Destinations | Where do travelers with children actually book? |
| 4 | Weekend vs. Weekday | How does booking behaviour shift across the calendar year? |
| 5 | Country → Destination | Per-country top destination (window function, true partitioned rank) |
| 6 | Reproducible Sample | Deterministic 10 % Bernoulli sample for fast iteration |
Originally built as a Hadoop MapReduce Java project in 2017 (CDH 5.11, Java 8, zero tests, several logic bugs). Fully rewritten for 2025 using DuckDB, pandas, and pytest.
| Legacy (Hadoop MapReduce) | This project (DuckDB) | |
|---|---|---|
| Infrastructure | YARN cluster + HDFS | Your laptop |
| Startup time | 5 – 30 min | < 1 s |
| Runtime on 803 K rows | 10 – 30 min | 2.1 s |
| Setup | Install Hadoop + JVM + Maven | pip install -e . |
| Tests | 0 | 30 (all passing) |
| Known bugs fixed | — | 6 (see below) |
# 1 — clone
git clone https://github.com/shrican/expedia-bigdata-analysis.git
cd expedia-bigdata-analysis
# 2 — install (creates the `expedia-run` CLI)
pip install -e ".[dev]"
# 3 — add data (download train.csv from Kaggle, rename to trainb.csv)
# https://www.kaggle.com/competitions/expedia-hotel-recommendations/data
cp ~/Downloads/train.csv input/trainb.csv
# 4 — run the full pipeline
expedia-run --data input/trainb.csv --output output/Results land in output/ — six CSV files and four PNG charts, generated in ~2 seconds.
Loading data from: input/trainb.csv
Registered 803,314 rows in 0.24s
Running analysis jobs:
[hotel_cluster_bookings] 100 rows (0.17s)
[top_n_hotel_clusters] 10 rows (0.17s)
[popular_family_destinations] 15 rows (0.17s)
[monthly_weekend_bookings] 24 rows (0.17s)
[country_popular_destinations] 1204 rows (0.17s)
[downsample (10%)] 80832 rows (0.57s)
Done. Total wall time: 2.10s
Cluster 91 dominates with 2,670 confirmed bookings — nearly 38 % more than runner-up cluster 48.
Destination 8250 leads family bookings by a wide margin, suggesting a resort-type property cluster.
Weekday bookings consistently outpace weekends 3:1. March and July peak — spring break and summer holiday patterns clearly visible.
Country 66 (USA) produces the most bookings at its #1 destination by a factor of ~10x, reflecting Expedia's US-heavy user base.
flowchart LR
CSV[("input/trainb.csv\n803K rows")] --> L["loader.load()\nDuckDB view: bookings"]
L --> J1["hotel_cluster_bookings\nGROUP BY hotel_cluster"]
L --> J2["popular_family_destinations\nWHERE srch_children_cnt > 0"]
L --> J3["monthly_weekend_bookings\nDAYOFWEEK + MONTH"]
L --> J4["country_popular_destinations\nROW_NUMBER() OVER PARTITION"]
L --> J5["downsample\nSAMPLE 10% (bernoulli, 42)"]
J1 --> T["top_n_hotel_clusters\nLIMIT 10"]
J1 --> CSV1[("hotel_cluster_bookings.csv")]
T --> CSV2[("top_n_hotel_clusters.csv")]
J2 --> CSV3[("popular_family_destinations.csv")]
J3 --> CSV4[("monthly_weekend_bookings.csv")]
J4 --> CSV5[("country_popular_destinations.csv")]
J5 --> CSV6[("downsample.csv")]
T --> P1[("top_clusters.png")]
J2 --> P2[("family_destinations.png")]
J3 --> P3[("monthly_bookings.png")]
J4 --> P4[("country_destinations.png")]
expedia-bigdata-analysis/
├── src/expedia/
│ ├── schema.py # typed column name constants
│ ├── loader.py # CSV → DuckDB view (+ computed columns)
│ ├── viz.py # matplotlib chart generators
│ ├── pipeline.py # click CLI: expedia-run
│ └── jobs/
│ ├── hotel_clusters.py
│ ├── top_clusters.py
│ ├── family_destinations.py
│ ├── monthly_bookings.py
│ ├── country_destinations.py
│ └── downsample.py
├── tests/
│ ├── conftest.py # DuckDB fixtures (200-row synthetic data)
│ └── test_jobs.py # 30 tests covering all 6 jobs + CLI
├── docs/assets/ # committed chart PNGs for the README
├── pyproject.toml # PEP 621, hatchling, ruff, pytest config
├── Makefile # install / test / lint / run / clean
├── Dockerfile
└── .github/workflows/ci.yml # matrix CI on Python 3.11 + 3.12
make install # pip install -e ".[dev]"
make test # pytest with coverage
make lint # ruff check
make run # expedia-run on input/trainb.csv
make clean # remove output/, __pycache__, .coveragedocker build -t expedia-analysis .
docker run --rm \
-v "$(pwd)/input:/app/input" \
-v "$(pwd)/output:/app/output" \
expedia-analysis| Bug | Original | Fixed |
|---|---|---|
Static counter in Downsample |
static int cnt reset per JVM, not globally |
SAMPLE … (bernoulli, seed) — globally correct |
CountryWisePopularDestinations |
Copy-paste of MonthWiseWeekendBookings mapper |
Real SQL ROW_NUMBER() OVER (PARTITION BY country) |
StateAnalysis |
Identical duplicate of HotelClusterBookingsAnalysis |
Removed; not a distinct job |
Wrong setJarByClass() |
Two jobs referenced sibling classes | N/A — no JARs |
| Header row fragility | key.get() != 0 breaks on multi-split input |
DuckDB parses header natively |
| Magic column indices | data[18], data[23] scattered everywhere |
Typed constants in schema.py |
Download from Kaggle (free account required):
https://www.kaggle.com/competitions/expedia-hotel-recommendations/data
Place train.csv (or any subset) at input/trainb.csv. The input/ directory is git-tracked via .gitkeep; .csv files are gitignored to keep the repo lightweight.
MIT © shrican