Sampreeth Avvari
Resume
Read the story

Architecture

Finds movie twins among 330,975 users.LSH in 37 seconds.

Spark on Dataproc splits 33.8 million MovieLens ratings by time, trains ALS and scores it against a popularity baseline. Beside it, MinHash and LSH in one Python process find users who rated the same movies, and a correlation check tests the top 100 pairs.

37 s

to index and query all 330,975 users, 10 s in and 27 s out

0.73

average rating correlation of the top 100 twins, against 0.03 for 100 random pairs

33.8M ratings

in MovieLens ml-latest, January 1995 to July 2023

22.2M pairs

candidate twins from LSH at a 0.9 threshold

New York --:-- simulated

One user's ratings
Recommender at scale Spark on Dataproc splits 33.8 million MovieLens ratings by time, trains ALS and scores it against a popularity baseline. Beside it, MinHash and LSH in one Python process find users who rated the same movies, and a correlation check tests the top 100 pairs. Popularity baseline damped mean, top 100 MovieLens 33.8M ratings HDFS CSV and Parquet Time split Spark, per user Spark ALS rank 200, 15 iterations Ranking metrics MAP and NDCG at 100 MinHash 30 permutations LSH index 6 bands, threshold 0.9 Candidate pairs 22.2M pairs Correlation check top 100 vs random 100 CSV Parquet train set top 100 each train set same 100 for all movie sets signatures query top 100 NYU Dataproc, Spark on HDFS NYU Greene HPC, pandas

Hover a part to trace what it touches. Click it for its notes.

  • Built by me
  • Managed by Google
  • Outside system
  • Main data path
  • On demand
  • Credentials cross

When it breaks

What each failure does, and what it does not do.

  1. Spark lost its workers

    The Spark MinHashLSH similarity join lost nodes on Dataproc even on a tenth of the users, so the twins search moved to datasketch in one Python process.

    Capstone Proj/Capstone_g69_Report.pdf

  2. A small dataset says little

    On the 600 user small set the same settings found 151 pairs, and their correlation barely beat random, 0.22 against 0.15.

    Capstone Proj/Capstone_g69_Report.pdf

  3. A user ALS never saw

    Cold start rows are dropped instead of scored, so ranking covers only users and movies with training history.

    Capstone Proj/ALS/als.py

What I tuned

Small choices that keep it fast and cheap.

  1. 30 permutations, 6 bands

    Settings from 15 to 500 permutations and 2 to 8 bands were tried. 30 and 6 gave the top 100 the highest correlation, 0.7348.

    Capstone Proj/Capstone_g69_Report.pdf

  2. Signatures in minutes

    30 permutations built every signature in 6 minutes 19 seconds. 500 took 33 minutes and scored worse.

    Capstone Proj/Capstone_g69_Report.pdf

  3. Damped popularity

    Adding 10,000 to each movie's rating count keeps rarely rated movies out of the baseline's top 100.

    Capstone Proj/ALS/popularity_baseline.py

Guards

Rules the code enforces, not rules people remember.

  1. Hold out the latest ratings

    Validation and test users keep their earliest 80 percent of ratings in training and are scored on the rest, in time order.

    Capstone Proj/ALS/traintestsplit.py

  2. Random pairs as a control

    The twins are compared with 100 random pairs scored the same way, so a high correlation has something to beat.

    pandas-greene/large.ipynb

  3. Splits written once

    The partitions are saved to Parquet once and reused, so every experiment reads the same train, validation and test rows.

    Capstone Proj/Capstone_g69_Report.pdf