The goal of this project is to build Movie Recommendation System with Spark using item-based colaborative filtering method.
- Python 3.7
- PySpark 2.4.5
- Numpy
- Matplotlib
- Seaborn
Data come from https://grouplens.org/datasets/movielens/. I used 100K movie ratings dataset from 2018 year.
- Create RDD of movie similarities
- Set similarity thresholds for movie ratings, category and co occurrence
- Reduce RDD of movie similarities based on similarity thresholds
- Set the rules for the recommendation system
- Create test cases
- Evaluate the recommendation system based on test cases
- Discuss the results
- Create a Movie Recommendation System Console Application
- Movie_Recommendation_System_Notebook.ipynb - notebook with calculations and descriptions
- Movie_Recommendation_System_Notebook.py - console application to run in the terminal
- MoviePairSimilarities - folder contains serialized movie pair similarities RDD
- movieNameDict.pickle - serialized lookup table of movie names
- movieList.pickle - serialized list of avaiable movie to rate in console application
- extract_available_movies.py - script to create list of available movies to rate
- ml-latest-small - folder contains datasets
User ratings:
- Lord of the Rings: The Two Towers, The (2002) - 5 stars
- Kill Bill: Vol. 1 (2003) - 5 stars
- Harry Potter and the Chamber of Secrets (2002) - 1 stars
Recomendation:
Recomended Movie #1
Title: Lord of the Rings: The Fellowship of the Ring, The (2001)
Similarity Score: 4.977
Similarity Types: 1.0
Cooccurence: 166
Recommended by: Lord of the Rings: The Two Towers, The (2002)
----------------------------------------------------------------------------------------------------
Recomended Movie #2
Title: Lord of the Rings: The Return of the King, The (2003)
Similarity Score: 4.9679
Similarity Types: 1.0
Cooccurence: 161
Recommended by: Lord of the Rings: The Two Towers, The (2002)
----------------------------------------------------------------------------------------------------
Recomended Movie #3
Title: Kill Bill: Vol. 2 (2004)
Similarity Score: 4.9676
Similarity Types: 0.6667
Cooccurence: 103
Recommended by: Kill Bill: Vol. 1 (2003)
----------------------------------------------------------------------------------------------------
Recomended Movie #4
Title: Army of Darkness (1993)
Similarity Score: 4.9421
Similarity Types: 0.2
Cooccurence: 23
Recommended by: Kill Bill: Vol. 1 (2003)
----------------------------------------------------------------------------------------------------
Recomended Movie #5
Title: Spirited Away (Sen to Chihiro no kamikakushi) (2001)
Similarity Score: 4.9276
Similarity Types: 0.3333
Cooccurence: 63
Recommended by: Lord of the Rings: The Two Towers, The (2002)
----------------------------------------------------------------------------------------------------
Recomended Movie #6
Title: Pulp Fiction (1994)
Similarity Score: 4.924
Similarity Types: 0.5
Cooccurence: 99
Recommended by: Kill Bill: Vol. 1 (2003)
----------------------------------------------------------------------------------------------------
Recomended Movie #7
Title: Who Framed Roger Rabbit? (1988)
Similarity Score: 4.9211
Similarity Types: 0.1429
Cooccurence: 39
Recommended by: Kill Bill: Vol. 1 (2003)
----------------------------------------------------------------------------------------------------
Recomended Movie #8
Title: Inglourious Basterds (2009)
Similarity Score: 4.9196
Similarity Types: 0.3333
Cooccurence: 56
Recommended by: Kill Bill: Vol. 1 (2003)
----------------------------------------------------------------------------------------------------
Recomended Movie #9
Title: Edward Scissorhands (1990)
Similarity Score: 4.9111
Similarity Types: 0.3333
Cooccurence: 52
Recommended by: Lord of the Rings: The Two Towers, The (2002)
----------------------------------------------------------------------------------------------------
Recomended Movie #10
Title: Toy Story 3 (2010)
Similarity Score: 4.8948
Similarity Types: 1.0
Cooccurence: 44
Recommended by: Lord of the Rings: The Two Towers, The (2002)
----------------------------------------------------------------------------------------------------
Discussion of results:
- As expected in recommended movies appear:
- Movies from the Lord of the Rings saga
- Kill Bill: Vol. 2
- No movies from Harry Potter saga
- More Quentin Tarantino (director of Kill Bill) movies like: Pulp Fiction and Inglourious Basterds
- Army of Darkness - I personally do not know this movie but from the sample photos and opinions it results from it has a very similar atmosphere to the Lord of the Rings
- Spirited Away - Japanese anime in the atmosphere of magic and gods, very high marks and good reviews
- Who Framed Roger Rabbit? - anime in the atmosphere of Kill Bill
- Toy Story 3 - belongs to the same categories like the Lord of the Rings: Adventure, Fantasy
- Edward Scissorhands - very good movie, but probably should not be on this list