← Back to files
Our RDS project, explained simply
Paper: Nassar, Jafar & Rahhal, "A novel deep multi-criteria collaborative filtering model", Knowledge-Based Systems (2020).
The whole project in one sentence:
"The paper guesses hotel ratings in two steps using sub-ratings. Most test users are brand new, so the model knows nothing about them. We gave new users a learned 'typical traveller' profile and trained both steps together, which cut the error by 5%."
1. The goal
TripAdvisor wants to suggest hotels you will like. To do that, it guesses the star rating (1–5) you would give a hotel you haven't stayed at. Then it shows you the hotels with the highest guesses.
2. The paper's idea
When people review a hotel, they don't only give overall stars. They also rate 7 things: value, rooms, location, cleanliness, check-in, service and business service.
The paper says: don't guess the overall stars directly. First guess the 7 small ratings, then combine them.
Like a teacher's final grade:
Step A: predict your marks in each of 7 subjects.
Step B: combine the 7 marks into one final grade.
Each step is a neural network. The only thing the model knows about you is your user number, and about the hotel its hotel number. From past reviews, it learns a short "profile" (a list of numbers) for every user and every hotel, and it guesses using those profiles.
3. First, we rebuilt the paper and checked it
- Same data. The paper's download link is dead, so we found an old copy on the Internet Archive: about 79,000 real TripAdvisor reviews (97% of the paper's data).
- Checked our setup. We ran 5 standard older methods. Our numbers matched the paper's almost exactly (within 0.005 stars), so our setup is right.
- Ran the paper's model. Its average mistake was 0.78 stars. That is slightly worse than the 0.76 the paper reports, but it is still the best of the methods in the paper.
4. The problem we found
84% of the reviews we test on are written by people who never wrote a review before. Most people review once and disappear.
For a brand-new person, the model has learned nothing, so their profile is just random numbers. That is like a teacher predicting a new student's marks from a random guess.
5. Our fix
Fix 1: one "typical traveller" profile for new people (biggest gain)
Before: a new person looking at Hotel X gets a guess that mixes Hotel X's profile (good, learned) with the new person's profile (random). The random half spoils the guess.
After: we keep one extra profile called "new person". Every new user gets this profile instead of random numbers.
How does it learn? During training, about 30% of the time we hide who the reviewer is and show the model only "a new person rated Hotel X 4 stars". After thousands of these, the "new person" profile learns things like "a typical traveller gives this beach hotel about 4.5, and this airport hotel about 3.5".
Like a shop with a customer who has no purchase history: instead of recommending random products, it shows what most customers buy.
Fix 2: train both steps together at the end
The paper trains Step B using the real 7 sub-ratings from reviews. But when the model is used, Step B gets Step A's guesses, which are rougher than the real ones.
Like a student who practises only with neatly printed questions, then gets messy handwritten ones in the exam.
So after the normal training, we train Step A and Step B together for a while. Step B now practises on the same kind of input it gets in the real test.
Fix 3: small clean-ups
- No impossible answers: predictions are kept between 1 and 5 stars (the paper's model could say 5.4).
- No "stuck" networks: the paper's network sometimes got stuck predicting 0 for everything. We changed its setup so this can't happen.
- Average of 5 runs: we train 5 times and average the answers, so one unlucky run can't spoil the result.
6. The result
For every test review, we compare the guess with the real stars. If someone gave 4★ and the model guessed 4.6, the mistake is 0.6. Averaged over about 16,000 test reviews:
| Average mistake (lower is better) |
|---|
| Paper's model | 0.78 stars |
| Our model | 0.74 stars (about 5% smaller) |
Why it's not luck: we split the data into 5 parts and tested 5 times, each time on reviews the model never saw. Ours won all 5 times. The chance of that by luck is about 1 in 1,000. It also beats the paper's own reported number (0.76).
Where the improvement comes from
| Who wrote the review | Paper's model | Ours |
|---|
| New people (84% of reviews) | 0.78 | 0.73, much better |
| Returning people (16%) | 0.74 | 0.78, a bit worse |
- New people improved a lot: they now get the "typical traveller" profile instead of random numbers.
- Returning people got slightly worse: because we hide identities during training, the model relies less on their one or two old reviews.
- Overall it's clearly better, because new people are most of the reviews.
7. What to say to the teacher (5 minutes)
- Goal (1 min): "We guess the stars a user would give a hotel they haven't visited, so we can recommend hotels."
- Paper's idea (1 min): "First guess 7 sub-ratings like rooms and service, then combine them into overall stars. Two neural networks."
- Our rebuild (1 min): "We rebuilt it on the same TripAdvisor data. Older methods matched the paper almost exactly. The paper's model got 0.78 error."
- Problem and fix (1 min): "84% of test reviewers are brand new, so the model gives them random profiles. We gave them a learned 'typical traveller' profile, and trained both steps together."
- Result (1 min): "Error dropped from 0.78 to 0.74, winning on all 5 test splits. The only cost: returning users, who are 16%, got slightly worse."
If the teacher asks…
- "Why is the paper's model worse in your test than in the paper?" The paper leaves out some training details, and our data copy is 97% of theirs. The older methods matched exactly, so our setup is right.
- "Did you cheat by tuning on the test data?" No. We chose every setting on a separate practice split of the training data. The test data was used only once, at the end.
- "Is 5% a big improvement?" For rating prediction, yes. It's about the same size as the improvement the paper itself claims over older methods, and it held on all 5 tests.
← Back to files