# Deep multi-criteria collaborative filtering: reproduction on TripAdvisor

A from-scratch PyTorch implementation of Nassar, Jafar & Rahhal, *A novel deep multi-criteria
collaborative filtering model for recommendation system*, Knowledge-Based Systems 187 (2020) 104811.
It uses the paper's own dataset (TripAdvisor hotel ratings, 7 criteria plus an overall rating) and
runs every experiment in Section 4.

## Files

| File | What it is |
|---|---|
| `DeepMCCF_TripAdvisor.ipynb` | The deliverable: executed, with every table, figure and the findings |
| `deepmccf.py` | The two DNNs, the single-DNN baseline, training, Surprise baselines, the five metrics |
| `prepare_data.py` | Parses the raw TripAdvisor files and builds `data/tripadvisor_mc.csv` |
| `run_experiments.py` | Runs Tables 3–6, Fig. 4 and the improved model over 5 folds; caches test predictions in `results/preds/` |
| `improved.py` | **DMCCF+**, the improved two-stage model |
| `accuracy_metrics.py` | Accuracy-type metrics (exact star, within 1 star, ROC-AUC, R², …) and the DMCCF+/SVD blend |
| `DeepMCCF_Presentation.pptx` | 9-slide summary deck |
| `dev_select.py` | Picks the DMCCF+ settings on an inner split of fold 0's training data (never the test folds) |
| `data/tripadvisor_mc.csv` | The cleaned ratings table (78,973 rows): all the experiments need |
| `data/raw/Review_Texts.zip` | The raw data (LARA TripAdvisor, Hongning Wang), 80 MB. Not in the shared zip: download it from [the Internet Archive's 2015 copy](http://web.archive.org/web/20150316061732/http://times.cs.uiuc.edu/~wang296/Data/LARA/TripAdvisor/Review_Texts.zip) only to rerun `prepare_data.py` |
| `results/` | Result tables (CSV), figures (PNG) and the run log |

## How to run

```bash
uv venv --python 3.11 .venv && uv pip install --python .venv/bin/python torch numpy pandas scikit-learn scikit-surprise matplotlib jupyter
```

```bash
.venv/bin/python prepare_data.py && .venv/bin/python run_experiments.py
```

The full run takes about 30 minutes on a laptop CPU, mostly the ablations; a GPU is not needed. After
that, the notebook runs in about 20 seconds because it reuses the cached predictions. Delete
`results/preds/` to recompute everything.

## The model (as in the paper)

1. **Criteria Ratings DNN:** user and item embeddings (64 each) → concatenate → ReLU `[128, 64, 32, 16, 8]` →
   7 ReLU outputs, one per criterion.
2. **Overall Rating DNN:** criteria z-scored with the training mean and std → ReLU `[64, 32, 16, 8]` → 1 ReLU output.
3. Trained **separately**: the second network learns from the true criteria. At prediction time, the
   criteria predicted by stage 1 go into stage 2.
4. Every weight starts from N(0, 0.05). Adam (lr 0.001), batch 512, MAE loss.
5. 5-fold cross-validation: 80 % training (including a 1 % validation set) and 20 % test.

## Data

| | Paper | This copy |
|---|---|---|
| Users | 72,119 | 70,523 |
| Hotels | 1,850 | 1,759 |
| Ratings | 81,085 | 78,973 |
| Sparsity | 99.94 % | 99.94 % |

The original host is offline. The only archived copy has 1,759 of the 1,850 hotel files. Every
rating bucket holds about 97.4 % of the paper's count, with the same distribution.

## Results (5-fold CV, mean)

| Method | MAE (paper) | MAE (ours) | F1 (paper) | F1 (ours) | F2 (paper) | F2 (ours) |
|---|---|---|---|---|---|---|
| SVD | 0.807 | 0.803 | 0.837 | 0.840 | 0.795 | 0.799 |
| SVD++ | 0.822 | 0.819 | 0.839 | 0.842 | 0.799 | 0.804 |
| SlopeOne | 0.916 | 0.912 | 0.867 | 0.868 | 0.925 | 0.926 |
| Baseline estimates | 0.809 | 0.807 | 0.839 | 0.841 | 0.799 | 0.801 |
| KNN with baseline | 0.812 | 0.807 | 0.839 | 0.841 | 0.798 | 0.801 |
| Single DNN | 0.786 | 0.811 | 0.857 | 0.855 | 0.918 | 0.838 |
| **Proposed (DMCCF)** | **0.755** | **0.777 ± 0.010** | **0.877** | **0.866** | **0.930** | **0.858** |

- **The baselines match the paper to within 0.005 MAE.**
- **The proposed model is still the best method**, 3.2 % better than SVD and 4.2 % better than the single DNN.
- **It is 0.021 MAE worse than the paper's number.**
- **Per-criterion error (Table 5) has the paper's pattern**, each criterion about 0.025 higher (mean 0.776 against 0.750).

### Ablations (whole-model MAE)

| Setting | Paper | Ours |
|---|---|---|
| Embedding size 8 / 64 | – / 0.755 | **0.752** / 0.777 |
| Layers `[8,16,32]` / `[8,…,128]` (paper's choice) | 0.948 / 0.755 | **0.765** / 0.777 |
| Optimizer Adadelta / Adamax / Adam | 0.911 / 0.792 / 0.755 | **0.751** / 0.753 / 0.777 |
| Layers `[8]`, optimizer SGD | 3.47, 3.28 | 0.816, 0.895 |

Here smaller and shallower networks do better than the paper's choice. The reason: 84 % of test
ratings come from users with no training rating, so their embeddings are never trained.

The paper's errors above 3 match a network whose ReLU output has died and predicts about 0. That
happened in some runs here too. This code re-initialises such runs (up to 5 times) instead of
keeping them.

## Improvement: DMCCF+

DMCCF+ keeps the paper's two stages (a criteria network, then an aggregation network) and fixes what
the reproduction found:

1. **Unknown-user token.** Every user without training ratings (84 % of test ratings) shares one learned
   embedding instead of a random untrained one. During training, each user ID is swapped for it with
   probability 0.3, so the token learns what a typical new reviewer gives a hotel.
2. **Joint fine-tuning.** After the paper's separate training, both stages are trained together on the
   overall rating, so the aggregation network learns from the *predicted* criteria it receives at test time.
3. **Trainable, regularised network.** He initialisation, linear outputs on standardised targets,
   embedding 16, a `[128, 64, 32]` tower, dropout and AdamW. This removes the dead-output restarts.
4. **Clip to 1–5 and average 5 seeds.**

Every setting was chosen on an inner split of fold 0's training data (`results/dev_selection.csv`);
the outer test folds were used once.

| | MAE | MAE, users seen in training (16 %) | MAE, unseen users (84 %) | Criteria MAE |
|---|---|---|---|---|
| SVD | 0.803 | 0.738 | 0.815 | – |
| Paper's model (reproduced) | 0.777 ± 0.010 | 0.738 | 0.784 | 0.776 |
| Paper's reported number | 0.755 | – | – | 0.750 |
| **DMCCF+** | **0.737 ± 0.004** | 0.782 | **0.729** | **0.732** |

- **5.0 % lower error than the paper's model**, better on 5/5 folds, paired t-test p = 0.001.
- **8.1 % lower error than SVD**, better on 5/5 folds, p < 0.001.
- **Also below the paper's own reported 0.755.**
- **Trade-off:** returning users get slightly worse predictions, because the model leans less on their
  one or two earlier reviews. F1 and F2 stay at the paper model's level (0.865 / 0.860).

### Accuracy, not just error (`accuracy_metrics.py`)

| 5-fold test | SVD | Paper model | **DMCCF+** | **DMCCF+ blend** (½ DMCCF+ + ½ SVD) |
|---|---|---|---|---|
| MAE ↓ | 0.803 | 0.777 | **0.737** | 0.766 |
| Exact star ↑ | 40.4 % | 47.9 % | **48.9 %** | 48.1 % |
| Within 1 star ↑ | 73.5 % | 74.9 % | **75.4 %** | 74.0 % |
| Balanced accuracy, good (≥ 4★) or not ↑ | **67.2 %** | 64.1 % | 62.5 % | 66.6 % |
| ROC-AUC ↑ | **0.732** | 0.710 | 0.708 | 0.729 |
| R² ↑ | **0.179** | 0.094 | 0.035 | 0.149 |

**No single model wins every metric.**
- **DMCCF+** gives the best error-type accuracy: exact star better on 5/5 folds vs the paper model, p = 0.03.
- **The blend** beats the paper model on ROC-AUC, balanced accuracy and R², each on 5/5 folds, with
  p ≤ 0.001, and still has a lower MAE.
- **Neither beats SVD on ranking.** SVD stays slightly ahead on ROC-AUC and R².

The cause: training on absolute error pulls predictions toward the typical 4–5 stars. Plain "good
hotel?" accuracy is not shown, because always answering "good" scores 77.5 %.

```bash
.venv/bin/python dev_select.py
```

That reruns the selection (about 6 minutes). `run_experiments.py` already includes DMCCF+ in the 5-fold comparison.

## Choices the paper does not specify

| Paper leaves open | Choice here |
|---|---|
| Data filtering | All 8 ratings present and in 1..5; no anonymous users; one rating per (user, hotel) |
| Epochs | Early stopping on the 1 % validation MAE, patience 10, best epoch restored |
| Top-K cutoff and relevance | K = 10; relevant = overall rating ≥ 4 |
| Precision/recall with an empty denominator | 1 (2018 scikit-surprise FAQ recipe); this reproduces the paper's baseline F1/F2 |
| KNN with baseline | Item-based, Pearson-baseline (a user-based matrix for 70k users needs ~40 GB) |
| Framework | PyTorch, not Keras 2 / TensorFlow |

## Caveats

- **MAP, MRR and FCP don't match the paper.** A test fold has about 15,800 ratings from 15,350 users;
  only about 400 users have 2 or more test ratings. Per-user ranking is therefore trivial: MAP and MRR
  come out at about 0.998 for every method, and FCP rests on about 300 pairs. The paper's values
  (MAP 0.30–0.46) must come from a ranking protocol it does not describe.
- **The data is 97.4 % of the paper's**, because 91 hotel files are missing from the archived copy.