Merge

.github/workflows/build.yml

@@ -2,12 +2,29 @@ name: build
 on: [push, pull_request]
 jobs:
   build:
-    if: "!contains(github.event.head_commit.message, '[skip ci]')"
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - ruby: 3.4
+            gemfile: Gemfile
+          - ruby: 3.3
+            gemfile: gemfiles/activerecord72.gemfile
+          - ruby: 3.2
+            gemfile: gemfiles/activerecord71.gemfile
+          - ruby: 3.1
+            gemfile: gemfiles/activerecord70.gemfile
     runs-on: ubuntu-latest
+    env:
+      BUNDLE_GEMFILE: ${{ matrix.gemfile }}
     steps:
-    - uses: actions/checkout@v2
-    - uses: ruby/setup-ruby@v1
-      with:
-        ruby-version: 2.7
-        bundler-cache: true
-    - run: bundle exec rake test
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+          bundler-cache: true
+      - uses: actions/cache@v4
+        with:
+          path: ~/.cache/disco
+          key: disco-v2
+      - run: bundle exec rake test

CHANGELOG.md

@@ -1,3 +1,76 @@
+## 0.5.1 (2024-12-29)
+
+- Removed dependency on `base64` gem for serialization
+
+## 0.5.0 (2024-10-22)
+
+- Changed dataset directory to match XDG Base Directory Specification
+- Dropped support for marshal serialization
+- Dropped support for Ruby < 3.1 and Rails < 7
+
+## 0.4.2 (2024-06-24)
+
+- Removed dependency on `csv` gem for `load_movielens`
+
+## 0.4.1 (2024-05-23)
+
+- Reduced memory for `item_recs` and `similar_users`
+
+## 0.4.0 (2023-01-30)
+
+- Fixed issue with `has_recommended` and inheritance with Rails < 6.1
+- Deprecated marshal serialization
+- Dropped support for Ruby < 2.7 and Rails < 6
+
+## 0.3.2 (2022-09-26)
+
+- Fixed issue when `fit` is called multiple times
+
+## 0.3.1 (2022-07-10)
+
+- Added support for JSON serialization
+
+## 0.3.0 (2022-03-22)
+
+- Changed `item_id` to `user_id` for `similar_users`
+- Changed warning to an error when `value` passed to `fit`
+- Changed to use Faiss over NGT for `optimize_item_recs` and `optimize_similar_users` when both are installed
+- Removed dependency on `wilson_score` gem for `top_items`
+- Dropped support for Ruby < 2.6
+
+## 0.2.9 (2022-03-22)
+
+- Fixed error with `load_movielens`
+
+## 0.2.8 (2022-03-13)
+
+- Fixed error with `top_items` with all same rating
+
+## 0.2.7 (2021-08-06)
+
+- Added warning for `value`
+
+## 0.2.6 (2021-02-24)
+
+- Improved performance
+- Improved `inspect` method
+- Fixed issue with `similar_users` and `item_recs` returning the original user/item
+- Fixed error with `fit` after loading
+
+## 0.2.5 (2021-02-20)
+
+- Added `top_items` method
+- Added `optimize_similar_users` method
+- Added support for Faiss for `optimize_item_recs` and `optimize_similar_users` methods
+- Added `rmse` method
+- Improved performance
+
+## 0.2.4 (2021-02-15)
+
+- Added `user_ids` and `item_ids` methods
+- Added `user_id` argument to `user_factors`
+- Added `item_id` argument to `item_factors`
+
 ## 0.2.3 (2020-11-28)
 
 - Added `predict` method

Gemfile

@@ -4,8 +4,10 @@ gemspec
 
 gem "rake"
 gem "minitest", ">= 5"
-gem "activerecord"
+gem "activerecord", "~> 8.0.0"
 gem "sqlite3"
 gem "daru"
+gem "matrix" # for daru
 gem "rover-df"
 gem "ngt", ">= 0.3.0"
+gem "faiss"

LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2019-2020 Andrew Kane
+Copyright (c) 2019-2024 Andrew Kane
 
 MIT License
 

README.md

@@ -6,14 +6,14 @@
 - Works with explicit and implicit feedback
 - Uses high-performance matrix factorization
 
-[![Build Status](https://github.com/ankane/disco/workflows/build/badge.svg?branch=master)](https://github.com/ankane/disco/actions)
+[![Build Status](https://github.com/ankane/disco/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/disco/actions)
 
 ## Installation
 
 Add this line to your application’s Gemfile:
 
 ```ruby
-gem 'disco'
+gem "disco"
 ```
 
 ## Getting Started
@@ -35,24 +35,24 @@ recommender.fit([
 
 > IDs can be integers, strings, or any other data type
 
-If users don’t rate items directly (for instance, they’re purchasing items or reading posts), this is known as implicit feedback. Leave out the rating, or use a value like number of purchases, number of page views, or time spent on page:
+If users don’t rate items directly (for instance, they’re purchasing items or reading posts), this is known as implicit feedback. Leave out the rating.
 
 ```ruby
 recommender.fit([
-  {user_id: 1, item_id: 1, value: 1},
-  {user_id: 2, item_id: 1, value: 1}
+  {user_id: 1, item_id: 1},
+  {user_id: 2, item_id: 1}
 ])
 ```
 
-> Use `value` instead of rating for implicit feedback
+> Each `user_id`/`item_id` combination should only appear once
 
-Get user-based (user-item) recommendations - “users like you also liked”
+Get user-based recommendations - “users like you also liked”
 
 ```ruby
 recommender.user_recs(user_id)
 ```
 
-Get item-based (item-item) recommendations - “users who liked this item also liked”
+Get item-based recommendations - “users who liked this item also liked”
 
 ```ruby
 recommender.item_recs(item_id)
@@ -99,18 +99,13 @@ recommender.item_recs("Star Wars (1977)")
 [Ahoy](https://github.com/ankane/ahoy) is a great source for implicit feedback
 
 ```ruby
-views = Ahoy::Event.
-  where(name: "Viewed post").
-  group(:user_id).
-  group("properties->>'post_id'"). # postgres syntax
-  count
+views = Ahoy::Event.where(name: "Viewed post").group(:user_id).group_prop(:post_id).count
 
 data =
-  views.map do |(user_id, post_id), count|
+  views.map do |(user_id, post_id), _|
     {
       user_id: user_id,
-      item_id: post_id,
-      value: count
+      item_id: post_id
     }
   end
 ```
@@ -181,26 +176,26 @@ user.update_recommended_products_v2(recs)
 user.recommended_products_v2
 ```
 
-For Rails < 6, speed up inserts by adding [activerecord-import](https://github.com/zdennis/activerecord-import) to your app.
-
 ## Storing Recommenders
 
 If you’d prefer to perform recommendations on-the-fly, store the recommender
 
 ```ruby
-bin = Marshal.dump(recommender)
-File.binwrite("recommender.bin", bin)
+json = recommender.to_json
+File.write("recommender.json", json)
 ```
 
-> You can save it to a file, database, or any other storage system
+The serialized recommender includes user activity from the training data (to avoid recommending previously rated items), so be sure to protect it. You can save it to a file, database, or any other storage system, or use a tool like [Trove](https://github.com/ankane/trove). Also, user and item IDs should be integers or strings for this.
 
 Load a recommender
 
 ```ruby
-bin = File.binread("recommender.bin")
-recommender = Marshal.load(bin)
+json = File.read("recommender.json")
+recommender = Disco::Recommender.load_json(json)
 ```
 
+Alternatively, you can store only the factors and use a library like [Neighbor](https://github.com/ankane/neighbor). See the [examples](https://github.com/ankane/neighbor/tree/master/examples/disco).
+
 ## Algorithms
 
 Disco uses high-performance matrix factorization.
@@ -226,16 +221,26 @@ recommender.fit(data, validation_set: validation_set)
 
 ## Cold Start
 
-Collaborative filtering suffers from the [cold start problem](https://www.yuspify.com/blog/cold-start-problem-recommender-systems/). It’s unable to make good recommendations without data on a user or item, which is problematic for new users and items.
+Collaborative filtering suffers from the [cold start problem](https://en.wikipedia.org/wiki/Cold_start_(recommender_systems)). It’s unable to make good recommendations without data on a user or item, which is problematic for new users and items.
 
 ```ruby
 recommender.user_recs(new_user_id) # returns empty array
 ```
 
 There are a number of ways to deal with this, but here are some common ones:
 
-- For user-based recommendations, show new users the most popular items.
-- For item-based recommendations, make content-based recommendations with a gem like [tf-idf-similarity](https://github.com/jpmckinney/tf-idf-similarity).
+- For user-based recommendations, show new users the most popular items
+- For item-based recommendations, make content-based recommendations with a gem like [tf-idf-similarity](https://github.com/jpmckinney/tf-idf-similarity)
+
+Get top items with:
+
+```ruby
+recommender = Disco::Recommender.new(top_items: true)
+recommender.fit(data)
+recommender.top_items
+```
+
+This uses [Wilson score](https://www.evanmiller.org/how-not-to-sort-by-average-rating.html) for explicit feedback and item frequency for implicit feedback.
 
 ## Data
 
@@ -257,45 +262,65 @@ Or a Daru data frame
 Daru::DataFrame.from_csv("ratings.csv")
 ```
 
-## Faster Similarity
+## Performance
 
-If you have a large number of users/items, you can use an approximate nearest neighbors library like [NGT](https://github.com/ankane/ngt) to speed up item-based recommendations and similar users.
+If you have a large number of users or items, you can use an approximate nearest neighbors library like [Faiss](https://github.com/ankane/faiss) to improve the performance of certain methods.
 
 Add this line to your application’s Gemfile:
 
 ```ruby
-gem 'ngt', '>= 0.3.0'
+gem "faiss"
+```
+
+Speed up the `user_recs` method with:
+
+```ruby
+recommender.optimize_user_recs
 ```
 
-Speed up item-based recommendations with:
+Speed up the `item_recs` method with:
 
 ```ruby
-model.optimize_item_recs
+recommender.optimize_item_recs
 ```
 
-Speed up similar users with:
+Speed up the `similar_users` method with:
 
 ```ruby
-model.optimize_similar_users
+recommender.optimize_similar_users
 ```
 
-This should be called after fitting or loading the model.
+This should be called after fitting or loading the recommender.
 
 ## Reference
 
+Get ids
+
+```ruby
+recommender.user_ids
+recommender.item_ids
+```
+
 Get the global mean
 
 ```ruby
 recommender.global_mean
 ```
 
-Get the factors
+Get factors
 
 ```ruby
 recommender.user_factors
 recommender.item_factors
 ```
 
+Get factors for specific users and items
+
+```ruby
+recommender.user_factors(user_id)
+recommender.item_factors(item_id)
+```
+
 ## Credits
 
 Thanks to:

Rakefile

@@ -5,5 +5,57 @@ task default: :test
 Rake::TestTask.new do |t|
   t.libs << "test"
   t.pattern = "test/**/*_test.rb"
-  t.warning = false
+  t.warning = false # for daru
+end
+
+# TODO use benchmark-ips
+def benchmark_user_recs(name, recommender)
+  ms = Benchmark.realtime do
+    recommender.user_ids.each do |user_id|
+      recommender.user_recs(user_id)
+    end
+  end
+  puts "%-8s %f" % [name, ms]
+end
+
+# TODO use benchmark-ips
+def benchmark_item_recs(name, recommender)
+  ms = Benchmark.realtime do
+    recommender.item_ids.each do |item_id|
+      recommender.item_recs(item_id)
+    end
+  end
+  puts "%-8s %f" % [name, ms]
+end
+
+namespace :benchmark do
+  task :user_recs do
+    require "bundler/setup"
+    Bundler.require
+    require "benchmark"
+
+    data = Disco.load_movielens
+    recommender = Disco::Recommender.new
+    recommender.fit(data)
+
+    benchmark_user_recs("none", recommender)
+    recommender.optimize_user_recs
+    benchmark_user_recs("faiss", recommender)
+  end
+
+  task :item_recs do
+    require "bundler/setup"
+    Bundler.require
+    require "benchmark"
+
+    data = Disco.load_movielens
+    recommender = Disco::Recommender.new
+    recommender.fit(data)
+
+    benchmark_item_recs("none", recommender)
+    recommender.optimize_item_recs(library: "ngt")
+    benchmark_item_recs("ngt", recommender)
+    recommender.optimize_item_recs(library: "faiss")
+    benchmark_item_recs("faiss", recommender)
+  end
 end