Respect negative conditions for collection associations

[avit]

When searching against attributes from a collection association (has_many / :through / has_and_belongs_to_many), a normal JOIN will include false positives when asking to exclude values (!=, NOT IN, IS NOT NULL) and another tuple is matched.

This is the basic case, see specs for details:

• A book exists with two tags: "medieval" and "fantasy"
• You search for books that are NOT "fantasy"
• The result would still include it, because it matches the other tag for NOT "fantasy"

In situations where this would happen, I've moved the LEFT OUTER JOIN into a subquery with inverted conditions. This is implemented with a correlated primary key which should help the query plan to avoid scanning entire tables in the subquery. The conditions now look like this:

WHERE root.id NOT IN (
  SELECT child.parent_id FROM child
   ... WHERE ... AND child.parent_id = root.id
)

Note that root.id is referenced in the inner query. The EXPLAIN query plan for this looks sensible, at least in MySQL where I explored it.

As far as I know this works with all standard ActiveRecord joins: I have not yet verified with composite keys or polymorphic joins or Nth-degree associations but it should be close to working if it isn't already.

I will start testing this against our own real-world data to see how it performs, please check it out too.

Known issues:

• Fix mongoid tests failures.
• Errors with not_null predicate: seems like a positive case and should use the original code branch.

[avit]

Yes I'm fixing mongoid... 🔨

[jonatack]

Awesome 🎉 great work @avit 💜

[jonatack]

Fixes #381.

[avit]

@jonatack Thanks for the merge on this.

I have one more thing to add: the "NOT NULL" predicate still needs to be handled specially. I will add another PR for this.

[janklimo]

Hey @avit thank you very much for your work on this. I was trying to use this functionality with conditional has_many ... through relationships with little success.

In my case:

• a Course has_many :user_assigned_content_skills, -> { where(source: 'user') }, class_name: "ContentSkill"
• and a ContentSkill belongs_to :skill and belongs_to :course

Then Course.ransack({user_assigned_content_skills_skill_name_not_eq: 'ruby'}).result.to_sql returns the following (i.e. false positives again if a course has multiple content_skills):

"SELECT courses.* FROM courses LEFT OUTER JOIN content_skills ON content_skills.course_id = courses.id AND content_skills.source = 'user' LEFT OUTER JOIN skills ON skills.id = content_skills.skill_id WHERE (skills.name != 'ruby')"

I'm sorry to hijack this old thread but I was wondering if:

a) you know of a way to make this query work
b) there is a plan to have this implemented in Ransack?

Many thanks for any insights!