Strange error with nil in hash_dot_syntax check?

[drsharp]

Hello!

I hope this is an ok place to ask about this. I have an app that has been using mongoid for a good long while. I recently upgraded to Ruby 3.1.3 and mongoid 8.0.3 and I'm occasionally seeing this error now:

ruby/3.1.0/gems/mongoid-8.0.3/lib/mongoid/attributes.rb:266:in 'hash_dot_syntax?': undefined method 'include?' for nil:NilClass (NoMethodError)

It looks to me like the method at

mongoid/lib/mongoid/attributes.rb

Lines 271 to 273 in cb7e0bd

	def hash_dot_syntax?(string)
	string.include?(".")
	end

assumes that the passed in string is an actual string, but somehow sometimes it seems to be nil.

Tracing the call stack it seems to be from read_raw_attribute here:

mongoid/lib/mongoid/attributes.rb

Lines 290 to 297 in cb7e0bd

	def read_raw_attribute(name)
	normalized = database_field_name(name.to_s)
	if attribute_missing?(normalized)
	raise Mongoid::Errors::AttributeNotLoaded.new(self.class, name)
	end
	if hash_dot_syntax?(normalized)

which is setting normalized via: normalized = database_field_name(name.to_s)

which is called from read_attribute here:

mongoid/lib/mongoid/attributes.rb

Lines 86 to 90 in cb7e0bd

	def read_attribute(name)
	field = fields[name.to_s]
	raw = read_raw_attribute(name)
	process_raw_attribute(name.to_s, raw, field)
	end

So then I trace it to the definition of database_field_name here:

mongoid/lib/mongoid/fields.rb

Lines 408 to 439 in cb7e0bd

	def database_field_name(name, relations, aliased_fields, aliased_associations)
	if Mongoid.broken_alias_handling
	return nil unless name
	normalized = name.to_s
	aliased_fields[normalized] || normalized
	else
	return nil unless name.present?
	key = name.to_s
	segment, remaining = key.split('.', 2)
	# Don't get the alias for the field when a belongs_to association
	# is not the last item. Therefore, get the alias when one of the
	# following is true:
	# 1. This is the last item, i.e. there is no remaining.
	# 2. It is not an association.
	# 3. It is not a belongs association
	if !remaining || !relations.key?(segment) || !relations[segment].is_a?(Association::Referenced::BelongsTo)
	segment = aliased_fields[segment]&.dup || segment
	end
	return segment unless remaining
	relation = relations[aliased_associations[segment] || segment]
	if relation
	k = relation.klass
	"#{segment}.#{database_field_name(remaining, k.relations, k.aliased_fields, k.aliased_associations)}"
	else
	"#{segment}.#{remaining}"
	end
	end
	end
	end

and I'm trying to figure out when it could return nil. It seems that if the string passed in to database_field_name is already nil, it'll return nil and then it breaks upstream as I mentioned above.

The entry of the call stack is here:

mongoid/lib/mongoid/association/proxy.rb

Lines 122 to 124 in cb7e0bd

	ruby2_keywords def method_missing(name, *args, &block)
	_target.send(name, *args, &block)
	end

My application was using mongoid 5.2.1 so going from 5.2.1 to 8.0.3 is a bit of a jump but it's interesting that this error is only occasionally cropping up. Also, this error is not consistent so I'm struggling how to figure out what is happening.

I think it's possible that there is a DB document that is missing an attribute and in 5.2.1 it just bubbled through as nil without any blow-ups, but now it's bubbling through as nil and blowing up in that hash_dot_syntax? check.

I thought I'd start with posting here in case anyone has encountered this or has some ideas. I'm happy to provide additional info if that helps, but I thought at least mentioning the problem may be a good first step.

Thank you!

[alexbevi]

Hi @drsharp, thanks for reporting this issue. As you've indicated this only happens intermittently and is hard to trace we'll keep this discussion opened for the time being in the event others can share a similar experience that may allow us to troubleshoot further.

If you can provide a self-contained reproduction that showcases the error (even occasionally) it would make further investigation a lot easier for the team.

Note - can you please include a full stack trace if you have it available? This may be of further assistance as well.

[drsharp]

Hi @alexbevi! Sorry it took me a while to get back to this. This is a side-project so I only work on it sporatically.

Here's some details that may help.

First, I have a Ruby class called Ifp::OriginalProperty that is just a pretty simple Mongoid document:

  class OriginalProperty
    include Mongoid::Document
    embedded_in :property_record

    field :title_en,       type: String
    field :title_fr,       type: String
    field :description_en, type: String
    field :description_fr, type: String
    field :detection_map,  type: Hash
  end

Then, in Ruby 2.3.5 using Mongoid 5.2.1, I can access this document like this:

[2] pry(#<Translator>)> db_property.original_property
=> #<Ifp::OriginalProperty _id: 63f3986d1baf6a6f288923bf, title_en: "Here is a cool description", title_fr: "", description_en: "", description_fr: "", detection_map: nil>

And I have a method that accesses the value of one of the attributes via a parameter called key. For example, when the key is :title_en it returns the expected data as follows:

[1] pry(#<Translator>)> key
=> :title_en
[2] pry(#<Translator>)> db_property.original_property
=> #<Ifp::OriginalProperty _id: 63f3986d1baf6a6f288923bf, title_en: "Here is a cool description", title_fr: "", description_en: "", description_fr: "", detection_map: nil>
[3] pry(#<Translator>)> db_property.original_property[key]
=> "Here is a cool description"

However, sometimes the key may be set to nil. It turns out that Mongoid 5.2.1 had no problems with that:

[4] pry(#<Translator>)> db_property.original_property[nil]
=> nil
[5] pry(#<Translator>)> key
=> nil
[6] pry(#<Translator>)> db_property.original_property[key].to_s
=> ""

It just returns nil, which is fine as it's handled downstream in my code.

Fast forward to Ruby 3.1.3 and Mongoid 8.0.3:

Same method, same usage, but this time with the error:

[1] pry(#<Translator>)> key
=> nil
[2] pry(#<Translator>)> db_property.original_property
=> #<Ifp::OriginalProperty _id: 63f3986d1baf6a6f288923bf, title_en: "Here is a cool description", title_fr: "", description_en: "", description_fr: "", detection_map: nil>
[3] pry(#<Translator>)> db_property.original_property[key]
NoMethodError: undefined method `include?' for nil:NilClass

      string.include?(".")
            ^^^^^^^^^
from /Users/dsharp/.rvm/gems/ruby-3.1.3@advert_import/gems/mongoid-8.0.3/lib/mongoid/attributes.rb:266:in `hash_dot_syntax?'

BTW, the full(er) backtrace looks like this:

/Users/dsharp/.rvm/gems/ruby-3.1.3@advert_import/gems/mongoid-8.0.3/lib/mongoid/attributes.rb:266:in `hash_dot_syntax?': undefined method `include?' for nil:NilClass (NoMethodError)

      string.include?(".")
            ^^^^^^^^^
	from /Users/dsharp/.rvm/gems/ruby-3.1.3@advert_import/gems/mongoid-8.0.3/lib/mongoid/attributes.rb:291:in `read_raw_attribute'
	from /Users/dsharp/.rvm/gems/ruby-3.1.3@advert_import/gems/mongoid-8.0.3/lib/mongoid/attributes.rb:88:in `read_attribute'
	from /Users/dsharp/.rvm/gems/ruby-3.1.3@advert_import/gems/mongoid-8.0.3/lib/mongoid/association/proxy.rb:123:in `method_missing'
	from /Users/dsharp/Code/-----.rb:88:in `previous_content'

I believe that trying to reference a field using the value nil is throwing this error. It's pretty obscure and certainly something I can just fix in my code, but the fact that it just silently returned nil in Mongoid 5 but throws an exception in Mongoid 8 made me think it's worth mentioning.

I hope that helps to investigate.

[danhealy]

Seeing the same with 8.1.4 - PR #5836

[alexbevi]

Thanks for the PR @danhealy, and apologies @drsharp as it seems I've overlooked following up on this discussion.

[drsharp]

No worries @alexbevi

The question I have now is: should the "nil bubbling up" be the expected behavior or as @danhealy PR'd should it raise an exception? I don't know which is more logical or not. I would think the explicit exception is probably a tighter API because it's saying: "if you try to access an attribute that isn't present, raise an exception", and that makes for more explicit, controlled usage.

However, this is the sort of thing that could benefit from some configuration option too. Something like allow_nil_field_names or something? Maybe it defaults to false but if set to true in some config, then it could work like it did in Mongoid 5: just silently let the nil bubble up through. So referencing a nil field name would just return a nil value.

I'm not steeped enough in Mongoid to know if this is a good idea or not, but it's worth considering.

[alexbevi]

I'm going to close this thread as further discussion is taking place on MONGOID-5789 and #5836