-
Notifications
You must be signed in to change notification settings - Fork 151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mongoid update #98
Mongoid update #98
Conversation
The goal is to match the original mongo-ruby-driver so that mongoid cna use em-mongo instead of the socket hack. removing unecessary commented out requires
wait until we have confirmation from others
still working on full testing regimen
Awesome! |
I can also share my gist of tweaks to have mongoid use em-mongo, but this is quite simillar. One downside of this is that mongoid would still use one connection regardless of synchrony pool size. |
I'd love to see it if you are able to post it. As for the single connection, are you saying that the code I posted will only use a single connection? Can you elaborate? I'd like to look into fixing that. Sujal On Dec 24, 2011, at 4:39 AM, Paweł [email protected] wrote:
|
Mongoid caches connections per class. I will provide an example soon. |
Ah, ok, just looked through that code. Is it caching the connection or what's returned from the master call? In that case, wouldn't it be caching the connection pool per class? I'll test this all out soon. Just need to fix some bugs so my test server starts up. Sujal On Dec 24, 2011, at 9:22 AM, Robert [email protected] wrote:
|
Unfortunately it is not caching a pool but one obtained connection from Fiber pool. So in most cases all you classes will end up caching and thus using later a first connection from the Pool. Different fibers queue requests to the same connection so it is almost like synchronously waiting for execution. It is possible that one Fiber grabs one connection from the pool, is yielded, and different fiber grabs a different connection for another class when executing its code. That way different classes will end up using different connection under the hood. Effective working with pool is only possible when directly executing queries using the driver.
Configuration:
|
name_resp.callback do |docs| | ||
names = docs.collect{ |doc| doc['name'] || '' } | ||
names = names.delete_if {|name| name.index(self.name).nil? || name.index('$')} | ||
names = names.map{ |name| name.sub(self.name + '.','')} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need all this extra processing here? Shouldn't we stick to drivers default response?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't have a better idea yet - basically, they have deep calls to other functions that have been made sync, so my choices were to redo these methods to call the async version or come up with another solution. (the main changes are the changes to call the adefer_as_a or the a* methods (in the first method).
Basically, what you're seeing is my attempt to get this working, at which point I'll step back and then refactor to clean things up, eliminate the copy/pasting and brute force stuff. For this one, though, I don't have a better idea yet.
Quick additional note: can we also add Also, let's drop old driver support, it'll make everyone's life easier (yes, laggards will have to upgrade :)) |
The good thing is most of what you pointed out is on my list of stuff to fix. There's a bit of cargo culting (though I did read all of it) i did to save time. Will take a look at everything in detail once I'm back on this task this afternoon. I'm working on getting a reliable authentication call working - the way I have it now, the auth call causes a stack too deep error. I have a lead on that, hope to have it sorted today. Sujal On Dec 27, 2011, at 11:20 AM, Ilya [email protected] wrote:
|
+1 .. I'd love to see mongoid+em-mongo |
needed to do this to get authentication working properly in a connection pool's block.
Not really sure why em-mongo does this - I assume it's a flag to indicate end of result set? No idea, but it's breaking Mongoid, so a quick patch
em-mongo insists that the document ID be :_id (a Symbol), not a "_id" (String) Will submit issue to em-mongo to ask if they should match the standard ruby driver's behavior which doesn't assume.
might be time to break this out into a separate project?
missing or misdirected aliases were the culprit
insert and update already return real values,
modeled after mongoid tests
includes additional finder tests - just the start
OK, now we're ready. Missed one callback case (an obvious one in hindsight, but one my API wasn't using often). :( It worked, and the next batch process that hammers the API will run in a few minutes. Will have validation that this is working enough for us then. Sujal |
|
||
# if selector.keys.length > 1 && RUBY_VERSION < '1.9' && selector.class != BSON::OrderedHash | ||
# raise MongoArgumentError, "DB#command requires an OrderedHash when hash contains multiple keys" | ||
# end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't need it, let's drop the dead code..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed.
A few pedantic questions / notes above.. plus, there is a bunch of dead code in the spec files - can we remove that? If the specs are all good, we can merge this and iterate from there.. since it's an almost unwieldy piece of code already. |
Thanks - most of the comments made sense, will take a closer look tomorrow. Was talking to someone about this - wondered if it made sense to make this part of em-synchrony directly, or to make it its own gem. On one hand, the bulk of the work is to the em-mongo driver, which has always been part of em-synchrony. The mongoid tweaks are just around the connections, mostly. On the other hand, the code does touch a lot of the em-mongo API... Anyway, something to think about -- I'll have an update on this tomorrow or Monday most likely. I have that connection caching thing to verify and sort out in mongoid on my list, just to make sure that the pool is being used properly. Sujal On Jan 7, 2012, at 4:52 PM, Ilya Grigorik wrote:
|
Sounds good. For where to keep the code: I'm good either way.. as long as we make sure its supported moving forward. :) |
Fixes typos in the finders spec for mongoid
At this point I don't think the spec problem I'm seeing in this branch is related to the connection pool. I've been talking to Sujal via email and hope to mind meld on this ASAP. At some point in the spec run, a request just doesn't get made and things halt. The reactor is still running, if I put a little "puts '.'" periodic timer in it shows output. But the request never gets made by em-mongo's connection and thing sit. I'm boggled by this, as if I move the specs around I can get the halt to change position, but it happens all the same. Really need some help on this, I think something is amiss in the em-mongo patching. |
Discovered last night by turning mongod logging way up that its having "9001 socket exception" errors that were not making it back through em-mongo in any visible way. Digging in on that front next. |
Ah, crap - I forgot we were supposed to look at this. I can possibly do this later today, but Monday would be a lot better for me. Shoot me an email off GitHub whenever you have time and let's just pick a time and get it done. Sujal On Feb 3, 2012, at 10:18 AM, Joshua Warchol wrote:
|
This was breaking on nested cursor enumeration, at least on our production service. I don't have a handy test case written yet. Will work on that next.
Conflicts: lib/em-synchrony/mongo.rb
Hey guys, apologies for dropping the ball on this. How do we look now in terms of completeness and functionality? |
Hi Ilya, thanks for picking this up. I will also review the code, run the specs and see how it goes |
There's still a broken testing setup somewhere in there which points to a bug in the underlying work i did with en-mongo/mongoid. At least, that's what it seems like. One test hangs, but it's always the 4th test to run (even if I reorder them, the 4th test hangs no matter which test it is). That's as far as I got. Maybe josh has other details to share. That being said, our API server is running in prod and working with limited traffic just fine. Sujal On Mar 27, 2012, at 10:27 AM, Luciano [email protected] wrote:
|
Sorry, I don't have anything more to add beyond what @sujal said. Unless we get someone with really deep Mongoid knowledge involved, I'd consider using it on top of em-mongo synchrony a "cutting edge" option. |
I have actually worked with MongoID internals quite a bit so I will give it a try |
Agreed. On Jul 21, 2012, at 2:55 PM, Ilya Grigorik [email protected] wrote:
|
Hi - Can i have someone take a look at this set of changes and review the approach?
I've basically updated the em-mongo support to cover more functionality, and I've then used that to update mongoid support to use em-mongo inside synchrony.
My goal is to get mongoid to use the connection pool that's part of synchrony and em-mongo, since that seems to be the preferred stack for synchrony and, thus, Goliath.
I'm still writing tests and will probably find things that aren't quite right, so this isn't ready to be merged yet. Thought this would be the easiest way to get a conversation going since there isn't a mailing list.
I'll be testing this for use in a production Goliath app.
(one additional note: The changes to break some compatibility with the existing em-mongo functionality to get it closer to the original non-synchrony em-mongo and mongo-ruby-driver.)
Thanks,
Sujal