This application shows how to use neo4j together with rails 3.1 by parsing Twitter feeds and creating a social graph. It first shows some basic operations which you are probably used to from the ActiveRecord/ActiveModel API. In the last section we take real advantage of the neo4j engine by implementing a recommendation algorithm for finding new twitter users.
Have fun and feel free to clone it.
Make sure you have Java JDK 1.6+ installed
Install latest JRuby, example
rvm use jruby-1.6.7
Install Rails (>= 3.1.1)
gem install rails
Use my rails template which will disable active record and enable neo4j instead
rails new kvitter -m http://andreasronge.github.com/neo4j/rails.rb
Edit the Gemfile and add the twitter gem
cd kvitter emacs Gemfile # and add gem 'twitter', '1.7.2'
Download all the dependencies
bundle
Run the following commands:
rails generate scaffold Tweet text:string link:string date:datetime tweet_id:string --indices tweet_id date text --has_n tags mentions links --has_one tweeted_by:tweeted rails generate scaffold User twid:string link:string --indices twid --has_n tweeted follows knows used_tags mentioned_from:mentions rails generate scaffold Link url:string --indices url --has_n tweets:links short_urls:redirected_link --has_one redirected_link rails generate scaffold Tag name:string --indices name --has_n tweets:tags used_by_users:used_tags
There is nothing magical happening here like neo4j configuration or migrations. It only creates plain new Ruby classes (models, controllers and views) and routing. The relationships and properties are only specified in the model classes.
Test the basic crud operations
rails s
Open browser: localhost:3000/tags
Add the following:
The following code does a twitter search and parses the result. It creates and connects the Tweet, Link, User and Tag model classes.
def search @tag = Tag.find(params[:id]) search = Twitter::Search.new result = search.hashtag(@tag.name) curr_page = 0 while curr_page < 2 do result.each do |item| parsed_tweet_hash = Tweet.parse(item) next if Tweet.find_by_tweet_id(parsed_tweet_hash[:tweet_id]) tweet = Tweet.create!(parsed_tweet_hash) twid = item['from_user'].downcase user = User.find_or_create_by(:twid => twid) user.tweeted << tweet user.save parse_tweet(tweet, user) end result.fetch_next_page curr_page += 1 end redirect_to @tag end def parse_tweet(tweet, user) tweet.text.gsub(/(@\w+|https?:\/\/[a-zA-Z0-9\-\.~\:\?#\[\]\!\@\$&,\*+=;,\/]+|#\w+)/).each do |t| case t when /^@.+/ t = t[1..-1].downcase next if t.nil? other = User.find_or_create_by(:twid => t) user.knows << other unless t == user.twid || user.knows.include?(other) user.save tweet.mentions << other when /#.+/ t = t[1..-1].downcase tag = Tag.find_or_create_by(:name => t) tweet.tags << tag unless tweet.tags.include?(tag) user.used_tags << tag unless user.used_tags.include?(tag) user.save when /https?:.+/ link = Link.find_or_create_by(:url => t) tweet.links << (link.redirected_link || link) end end tweet.save! end
Change index on text to fulltext lucene:
property :text, :type => String, :index => :fulltext
Add a to_s and parse method and change the index type of text to fulltext (we need that later on, see below).
def to_s text.gsub(/(@\w+|https?\S+|#\w+)/,"").strip end def self.parse(item) {:tweet_id => item['id_str'], :text => item['text'], :date => Time.parse(item['created_at']), :link => "http://twitter.com/#{item['from_user']}/statuses/#{item['id_str']}" } end
Notice : in Neo4j it is not necessary to specify the types of properties. By setting :type => String we force that each Tweet object will store the property ‘type’ as strings.
change
resource :tags
to
resources :tags do get :search, :on => :member end
Add a button to the view:
<%= button_to "Search", [:search, @tag], :method => :get %>
Test the application now by opening a browser localhost:3000/tags Create a new tag and press the button ‘search’ You will now found tweets, users and links
When you looked at all the links, most of them are short urls like t.co Since we are more interested in the real link and who has tweeted about them we must follow the URL by doing a HTTP head request.
We use a before save callback to create a the real link which correspond to where the short url is directed to. The short link and the real link are connected in a redirected_link relationship.
class Link < Neo4j::Rails::Model
property :url, :type => String, :index => :exact
has_n(:tweets).from(:links)
has_n(:short_urls).from(:redirected_link)
has_one :redirected_link
# Add the following:
before_save :create_redirect_link
SHORT_URLS = %w[t.co bit.ly ow.ly goo.gl tiny.cc tinyurl.com doiop.com readthisurl.com memurl.com tr.im cli.gs short.ie kl.am idek.net short.ie is.gd hex.io asterl.in j.mp].to_set
def to_s
url
end
private
def self.short_url?(url)
domain = url.split('/')[2]
domain && SHORT_URLS.include?(domain)
end
def create_redirect_link
return if !self.class.short_url?(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.read_timeout = 200
req = Net::HTTP::Head.new(uri.request_uri)
res = http.request(req)
redirect = res['location']
if redirect && url != redirect
self.redirected_link = Link.find_or_create_by(:url => redirect.strip)
end
rescue Timeout::Error
puts "Can't acccess #{url}"
rescue Error
puts "Can't call #{url}"
rescue Net::HTTPBadResponse
puts "Bad response for #{url}"
end
In order to show both outgoing redirected_link and incoming redirected_link (by using short_urls method) add the following:
<p>
<b>Short Urls:</b>
<% @link.short_urls.each do |link| %>
<%= link_to link, link %> <br/>
<% end %>
</p>
<% if @link.redirected_link %>
<p>
<b>Redirects to</b>
<%= link_to @link.redirected_link, @link.redirected_link %>
</p>
<% end %>
The index page shows all the links, including links like bit.ly and the the real one. To only return the real URLs we can use rules, which is a bit similar to scope in ActiveRecord.
rule(:real) { redirected_link.nil?}
This means that it will group all links under the rule :real which does not have a redirected_link To return all those nodes, use the class method #real. Notice you can do some really interesting queries using rules and cypher together, see “Rules-Cypher”:github.com/andreasronge/neo4j/wiki/Neo4j%3A%3AWrapper-Rules-and-Functions
Btw, the Neo4j::Rails::Model#all method is also implemented as a rule. You can also chain rules, just like scopes in Active Record.
def index @links = Link.real respond_to do |format| format.html # index.html.erb format.json { render :json => @links } end end
Since the rules do work by creating relationships when new nodes are created/updated/deleted we must do another search or stop the rails server and delete the database ‘db/neo4j-developement’
The list of all tweets (localhost:3000/tweets ) does not look good. It needs some pagination. Add the two gem in Gemfile:
gem 'neo4j-will_paginate', :git => 'git://github.com/andreasronge/neo4j-will_paginate.git' gem 'will_paginate'
Neo4j.rb comes included with the will_paginate gem. Change the index method
@tweets = Tweet.all
to
@tweets = Tweet.all.paginate(:page => params[:page], :per_page => 10)
Pagination is support for all traversals and lucene queries.
Add the following line before the table:
<%= will_paginate(@tweets) %>
Lets say we want to sort the tweets by the text. Lucene has two types of indexes: exact and fulltext. The exact index is perfect for keywords while the fulltext is for longer texts. We have already changed the index of text to fulltext for the app/models/tweets.rb See lucene.apache.org/java/3_0_0/queryparsersyntax.html for the query syntax.
Add the following form before the table
<%= form_for(:tweets, :method => :get) do |f| %>
<div class="field">
<%= text_field_tag :query %>
<%= f.submit "Search" %>
</div>
<% end %>
In the index method we now should handle the query parameter. Add the following before the respond_to.
def index query = params[:query] if query.present? @tweets = Tweet.all("text:#{query}", :type => :fulltext).paginate(:page => params[:page], :per_page => 10) else @tweets = Tweet.all.paginate(:page => params[:page], :per_page => 10) end # respond_to ... end
Test it !
Would it not be cool to visualize who knows who ?
Download the D3 version 2.4.6 javascript library from github.com/mbostock/d3/archives/master.
Unzip it and move the following files to the app/assets/javascript folder
-
d3.geom.js
-
d3.js
-
d3.layout.js
Create a JSON API needed by the javascript
def index @users = User.all respond_to do |format| format.html # index.html.erb format.json do nodes = @users.map{|u| {:name => u.twid, :value => u.tweeted.size}} links = [] @users.each do |user| links += user.knows.map {|other| { :source => nodes.find_index{|n| n[:name] == user.twid}, :target => nodes.find_index{|n| n[:name] == other.twid}}} end render :json => {:nodes => nodes, :links => links} end end end
Test the API, open a browser localhost:3000/users.json it should return something like this:
{"nodes":[{"name":"geoaxis","value":1},{"name":"peterneubauer","value":1},{"name":"dolugen","value":1},{"name":"neo4j","value":8},{"name":"emileifrem","value":1},{"name":"linnbar_consult","value":1},{"name":"skillsmatter","value":0},{"name":"swgoof","value":1},{"name":"bytor99999","value":0},{"name":"teropaananen","value":1},{"name":"tobyorourke","value":1},{"name":"sannegrinovero","value":1},{"name":"emmanuelbernard","value":0},{"name":"jimwebber","value":0},{"name":"ianrobinson","value":0},{"name":"mesirii","value":1},{"name":"kings13y","value":0},{"name":"chitrasen","value":1},{"name":"einarhreindal","value":1},{"name":"smunchang","value":1},{"name":"technige","value":1},{"name":"josh_adell","value":1},{"name":"mmuekk","value":1},{"name":"asbkar","value":1},{"name":"noppanit","value":1},{"name":"roddare","value":1},{"name":"tanyaespe","value":1},{"name":"erikusaj","value":1},{"name":"druidjaidan","value":1}],"links":[{"source":0,"target":1},{"source":2,"target":3},{"source":2,"target":4},{"source":3,"target":7},{"source":3,"target":4},{"source":3,"target":8},{"source":3,"target":9},{"source":3,"target":10},{"source":3,"target":11},{"source":3,"target":12},{"source":3,"target":13},{"source":3,"target":14},{"source":3,"target":1},{"source":3,"target":16},{"source":4,"target":9},{"source":5,"target":6},{"source":7,"target":4},{"source":9,"target":4},{"source":10,"target":4},{"source":11,"target":12},{"source":11,"target":13},{"source":11,"target":14},{"source":15,"target":9},{"source":15,"target":4},{"source":17,"target":9},{"source":17,"target":4},{"source":18,"target":6},{"source":19,"target":4},{"source":19,"target":3},{"source":20,"target":21},{"source":22,"target":6},{"source":23,"target":6},{"source":24,"target":1},{"source":25,"target":1},{"source":26,"target":4},{"source":26,"target":3},{"source":27,"target":6},{"source":28,"target":6}]}
Add the following coffeescript (be careful with the indentation):
# Place all the behaviors and hooks related to the matching controller here.
# All this logic will automatically be available in application.js.
# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/
$(document).ready ->
w = 1260
h = 1300
fill = d3.scale.category20()
vis = d3.select("#graph").append("svg:svg").attr("width", w).attr("height", h)
d3.json("/users.json", (json) ->
force = d3.layout.force()
.charge(-320)
.linkDistance(160)
.nodes(json.nodes)
.links(json.links)
.size([w, h])
.start()
link = vis.selectAll("line.link")
.data(json.links)
.enter().append("svg:line")
.attr("class", "link")
.style("stroke-width", (d) -> Math.sqrt(d.value))
.attr("x1", (d) -> d.source.x)
.attr("y1", (d) -> d.source.y)
.attr("x2", (d) -> d.target.x)
.attr("y2", (d) -> d.target.y)
node = vis.selectAll("g.node")
.data(json.nodes)
.enter().append("svg:g")
.attr("transform", (d) -> "translate(" + d.x + "," + d.y + ")")
.attr("class", "node")
.call(force.drag)
node.append("svg:circle")
.attr("r", (d) -> if d.value > 25 then 50 else d.value*2 + 5)
.style("fill", (d) -> '#fea')
node.append("svg:title").text((d) -> d.name)
node.append("svg:text")
.attr("text-anchor", "middle")
.attr("dy", ".3em")
.text((d) -> d.name)
vis.style("opacity", 1e-6)
.transition()
.duration(0)
.style("opacity", 1)
force.on("tick", ->
link.attr("x1", (d) -> d.source.x).attr("y1", (d) -> d.source.y).attr("x2", (d) -> d.target.x).attr("y2", (d) -> d.target.y)
node.attr("transform", (d) -> "translate(" + d.x + "," + d.y + ")")
)
)
Add the following at the bottom of the file
<div id='graph'> </div>
Make sure things are loading in the correct order.
//= require jquery //= require jquery_ujs //= require d3 //= require d3.geom //= require d3.layout //= require users
circle.node {
stroke: #fff;
stroke-width: 1.5px;
}
line.link {
stroke: #999;
stroke-opacity: .6;
}
Open a browser localhost:3000/users and scroll down
Add the following before the link_to lines
<p>
<b>Used tags</b>
<% @user.used_tags.each do |tag| %>
<%= link_to tag.name, tag %> <br/>
<% end %>
</p>
<p>
<b>Knows:</b><br/>
<% @knows.each do |user| %>
<%= link_to user, user %> <br/>
<% end %>
<%= will_paginate(@knows) %>
</p>
<p>
<b>Mentioned from:</b><br/>
<% @mentioned_from.each do |tweet| %>
<%= link_to tweet, tweet %> <br/>
<% end %>
</p>
<p>
<b>Tweets</b><br/>
<% @user.tweeted.each do |tweet| %>
<%= link_to tweet, tweet %> <br/>
<% end %>
</p>
Change the show function to:
def show @user = User.find(params[:id]) @knows = @user.knows.paginate(:page => params[:page], :per_page => 10) @mentioned_from = @user.mentioned_from respond_to do |format| format.html # show.html.erb format.json { render :json => @user } end end
Notice that we do pagination of a traversal (knows).
Show the relationships from and to a tweet:
<p>
<b>Text:</b>
<%= @tweet.text %>
</p>
<p>
<b>Link:</b>
<%= link_to @tweet.link, @tweet.link %>
</p>
<p>
<b>Date:</b>
<%= @tweet.date %>
</p>
<p>
<b>Tweet:</b>
<%= @tweet.tweet_id %>
</p>
<p>
<b>Tweeted by</b>
<%= link_to @tweet.tweeted_by, @tweet.tweeted_by %>
</p>
<p>
<b>Tags:</b><br/>
<% @tweet.tags.each do |tag| %>
<%= link_to tag.name, tag %> <br/>
<% end %>
</p>
<p>
<b>Links:</b><br/>
<% @tweet.links.each do |link| %>
<%= link_to link, link %> <br/>
<% end %>
</p>
<p>
<b>Tweets:</b><br/> <% @tweets.each do |tweet| %> <%= link_to tweet, tweet %> <br/> <% end %> <%= will_paginate(@tweets) %>
</p>
Add the following line in the show method after the Tag.find line
@tweets = @tag.tweets.paginate(:page => params[:page], :per_page => 10)
The following algorithm works like this:
1. Get all the users who are also using my tags. 2. For each of those users get their used tags and compare with mine. 3. Select the user who has the most similar tags to mine.
Add the following at the bottom of the file user_controllers.rb
private def recommend(user) my_tags = user.used_tags.to_a my_friends = user.knows.to_a # we are here using the raw java API - that's why using _java_node, raw and wrapper other_users = user._java_node.outgoing(:used_tags).incoming(:used_tags).raw.depth(2).filter{|path| path.length == 2 && !my_friends.include?(path.end_node)} # for all those users, find the person who has the max number of same tags as I have found = other_users.max_by{|friend| (friend.outgoing(:used_tags).raw.map{|tag| tag[:name]} & my_tags).size } found && found.wrapper # load the ruby wrapper around the neo4j java node end
Add one line to the show method:
def show @user = User.find(params[:id]) @recommend = recommend(@user) respond_to do |format| format.html # show.html.erb format.json { render :json => @user } end end
Display a recommendation if available
<% if @recommend %> <p> <b>Recommend</b> <%= link_to @recommend.twid, @recommend %> </p> <% end %>