-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Magnet is a Ruby-On-Rails application that collects, filters (content moderation) and distributes contents posted on Social Networks. This wiki is the main source of documentation for developers working with (or contributing to) the Magnet project.
Posts are organized in collections, characterized by a set of communication channels (or content feeds) used to fetch contents from Social Networks.
- Facebook channels: can fetch contents directly posted in a page (or an page's album), or containing a mention to it
- Twitter channels: can fetch tweets containing an #hashtag, or posted by a particolar @user
- Instagram channels: can fetch posts containing an #hashtag, or posted by a particolar @user
- Tumblr channels: can fetch contents directly posted in a blog, or posts containing an #hashtag
- RSS channels: can fetch contents from a public RSS source
Each collection can optionally have a distinct set of rules, used to tell the system whether each post should be directly accepted, rejected, or marked for human acknowledgement before being published. These are some of the existing options:
- Require explicit manual approval for any post
- Discard text-only posts
- Discard posts containing an identical image
- Discard posts containing more than a certain number of hashtags
- Discard posts containing portions of text listed in a stopword set
- Set blacklists and whitelists of users
Upon fetching and filtering, contents are then distributed to a number of output channels:
- Web social walls
- Mobile webviews
- LCD walls
- Rest APIs (custom embedding or native apps integrations)
Magnet is conceptually a pretty simple application, but needs a number of supporting services to run:
- A relational database, used to store users' credentials and collections metadata: MySQL 5 is the platform used both in development and production deployments. That said, any RDBMs supported by ActiveRecord should work out of the box (no custom queries in the code, as far as I can remember; I tried using PostgreSQL, with no issues)
- Redis, used to execute asynchronous jobs: Magnet uses Sidekiq for async job processing, and Sidekiq-Cron for batch scheduling
- MongoDB, used to store the actual posts fetched from the various channels (feeds): MongoDB 3.2 is the platform used both in development and production deployments, right now. That said, MongoDB 2.6 is the minimum supported (tested) version, since posts belonging to each content collection are stored in a separate MongoDB collection, to avoid locking and keep indexes as small as possible
- Elasticsearch, optionally used to index posts and perform queries-at-speed-of-light: this is entirely optional, and is useful in cases of heavy manual content moderation, to speed up full-text queries against a large number of contents
Next: Basic setup