Understanding Intrigue Core: Key System Files

This document is a quick rundown of where functionality lives in the system, designed to give new developers a head start.

In the root directory:

core.rb - this is the first system file which loads in libraries for the various components... sinatra, redis, sidekiq, database (sequel), etc.

core-cli.rb - this is a simple cli utility that utilizes the API client (in most cases) to expose functionality on the command line.

Dockerfile - contains the specification for the all-in-one Docker image. Note that this relies heavily on the util/bootstrap.sh system and is based on Ubuntu.

Procfile - contains the processes that should be started using foreman start in development mode

Rakefile - contains the rake tasks to setup the system; $ rake db:migrate ... migrates the database $ rake setup ... copies the config files, downloads the latest data and

Vagrantfile - contains the specification for the development Vagrant image. Note that this also relies heavily on util/bootstrap.sh

Details on system files on a directory-by-directory basis:

  |-api_client 

This is an api client for the system, packaged as a ruby gem that is used to interact with the API. The built-in UI uses this client as well.

  |-app
  |---models
  |---routes
  |---views
  |---workers

This section contains the sinatra API routes, ERB views and Sequel-based data models. The graph generation is also a background job, and it can be found in the workers section

  |-config

The config directory is where the system's config files are stored. Key files:

• config.json - main config file for the system and tasks
• redis.yml - configuration file for redis client
• sidekiq.yml - configuration file for sidekiq (background workers)
• puma.rb - configuration file for puma (application server)

  |-data

Data files such as GeoCity geolocation data and NVD vulnerability data. Some files are downloaded via the $rake setup command (Rake is a task runner, and tasks are specfiied in the Rakefile in the root directory)

  |-db

Database migrations. Sequel is used as the ORM layer.

  |-lib 

The lib directory contains all the core functionality of the system.

    |---client
    |-----search
    |-----social

The lib/client directory contains api clients used by intrigue tasks.

    |---checks

The lib/checks directory contains Exposure Checks, a special type of task+issue combination that work exactly like a normal task and issue, except that they're defined in a single file, and the return value of the TASK, if truthy (ex: 1, true, "positive result"), will ensure that the issue is created and linked to the entity it is called upon.

    |---config

The lib/config directory contains libraries used for global configuration (as specified in the config/config.json documented above)

    |---entities

The lib/entities directory contains specifications for entities. Entities are the "types" that the systems knows about. All entities are subclassed off Intrigue::Core::Model::Entity.

    |---handlers

The lib/handlers directory contains specifications for handlers, which can be configured to trigger after a task or scan run and typically export data out of the system.

    |---initialize

The lib/initialize directory contains monkeypatches for core Ruby libraries.

    |---workflows

The lib/workflows directory contains specifications for workflows. Workflows are specified in a format that defines recursively iterative ... lists of tasks... when configured to run after any given task, a workflow will tell the system how to iteratively which new tasks should be run on any new entity

    |---notifiers

The lib/notifiers directory contains specifications for notifiers. Notifiers, when configured via config/config.json can trigger notifications to the user through the _notify() method. Slack is the only supported notifier today.

    |---system

The lib/system directory contains a variety of libraries specific to the system, such at the global configuration manager, and lists of disallowed entity patterns.

    |---tasks

The lib/tasks directory contains specifications for tasks, which are units of functionality of the system. Tasks must have a metdata method and a run method. Tasks are automatically run in the background via Sidekiq and always accept a single entity and options as input. Task have many helper methods such as the _log* methods, _notify* methods and _create_entity. See lib/tasks/example.rb for some examples.

    |-----enrich

The lib/tasks/enrich directory contains a specific type of task, the enrichment task. Enrichment tasks are run upon creation of a new entity. If no enrichment task is specified for an entity (as defined in the lib/entities directory), the lib/tasks/enrich/generic.rb enrichment is scheduled and run for that entity.

    |-----helpers

This directory, lib/helpers, contains a variety of helpers used in the lib/tasks/*.rb files.

    |-----import

The lib/tasks/import directory contains a specific type of task, the import task. Import tasks are designed to bring many entities into the system at once.

    |-----vulns

The lib/tasks/vulns directory contains a specific type of task, the vuln check tasks. Vuln Check tasks are designed to check for a specific vulnerability and will often create an "Issue" when a vulnerability is discovered.

 |-log

All system logs are stored in the log directory.

 |-public
 |---bower_components
 |---css
 |---img
 |---js
 |-----sigma
 |-------plugins

All javascript and css assets are stored in the public directory.

 |-spec
 |---integration

System tests, written with rspec, are stored in the spec directory. Tests need work.

 |-tmp
 |---pids

Tmp directory is just temporary storage for the system. Pid files will be stored in tmp/pids in production deployments.

 |-util
 |---god

Util directory contains a variety of utilities used by the system. Note that the util/god directory contains the process management used in production.