The Holmes-Totem Planner is responsible for turning data into information by performing feature extraction against submitted objects. When tasked, Holmes-Totem schedules the execution of its services which are capable of performing static and dynamic analysis as well as gather data from third parties.
The Holmes-Totem Investigation Planner is optimized for executing extraction services that complete in a few seconds, i.e. static analysis and 3rd party queries. When dealing with services that take longer to complete, we recommend pairing the Holmes-Totem Planner with Holmes-Totem-Dynamic.
Holmes-Totem is pre-packaged with with a number of static analysis and query Services to help you get started. A few highlights:
| Name | Object Type | Description |
|---|---|---|
| ASNMeta | IPv4/6 | Collects ASN information from Team Cymru |
| DNSMeta | Domain | Collects current DNS information from your local resolver and authoritative NS |
| PassiveTotal | IPv4/6, Domain, Email | Returns available Passive Total information |
| GoGadget | Binary Executable | Extracts the major gadgets of an executable |
| Objdump | Binary Executables | Provides a parsed version of the objdump output |
| PEiD | File | Provides PEiD information (packer, compiler, etc) for files |
| PEInfo | PE32 | Extracts information about PE file and is based on pefile. Most of the information contained in the PE headers is accessible as well as all sections' details and their data. |
| PEMeta | PE32 | Extracts information about PE file and is based on PEV. |
| RichHeader | PE32 | Rich Header extraction for PE32 files |
| VirusTotal | Binary | Returns available VirusTotal information with a public or private key |
| Yara | File | Performs Yara signature matching with a default rule pack or custom rule |
| ZipMeta | Zip | Provides meta information contained in zip files |
| PDFparse | This service explores the structure of the pdf file and dumps the object content into JSON format | |
| Shodan | IPv4/IPv6 | A service for gathering Shodan information about an IP address |
Holmes-TOTEM is built with the Akka Toolkit and performs best with Oracle's Java 8. When executing tasks, Holmes-TOTEM requires an HTTP complaint server for delivering files, a database for storing results, and a queuing server for organizing tasking.
Holmes-Totem requires Java 8 and we used the SBT build tool for dependency management and assembling.
RabbitMQ is the queuing server of choice for Holmes Processing. Other AMQP complaint services should work but are untested by Holmes Processing. For sending tasking to the queuing server, we recommend using Holmes-Gateway for optimizing the tasking and handling user authentication.
Holmes-Storage is the Holmes Processing recommendation for managing the sample repository and storing Holmes-TOTEM results. While not strictly required, Holmes-Storage will ease the creation of the databases and supply the information in the expected format for other Holmes Processing solutions.
The supplied services rely on Docker and Docker-Compose.
Work In Progress
- Clone the Git Repository and Change Directory
$ git clone https://github.com/HolmesProcessing/Holmes-Totem.git
$ cd Holmes-Totem- Compile Holmes-TOTEM Holmes-TOTEM uses SBT to download all dependencies and compile the source into a working JAR file.
$ sbt assemblyThe assembled jar file will be located in ./target/scala-2.11/totem-assembly-0.5.0.jar
- Perform Totem Configuration
Holmes-TOTEM is packaged with sane configuration defaults for Holmes-TOTEM and Docker-Compose. These configuration settings will configure the system to use all available Holmes-TOTEM services. These default configuration can be used by removing the .example tag at the end of the file name.
$ cp ./config/totem.conf.example ./config/totem.conf
$ cp ./config/docker-compose.yml.example ./config/docker-compose.ymlAfter the files are created, please perform any adjustments to the configuration to match your environment and needs. You will most likely need to adjust the values for rabbit_settings.
- Perform Service Configuration
Holmes-TOTEM provides a number of standard services that are packaged as Docker containers. These containers will manage all dependencies but configuration is still required. In most cases this should be as simple as renaming the service.conf.example file to service.conf for each of the available services. However, some services will require an API key or additional information to execute. For more information and details on the options available, please visit the directory and read the README.md for each service ./src/main/scala/org/holmesprocessing/totem/services/
Holmes-TOTEM allows you to upload the configuration files for the individual services to Holmes-Storage. You can upload individual files manually:
$ curl -F config=@${confPath} http://${storageIP}:${storagePort}/config/${serviceName}/service.confor you can use the script to automatically upload all your service.conf-files:
$ cd config
$ ./upload_configs.sh ${storageIP}:${storagePath}This script will go through each service and look for the file "service.conf" and upload it. If there is no such file, it will upload the file "service.conf.example", if it exists. Before uploading, the script asks for each file individually, since existing configurations on storage are overwritten.
In order to use these uploaded configuration files, use the script
$ cd config
$ ./compose_download_conf.sh ${storageIP}:${storagePath}- Start the Services
$ docker-compose -f ./config/docker-compose.yml up -d- Execute Holmes-TOTEM
$ java -jar ./target/scala-2.11/totem-assembly-0.5.0.jarWe recommend using Holmes-Gateway for optimizing the tasking and handling user authentication. Please visit the Holmes-Gateway repository for further information.
Holmes-Toolbox provides a Command Line Interface (CLI) for sending tasking to Holmes-TOTEM.
Holmes-TOTEM can be manually tasked using custom created AMQP message using JSON as the message body. The following minimal example will task Holmes-TOTEM to execute PEID, YARA, and PEINFO against a sample.
SAMPLE = <sample>
URI = "<Storage URL>" + SAMPLE
jdict = {
"download": True, #specifies if TOTEM should download the file
"primaryURI": URI, #first URI to download the file
"secondaryURI": URI, #back up URI to download the file
"filename": SAMPLE, #filename of the sample, or identifier (i.e. domain name) if !download
"tasks": { #list or tasks to execute with optional arguments
"PEID": [],
"YARA": [],
"PEINFO": [],
},
"tags": [ #add an optional source tag
"totem-test"
],
"attempts": 0
}Holmes-Totem is derived from the Novetta open source project Totem and Holmes Group LLC is not related or endorsed by Novetta. We gracelessly thank Novetta for their contribution and we could not have created this project without their support.
Holmes Processing would also like to thank the CRITs team for their valuable discussions and support they provided.