diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 31775712..1c9d4039 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,14 +1,77 @@ -##Bugs -We use Github Issues for our bug reporting. Please make sure the bug isn't already listed before opening a new issue. +# Contributing -##Development -All work on Haystack happens directly on Github. Core Haystack team members will review opened pull requests. +Code contributions are always welcome! -##Requests -If you see a feature that you would like to be added, please open an issue in the respective repository or in the general Haystack repo. +* Open an issue in the repo with defect/enhancements +* We can also be reached @ https://gitter.im/expedia-haystack/Lobby +* Fork, make the changes, build and test it locally +* Issue a PR - watch the PR build in [travis-ci](https://travis-ci.org/ExpediaDotCom/haystack-traces) +* Once merged to master, travis-ci will build and release the artifacts to [docker hub] -##Contributing to Documentation -To contribute to documentation, you can directly modify the corresponding .md files in the docs directory under the base haystack repository, and submit a pull request. Once your PR is merged, the documentation is automatically built and deployed to https://expediadotcom.github.io/haystack. -##License -By contributing to Haystack, you agree that your contributions will be licensed under its Apache License. \ No newline at end of file +## Building + +####Prerequisite: + +* Make sure you have Java 1.8 +* Make sure you have maven 3.3.9 or higher +* Make sure you have docker 1.13 or higher + + +Note : For mac users you can download docker for mac to set you up for the last two steps. + +####Build + +For a full build, including unit tests and integration tests, docker image build, you can run - +``` +make all +``` + +####Integration Test + +####Prerequisite: +1. Install docker using Docker Tools or native docker if on mac +2. Verify if docker-compose is installed by running following command else install it. +``` +docker-compose + +``` + +Run the build and integration tests for individual components with +``` +make indexer + +``` + +&& + +``` +make reader + +``` + + +``` +make backends + +``` + + +## Releasing the artifacts + +Currently we publish the repo to docker hub and nexus central repository. + +* Git tagging: + +``` +git tag -a -m "Release description..." +git push origin +``` + +`` must follow semantic versioning scheme. + +Or one can also tag using UI: https://github.com/ExpediaDotCom/haystack-traces/releases + +It is preferred to create an annotated tag using `git tag -a` and then use the release UI to add release notes for the tag. + +* After the release is completed, please update the `pom.xml` files to next `-SNAPSHOT` version to match the next release \ No newline at end of file diff --git a/README.md b/README.md index 7a5e7308..574b75b0 100644 --- a/README.md +++ b/README.md @@ -1,51 +1,61 @@ [![Build Status](https://travis-ci.org/ExpediaDotCom/haystack-traces.svg?branch=master)](https://travis-ci.org/ExpediaDotCom/haystack-traces) [![License](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg)](https://github.com/ExpediaDotCom/haystack/blob/master/LICENSE) -# haystack-traces -This repo contains the haystack components that build the traces, store them in Cassandra and ElasticSearch(for indexing) and provide a grpc endpoint for accessing them +# Haystack Traces +Traces is a subsystem included in Haystack that provides a distributed tracing system to troubleshoot problems in microservice architectures. Its design is based on the [Google Dapper](http://research.google.com/pubs/pub36356.html) paper. -## Building -#### -Since this repo contains haystack-idl as the submodule, so use the following to clone the repo -* git clone --recursive git@github.com:ExpediaDotCom/haystack-traces.git . +This repo contains the haystack components that build the traces. It uses ElasticSearch for indexing and a storage backend for persistence -####Prerequisite: +## Architecture +Please see the [architecture document](https://expediadotcom.github.io/haystack/docs/subsystems/subsystems_traces.html) for the high level architecture of the traces subsystem -* Make sure you have Java 1.8 -* Make sure you have maven 3.3.9 or higher -* Make sure you have docker 1.13 or higher +## Components -Note : For mac users you can download docker for mac to set you up for the last two steps. +### haystack-trace-indexer -####Build +Trace Indexer is the component which reads spans from a kafka topic and writes to elasticsearch(for indexing) +and the storage backend for persistence. Please see the [indexer app](indexer/) for more details -For a full build, including unit tests and integration tests, docker image build, you can run - -``` -make all -``` +### haystack-trace-reader -####Integration Test +Trace Reader is the component which retrieves the trace-ids from elastic search based on the given queries and then fetches the spans from +the storage backend. Please see the [reader app](reader/) for more details -####Prerequisite: -1. Install docker using Docker Tools or native docker if on mac -2. Verify if docker-compose is installed by running following command else install it. -``` -docker-compose +### Storage Backend -``` +Haystack Traces multiple storage backend apps, used to store and query spans. The Storage backend apps are +grpc apps which are expected to implement this [grpc contract](https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/backend/storageBackend.proto) +The [reader](reader/src/main/scala/com/expedia/www/haystack/trace/reader/stores/readers/grpc/GrpcTraceReader.scala) and [indexer](indexer/src/main/scala/com/expedia/www/haystack/trace/indexer/writers/grpc/GrpcTraceWriter.scala) components read and write to the underlying datastore using this service and the default configuration expects the storage backend app to run on the same host(localhost) as the indexer and reader app. -Run the build and integration tests for individual components with -``` -make indexer +By default the traces subsystem comes bundled with the following backends. You can always run your custom backends as long as it implements the [grpc contract](https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/backend/storageBackend.proto). -``` +#### In-Memory +The in-memory storage backend app keeps the spans in memory. It +is neither persistent, nor viable for realistic work loads. Please see the [memory backend app](backends/memory) for more details -&& -``` -make reader +#### Cassandra +The Cassandra storage-backend app is tested against [Cassandra 3.11.3+](http://cassandra.apache.org/). It is designed for production scale. Please see the [cassandra backend app](backends/cassandra) for more details -``` +#### Mysql +The Mysql storage-backend app is tested against [Mysql 5.6++](https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-13.html) and [amazon aurora mysql](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMySQL.Overview.html). +It is designed for production scale. Please see the [mysql backend app](backends/mysql) for more details + + +## Contributing to this codebase +Please see [CONTRIBUTING.md](CONTRIBUTING.md) + + +## Bugs, Feature Requests, Documentation Updates +Please see the [contributing page](https://expediadotcom.github.io/haystack/docs/contributing.html) on our website + +## Contact Info + +Interested in haystack? Want to talk? Have questions, concerns or great ideas? +Please join us on [gitter](https://gitter.im/expedia-haystack/Lobby) + +##License +By contributing to Haystack, you agree that your contributions will be licensed under its Apache License. \ No newline at end of file diff --git a/Release.md b/Release.md deleted file mode 100644 index b26e745c..00000000 --- a/Release.md +++ /dev/null @@ -1,19 +0,0 @@ -#Releasing -Currently we publish the repo to docker hub and nexus central repository. - -#How to release and publish - -* Git tagging: - -``` -git tag -a -m "Release description..." -git push origin -``` - -`` must follow semantic versioning scheme. - -Or one can also tag using UI: https://github.com/ExpediaDotCom/haystack-traces/releases - -It is preferred to create an annotated tag using `git tag -a` and then use the release UI to add release notes for the tag. - -* After the release is completed, please update the `pom.xml` files to next `-SNAPSHOT` version to match the next release \ No newline at end of file diff --git a/backends/Makefile b/backends/Makefile index 04d475ee..20b65196 100644 --- a/backends/Makefile +++ b/backends/Makefile @@ -1,4 +1,4 @@ -.PHONY: all cassandra memory release +.PHONY: all cassandra mysql memory release PWD := $(shell pwd) @@ -11,6 +11,13 @@ build_cassandra: cd ../ && ./mvnw package -DfinalName=haystack-trace-backend-cassandra -pl backends/cassandra -am +mysql: build_mysql + cd mysql && $(MAKE) integration_test + +build_mysql: + cd ../ && ./mvnw package -DfinalName=haystack-trace-backend-mysql -pl backends/mysql -am + + memory: build_memory cd memory && $(MAKE) integration_test @@ -21,3 +28,4 @@ build_memory: release: cd cassandra && $(MAKE) docker_build && $(MAKE) release cd memory && $(MAKE) docker_build && $(MAKE) release + cd mysql && $(MAKE) docker_build && $(MAKE) release diff --git a/backends/cassandra/README.md b/backends/cassandra/README.md index ff9588c3..9f5b8196 100644 --- a/backends/cassandra/README.md +++ b/backends/cassandra/README.md @@ -6,9 +6,17 @@ Grpc service which can read a write spans to a cassandra cluster ##Technical Details In order to understand this service, we recommend to read the details of [haystack](https://github.com/ExpediaDotCom/haystack) project. -This service reads from [Cassandra](http://cassandra.apache.org/). API endpoints are exposed as [GRPC](https://grpc.io/) endpoints. +This service reads from [Cassandra](http://cassandra.apache.org/). API endpoints are exposed as [GRPC](https://grpc.io/) endpoints based on [this]((https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/backend/storageBackend.proto)) contract. + +The Schema for the cassandra table is created by the code when it starts up if it doesn't exist using the following command + +` +CREATE KEYSPACE IF NOT EXISTS haystack WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor' : 1 } AND durable_writes = false; CREATE TABLE IF NOT EXISTS haystack.traces (id varchar, ts timestamp, spans blob, PRIMARY KEY ((id), ts)) WITH CLUSTERING ORDER BY (ts ASC) AND compaction = { 'class' : 'DateTieredCompactionStrategy', 'max_sstable_age_days': '3' } AND gc_grace_seconds = 86400; +` + +## Deployments +The reader and the indexer app expects the storage-backend app as a sidecar container and sample deployment topology using docker compose is shared [here](https://github.com/ExpediaDotCom/haystack-docker) -Will fill in more details as we go.. ## Building -Check the details on [Build Section](../README.md) \ No newline at end of file +Check the details on [Build Section](../../CONTRIBUTING.md) \ No newline at end of file diff --git a/backends/cassandra/src/main/scala/com/expedia/www/haystack/trace/storage/backends/cassandra/client/CassandraTableSchema.scala b/backends/cassandra/src/main/scala/com/expedia/www/haystack/trace/storage/backends/cassandra/client/CassandraTableSchema.scala index 8e8490ae..07961ae7 100644 --- a/backends/cassandra/src/main/scala/com/expedia/www/haystack/trace/storage/backends/cassandra/client/CassandraTableSchema.scala +++ b/backends/cassandra/src/main/scala/com/expedia/www/haystack/trace/storage/backends/cassandra/client/CassandraTableSchema.scala @@ -26,8 +26,6 @@ object CassandraTableSchema { val ID_COLUMN_NAME = "id" val TIMESTAMP_COLUMN_NAME = "ts" val SPANS_COLUMN_NAME = "spans" - val SERVICE_COLUMN_NAME = "service_name" - val OPERATION_COLUMN_NAME = "operation_name" /** diff --git a/backends/memory/README.md b/backends/memory/README.md index a53728b9..5f82a53b 100644 --- a/backends/memory/README.md +++ b/backends/memory/README.md @@ -5,9 +5,9 @@ Grpc service which can read a write spans to a an in memory map ##Technical Details In order to understand this service, we recommend to read the details of [haystack](https://github.com/ExpediaDotCom/haystack) project. -This service reads from an in memory map. API endpoints are exposed as [GRPC](https://grpc.io/) endpoints. +This service reads from an in memory map. API endpoints are exposed as [GRPC](https://grpc.io/) endpoints based on [this]((https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/backend/storageBackend.proto)) contract. -Will fill in more details as we go.. +* Note : Its purpose is for testing, for example starting a server on your laptop without any database needed. This only works if the reader and indexer apps are running locally and talk to the same in-memory backend server. -## Building -Check the details on [Build Section](../README.md) \ No newline at end of file +# Building +Check the details on [Build Section](../../CONTRIBUTING.md) \ No newline at end of file diff --git a/backends/mysql/Makefile b/backends/mysql/Makefile new file mode 100644 index 00000000..227d9282 --- /dev/null +++ b/backends/mysql/Makefile @@ -0,0 +1,24 @@ +.PHONY: docker_build prepare_integration_test_env integration_test release + +export DOCKER_ORG := ayansen89 +export DOCKER_IMAGE_NAME := ayansen89/haystack-trace-backend-mysql +PWD := $(shell pwd) +SERVICE_DEBUG_ON ?= false + +docker_build: + # build docker image using existing app jar + docker build -t $(DOCKER_IMAGE_NAME) -f build/docker/Dockerfile . + +prepare_integration_test_env: docker_build + # prepare environment to run integration tests against + docker-compose -f build/integration-tests/docker-compose.yml -p sandbox up -d + sleep 30 + +integration_test: prepare_integration_test_env + cd ../../ &&./mvnw integration-test -pl backends/mysql -am + docker-compose -f build/integration-tests/docker-compose.yml -p sandbox stop + docker rm $(shell docker ps -a -q) + docker volume rm $(shell docker volume ls -q) + +release: + ../../deployment/scripts/publish-to-docker-hub.sh diff --git a/backends/mysql/README.md b/backends/mysql/README.md new file mode 100644 index 00000000..b35fc782 --- /dev/null +++ b/backends/mysql/README.md @@ -0,0 +1,21 @@ +# Storage Backend - Mysql + +Grpc service which can read a write spans to a mysql cluster + +##Technical Details + +In order to understand this service, we recommend to read the details of [haystack](https://github.com/ExpediaDotCom/haystack) project. +This service reads from [Mysql](https://www.mysql.com/). API endpoints are exposed as [GRPC](https://grpc.io/) endpoints based on [this]((https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/backend/storageBackend.proto)) contract. + +The Schema for the sql table is created by the code when it starts up if it doesn't exist using the following command + +` +CREATE DATABASE IF NOT EXISTS haystack; USE haystack; create table IF NOT EXISTS spans (id varchar(255) not null, spans LONGBLOB not null, ts timestamp default CURRENT_TIMESTAMP, PRIMARY KEY (id, ts)) +` + +## Deployments +The reader and the indexer app expects the storage-backend app as a sidecar container and sample deployment topology using docker compose is shared [here](docker-compose.yml) + + +## Building +Check the details on [Build Section](../../CONTRIBUTING.md) \ No newline at end of file diff --git a/backends/mysql/build/docker/Dockerfile b/backends/mysql/build/docker/Dockerfile new file mode 100644 index 00000000..1cd73feb --- /dev/null +++ b/backends/mysql/build/docker/Dockerfile @@ -0,0 +1,21 @@ +FROM openjdk:8-jre +MAINTAINER Haystack + +ENV APP_NAME haystack-trace-backend-mysql +ENV APP_HOME /app/bin +ENV JMXTRANS_AGENT jmxtrans-agent-1.2.6 + +RUN mkdir -p ${APP_HOME} + +COPY target/${APP_NAME}.jar ${APP_HOME}/ +COPY build/docker/start-app.sh ${APP_HOME}/ +RUN chmod +x ${APP_HOME}/start-app.sh + +COPY build/docker/jmxtrans-agent.xml ${APP_HOME}/ +ADD https://github.com/jmxtrans/jmxtrans-agent/releases/download/${JMXTRANS_AGENT}/${JMXTRANS_AGENT}.jar ${APP_HOME}/ + +WORKDIR ${APP_HOME} + +EXPOSE 8090 + +ENTRYPOINT ["./start-app.sh"] diff --git a/backends/mysql/build/docker/jmxtrans-agent.xml b/backends/mysql/build/docker/jmxtrans-agent.xml new file mode 100644 index 00000000..0f9ecbc5 --- /dev/null +++ b/backends/mysql/build/docker/jmxtrans-agent.xml @@ -0,0 +1,30 @@ + + + + + + + + + + + + + + + + + + ${HAYSTACK_GRAPHITE_HOST:monitoring-influxdb-graphite.kube-system.svc} + ${HAYSTACK_GRAPHITE_PORT:2003} + ${HAYSTACK_GRAPHITE_ENABLED:false} + haystack.traces.backend-mysql.#hostname#. + + 30 + diff --git a/backends/mysql/build/docker/start-app.sh b/backends/mysql/build/docker/start-app.sh new file mode 100755 index 00000000..a1d956bf --- /dev/null +++ b/backends/mysql/build/docker/start-app.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +[ -z "$JAVA_XMS" ] && JAVA_XMS=256m +[ -z "$JAVA_XMX" ] && JAVA_XMX=256m +[ -z "$JAVA_GC_OPTS" ] && JAVA_GC_OPTS="-XX:+UseG1GC" + +set -e +JAVA_OPTS="${JAVA_OPTS} \ +-javaagent:${APP_HOME}/${JMXTRANS_AGENT}.jar=${APP_HOME}/jmxtrans-agent.xml \ +${JAVA_GC_OPTS} \ +-Xmx${JAVA_XMX} \ +-Xms${JAVA_XMS} \ +-XX:+ExitOnOutOfMemoryError \ +-Dapplication.name=${APP_NAME} \ +-Dapplication.home=${APP_HOME}" + +if [[ -n "$SERVICE_DEBUG_ON" ]] && [[ "$SERVICE_DEBUG_ON" == true ]]; then + JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=5005,server=y" +fi + +exec java ${JAVA_OPTS} -jar "${APP_HOME}/${APP_NAME}.jar" diff --git a/backends/mysql/build/integration-tests/docker-compose.yml b/backends/mysql/build/integration-tests/docker-compose.yml new file mode 100644 index 00000000..62dc8fcf --- /dev/null +++ b/backends/mysql/build/integration-tests/docker-compose.yml @@ -0,0 +1,12 @@ +version: '3' +services: + mysql: + image: mysql:8.0.13 + environment: + MAX_HEAP_SIZE: 256m + HEAP_NEWSIZE: 256m + MYSQL_ROOT_PASSWORD: root + MYSQL_DATABASE: haystack-spans + ports: + - "3306:3306" + command: --default-authentication-plugin=mysql_native_password diff --git a/backends/mysql/docker-compose.yml b/backends/mysql/docker-compose.yml new file mode 100644 index 00000000..c06de6f5 --- /dev/null +++ b/backends/mysql/docker-compose.yml @@ -0,0 +1,107 @@ +# +# Copyright 2019 Expedia, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +version: "3" +services: + mysql: + image: mysql:8.0.13 + environment: + MAX_HEAP_SIZE: 256m + HEAP_NEWSIZE: 256m + MYSQL_ROOT_PASSWORD: root + MYSQL_DATABASE: haystack-spans + ports: + - "3306:3306" + command: --default-authentication-plugin=mysql_native_password + + storage-backend: + image: expediadotcom/haystack-trace-backend-mysql:1.0.8 + environment: + HAYSTACK_GRAPHITE_ENABLED: "false" + HAYSTACK_LOG_LEVEL: "DEBUG" + JAVA_XMS: 128m + depends_on: + - "mysql" + restart: always + # uncomment below port mapping to expose and connect to this application out of local docker container network +# ports: +# - "8090:8090" + + trace-reader: + image: expediadotcom/haystack-trace-reader:1.0.8 + environment: + HAYSTACK_GRAPHITE_ENABLED: "false" + HAYSTACK_PROP_BACKEND_CLIENT_HOST: "storage-backend" + JAVA_XMS: 128m + restart: always + + trace-indexer: + image: expediadotcom/haystack-trace-indexer:1.0.8 + environment: + HAYSTACK_GRAPHITE_ENABLED: "false" + HAYSTACK_PROP_BACKEND_CLIENT_HOST: "storage-backend" + HAYSTACK_PROP_SERVICE_METADATA_ENABLED: "true" + HAYSTACK_PROP_KAFKA_MAX_WAKEUPS: "100" + HAYSTACK_PROP_SERVICE_METADATA_FLUSH_INTERVAL_SEC: "0" + JAVA_XMS: 128m + depends_on: + - "elasticsearch" + restart: always + + elasticsearch: + image: elastic/elasticsearch:6.0.1 + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + xpack.security.enabled: "false" + ports: + - "9200:9200" + restart: always + + zookeeper: + image: wurstmeister/zookeeper + ports: + - "2181:2181" + + kafkasvc: + image: wurstmeister/kafka:2.11-1.1.1 + depends_on: + - zookeeper + environment: + KAFKA_BROKER_ID: 1 + KAFKA_ADVERTISED_LISTENERS: INSIDE://kafkasvc:9092,OUTSIDE://localhost:19092 + KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:19092 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT + KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE + KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 + KAFKA_CREATE_TOPICS: "proto-spans:1:1,metricpoints:1:1,metric-data-points:1:1,mdm:1:1,metrics:1:1,graph-nodes:1:1,service-graph:1:1,mapped-metrics:1:1,anomalies:1:1,alerts:1:1" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + ports: + - "9092:9092" + - "19092:19092" + + + ui: + image: expediadotcom/haystack-ui:1.1.4 + volumes: + - ./:/data + ports: + - "8080:8080" + environment: + HAYSTACK_OVERRIDES_CONFIG_PATH: /data/connectors.json + HAYSTACK_PROP_CONNECTORS_TRACES_CONNECTOR__NAME: "haystack" + HAYSTACK_PROP_CONNECTORS_TRACES_SERVICE__REFRESH__INTERVAL__IN__SECS: "0" + HAYSTACK_PROP_CONNECTORS_TRACES_HAYSTACK__HOST: "trace-reader" + HAYSTACK_PROP_CONNECTORS_TRACES_HAYSTACK__PORT: "8088" \ No newline at end of file diff --git a/backends/mysql/pom.xml b/backends/mysql/pom.xml new file mode 100644 index 00000000..4dfa40ce --- /dev/null +++ b/backends/mysql/pom.xml @@ -0,0 +1,164 @@ + + + + + haystack-trace-backends + com.expedia.www + 1.0.0-SNAPSHOT + ../pom.xml + + + 4.0.0 + haystack-trace-backend-mysql + jar + + + com.expedia.www.haystack.trace.storage.backends.mysql.Service + ${project.artifactId}-${project.version} + 8.0.13 + 1.4 + + + + + com.google.protobuf + protobuf-java + + + mysql + mysql-connector-java + ${mysql.connector.version} + + + commons-dbcp + commons-dbcp + ${apache.commons.dhcp.version} + + + + io.grpc + grpc-protobuf + + + + io.grpc + grpc-stub + + + + io.grpc + grpc-services + + + + io.grpc + grpc-netty + + + + io.netty + netty-handler + + + + org.apache.commons + commons-lang3 + + + + org.apache.httpcomponents + httpclient + + + + com.amazonaws + aws-java-sdk-ec2 + + + + + + ${finalName} + + + org.scalatest + scalatest-maven-plugin + + + test + + test + + + com.expedia.www.haystack.trace.storage.backends.mysql.unit + + + + integration-test + integration-test + + test + + + com.expedia.www.haystack.trace.storage.backends.mysql.integration + + + + + + + org.apache.maven.plugins + maven-shade-plugin + + true + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + package + + shade + + + + + reference.conf + + + ${mainClass} + + + + + + + + + net.alchim31.maven + scala-maven-plugin + + + + org.scalastyle + scalastyle-maven-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + + + + + diff --git a/backends/mysql/src/main/resources/config/base.conf b/backends/mysql/src/main/resources/config/base.conf new file mode 100644 index 00000000..cffe9371 --- /dev/null +++ b/backends/mysql/src/main/resources/config/base.conf @@ -0,0 +1,48 @@ +health.status.path = "/app/isHealthy" + +service { + port = 8090 + ssl { + enabled = false + cert.path = "" + private.key.path = "" + } +} + +mysql { + # multiple endpoints can be provided as comma separated list + endpoints = "jdbc:mysql://mysql/haystack" + driver = "com.mysql.cj.jdbc.Driver" + + + + connections { + max.per.host = 50 + read.timeout.ms = 30000 + conn.timeout.ms = 10000 + keep.alive = true + } + + credentials { + username: "root" + password : "root" + } + + retries { + max = 10 + backoff { + initial.ms = 100 + factor = 2 + } + } + + database: { + # auto creates the table in mysql(if absent) + # if schema field is empty or not present, then no operation is performed + auto.create.schema = "CREATE DATABASE IF NOT EXISTS haystack; USE haystack; create table IF NOT EXISTS spans (id varchar(255) not null, spans LONGBLOB not null, ts timestamp default CURRENT_TIMESTAMP, PRIMARY KEY (id, ts))" + + name: "spans" + + ttl.sec = 259200 + } +} diff --git a/backends/mysql/src/main/resources/logback.xml b/backends/mysql/src/main/resources/logback.xml new file mode 100644 index 00000000..dfaf958d --- /dev/null +++ b/backends/mysql/src/main/resources/logback.xml @@ -0,0 +1,27 @@ + + + + + + + true + + + + + + %d{yyyy-MM-dd HH:mm:ss:SSS} %thread, %level, %logger{70}, "%msg" %replace(%ex){'[\n]+', '\\n'}%nopex%n + + + + + + ${HAYSTACK_LOG_QUEUE_SIZE:-500} + ${HAYSTACK_LOG_DISCARD_THRESHOLD:-0} + + + + + + + \ No newline at end of file diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/Service.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/Service.scala new file mode 100644 index 00000000..4ed5cf66 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/Service.scala @@ -0,0 +1,92 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.expedia.www.haystack.trace.storage.backends.mysql + +import java.io.File + +import com.codahale.metrics.JmxReporter +import com.expedia.www.haystack.commons.logger.LoggerUtils +import com.expedia.www.haystack.commons.metrics.MetricsSupport +import com.expedia.www.haystack.trace.storage.backends.mysql.client.{MysqlTableSchema, SqlConnectionManager} +import com.expedia.www.haystack.trace.storage.backends.mysql.config.ProjectConfiguration +import com.expedia.www.haystack.trace.storage.backends.mysql.services.{GrpcHealthService, SpansPersistenceService} +import com.expedia.www.haystack.trace.storage.backends.mysql.store.{MysqlTraceRecordReader, MysqlTraceRecordWriter} +import io.grpc.netty.NettyServerBuilder +import org.slf4j.{Logger, LoggerFactory} + +object Service extends MetricsSupport { + private val LOGGER: Logger = LoggerFactory.getLogger("MysqlBackend") + + // primary executor for service's async tasks + implicit private val executor = scala.concurrent.ExecutionContext.global + + def main(args: Array[String]): Unit = { + startJmxReporter() + startService() + } + + private def startJmxReporter(): Unit = { + JmxReporter + .forRegistry(metricRegistry) + .build() + .start() + } + + private def startService(): Unit = { + try { + val config = new ProjectConfiguration + val serviceConfig = config.serviceConfig + val sqlConnectionManager = new SqlConnectionManager(config.mysqlConfig.clientConfig) + + MysqlTableSchema.ensureExists(config.mysqlConfig.databaseConfig.autoCreateSchema, sqlConnectionManager.getConnection) + + val tracerRecordWriter = new MysqlTraceRecordWriter(config.mysqlConfig,sqlConnectionManager) + val tracerRecordReader = new MysqlTraceRecordReader(config.mysqlConfig.clientConfig,sqlConnectionManager) + + val serverBuilder = NettyServerBuilder + .forPort(serviceConfig.port) + .directExecutor() + .addService(new GrpcHealthService()) + .addService(new SpansPersistenceService(reader = tracerRecordReader, writer = tracerRecordWriter)(executor)) + + // enable ssl if enabled + if (serviceConfig.ssl.enabled) { + serverBuilder.useTransportSecurity(new File(serviceConfig.ssl.certChainFilePath), new File(serviceConfig.ssl.privateKeyPath)) + } + + val server = serverBuilder.build().start() + + LOGGER.info(s"server started, listening on ${serviceConfig.port}") + + Runtime.getRuntime.addShutdownHook(new Thread() { + override def run(): Unit = { + LOGGER.info("shutting down gRPC server since JVM is shutting down") + server.shutdown() + LOGGER.info("server has been shutdown now") + } + }) + + server.awaitTermination() + } catch { + case ex: Throwable => + ex.printStackTrace() + LOGGER.error("Fatal error observed while running the app", ex) + LoggerUtils.shutdownLogger() + System.exit(1) + } + } +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/client/MysqlTableSchema.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/client/MysqlTableSchema.scala new file mode 100644 index 00000000..492f56d2 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/client/MysqlTableSchema.scala @@ -0,0 +1,65 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.client + +import java.sql.Connection + +import org.slf4j.LoggerFactory + +object MysqlTableSchema { + private val LOGGER = LoggerFactory.getLogger(MysqlTableSchema.getClass) + + val ID_COLUMN_NAME = "id" + val TIMESTAMP_COLUMN_NAME = "ts" + val SPANS_COLUMN_NAME = "spans" + val SERVICE_COLUMN_NAME = "service_name" + val OPERATION_COLUMN_NAME = "operation_name" + + /** + * ensures the keyspace and table name exists in com.expedia.www.haystack.trace.storage.backends.mysql + * + * @param connection com.expedia.www.haystack.trace.storage.backends.mysql client connection + * @param autoCreateSchema if present, then apply the sql schema that should create the keyspace and com.expedia.www.haystack.trace.storage.backends.mysql table + * + */ + def ensureExists(autoCreateSchema: Option[String], connection: Connection): Unit = { + autoCreateSchema match { + case Some(schema) => applySqlSchema(connection, schema) + case _ => + } + } + + /** + * apply the sql schema + * + * @param connection session object to interact with com.expedia.www.haystack.trace.storage.backends.mysql + * @param schema schema data + */ + private def applySqlSchema(connection: Connection, schema: String): Unit = { + val statement = connection.createStatement() + try { + for (cmd <- schema.split(";")) { + if (cmd.nonEmpty) statement.execute(cmd) + } + } catch { + case ex: Exception => + LOGGER.error(s"Failed to apply sql $schema with following reason:", ex) + throw new RuntimeException(ex) + } + } +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/client/SqlConnectionManager.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/client/SqlConnectionManager.scala new file mode 100644 index 00000000..8cc0de8f --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/client/SqlConnectionManager.scala @@ -0,0 +1,35 @@ +package com.expedia.www.haystack.trace.storage.backends.mysql.client + +import java.sql.Connection + +import com.expedia.www.haystack.trace.storage.backends.mysql.config.entities.ClientConfiguration +import org.apache.commons.dbcp.BasicDataSource +import org.slf4j.{Logger, LoggerFactory} + +class SqlConnectionManager(config: ClientConfiguration) extends AutoCloseable { + + private val LOGGER: Logger = LoggerFactory.getLogger(this.getClass) + + private lazy val connectionPool: BasicDataSource = { + + val basicDataSource = new BasicDataSource + basicDataSource.setDriverClassName(config.driver) + basicDataSource.setUrl(config.endpoints) + basicDataSource.setMaxActive(config.socket.maxConnectionPerHost) + config.plaintextCredentials.foreach { credentials => + basicDataSource.setPassword(credentials.password) + basicDataSource.setUsername(credentials.username) + } + basicDataSource + } + + def getConnection: Connection = { + connectionPool.getConnection + } + + override def close(): Unit = { + connectionPool.close() + } + + +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/ProjectConfiguration.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/ProjectConfiguration.scala new file mode 100644 index 00000000..d5d91edf --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/ProjectConfiguration.scala @@ -0,0 +1,90 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.config + +import com.expedia.www.haystack.commons.config.ConfigurationLoader +import com.expedia.www.haystack.commons.retries.RetryOperation +import com.expedia.www.haystack.trace.storage.backends.mysql.config.entities._ +import com.typesafe.config.Config +import org.apache.commons.lang3.StringUtils + +class ProjectConfiguration { + private val config = ConfigurationLoader.loadConfigFileWithEnvOverrides() + + val healthStatusFilePath: String = config.getString("health.status.path") + + val serviceConfig: ServiceConfiguration = { + val serviceConfig = config.getConfig("service") + + val ssl = serviceConfig.getConfig("ssl") + val sslConfig = SslConfiguration(ssl.getBoolean("enabled"), ssl.getString("cert.path"), ssl.getString("private.key.path")) + + ServiceConfiguration(serviceConfig.getInt("port"), sslConfig) + } + /** + * + * mysql configuration object + */ + val mysqlConfig: MysqlConfiguration = { + + + def databaseConfig(databaseConfig: Config): DatabaseConfiguration = { + val autoCreateSchemaField = "auto.create.schema" + val autoCreateSchema: Option[String] = if (databaseConfig.hasPath(autoCreateSchemaField) + && StringUtils.isNotEmpty(databaseConfig.getString(autoCreateSchemaField))) { + Some(databaseConfig.getString(autoCreateSchemaField)) + } else { + None + } + + DatabaseConfiguration(databaseConfig.getString("name"), databaseConfig.getInt("ttl.sec"), autoCreateSchema) + } + + val mysqlConfig = config.getConfig("mysql") + + + val credentialsConfig: Option[CredentialsConfiguration] = + if (mysqlConfig.hasPath("credentials")) { + Some(CredentialsConfiguration(mysqlConfig.getString("credentials.username"), mysqlConfig.getString("credentials.password"))) + } else { + None + } + + val socketConfig = mysqlConfig.getConfig("connections") + + val socket = SocketConfiguration( + socketConfig.getInt("max.per.host"), + socketConfig.getBoolean("keep.alive"), + socketConfig.getInt("conn.timeout.ms"), + socketConfig.getInt("read.timeout.ms")) + + MysqlConfiguration( + clientConfig = ClientConfiguration( + mysqlConfig.getString("endpoints"), + mysqlConfig.getString("driver"), + credentialsConfig, + socket), + retryConfig = RetryOperation.Config( + mysqlConfig.getInt("retries.max"), + mysqlConfig.getLong("retries.backoff.initial.ms"), + mysqlConfig.getDouble("retries.backoff.factor")), + databaseConfig = databaseConfig(mysqlConfig.getConfig("database")) + ) + } + +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/ClientConfiguration.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/ClientConfiguration.scala new file mode 100644 index 00000000..b482b7b6 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/ClientConfiguration.scala @@ -0,0 +1,55 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.config.entities + +import com.expedia.www.haystack.commons.retries.RetryOperation +import org.apache.commons.lang3.StringUtils + + +/** define the table information in mysql + * + * @param name : name of mysql table + * @param recordTTLInSec : ttl of record in sec + * @param autoCreateSchema : apply sql and create table if not exist, optional + */ +case class DatabaseConfiguration(name: String, + recordTTLInSec: Int = -1, + autoCreateSchema: Option[String] = None) { + require(StringUtils.isNotEmpty(name)) +} + +/** + * defines the configuration parameters for mysql client + * + * @param endpoints : list of mysql endpoints + * @param driver : Sql Driver Implementation Class + * @param socket : socket configuration like maxConnections, timeouts and keepAlive + */ +case class ClientConfiguration(endpoints: String, + driver: String, + plaintextCredentials: Option[CredentialsConfiguration], + socket: SocketConfiguration) + +/** + * @param retryConfig retry configuration if writes fail + */ +case class MysqlConfiguration(clientConfig: ClientConfiguration, + retryConfig: RetryOperation.Config, + databaseConfig: DatabaseConfiguration + ) + diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/CredentialsConfiguration.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/CredentialsConfiguration.scala new file mode 100644 index 00000000..b93364b1 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/CredentialsConfiguration.scala @@ -0,0 +1,21 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.config.entities + +case class CredentialsConfiguration(username: String, + password: String) diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/ServiceConfiguration.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/ServiceConfiguration.scala new file mode 100644 index 00000000..b17afe89 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/ServiceConfiguration.scala @@ -0,0 +1,23 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.config.entities + +/** + * @param port port to start grpc servicer on + */ +case class ServiceConfiguration(port: Int, ssl: SslConfiguration) +case class SslConfiguration(enabled: Boolean, certChainFilePath: String, privateKeyPath: String) diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/SocketConfiguration.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/SocketConfiguration.scala new file mode 100644 index 00000000..61eb083d --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/config/entities/SocketConfiguration.scala @@ -0,0 +1,23 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.config.entities + +case class SocketConfiguration(maxConnectionPerHost: Int, + keepAlive: Boolean, + connectionTimeoutMillis: Int, + readTimeoutMills: Int) diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/metrics/AppMetricNames.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/metrics/AppMetricNames.scala new file mode 100644 index 00000000..d10e8c26 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/metrics/AppMetricNames.scala @@ -0,0 +1,26 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.metrics + +object AppMetricNames { + val MYSQL_READ_TIME = "mysql.read.time" + val MYSQL_READ_FAILURES = "mysql.read.failures" + val MYSQL_WRITE_TIME = "mysql.write.time" + val MYSQL_WRITE_FAILURE = "mysql.write.failure" + val MYSQL_WRITE_WARNINGS = "mysql.write.warnings" +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/GrpcHandler.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/GrpcHandler.scala new file mode 100644 index 00000000..c78fe26a --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/GrpcHandler.scala @@ -0,0 +1,63 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.services + +import com.expedia.www.haystack.commons.metrics.MetricsSupport +import com.expedia.www.haystack.trace.storage.backends.mysql.services.GrpcHandler._ +import com.google.protobuf.GeneratedMessageV3 +import io.grpc.Status +import io.grpc.stub.StreamObserver +import org.slf4j.{Logger, LoggerFactory} + +import scala.concurrent.{ExecutionContextExecutor, Future} +import scala.util.{Failure, Success} + +object GrpcHandler { + protected val LOGGER: Logger = LoggerFactory.getLogger(classOf[GrpcHandler]) +} + +/** + * Handler for Grpc response + * populates responseObserver with response object or error accordingly + * takes care of corresponding logging and updating counters + * + * @param operationName : name of operation + * @param executor : executor service on which handler is invoked + */ + +class GrpcHandler(operationName: String)(implicit val executor: ExecutionContextExecutor) extends MetricsSupport { + private val metricFriendlyOperationName = operationName.replace('/', '.') + private val timer = metricRegistry.timer(metricFriendlyOperationName) + private val failureMeter = metricRegistry.meter(s"$metricFriendlyOperationName.failures") + + def handle[Rs](request: GeneratedMessageV3, responseObserver: StreamObserver[Rs])(op: => Future[Rs]): Unit = { + val time = timer.time() + op onComplete { + case Success(response) => + responseObserver.onNext(response) + responseObserver.onCompleted() + time.stop() + LOGGER.debug(s"service invocation for operation=$operationName and request=${request.toString} completed successfully") + + case Failure(ex) => + responseObserver.onError(Status.fromThrowable(ex).asRuntimeException()) + failureMeter.mark() + time.stop() + LOGGER.debug(s"service invocation for operation=$operationName and request=${request.toString} failed with error", ex) + } + } +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/GrpcHealthService.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/GrpcHealthService.scala new file mode 100644 index 00000000..426bdf1b --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/GrpcHealthService.scala @@ -0,0 +1,31 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.services + +import io.grpc.health.v1.{HealthCheckRequest, HealthCheckResponse, HealthGrpc} +import io.grpc.stub.StreamObserver + +class GrpcHealthService extends HealthGrpc.HealthImplBase { + + override def check(request: HealthCheckRequest, responseObserver: StreamObserver[HealthCheckResponse]): Unit = { + responseObserver.onNext(HealthCheckResponse + .newBuilder() + .setStatus(HealthCheckResponse.ServingStatus.SERVING) + .build()) + responseObserver.onCompleted() + } +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/SpansPersistenceService.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/SpansPersistenceService.scala new file mode 100644 index 00000000..4a4db7c4 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/services/SpansPersistenceService.scala @@ -0,0 +1,58 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.services + +import com.expedia.open.tracing.backend.WriteSpansResponse.ResultCode +import com.expedia.open.tracing.backend._ +import com.expedia.www.haystack.trace.storage.backends.mysql.store.{MysqlTraceRecordReader, MysqlTraceRecordWriter} +import io.grpc.stub.StreamObserver + +import scala.collection.JavaConverters._ +import scala.concurrent.ExecutionContextExecutor + +class SpansPersistenceService(reader: MysqlTraceRecordReader, + writer: MysqlTraceRecordWriter) + (implicit val executor: ExecutionContextExecutor) extends StorageBackendGrpc.StorageBackendImplBase { + + private val handleReadSpansResponse = new GrpcHandler(StorageBackendGrpc.METHOD_READ_SPANS.getFullMethodName) + private val handleWriteSpansResponse = new GrpcHandler(StorageBackendGrpc.METHOD_WRITE_SPANS.getFullMethodName) + + override def writeSpans(request: WriteSpansRequest, responseObserver: StreamObserver[WriteSpansResponse]): Unit = { + handleWriteSpansResponse.handle(request, responseObserver) { + writer.writeTraceRecords(request.getRecordsList.asScala.toList) map (_ => + WriteSpansResponse.newBuilder().setCode(ResultCode.SUCCESS).build()) + } + } + + /** + * 
+    * read buffered spans from backend
+    *
+ */ + override def readSpans(request: ReadSpansRequest, responseObserver: StreamObserver[ReadSpansResponse]): Unit = { + + handleReadSpansResponse.handle(request, responseObserver) { + reader.readTraceRecords(request.getTraceIdsList.iterator().asScala.toList).map { + records => { + ReadSpansResponse.newBuilder() + .addAllRecords(records.asJava) + .build() + } + } + } + } +} diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/store/MysqlTraceRecordReader.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/store/MysqlTraceRecordReader.scala new file mode 100644 index 00000000..dc6cb735 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/store/MysqlTraceRecordReader.scala @@ -0,0 +1,77 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.store + +import java.sql.Connection + +import com.expedia.open.tracing.backend.TraceRecord +import com.expedia.www.haystack.commons.metrics.MetricsSupport +import com.expedia.www.haystack.trace.storage.backends.mysql.client.MysqlTableSchema._ +import com.expedia.www.haystack.trace.storage.backends.mysql.client.SqlConnectionManager +import com.expedia.www.haystack.trace.storage.backends.mysql.config.entities.ClientConfiguration +import com.expedia.www.haystack.trace.storage.backends.mysql.metrics.AppMetricNames +import com.google.protobuf.ByteString +import org.slf4j.LoggerFactory + +import scala.concurrent.{ExecutionContextExecutor, Future, Promise} + +class MysqlTraceRecordReader(config: ClientConfiguration, sqlConnectionManager: SqlConnectionManager) + (implicit val dispatcher: ExecutionContextExecutor) extends MetricsSupport { + private val LOGGER = LoggerFactory.getLogger(classOf[MysqlTraceRecordReader]) + val readRecordSql = "SELECT * FROM spans WHERE id = ?" + private lazy val readTimer = metricRegistry.timer(AppMetricNames.MYSQL_READ_TIME) + private lazy val readFailures = metricRegistry.meter(AppMetricNames.MYSQL_READ_FAILURES) + + //We currently don't have a way to make an async jdbc call, we should investigate this further to see if its possible to make this call async. + def readTraceRecords(traceIds: List[String]): Future[Seq[TraceRecord]] = { + val timer = readTimer.time() + val promise = Promise[Seq[TraceRecord]] + var connection: Connection = null + try { + connection = sqlConnectionManager.getConnection + val statement = connection.prepareStatement(readRecordSql) + statement.setString(1, traceIds.head) + statement.execute() + val results = statement.getResultSet + var records: List[TraceRecord] = List() + + while (results.next()) { + val spans = results.getBlob(SPANS_COLUMN_NAME) + val record = TraceRecord.newBuilder() + .setTraceId(results.getString(ID_COLUMN_NAME)) + .setSpans(ByteString.copyFrom(spans.getBytes(1, spans.length().toInt))) + .setTimestamp(results.getTimestamp(TIMESTAMP_COLUMN_NAME).getTime) + .build() + records = record :: records + + } + promise.success(records) + promise.future + } + catch { + case ex: Exception => + readFailures.mark() + timer.stop() + LOGGER.error("Failed to read raw traces with exception", ex) + Future.failed(ex) + } + finally { + if (connection != null) connection.close() + } + } + +} \ No newline at end of file diff --git a/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/store/MysqlTraceRecordWriter.scala b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/store/MysqlTraceRecordWriter.scala new file mode 100644 index 00000000..71a29325 --- /dev/null +++ b/backends/mysql/src/main/scala/com/expedia/www/haystack/trace/storage/backends/mysql/store/MysqlTraceRecordWriter.scala @@ -0,0 +1,105 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.store + +import java.io.ByteArrayInputStream +import java.sql.{Connection, Timestamp} +import java.util.concurrent.atomic.AtomicInteger + +import com.expedia.open.tracing.backend.TraceRecord +import com.expedia.www.haystack.commons.metrics.MetricsSupport +import com.expedia.www.haystack.commons.retries.RetryOperation._ +import com.expedia.www.haystack.trace.storage.backends.mysql.client.SqlConnectionManager +import com.expedia.www.haystack.trace.storage.backends.mysql.config.entities.MysqlConfiguration +import com.expedia.www.haystack.trace.storage.backends.mysql.metrics.AppMetricNames +import org.slf4j.LoggerFactory + +import scala.concurrent.{ExecutionContextExecutor, Future, Promise} +import scala.util.{Failure, Success} + +class MysqlTraceRecordWriter(config: MysqlConfiguration, sqlConnectionManager: SqlConnectionManager)(implicit val dispatcher: ExecutionContextExecutor) + extends MetricsSupport { + + private val LOGGER = LoggerFactory.getLogger(classOf[MysqlTraceRecordWriter]) + private lazy val writeTimer = metricRegistry.timer(AppMetricNames.MYSQL_WRITE_TIME) + private lazy val writeFailures = metricRegistry.meter(AppMetricNames.MYSQL_WRITE_FAILURE) + + val writeRecordSql = "insert into spans values(?,?,?)" + + //We currently don't have a way to make an async jdbc call, we should investigate this further to see if its possible to make this call async. + private def execute(record: TraceRecord): Future[Unit] = { + + val promise = Promise[Unit] + + // execute the request async with retry + withRetryBackoff(retryCallback => { + var connection:Connection = null + try { + val timer = writeTimer.time() + connection = sqlConnectionManager.getConnection + val statement = connection.prepareStatement(writeRecordSql) + statement.setString(1, record.getTraceId) + statement.setBlob(2, new ByteArrayInputStream(record.getSpans.toByteArray)) + statement.setTimestamp(3, new Timestamp(System.currentTimeMillis())) + statement.execute() + retryCallback.onResult(statement.getResultSet) + } catch { + case ex: Exception => retryCallback.onError(ex, retry = true) + } + finally { + if (connection != null) connection.close() + } + }, + config.retryConfig, + onSuccess = (_: Any) => promise.success(), + onFailure = ex => { + writeFailures.mark() + LOGGER.error(s"Fail to write to mysql after ${config.retryConfig.maxRetries} retry attempts for ${record.getTraceId}", ex) + promise.failure(ex) + }) + promise.future + } + + /** + * writes the traceId and its spans to mysql. Use the current timestamp as the sort key for the writes to same + * TraceId. Also if the parallel writes exceed the max inflight requests, then we block and this puts backpressure on + * upstream + * + * @param traceRecords : trace records which need to be written + * @return + */ + def writeTraceRecords(traceRecords: List[TraceRecord]): Future[Unit] = { + val promise = Promise[Unit] + val writableRecordsLatch = new AtomicInteger(traceRecords.size) + traceRecords.foreach(record => { + /* write spanBuffer for a given traceId */ + execute(record).onComplete { + case Success(_) => if (writableRecordsLatch.decrementAndGet() == 0) { + promise.success() + } + case Failure(ex) => + //TODO: We fail the response only if the last mysql write fails, ideally we should be failing if any of the mysql writes fail + if (writableRecordsLatch.decrementAndGet() == 0) { + promise.failure(ex) + } + } + }) + promise.future + + } +} diff --git a/backends/mysql/src/test/resources/config/base.conf b/backends/mysql/src/test/resources/config/base.conf new file mode 100644 index 00000000..5a7d842a --- /dev/null +++ b/backends/mysql/src/test/resources/config/base.conf @@ -0,0 +1,44 @@ +haystack.graphite.host = "monitoring-influxdb-graphite.kube-system.svc" + +service { + port = 8090 + ssl { + enabled = false + cert.path = "/ssl/cert" + private.key.path = "/ssl/private-key" + } +} + +mysql { + # multiple endpoints can be provided as comma separated list + endpoints = "jdbc:mysql://mysql/haystack" + driver = "com.mysql.cj.jdbc.Driver" + + connections { + max.per.host = 100 + read.timeout.ms = 5000 + conn.timeout.ms = 10000 + keep.alive = true + } + + retries { + max = 2 + backoff { + initial.ms = 100 + factor = 2 + } + } + + credentials { + username: "root" + password : "root" + } + + database: { + # auto creates the table in mysql(if absent) + # if schema field is empty or not present, then no operation is performed + auto.create.schema = "CREATE DATABASE IF NOT EXISTS haystack; USE haystack; create table IF NOT EXISTS spans (id varchar(255) not null, spans LONGBLOB not null, ts timestamp default CURRENT_TIMESTAMP, PRIMARY KEY (id, ts))" + name: "spans" + ttl.sec = 86400 + } +} diff --git a/backends/mysql/src/test/resources/logback-test.xml b/backends/mysql/src/test/resources/logback-test.xml new file mode 100644 index 00000000..298193e0 --- /dev/null +++ b/backends/mysql/src/test/resources/logback-test.xml @@ -0,0 +1 @@ + diff --git a/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/integration/BaseIntegrationTestSpec.scala b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/integration/BaseIntegrationTestSpec.scala new file mode 100644 index 00000000..c45423b7 --- /dev/null +++ b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/integration/BaseIntegrationTestSpec.scala @@ -0,0 +1,66 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.integration + +import java.util.UUID +import java.util.concurrent.Executors + +import com.expedia.open.tracing.backend.{StorageBackendGrpc, TraceRecord} +import com.expedia.www.haystack.trace.storage.backends.mysql.Service +import com.google.protobuf.ByteString +import io.grpc.ManagedChannelBuilder +import io.grpc.health.v1.HealthGrpc +import org.scalatest._ + +trait BaseIntegrationTestSpec extends FunSpec with GivenWhenThen with Matchers with BeforeAndAfterAll with BeforeAndAfterEach { + protected var client: StorageBackendGrpc.StorageBackendBlockingStub = _ + + protected var healthCheckClient: HealthGrpc.HealthBlockingStub = _ + + private val executors = Executors.newSingleThreadExecutor() + + + override def beforeAll() { + + + + executors.submit(new Runnable { + override def run(): Unit = Service.main(null) + }) + + Thread.sleep(5000) + + client = StorageBackendGrpc.newBlockingStub(ManagedChannelBuilder.forAddress("localhost", 8090) + .usePlaintext(true) + .build()) + + healthCheckClient = HealthGrpc.newBlockingStub(ManagedChannelBuilder.forAddress("localhost", 8090) + .usePlaintext(true) + .build()) + } + + + protected def createTraceRecord(traceId: String = UUID.randomUUID().toString, + ): TraceRecord = { + val spans = "random span".getBytes + TraceRecord + .newBuilder() + .setTraceId(traceId) + .setTimestamp(System.currentTimeMillis()) + .setSpans(ByteString.copyFrom(spans)).build() + } +} diff --git a/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/integration/StorageBackendServiceIntegrationTestSpec.scala b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/integration/StorageBackendServiceIntegrationTestSpec.scala new file mode 100644 index 00000000..92a8c9eb --- /dev/null +++ b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/integration/StorageBackendServiceIntegrationTestSpec.scala @@ -0,0 +1,47 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.integration + +import java.util.UUID + +import com.expedia.open.tracing.backend.{ReadSpansRequest, WriteSpansRequest} + +class StorageBackendServiceIntegrationTestSpec extends BaseIntegrationTestSpec { + + + describe("Mysql Persistence Service read trace records") { + + it("should read and write trace records for given traceID") { + Given("trace") + val traceId = UUID.randomUUID().toString + val record = createTraceRecord(traceId) + val writeSpansRequest = WriteSpansRequest.newBuilder().addRecords(record).build() + + When("writespans is invoked") + val traceRecords = client.writeSpans(writeSpansRequest) + + Then("should write the trace") + val readSpansRequest = ReadSpansRequest.newBuilder().addTraceIds(traceId).build() + val retrievedRecord = client.readSpans(readSpansRequest) + + retrievedRecord.getRecordsList should not be empty + retrievedRecord.getRecordsCount shouldEqual 1 + retrievedRecord.getRecordsList.get(0).getTraceId shouldEqual traceId + } + + } +} diff --git a/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/unit/BaseUnitTestSpec.scala b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/unit/BaseUnitTestSpec.scala new file mode 100644 index 00000000..b81cf527 --- /dev/null +++ b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/unit/BaseUnitTestSpec.scala @@ -0,0 +1,22 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.expedia.www.haystack.trace.storage.backends.mysql.unit + +import org.scalatest.{FunSpec, GivenWhenThen, Matchers} +import org.scalatest.easymock.EasyMockSugar + +trait BaseUnitTestSpec extends FunSpec with GivenWhenThen with Matchers with EasyMockSugar diff --git a/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/unit/config/ConfigurationLoaderSpec.scala b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/unit/config/ConfigurationLoaderSpec.scala new file mode 100644 index 00000000..9bbbd22d --- /dev/null +++ b/backends/mysql/src/test/scala/com/expedia/www/haystack/trace/storage/backends/mysql/unit/config/ConfigurationLoaderSpec.scala @@ -0,0 +1,53 @@ +/* + * Copyright 2019 Expedia, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.expedia.www.haystack.trace.storage.backends.mysql.unit.config + +import com.expedia.www.haystack.trace.storage.backends.mysql.config.ProjectConfiguration +import com.expedia.www.haystack.trace.storage.backends.mysql.config.entities.ServiceConfiguration +import com.expedia.www.haystack.trace.storage.backends.mysql.unit.BaseUnitTestSpec + +class ConfigurationLoaderSpec extends BaseUnitTestSpec { + describe("ConfigurationLoader") { + val project = new ProjectConfiguration() + it("should load the service config from base.conf") { + val serviceConfig: ServiceConfiguration = project.serviceConfig + serviceConfig.port shouldBe 8090 + serviceConfig.ssl.enabled shouldBe false + serviceConfig.ssl.certChainFilePath shouldBe "/ssl/cert" + serviceConfig.ssl.privateKeyPath shouldBe "/ssl/private-key" + } + it("should load the mysql config from base.conf and few properties overridden from env variable") { + val mysqlConfig = project.mysqlConfig + val clientConfig = mysqlConfig.clientConfig + + // this will fail if run inside an editor, we override this config using env variable inside pom.xml + clientConfig.endpoints shouldBe "jdbc:mysql://mysql/haystack" + + clientConfig.socket.keepAlive shouldBe true + clientConfig.socket.maxConnectionPerHost shouldBe 100 + clientConfig.socket.readTimeoutMills shouldBe 5000 + clientConfig.socket.connectionTimeoutMillis shouldBe 10000 + mysqlConfig.retryConfig.maxRetries shouldBe 2 + mysqlConfig.retryConfig.backOffInMillis shouldBe 100 + mysqlConfig.retryConfig.backoffFactor shouldBe 2 + + mysqlConfig.databaseConfig.autoCreateSchema should not be None + mysqlConfig.databaseConfig.name shouldBe "spans" + mysqlConfig.databaseConfig.recordTTLInSec shouldBe 86400 + } + + } +} diff --git a/backends/pom.xml b/backends/pom.xml index f152a902..d04e9719 100644 --- a/backends/pom.xml +++ b/backends/pom.xml @@ -18,6 +18,7 @@ cassandra memory + mysql diff --git a/indexer/README.md b/indexer/README.md index a46d310e..aceb5344 100644 --- a/indexer/README.md +++ b/indexer/README.md @@ -23,8 +23,5 @@ In order to understand the haystack, we recommend to read the details of [haysta Its written in kafka-streams(http://docs.confluent.io/current/streams/index.html) and hence some prior knowledge of kafka-streams would be useful. -##Technical Details -Fill this as we go along.. - ## Building -Check the details on [Build Section](../README.md) \ No newline at end of file +Check the details on [Build Section](../CONTRIBUTING.md) \ No newline at end of file diff --git a/reader/README.md b/reader/README.md index 3ab07130..d6e32640 100644 --- a/reader/README.md +++ b/reader/README.md @@ -7,7 +7,5 @@ Service for fetching traces and fields from persistent storages. In order to understand this service, we recommend to read the details of [haystack](https://github.com/ExpediaDotCom/haystack) project. This service reads from [TraceBackend]() and [ElasticSearch](https://www.elastic.co/) stores. API endpoints are exposed as [GRPC](https://grpc.io/) endpoints. -Will fill in more details as we go.. - ## Building -Check the details on [Build Section](../README.md) \ No newline at end of file +Check the details on [Build Section](../CONTRIBUTING.md) \ No newline at end of file