media-search is a service for searching for media such as on YouTube by queries, accessible both on the commandline and on a web user interface, via HTTP requests. It is instrumented with OpenCensus which gives us distributed tracing and monitoring for visibility into our entire system as a call propagates through the various microservices.
It consists of:
- A frontend service (OFE) which is accessible by HTTP requests
- Clients in Python3, Go, Javascript+HTML(accessible via webpage)
- A database to store information about already seen queries(caching)
- A backend for searching for content on YouTube's servers
- A backend for getting Ids
- A backend for asynchronous detail retrieval of searched queries so that later media inspections quickly pull up information on searched media
- An instrumented Mongo Go driver from https://github.com/orijtech/mongo-go-driver
Normal access is by hitting the frontend service (OFE) via HTTP requests to /search with either GET or POST methods. OFE then invokes a gRPC transport based connection to the search backend(SB) which then searches for content from YouTube. Because content search from YouTube incurs API quota costs as well as time expense having to fetch from YouTube's servers, it helps to cache results so that subsequent repetitions will return cached content in very little time. Once results have been returned during a cache-miss, they are cached to MongoDB and a subsequent asynchronous call is made to a gRPC accessible service that then fetches individual meta information about each video and also caches that to MongoDB.
The architectural diagram looks something like this:
After tracing through requests, the service map might look like this:
go get -u -v github.com/orijtech/media-search/...
Assuming you already set the credentials in Requirements, run
make run-microservices
Also don't forget to run Prometheus like this
prometheus --config.file=prometheus.yml
If you'd like to terminate all the running binaries/microservices, and have pkill
in your shell, you can run:
make kill-microservices
Client | Language | Running it |
---|---|---|
Web UI | Javascript+HTML | Visit http://localhost:9778 |
clients/client.go | Go | go run clients/client.go |
clients/client.py | Python3 | python3 clients/client.py |
clients/client.sh | Shell | ./clients/client.sh |
The WebUI looks something like this
You can examine traces and metrics by visiting
- AWS X-Ray
- Stackdriver
The clients' HTTP requests propagate their traces through to the server and back, and then to the exporters yielding insights such as: