This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 857201. The European Commission assumes no responsibility for any content of this repository.

Cache Monitor

Monitors availability of media items in Cache

What is this?

The Cache Monitor service is part of the platform for media caching on trains. It is responsible for observing the cache directory of the Cache to determine availability of content. Changes in the availability of content items are reported to the State API which adapts the cache state in the State DB accordingly. The cache state is queried by the Prefetcher which will request missing content items from the cache in order for it to load the content from the upstream location.

How does it work?

Below architecture diagram shows the software modules that implement the essential functionalities of the Cache Monitor. These include clients for HTTP communication with the State API and communication with the Message Streamer. A core logic module manages the file system observer and coordinates reporting of content item availability. The file system oberver notifies the whenever a file is added to or removed from the observed cache directory. The NGINX based cache uses the md5 hash of hostname and path of the upstream location of a given resource as a filename. We use thsi knowledge on NGINX' naming convention to detect whether a content item is available in the cache or not.

The basic program flow of the core logic in monitor.service.ts is as follows:

• Set listener for messages of type 'new-cache-state'
• On 'new-cache-state' message: load cache state (this is very likely an iterative process, as the cache state comprises likely a couple of thousend documents. State API supports pagination which is used in those cases.)
• For each content item in the cache state (/on a cache state item page): if the content item's urlHash property equals one of the names of the files in the cache directory, report the content item to be available, else to be missing
• Set listener for file change events of type 'add' and 'remove' (i.e. 'unlink'in the used chokidar module)
• On file 'add': Set availability of content item with property 'urlHash' equal to file name (cache hash) to true:
  • Will through an error if content item with this hashUrl is not known
  • Intermediate files (e.g. file for which the download has not completed) in the cache directory will be ingnored as those do not have the final name as expected by the cache state. Atempts to update the cache state for those files would result in errors in any case.
• On file 'add': Set availability of content item with property 'urlHash' equal to file name (cache hash) to false

Technologie used

• Nest.js

Install, build, run

Note: Typically you would use the up.sh script from the Platform project to install, build and run this service as part of a composite of docker services. Read on if you intend to run the service directly on your host system.

Prerequestits: Following software needs to be installed on your host machine in order to execute the subsequent steps.

• Node.js
• NPM

First, git clone this project and change into its root directory. Than run the following command to install its dependencies:

$ npm install

You can than run the service in three different modes.

# development
$ npm run start

# watch mode
$ npm run start:dev

# production mode
$ npm run start:prod

With following command you can build a docker image for this service. But again, typically you use the startup script up.sh of the Platform project to do the job.

$ docker build -t 5gv-cache-monitor .