-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature]GeoIP datasource implementation #6559
Comments
For GeoIP datasource implementation, the original plan was having n+1 system index. For example, we will have one system index to store datasource metadata and one system indice for each of n datasource. During update, it will create a new system index and delete the old system index for a datasource. Other options to reduce the operation of create/delete index are,
The problem that I encountered with above three options are that,
Therefore, unless there are hidden risk of continuous creation/deletion of system index with frequency of twice a week per datasource, the original approach is better than others in terms of latency and success rate. Another option I can think of is utilizing post filtering after getting result from an index to avoid circuit breaking exception. |
|
|
Implementation is completed. |
Description
This document contains implementation detail on GeoIP datasource as part of #5856
Tasks
Tasks are listed here to track a progress in the implementation. One PR can cover multiple tasks if code change is small.
Create datasource
Update datasource
Read datasource
Delete datasource
Update GeoIP database
Delete GeoIP database
User scenarios
Create/Update of GeoIP data source
Datasource API signature
GeoIP database in an index
Manifest.json
Deletion of GeoIP data source
Cluster manager node failure
All of the works related with GeoIP datasource will be executed in a cluster manager node. The cluster manager node maintains scheduled tasks in memory. When cluster manager node fails, it will fail over to the one of cluster eligible node. The new cluster manager node will scan all existing GeoIP datasource and schedule tasks again accordingly. It use "next_update" field in GeoIP datasource to set correct time to update GeoIP databases.
The text was updated successfully, but these errors were encountered: