-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Logstash persistent queue improvement #2744
Conversation
2141194
to
f2180ad
Compare
@PMazarovich Could you please merge current develop changes (or rebase, as you prefer) to fix CI build? |
f2180ad
to
6d9d995
Compare
Rebase done |
'host': os.getenv('DJANGO_LOG_SERVER_HOST', 'localhost'), | ||
'port': os.getenv('DJANGO_LOG_SERVER_PORT', 5000), | ||
'port': os.getenv('DJANGO_LOG_SERVER_PORT', 8080), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the reason to change the port from 5000 to 8080?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For http protocol, port 8080 is standard one, and logstash now works via http because of persistent queue
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@azhavoro , we use 8080 for cvat_proxy. Can it be a problem?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not open port, I don't see any problem here. We use the same port number for cvat_server https://github.com/openvinotoolkit/cvat/blob/develop/Dockerfile#L157
@PMazarovich , thanks for the contribution! |
Co-authored-by: Andrey Zhavoronkov <[email protected]>
HI, the log in the rq like INFO:cvat.server_33:Create Task #33 ,Uploadpath ... is not into logstash |
Motivation and context
In current implementation of ELK there is no persistent queue in logstash. This can lead to the loss of important analytics data. With this improvement any data, that was sent by cvat will eventually be sent into logstash if logstash instance was unavaliable for some time. Or to elasticsearch instance if it was unavaliable. So, all of features of persistent queue is now ready for working with cvat.
As a side note, there is a metadata with year and month added. This is done for better flexibility in index management in elasticsearch.
How has this been tested?
It was tested manually
Checklist
develop
branchcvat-core, cvat-data and cvat-ui)
License
Feel free to contact the maintainers if that's a concern.