Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stabilize REST API mode #43

Closed
michaelweiser opened this issue Sep 21, 2018 · 1 comment
Closed

Stabilize REST API mode #43

michaelweiser opened this issue Sep 21, 2018 · 1 comment
Assignees
@michaelweiser
Labels
enhancement needs investigation
Milestone
2.0

Comments

@michaelweiser
Copy link
Contributor

The current REST API mode works but needs additional error handling and testing for exceptional conditions and erros.
@michaelweiser michaelweiser added this to the 1.7 milestone Sep 21, 2018
@michaelweiser michaelweiser self-assigned this Sep 21, 2018
@michaelweiser
Copy link
Contributor Author

Peekaboo fails with the following error if the connection to the API cannot be established. Afterwards Peekaboo is in an unresponsive state and does not retry or at least shut down.

2018-11-23 15:57:06,345 - urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTTP connection (1): localhost:8090
HTTPConnectionPool(host='localhost', port=8090): Max retries exceeded with url: /cuckoo/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f270c0aaa50>: Failed to establish a new connection: [Errno 111] Connection refused',))
2018-11-23 15:57:06,349 - urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTTP connection (1): localhost:8090
HTTPConnectionPool(host='localhost', port=8090): Max retries exceeded with url: /cuckoo/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f270c0c61d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2018-11-23 15:57:06,356 - urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTTP connection (1): localhost:8090
HTTPConnectionPool(host='localhost', port=8090): Max retries exceeded with url: /cuckoo/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f270c0c60d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/opt/peekaboo/bin/peekaboo", line 11, in <module>
    load_entry_point('PeekabooAV==1.6.1', 'console_scripts', 'peekaboo')()
  File "build/bdist.linux-x86_64/egg/peekaboo/daemon.py", line 297, in run
  File "build/bdist.linux-x86_64/egg/peekaboo/toolbox/cuckoo.py", line 202, in __init__
  File "build/bdist.linux-x86_64/egg/peekaboo/toolbox/cuckoo.py", line 240, in __status
  File "build/bdist.linux-x86_64/egg/peekaboo/toolbox/cuckoo.py", line 236, in __get
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8090): Max retries exceeded with url: /cuckoo/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f270c0c60d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

@michaelweiser michaelweiser modified the milestones: 1.7, 2.0 Apr 23, 2019
michaelweiser added a commit to michaelweiser/PeekabooAV that referenced this issue May 8, 2019
@michaelweiser
More robustly poll Cuckoo REST API jobs
a9c4d3c 
Downloading the full list of jobs gets less and less efficient as the
number of current and past jobs increases. There is no way to filter
down to specific job IDs. The limit and offset parameters of the list
action of the API cannot be used to achieve a similar effect because the
job list is not sorted by job/task ID and the parameters seem only meant
to iterate over the whole list in blocks, not to extract specific jobs
from it.

The previous logic of determining the highest job ID at startup and
requesting the next million entries from that offset on was therefore
likely not working as expected and making us "blind" to status changes
of jobs which end up below our offset in the job list.

This change adjusts the CuckooAPI to make use of the list of running
jobs we've had for some time now to work around this. Instead of getting
a list of all jobs starting from the highest job id we saw at startup we
just get each job's status individually. While this makes for more
requests, it should over a longer runtime make for less network traffic
and reliably get us the data we need about our jobs.

Also, turn the shutdown_requested flag into an event so we can use its
wait() method to also implement the poll interval and get immediate
reaction to a shutdown request.

Finally, we switch to endless retrying of failed job status requests
paired with the individual request retry logic introduced earlier. On
submission we still fail the submission process after timeouts or
retries on the assumption that without the job being submitted to
Cuckoo, our feedback to the client that analysis fail will cause it to
resubmit and still avoid duplicates.

Closes scVENUS#43.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michaelweiser michaelweiser

Labels
enhancement needs investigation
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
1 participant
@michaelweiser