Refactored operation polling to only request active operation

[elegantmoose]

Potentially:

• addresses Quick fix for high network use with large operation sets #52
• fixes Fix for high network utilization from operation api endpoint #51

[elegantmoose]

@timbrigham-oc - Can you try this branch for me and see if it solves your problem?

i.e.

cd caldera/plugins/magma
git fetch --all
git checkout fix/operation_data_network_usage
cd caldera
python server.py --build

and let me know how it goes

[timbrigham-oc]

@elegantmoose, I just looked at this. I do definitely see a change in behavior, it looks like it's now only pulling the specs from the currently executing operation. Very good change IMO, something I was looking at how to do myself. :)

If I pick one of my larger lateral movement tests it's around 1.5 megabytes each pull, still much better then it has been. I am still seeing refreshes occurring on finished operations. Know that wasn't the intent of that branch, just noting its happening.

This has dropped my network usage by an order of magnitude.

[elegantmoose]

Awesome. Ill look at the finished operations polling and see if also quick fix. Either way Ill merge this partial fix today.

[elegantmoose]

@timbrigham-oc Pull branch again and see if network traffic gets more reduced. (added check for finished operations)

[timbrigham-oc]

@elegantmoose , I'm in the middle of a test sequence now, I'll check back on this in the morning.

[elegantmoose]

@timbrigham-oc no rush 👍

[timbrigham-oc]

Ok, following up. :)

On a preliminary look I'm still seeing operations updating on running tasks where the data refreshing only for the active operation, so that behavior still looks good. The refreshes on finished operations are also not firing, so that should be good as well.

[timbrigham-oc]

@elegantmoose, I'm seeing an issue that might (I haven't made any other code changes, and just noticed this now) be caused by this update. The 'Filters' view is not being restricted to only only host / agent IDs and abilities that are relevant to a given operation.

You can see that this operation ran against one agent and only used three abilities (nma, reverse lookup ip, dns match)

The abilities, in the filter view are showing every possible ability from the adversary definition, and every agent ID.

[timbrigham-oc]

It's strange.. the same exact contents on that dialog will also show, regardless of which operation I have selected.
If I force reload the page it clears up temporarily.

I'm guessing it might be something along the line of whichever operation I select first - if I then select another operation - isn't getting updated.

[elegantmoose]

Yea, that was probably me that introduced those. Ill take a look at ASAP. Opened new issue - mitre/caldera#3004