fields could be missing in Arkime due to a large number of concurrent requests

[mmguero]

Malcolm's is notorious for pushing the boundaries of Arkime in terms of the number of custom fields we create (like 3k+). This occasionally causes issues in Arkime surrounding creating and retrieving those fields. Totally not bagging on Arkime, they're fantastic and always helpful to find the solution, it's just the nature of integrating with another tool.

Recently I discovered that some fields weren't getting created correctly on the initial statup of Arkime. Andy Wick helped point in the direction of the debug I needed to do and I saw this:

malcolm-arkime-1  | Aug 11 14:46:40 config.c:115 moloch_config_section_str(): custom-fields.zeek.rdp.security_protocol=db:zeek.rdp.security_protocol;group:zeek_rdp;kind:termfield;friendly:Security Protocol;help:Security Protocol
malcolm-arkime-1  | Aug 11 14:46:40 field.c:131 moloch_field_define_text_full(): Parsing db:zeek.rdp.security_protocol;group:zeek_rdp;kind:termfield;friendly:Security Protocol;help:Security Protocol
malcolm-arkime-1  | Aug 11 14:46:40 config.c:115 moloch_config_section_str(): custom-fields.zeek.rdp.client_channels=db:zeek.rdp.client_channels;group:zeek_rdp;kind:termfield;friendly:Channel;help:Channel
malcolm-arkime-1  | Aug 11 14:46:40 field.c:131 moloch_field_define_text_full(): Parsing db:zeek.rdp.client_channels;group:zeek_rdp;kind:termfield;friendly:Channel;help:Channel
malcolm-arkime-1  | Aug 11 14:46:40 http.c:763 moloch_http_schedule(): ERROR - Dropping request (https://arkime.com/faq#error-dropping-request) /arkime_fields/_doc/zeek.rdp.client_channels of size 128 queue 1001 is WAY too big

The (temporary, at least) fix is to set the requests for field creation in Arkime to be higher priority (see mmguero-dev/Malcolm@3ad2f9c). Andy says he's going to look at better solutions in Arkime in the future.