If a query is successfully processed via Buddy log it in the original form

[donhardman]

Proposal:

We are currently logging all queries, both those with errors and those without. However, when a query is proxied to the Buddy and the Buddy processes it successfully, we do not log those queries at all. We only log when an error is returned.

We should log all successfully processed queries that were originally erroneous but were proxied to the Buddy. If the Buddy returns an OK result for a query that initially had an error, we should log the ORIGINAL query as successful since it was processed without error with the help of the Buddy.

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Task estimated
• Specification created, reviewed and approved
• Implementation completed
• Tests developed
• Documentation updated
• Documentation proofread
• Changelog updated
• OpenAPI YAML updated and issue created to rebuild clients

[tomatolog]

I need concrete example what queries logs wrong as search, update, replace, system handles errors differently. SphinxQL and HTTP JSON handler also has different error processing.

For example for search query it is hard to do for SphinxQL handler as:

• query got processed by handler that logs error then up the stack daemon takes decision send reply to client or send request to buddy. And error already got put into query log for that query.
• if query got fixed by buddy and good result returned there is no internal info about fixed query such as timings \ io and cpu counters \ total counters that is why it is not clear what to log in this case.

[donhardman]

For queries that will be passed to the Buddy, we can use the following, which is valid and should be handled by Buddy:

SELECT * FROM table WHERE match('test') OPTION fuzzy=1, layouts=''

We need to track timings and all necessary information for logging to maintain a consistent format on the daemon side.

[tomatolog]

daemon does not have any timings \ agent timings \ io and cpu counters \ total counters when got reply from buddy. If these should be put into log to query looks like original buddy should collect these metrics and provide them into daemon.

Another case is that daemon passes all the failed requests into buddy but query_log has only search requests if daemon will log all source strings after buddy handles these well query_log will have inserts \ updates \ bulk \ call keywords and so on. Buddy should reply into daemon with the handled request type to filter out requests and log only search requests.

The only daemon has is the source query string (SphinxQL or JSON) and it can log it as is, ie if the query_log_format option is plain daemon still logs the source SphinxQL or JSON string that will look unusual.

[tomatolog]

after the discussion with the @sanikolaev it worth to change response from Buddy to Manticore Search to include the new object query_log that will be a flag for daemon to store the current request string as is into query log and also include the following metrics into that object:

• total_found
• time

These could be get by buddy by the SHOW META query for every SELECT it need to log by the daemon. If the SHOW META output has cpu_time agents_cpu_time these also could be provided and logged by the daemon to make sure the query log entry is similar to Query-logging

{
  "type": "sql response",
  "message": [
    {
      "total": 6,
      "error": "",
      "warning": ""
    }
  ],
  "error_code": 0,
  "version": 2,
"query_log" : {
"total_found": 1000,
"time": 0.01,
"cpu_time":18.004,
"agents_cpu_time":0.00
}
}

Without the new flag \ object daemon has no information is it the search query and should be log into query.log or the bulk \ insert \ ALTER or some system query and should be skipped

[donhardman]

As discussed before, instead of using SHOW META all the time, it seems to be a better idea to add meta information to EACH response, in case it's requested with Buddy at least. This may contain a "meta" field with all information. In that case, when Buddy performs multiple queries to solve a request, we do not need to add "SHOW META" manually all the time. Instead, we can modify the core to aggregate the meta responses, accumulate them, and return them to the daemon.

[donhardman]

We should process and try to log it with daemon.

We can validate it by using the following MRE:

mysql> create table test(name text) min_infix_len='1';
mysql> select * from test where match('hello') option fuzzy = 1;

And we will get response from buddy as follows:

[BUDDY] [671a23e68c8717.40395458] response data: {"version":3,"type":"sql response","message":[{"total":0,"error":"","warning":"","columns":[{"id":{"type":"long long"}},{"name":{"type":"string"}}]}],"meta":{"total":"0","total_found":"0","total_relation":"eq","time":"0.008"},"error_code":200}

Let's proceed with logging such queries. In case we have no meta info it will be null.

[tomatolog]

merged at 210b06c and now if SphinxQL request to buddy passed well it logs into query log

[tomatolog]

not sure if @PavelShilin89 should add test for this case or not

[sanikolaev]

not sure if @PavelShilin89 should add test for this case or not

Absolutely, let's add a CLT test which tests that various queries processed through Buddy are logged properly.

[sanikolaev]

@donhardman as we discussed, the remaining issue is that the daemon currently logs all queries processed through Buddy. Instead, we only want to log SELECT queries. Let's handle this by introducing a new log flag.

[donhardman]

Task: manticoresoftware/manticoresearch-buddy#394

[tomatolog]

wrong commit reference

[sanikolaev]

Blocked by manticoresoftware/manticoresearch-buddy#394

[donhardman]

Let's check the meta property, and when we find a total value in it, we can confirm this is a select operation and log it.

[tomatolog]

should be fixed at 9222c03 now daemon does not log into query.log if the buddy reply has no meta.total

[sanikolaev]

should be fixed at 9222c03 now daemon does not log into query.log if the buddy reply has no meta.total

Looks good to me.

Reopening since there are non-checked items in the checklist.

Assigning to @PavelShilin89 to prepare tests. These should include SQL/JSON select queries that are processed via Buddy and sent via MySQL/HTTP. The queries should be logged according to the documentation here: https://manual.manticoresearch.com/Logging/Query_logging#Query-logging

[PavelShilin89]

Tests have already been developed previously covering these cases.