get_impala_queries does not return all records

[funes79]

When I run a get_impala_queries from python it returns just 2 records, even if I use the same date range and filter.
When I filter it in the CM UI, the first two records appears immediately, and then after a second the rest of those queries.
Is it possible to get the next result somehow?

[Liuzhj]

hi @funes79

do you have some code to show ?

[funes79]

Running the query for expensive queries in the last 7 days returns just 2 record:

from datetime import datetime, timedelta
api = ApiResource(cm_host, username="reader", password="cmreader", version=18)
c = api.get_all_clusters()[0]

for s in c.get_all_services():
    if s.type == 'IMPALA':
        impala = s

now = datetime.utcnow()
daysback = 7
start = now - timedelta(days=daysback)
end = now
print('> Scanning last %s days, from %s till %s ' % ( daysback, start, end) )
filterStr = 'memory_aggregate_peak >= 60GB'
queries = impala.get_impala_queries(start_time=start, end_time=end, filter_str=filterStr, limit=1000, offset=0)
for query in queries.queries:
    print '> queyrid = '+query.queryId

Result:

Scanning last 7 days, from 2018-04-14 19:14:30.128989 till 2018-04-21 19:14:30.128989
queyrid = 4b4732a957d53eff:36aed14500000000
queyrid = 284a368120d0f55f:10950c0f00000000

But looping through and calling the get_impala_queries for shorter intervals returns more results:

for i in xrange(1,7):
    start = now - timedelta(days=i)
    end = now - timedelta(days=i-1)
    print('> Scanning %s days ago, from %s till %s ' % ( daysback, start, end) )    
    filterStr = 'memory_aggregate_peak >= 60GB'
    queries = impala.get_impala_queries(start_time=start, end_time=end, filter_str=filterStr, limit=1000, offset=0)
    for query in queries.queries:
        print '> queryid = '+query.queryId

Result:

Scanning 7 days ago, from 2018-04-20 19:14:30.128989 till 2018-04-21 19:14:30.128989
Scanning 7 days ago, from 2018-04-19 19:14:30.128989 till 2018-04-20 19:14:30.128989
queryid = 4b4732a957d53eff:36aed14500000000
queryid = 284a368120d0f55f:10950c0f00000000
Scanning 7 days ago, from 2018-04-18 19:14:30.128989 till 2018-04-19 19:14:30.128989
queryid = f4a9f149c0fc14d:520bc9ec00000000
queryid = 6a4b66e3cb8f1778:32ed8d7d00000000
queryid = bb488109a374db59:4eb2f8f700000000
queryid = b4963e9b6f9d1ea:9f55733800000000
.... and much more

[Liuzhj]

hi @funes79
ok ,i get it , i will try it tomorrow .

[Liuzhj]

hi @funes79

i am try to execute , it's execute normally,
this it my code

import ...

def impala_query(cluster):
    end = datetime.now()
    start = end - timedelta(days=7)
    print start, end
    for s in cluster.get_all_services():
        if s.type == 'IMPALA':
            impala = s
            q =  impala.get_impala_queries(start_time=start, end_time=end, filter_str='database=xxx')
            for i in q.queries:
                print i.queryId

if __name__ == '__main__':
    try:
        cm_host = 'xx'
        api = ApiResource(cm_host, username='reader', password='cmreader', version=6)
        clusterName = api.get_all_clusters()[0]
        impala_query(clusterName)

[funes79]

Cant be it somehow related to the fact, that the filtering takes some time, and in the GUI the first two rows appears immeadiately and then the others are fetched? So I think if the gui uses Solr or some other kind of index, then results are not available immeadiately, but in several steps.. This is typical for wiki search, where you get the first set of result and then a token, and with that token you continue to fetch more results

…

On Mon, Apr 23, 2018 at 4:33 AM, 刘志杰 ***@***.***> wrote: hi @funes79 <https://github.com/funes79> i am try to execute , it's execute normally, this it my code import ... def impala_query(cluster): end = datetime.now() start = end - timedelta(days=7) print start, end for s in cluster.get_all_services(): if s.type == 'IMPALA': impala = s q = impala.get_impala_queries(start_time=start, end_time=end, filter_str='database=xxx') for i in q.queries: print i.statement if __name__ == '__main__': try: cm_host = 'xx' api = ApiResource(cm_host, username='reader', password='cmreader', version=6) clusterName = api.get_all_clusters()[0] impala_query(clusterName) [image: image] <https://user-images.githubusercontent.com/14147011/39104190-bf8489f2-46e1-11e8-9041-11e8ba1406e6.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATRCidza9SCuO5RSlOB8yt_eND3MK2V1ks5trT1-gaJpZM4TdKZ6> .