Queries to IRSA TAP failing intermittently

[federicomarocco]

While trying to pull data from IRSA via TAP, I get the following error:

Traceback (most recent call last):
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connection.py", line 464, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/urllib3/connection.py", line 464, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/query.py", line 258, in execute_votable
    return votableparse(self.execute_stream(post=post).read)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 1101, in execute_stream
    return super().execute_stream(post=post)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/utils/decorators.py", line 9, in wrapper
    raw = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/query.py", line 207, in execute_stream
    response = self.submit(post=post)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 1134, in submit
    response = self._session.post(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fedde/NEOSurveyor/NEOSRef/scripts/make_neosref.py", line 18, in <module>
    catwise = get_catwise(ra,dec)
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/NEOSurveyor/NEOSRef/scripts/get_catwise.py", line 27, in get_catwise
    catwise = Irsa.query_tap(query=query).to_table()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/astroquery/ipac/irsa/core.py", line 78, in query_tap
    return self.tap.search(query, language='ADQL', maxrec=maxrec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 278, in run_sync
    **keywords).execute()
                ^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/tap.py", line 1117, in execute
    return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/fedde/anaconda3/envs/astro/lib/python3.11/site-packages/pyvo/dal/query.py", line 261, in execute_votable
    raise DALFormatError(e, self.queryurl)
pyvo.dal.exceptions.DALFormatError: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) 

The error is intermittent, but happens often enough to be a nuisance. The query I was trying to run is:

columns = 'source_id,ra,sigra,dec,sigdec,sigradec,pmra,sigpmra,pmdec,sigpmdec,w1mpro,w1sigmpro,w1snr,w1rchi2,w1sat,w2mpro,w2sigmpro,w2snr,w2rchi2,w2sat'
    
query = ("select "+columns+" from catwise_2020 where ra>="+str(minra)+" and ra<"+str(maxra)+" and dec>="+str(mindec)+" and dec<"+str(maxdec)+" and nb=1 and ab_flags='00' and cc_flags='0000'")`

catwise = Irsa.query_tap(query=query).to_table()

Where minra,maxra,mindec,maxdec are set iteratively within a for loop. Entering debug mode and doing votableparse(self.execute_stream(post=post).read) returns a valid VO table <class 'astropy.io.votable.tree.VOTableFile'>, which is very puzzling.

The code never seems to fail twice on the same set of minra,maxra,mindec,maxdec, so I would think that getting astroquery to retry instead of crashing would fix the issue.

[bsipocz]

I was trying to reproduce and debug this yesterday with Federico. Given that we couldn't pin-point this down, I asked him to open an issue with the traceback hoping more pairs of eyes would get further.

My first thought was that this was a server timeout, something similar I saw last week where a sync query didn't go through but running it async succeeded (for that one I got a 502 error coming from the same line, but overall a similarly unuseful error message).

But this case is very puzzling as jumping into debug mode the response votable parses just fine.

So, at the very least, this is a report along the same line as a couple other issues that are calling for better error messages, and preferably some internal simplifications, so it will become much easier to debug and see the actual problems.