Data not showing up

[ReporterMarina]

Hi I'm trying to scrape this URL. Normally I just click Download, Data, Show All... but that options has been greyed out lately.

https://covid19tracker.health.ny.gov/views/DailyHospitalizationSummary/Reopening-DailyHospitalization?%3Aembed=y&%3AshowVizHome=y&%3Atabs=y&%3Atoolbar=y&%3Adevice=desktop&%3AapiID=host0#navType=1&navSrc=Parse

When I can download it, the Worksheet is called: Chart (2)

Column headings are: COUNTY | CURRENT_ICU_PATIENTS | CURRENT_INTUBATED_PATIENTS | CURRENT_PATIENTS | Header | RegionFilter | EconRegion | Last Updated Date | DAILY_PATIENTS | ECON_REGION | FAC_LAT | FAC_LONG | HOSPITAL | HOSPITAL_NETWORK | NYC_IND | Number of Records | REGION | RPT_DT | SUB_REGION | TOTAL_DEATH | TOTAL_DISCHARGE

I used your code and added:

t.data.to_csv(t.name + '.csv')

But I only got a few column heads:

RPT_DT-value RPT_DT-alias SUM(CURRENT_PATIENTS)-value SUM(CURRENT_PATIENTS)-alias Measure Names-alias SUM(CURRENT_ICU_PATIENTS)-alias

How do I get all of the column heads?

When I CAN download the data, it brings me to this URL:

https://covid19tracker.health.ny.gov/vizql/w/DailyHospitalizationSummary/v/Reopening-DailyHospitalization/viewData/sessions/769B5BD96B944D4AB3F118C46FDE92A3-4:0/views/12302628778901485932_5034431464735168763?maxrows=200&viz=%7B%22worksheet%22%3A%22Chart%20(2)%22%2C%22dashboard%22%3A%22Reopening%20-%20Daily%20Hospitalization%22%7D

Then I click Full Data, Show All Columns, then Download All Rows As a Text File.

[bertrandmartel]

@subwaysurfing Hello, It seems it's a problem on the server side. I've tried to implement the api call used to get those views and sometimes they work, sometimes not. Exactly like in the browser. From time to time, the download button is greyed out, and when this happens, the api calls don't work anymore. I'm not sure what the problem is for the download being unavailable at moments.

What I can do is to implement the "show all column" api which is :

POST https://covid19tracker.health.ny.gov/vizql/w/DailyHospitalizationSummary/v/Reopening-DailyHospitalization/sessions/{session_id}/commands/tabdoc/get-underlying-data

but it will only work when the download button is not greyed out, so it won't solve the initial issue.

[ReporterMarina]

Thanks, interesting! Are the session IDs randomly generated and then they won't work after one 'use'? Or can they be 'reused' again? I'll try using that URL with the Page Refresh extension in hopes of not having to spend all day hitting refresh anymore.

…

On Fri, Jun 18, 2021 at 10:20 AM Bertrand Martel ***@***.***> wrote: @subwaysurfing <https://github.com/subwaysurfing> Hello, It seems it's a problem on the server side. I've tried to implement the api call used to get those views and sometimes they work, sometimes not. Exactly like in the browser. From time to time, the download button is greyed out, and when this happens, the api calls don't work anymore. I'm not sure what the problem is for the download being unavailable at moments. What I can do is to implement the "show all column" api which is : POST https://covid19tracker.health.ny.gov/vizql/w/DailyHospitalizationSummary/v/Reopening-DailyHospitalization/sessions/B8CA56840F464E83BA511E7F4C77B167-7:5/commands/tabdoc/get-underlying-data but it will only works when the download button is not greyed out, so it won't solve the initial issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUQ2S44QEW4OPIYX5UFEQ33TTNI3VANCNFSM4644JYWA> .

[bertrandmartel]

@subwaysurfing The session ID is extracted from a JSON embedded in the html (in a textarea with id tsConfigContainer). Unless the page is refreshed, the session id is the same when you navigate on Tableau filters/selection etc...

I will try to implement the API so that you can write a script that will automate this

[bertrandmartel]

I've created #19

[bertrandmartel]

@subwaysurfing I've released v0.1.13 with the following features:

• get download summary
• get download underlying data
• get csv data

The last one is the one you want to use :

from tableauscraper import TableauScraper as TS
from tableauscraper.TableauScraper import TableauException
import time

loop = True
schedulingSeconds = 5

url = 'https://covid19tracker.health.ny.gov/views/DailyHospitalizationSummary/Reopening-DailyHospitalization'

def schedule():
  print("scheduling")
  time.sleep(schedulingSeconds)

while (loop):
    ts = TS()
    try:
      ts.loads(url)
      wb = ts.getWorkbook()
      data = wb.getCsvData(sheetName="Chart (2)", prefix="vud")
      if data is not None:
          loop = False
          print(data)
      else:
          schedule()
    except TableauException:
      schedule()

try this on repl.it

Checkout readme for getCsvData usage

You have also an example here for a more generic way to use it. In the latter link it gets the Wyoming covid dashboard data on public.tableau.com.

I've noticed that public.tableau.com and your Tableau server are not using the same path prefix for the csv api:

dataUrl = f'{scraper.host}{scraper.tableauData["vizql_root"]}/{prefix}/sessions/{scraper.tableauData["sessionid"]}/views/{viewId}'

The prefix variable is vud in your Tableau server URL but it's vudcsv in the official public.tableau.com. This is why I've added a prefix argument that must be set in your case as this prefix value is hardcoded in JS

Also note that the above code schedules the API calls every 5 seconds which sometimes works directly, sometimes after a few minutes. The Tableau server seems to timeout sometimes so I've added TableauException exception case