…getredash#4262)

…4256)

* track general RQ queues (default, periodic and schemas)

* get all active RQ queues

* call get_celery_queues in another place

…o continue running periodically even after longer executions (getredash#4283)

* Make core app compatible with Python 3

No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.

* Use Python 3.7 as base docker image

Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.

* Upgrade some requirements to newest versions

Some of the older versions were not compatible with Python 3.

* Migrate tests to Python 3

* Build frontend on Python 3

* Make the HMAC sign function compatible with Python 3

In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.

* Use assertCountEqual instead of assertItemsEqual

The latter is not available in Python 3.
See https://bugs.python.org/issue17866

* Remove redundant encoding header for Python 3 modules

* Remove redundant string encoding in CLI

* Rename list() functions in CLI

These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().

Only the Python function is renamed, from the perspective of the
CLI nothing changes.

* Replace usage of Exception.message in CLI

`message` is not available anymore, instead use the string
representation of the exception.

* Adapt test handlers to Python 3

* Fix test that relied on dict ordering

* Make sure test results are always uploaded (getredash#4215)

* Support encoding memoryview to JSON

psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See getredash#3156

* Fix test relying on object address ordering

* Decode bytes returned from Redis

* Stop using e.message for most exceptions

Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.

* Fix writing XLSX files in Python 3

The buffer for the file should be made of bytes and the actual
content written to it strings.

Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.

* Fix test by comparing strings to strings

* Fix another exception message unavailable in Python 3

* Fix export to CSV in Python 3

The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.

* (Python 3) Use Redis' decode_responses=True option (getredash#4232)

* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)

* Make sure Redis connection uses decoded_responses option

* Remove unused imports.

* Use Redis' decode_responses option

* Remove cases of explicit Redis decoding

* Rename helper function and make sure it doesn't apply twice.

* Don't add decode_responses to Celery Redis connection URL

* Fix displaying error while connecting to SQLite

The exception message is always a string in Python 3, so no
need to try to decode things.

* Fix another missing exception message

* Handle JSON encoding for datasources returning bytes

SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.

This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.

* Fix Python 3 compatibility with RQ

* Revert some changes 2to3 tends to do (getredash#4261)

- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.

- Also fixed some naming issues with one character variables in list comprehensions.

- Fix Flask warning.

* Upgrade dependencies

* Remove useless `iter` added by 2to3

* Fix get_next_path tests (getredash#4280)

* Removed setting SERVER_NAME in tests setup to avoid a warning.

* Change get_next_path to not return empty string in case of a domain only value.

* Fix redirect tests:

Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.

* Remove explicit dependency for Werkzeug

* Switched pytz and certifi to unbinded versions.

* Switch to new library for getting country from IP

`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .

* Python 3 RQ modifications (getredash#4281)

* show current worker job (alongside with minor cosmetic column tweaks)

* avoid loading entire job data for queued jobs

* track general RQ queues (default, periodic and schemas)

* get all active RQ queues

* call get_celery_queues in another place

* merge dicts the Python 3 way

* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions

* Remove legacy Python flake8 tests

* Start draft for new Parameter structure

* Add the rest of the methods

* EnumParameter

* QueryBasedDropdownParameter

* DateParameter

* DateRangeParameter

* Update Parameter usage on code

* Merge dynamicValue into normalizedValue

* Add updateLocals and omit unwanted props

* Allow null NumberParameter and omit parentQueryId

* Rename parameter getValue to getExecutionValue

* Update $$value to normalizedValue + omit on save

* Add a few comments

* Remove ngModel property from Parameter

* Use value directly in DateRangeParameter

* Use simpler separator for DateRange url param

* Add backward compatibility

* Use normalizeValue null value for isEmpty

* Start creating jest tests

* Add more tests

* Normalize null value for multi mode in Enum

* Use saved value for param isEmpty

* Migrate table editor to React: skeleton, Grid tab

* Columns tab

* Cleanup

* Columns tab: DnD column sorting

* Columns types should be JSX

* New Columns tab UI/X

* Use Sortable component on Columns tab

* Tests: Grid Settings

* Tests: Columns Settings

* Tests: Editors for Text, Number, Boolean and Date/Time columns

* Tests: Editors for Image and Link columns

* Minor UI fix

* Trigger build

* Debounce inputs

* CircleCI workflow improvements

- Don't automatically build the Docker image.
- Make the Python lint step requirement for the follow up steps. When it fails it usually means there is a code error which will prevent the next steps anyway.

* Fix YAML syntax error.

* Add separate build Docker image step for master branch

…#4300)

Co-Authored-By: Arik Fraimovich <[email protected]>

…rker process (getredash#4314)

* New feature - Alert muting

* pep8 fix

* Fixed backend api update

* whoops semicolon

* Implemented mute

* Add last 14, 30, 60, and 90 days to DRP.js

Date Range Parameter (DRP)

* Add last 14, 30, 60, and 60 day params to DRP.jsx

DateRangeParameters (DRP)

* prioritize periodic jobs

* declare default queues in inside worker()

* separate send_email to its own queue

* add some logging to scheduler

* clean failed RQ job data from Redis

* move stale job purging to tasks/general.py

* provide better documentation on why we don't reject keys in FailedJobRegistry at the moment

* pleasing the CodeClimate overlords

* simplified clenaup by deleting both job data and registry entry

* use FailedJobRegistry as source of truth for purging

* remove redundant key deletion

* Update redash/settings/__init__.py

Co-Authored-By: Arik Fraimovich <[email protected]>

* Migrate settings-screen to React

* Use black instead of blue color for active item

* Revert "Use black instead of blue color for active item"

This reverts commit 0e4ecec.

* Add selectable=false to the Menu

* add some logging to scheduler

* schedule jobs only if they are not already scheduled

* jobs scheduled with an interval over 24 hours were not repeated

* schedule version_check using standard scheduling

* clean up old jobs that are not part of the definition anymore

* add some tests

* add one more test to verify that reschedules are not done when not neccesary

* no need to check for func existence - all jobs have a func to run

* handle a console warning about passing in string page options

* preserve selected tab in the location hash

…S credentials. (getredash#4344)

Without this upgrades from at least v5 (and earlier) won't work.

* Migrate Cohort to React: Editor

* Extract prepareData and getOptions to own files

* Refine CohortRenderer Angular component (js, less, prepareData) for easier migration

* Migrate Cohort to React: Renderer

* Migrate Cornelius to React: styles

* Migrate Cohort to React: Cornelius library

* Cornelius: add licence info; remove unused style

* Cornelius: use numeral to format numbers; revisit styles

* Cornelius: use moment to format date labels

* Cornelius: use chroma for cell backgrounds; update options; update proptypes; minor fixes

* Tidy up

* Tests

* Fix: correctly evaluate numeric thresholds

* Missing import

* More missing imports

* Alert evaluation: support for booleans

* Script to reencrypt data source options.

* Implement reencrypt sub command under database command.

* use  to set the hash instead of directly manipulating it

* Update Jobs.jsx

* Migrate Choropleth to React: skeleton

* Migrate Choropleth to React: Editor - skeleton

* Choropleth Editor: Bounds tab

* Choropleth Editor: Colors tab

* Choropleth Editor: Format tab

* Choropleth Editor: General tab

* Some refinements

* Migrate Choropleth to React: Renderer

* Refine code

* CR1

* Migrate Funnel visualization to React: Editor

* Migrate Funnel visualization to React: Renderer

* Replace Auto sort options with Sort Column + Reverse Order

* Add option for items limit (instead of hard-coded value)

* Add number formatting options

* Replace d3.max with lodash.maxBy; fix bug in prepareData

* Add options for min/max percent values

* Debounce inputs

* Tests

* Refine Renderer: split components, use Ant Table for rendering, fix data handling

* Extract utility function to own file

* Fix tests

* Fix: sometimes after updating options, funnel shows "ghost" rows from previous dataset

* Sort by value column by default

* Remove ui-ace.

* Remove ui-sortable.

* Remove angular-base64-upload.

* Remove angular-messages.

* Remove jquery-ui.

* Update package-lock.json.

* Hide deprecated visualizations from query editor

* Fix Map tests

…h#4389)

…etredash#4394)

These few builtins were available in Python 2.7 but not anymore
in Python 3, making the runner fail to start.

* CloudWatch Metrics query runner

* Add: query runner for CloudWatch Logs Insights

* Add logos

* Update Insights type

* Basic test connection

* Format files

It was accidentally upgraded as part of the dependencies upgrade we did recently, but 4.6.5 has a bug...

…ipt (getredash#4378)

…h Azure dependencies (getredash#4407)

* refreshing snowflake schema w/o waking cluster

Have also added a new internal method to not select a
warehouse while executing query
Using 'show columns' to fetch database schema instead of
executing a select query in information schema
show columns does not require a warehouse to run

* modularising snowflake code to avoid repetitions

fixing internal function syntax and avoiding
code repetition

* removing user object in snowflake schema query

…ration (getredash#4382)

* Add support for configuring a Username/Password for the connection to Druid

* Bump pydruid version for username/password support

* Deal with missing/empty configuration parameters

Can use rds-ca-2019 and etc
`$ curl https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem \
> redash/query_runner/files/rds-combined-ca-bundle.pem`

* Remove --max-old-space-size=4096 from build

Looks like it's no longer needed.

* Update to node v12.

* Add build:old-node-version for those who have Node < 12.

* crude unit tests for counter visualisation utils

* improve type safety with default param values for getCounterData()

* fix count rows never shows zero

* remove default values for getCounterData() params

* Remove app/service/query-string (unused) and its dependency.

* Fix usage of mixed operators.

* eslint --fix fixes for missing dependencies for react hooks

* Fix: useCallback dependency passed to $http's .catch.

* Satisfy react/no-direct-mutation-state.

* Fix no-mixed-operators violations.

* Move the decision of whether to render Custom chart one level up to make sure hooks are called in the same order.

* Fix: name was undefined. It wasn't detected before because there is such global.

* Simplify eslint config and switch to creat-react-app's eslint base.

* Add prettier config.

* Make sure eslint doesn't conflict with prettier

* A few updates post eslint (getredash#4425)

* Prettier command in package.json

* Apply black formatting

* Add auto formatting when committing to master

* Update CONTRIBUTING.md re. Black & Prettier

* Prettier all the JS files

* Add GitHub Action to autoformat code pushed to master

* Fix eslint violation due to formatting.

* Remove GitHub actions for styling

* Add restyled.io config

This is to allow for better metrics collection and tracking on the service side.
More info can be found [here](https://docs.microsoft.com/en-us/azure/kusto/api/netfx/request-properties#the-application-x-ms-app-named-property)

…4449)

* Initial React Rendering with useDashboard

* Make sure widgets refresh + useCallback

* Rename collectFilters and add refreshRate

* Fix error updates not being rendered

* Only render widget bottom when queryResults exists

* Cleanup

* Add useCallback to refreshDashboard

* Make sure Promise.all have all promises done

* Start migrating Dashoard to React
- initial rendering
- some actions
- temporary updated less file

* Fullscreen handler added

* Separate refreshRateHandler hook

* Add a few tooltips

* Separate DashboardControl and normalize btn width

* Share Button

* Fix serach params not updating

* Enumerate More Options

* Toggle Publish options

* Archive Dashboard

* Parameters + Filters

* Prepare Manage Permissions

* Start to create edit mode

* Add Edit Mode functionalities

* Use previous state when updating dashboard

* Mobile adjustments

* PermissionsEditorDialog + Dashboard page title

* Update Dashboard spec

* Fix other specs

* Break dashboard.less

* Hide publish button on mobile

* Angular Cleaning

* Keep edit state when changing resolution

* Bug fix: Dashboard Level Filters not updating

* Remove prepareWidgetsForDashboard

* Revert "Remove prepareWidgetsForDashboard"

This reverts commit b434f03.

* Avoid saving layout changes out of editing mode

* Apply policy for enabled refresh rates

* Disable loadDashboard deps

* Restyled by prettier (getredash#4459)

* Update title when dashboard name updates

Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>

* Migrate VisualizationEmbed to React

* Angular cleanup

* Remove onClick event from TimeAgo

* Check Table exists before taking snapshot

* Apply Prettier

…4481)

… as of Python 2.6 and is removed in Python 3. Use str(e) to access the user-readable message. Use e.args to access arguments passed to the exception. (getredash#4482)

Without those the values might be miscategorized in the UI.

…ash#4486)

…ash#4487)

…er (getredash#4488)