Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frontend Status Badges #224

Open
wants to merge 175 commits into
base: master
Choose a base branch
from
Open

Conversation

JVickery-TBS
Copy link
Contributor

feat(templates): status badges;

  • Added status badges.

As promised back in February (here: #200 (comment)), this adds frontend badges stating the status of the xloadering and availability in the datastore.

Threw in the French (at least French Canadian) badges as well.

dependabot-preview bot and others added 30 commits April 28, 2021 22:14
Develop to master - merge from upstream
Develop to master - make datastore more robust
- If all types have been rejected, ensure that the fallback flag is correctly set
- replace empty strings with None if they have types that will choke on empty string
- Column that has some rows with free text and others with numeric data
- Column that has some rows with timestamp and others with empty string
- Remove unused imports, or tag those that serve a purpose (testing what can be imported)
- Remove obsolete exclusions from Flake8 config
- Show the first 100 and last 100 logs, with a message between to say how many were skipped
- Start with the first and last 50 rows, provide a link to double it
- Add a notice at the top of the page when logs have been hidden.
- Clarify that we are hiding logs, not skipping actual processing of data.
- Add 'Show All' link to show all logs if the user is confident they can handle it.
…imeout

[QOLSVC-1863] truncate on-page XLoader logs if there are too many
Develop to master - merge and fix upstream changes
- This is more efficient than setting it on each call,
and applies even to code that just reads the config without accepting an override.
snyk-bot and others added 19 commits June 22, 2024 03:23
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-URLLIB3-7267250
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-URLLIB3-7267250
…8b90d49c45dc0e5

[Snyk] Security upgrade urllib3 from 2.0.7 to 2.2.2
…6d11665f68574a7f

[Snyk] Security upgrade urllib3 from 2.0.7 to 2.2.2
…c73c00548908e25

[Snyk] Security upgrade requests from 2.31.0 to 2.32.0
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-ZIPP-7430899
…ae117c430583c679

[Snyk] Security upgrade zipp from 3.15.0 to 3.19.1
…771577aa8dcf9ec4

[Snyk] Fix for 2 vulnerabilities
- Implement experimental `IPipeValidation` implement class from ckanext-validation.
…fied/validation-support

Validation Extension Support (Minified)
Develop -> Master: feat(dev): implement IPipeValidation + updates
- Added status badges.
ckanext/xloader/helpers.py Outdated Show resolved Hide resolved
- Reverse debug badge logic.
…GovAuToCkan)

Conflicts Fixed:
	ckanext/xloader/jobs.py
	ckanext/xloader/loader.py
	ckanext/xloader/plugin.py
	ckanext/xloader/templates/package/resource_read.html
	ckanext/xloader/tests/samples/mixed_numeric_string_sample.csv
	ckanext/xloader/tests/samples/sample_with_blanks.csv
	ckanext/xloader/tests/test_loader.py
	ckanext/xloader/tests/test_plugin.py
	ckanext/xloader/utils.py
	requirements.txt

Changes to be committed:
	modified:   ckanext/xloader/config_declaration.yaml
	modified:   ckanext/xloader/jobs.py
	modified:   ckanext/xloader/loader.py
	modified:   ckanext/xloader/plugin.py
	modified:   ckanext/xloader/templates/package/resource_read.html
	modified:   ckanext/xloader/tests/samples/sample_with_blanks.csv
	new file:   ckanext/xloader/tests/samples/sample_with_extra_blank_cells.csv
	modified:   ckanext/xloader/tests/test_loader.py
	modified:   ckanext/xloader/tests/test_plugin.py
	modified:   ckanext/xloader/utils.py
	modified:   dev-requirements.txt
	modified:   requirements.txt
@ThrawnCA ThrawnCA changed the base branch from master to develop October 15, 2024 05:10
@ThrawnCA ThrawnCA changed the base branch from develop to master October 15, 2024 05:11
Copy link
Collaborator

@duttonw duttonw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readme needs to be updated with the new config options:

ckanext.xloader.show_badges : boolean (default True)
ckanext.xloader.debug_badges : boolean (default False)

@duttonw
Copy link
Collaborator

duttonw commented Oct 15, 2024

running it via qld-gov-au data.qld.gov.au site for end to end testing via qld-gov-au#108

@duttonw
Copy link
Collaborator

duttonw commented Oct 16, 2024

Hi @JVickery-TBS ,

We tried this in dev.data.qld.gov.au and we already run into a bug, a i18n bug ;)

Carl @ThrawnCA has raised a discussion point on having better 404 handling if a i18n file is not found (use default)
ckan/ckan#8487

https://dev.data.qld.gov.au/static/badges/en_AU/datastore-running.svg
but
https://dev.data.qld.gov.au/static/badges/en/datastore-running.svg
exists

My suggestion is to ditch the images and use i18n in text form only. I.e. borrow from the validations plugin.

from

<a href="/dataset/7795b370-eef5-4808-9844-812d7c6b302a/resource_data/550bcdfe-3ced-48a0-abcb-65d7e74c91ea" 
class="loader-badge">
<img src="/static/badges/en_AU/datastore-running.svg" 
alt="Loading data into DataStore" title="October 16, 2024, 13:37 (AEST)">
</a>

to

<a href="/dataset/7795b370-eef5-4808-9844-812d7c6b302a/resource_data/550bcdfe-3ced-48a0-abcb-65d7e74c91ea" 
class="validation-badge" 
title="Loading data into DataStore: October 16, 2024, 13:38 (AEST)" >
<span class="prefix">Datastore</span>
<span class="status success">Running</span>
</a>

image

What do you think?

I'll see if I can get round to this update as what you have is very nice. (if you don't beat me to it).

Regards

@duttonw

p.s. I've just built a pypi github actions deployment pipeline (still needs improvements) but its now on ckna/ckanext-xloader :) You can read a bit more about it here: #229

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants