Testing op-admin-dashboard #21

swastis10 · 2023-03-16T22:21:59Z

I have started testing op-admin-dashboard. I checked out dev branch found here.

I loaded data in the dashboard using the script found here. The tar files loaded were -

vail_2023-01-01.tar
4c_2023-01-01.tar
sc_2023-01-01.tar
fc_2023-01-01.tar

The dashboard has started to look something like the screenshot attached below.

The text was updated successfully, but these errors were encountered:

swastis10 · 2023-03-16T22:34:56Z

Testing Home Tab

Information displayed:

Boxes

Number of users: Number of users with trip:
I am able to see 87 number of users
Number of active users: Users with at least one PUT request today:
Unable to see any active users
Number of trips (new today):
Unavailable
Days since first user:
Unavailable
Confirmed trips:
Can see 62735 confirmed trips

Plot

Sign up trend: A bar chart where the x-axis shows date and the y-axis shows the number of signups.
Available
Trip trend: A bar chart where the x-axis shows date and the y-axis shows the number of trips.
Available
Participation period: A histogram of days participated which is the number of days from the first day to the date of the latest PUT request.
This is unavailable

swastis10 · 2023-03-16T22:53:49Z

Testing Data tab

Tables displayed:

Participant table:
Expected: user ID, labelling stats, first and last PUT and GET calls, trip stats.
Available: user_id and user_token

Trip table:
Expected - trip attributes, label data.
Available - trip attributes, label data

Demographic table: Unavailable

-Export Button
This is working well. On UUID, I am able to download csv which has data for all 87 users.
On trips tab, I am able to download data for 62735 users.

Errors encountered:

While trying to filter this data using the date filter provided above I encountered some errors. For example: I applied a filter from date 1 Jan to 10 Jan and it did not filter anything and gave the following errors -

Error resolution:

Started mongodb container again and the filter started working fine.

swastis10 · 2023-03-16T23:53:10Z

Testing Token tab

Download tokens as a file:
To potentially associate them with other info

I tried generating the token which resulted in creating the following error -

Traceback (most recent call last):
  File "/usr/src/app/pages/tokens.py", line 113, in generate_tokens
    insert_many_tokens(tokens)
  File "/usr/src/app/emission/storage/decorations/token_queries.py", line 19, in insert_many_tokens
    insert_many_entries(entry_list)
  File "/usr/src/app/emission/storage/decorations/token_queries.py", line 15, in insert_many_entries
    edb.get_token_db().insert_many(entry_list)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/collection.py", line 761, in insert_many
    blk.execute(write_concern, session=session)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/bulk.py", line 528, in execute
    return self.execute_command(generator, write_concern, session)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/bulk.py", line 358, in execute_command
    with client._tmp_session(session) as s:
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1829, in _tmp_session
    s = self._ensure_session(session)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1816, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
    server_session = self._get_server_session()
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1802, in _get_server_session
    return self._topology.get_server_session()
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
    None)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: db:27017: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 64135f321bca3dd42a198a92, topology_type: Single, servers: [<ServerDescription ('db', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('db:27017: [Errno -2] Name or service not known')>]>

Generate QR codes to download:
This is particularly important for programs that want to print out the QR codes and hand them out, or want to send them to participants to re-login

Because no token was generated the exported QRCodes.zip file was empty

Screenshot:

Error resolution:

Started mongodb container again and the filter started working fine.

Generated QR codes also downloaded fine

If I click on For testing checkbox, the token string generated does contain the substring _test_ in it.

swastis10 · 2023-03-17T00:22:07Z

Testing Maps Tab

We have 3 different map features which are as follows:

Density
Bubble
Trip lines

Screenshots:
Trip lines -

User_id and email tab can be used to render the trip line graph

Bubble Heatmap - User_id and email tab remain disabled

Density Heatmap - User_id and email tab remain disabled

shankari · 2023-03-18T04:20:28Z

@AlirezaRa94 FYI, in case you want to get a head start on fixing these issues

shankari · 2023-03-20T00:07:50Z

@swastis10

this error indicates that the mongodb container was not actually up, or you were unable to connect to it.

  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
    None)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: db:27017: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 64135f321bca3dd42a198a92, topology_type: Single, servers: [<ServerDescription ('db', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('db:27017: [Errno -2] Name or service not known')>]>

Can you verify that the container was running at the time you ran that test?

AlirezaRa94 · 2023-03-20T00:56:04Z

I guess other errors are related to the database connection as well. Please let me know if this is not the case.

shankari · 2023-03-20T01:11:31Z

@AlirezaRa94 it looks like the trip table was working, so not sure this is the problem on all tabs

AlirezaRa94 · 2023-03-20T04:21:33Z

It seems that at the beginning, the connection was proper, but after that, it lost its connection. When you filter data using those date filters, it tries to get data from the database again, and I think the first error is because of the lost connection.

shankari · 2023-03-21T17:50:02Z

@swastis10 I was able to run the dashboard yesterday without running into the errors that you did, but I only loaded one program. Have you been able to retry with checking whether the database was running correctly?

swastis10 · 2023-03-21T19:17:04Z

@shankari @AlirezaRa94 all the above mentioned issues got resolved once I restarted the mongodb container

swastis10 · 2023-03-21T19:18:46Z

Testing Push notification tab

Some use cases of this Push notification tab are :

When there is a new update to the app, one program had us send a notification to upgrade the app
When we get to the end of their data collection period, one program wanted us to send a notification to remind people to uninstall the app
One of the programs had us send out training videos

swastis10 · 2023-03-21T19:21:54Z

Testing settings tab

The settings tab is empty as of now

swastis10 · 2023-03-21T19:28:08Z

Test Dash basic Authentication

As soon as I open the dashboard on Chrome, it prompts me to put username and password.

Once, I enter the username and password, the dashboard loads fine.

swastis10 · 2023-03-21T20:37:21Z

Testing with AWS Cognito

I have changed docker-compose-dash-app.yml to AUTH_TYPE: "cognito" and set config.py using my NREL credentials and I can see the following Cognito login on my screen.

I am prompted to create an account with cognito:

I enter my email id and password and it sends out a verification code to my email:

I also set up an authenticator email:

Once done, I am also to see the openpath-stage dashboard:

I can also see my details in the user-pool:

shankari · 2023-03-21T21:52:12Z

@swastis10 @AlirezaRa94
All this looks good except for

Once done, I am also to see the openpath-stage dashboard:

Once done, you should see the admin dashboard, not the public dashboard.

swastis10 · 2023-03-21T23:41:55Z

@swastis10 @AlirezaRa94 All this looks good except for

Once done, I am also to see the openpath-stage dashboard:

Once done, you should see the admin dashboard, not the public dashboard.

@shankari In cognito, when I go to our NREL user pool, I see public dashboard URL as the hosted UI (See attached screenshot). We either need to change the callback to localhost link or need to host admin dashboard on Stage so that we can view that UI once we sign in through cognito.

shankari · 2023-03-21T23:46:18Z

@AlirezaRa94 how did you configure this for your testing?
@swastis10 once you work with cloud services to get it to work in the staging environment, we can change the URL to the staging admin URL

AlirezaRa94 · 2023-03-22T00:00:42Z

@shankari @swastis10
I set the callback URL (redirect URL) to the localhost (http://localhost:8050), both in the user pool and the config.py file.

swastis10 · 2023-03-22T02:37:10Z

After setting callback URL to http://localhost:8050, I was able to load admin dashboard after Cognito login.

There was another error I encountered:

Logs:
Traceback (most recent call last):
File "/usr/src/app/app_sidebar_collapsible.py", line 212, in display_page
raise e
File "/usr/src/app/app_sidebar_collapsible.py", line 205, in display_page
is_authenticated = authenticate_user(search)
File "/usr/src/app/utils/cognito_utils.py", line 82, in authenticate_user
user_data = get_tokens(query_params['code'])
File "/usr/src/app/utils/cognito_utils.py", line 32, in get_tokens
response = requests.post(token_endpoint, data=body, headers=headers)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/sessions.py", line 573, in request
prep = self.prepare_request(req)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/sessions.py", line 496, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/models.py", line 368, in prepare
self.prepare_url(url, params)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/models.py", line 440, in prepare_url
f"Invalid URL {url!r}: No scheme supplied. "
requests.exceptions.MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant https://?

Another error:
Everytime I clicked on the sidebar to go to another tab e.g home, maps, data - It moved me to cognito login screen everytime

AlirezaRa94 · 2023-03-22T04:38:31Z

@swastis10
See if the TOKEN_ENDPOINT in the config.py file is set correctly. You can also find more information about them in the updated version of the Readme file.

shankari · 2023-03-22T16:03:38Z

@AlirezaRa94 it looks like I can't regenerate new tokens?
I first generated tokens with program = nrelop_smart_commute_ebike_. It generated 50 tokens with the nrelop_nrelop_smart_commute_ebike prefix and displayed them.

I then tried to regenerate with program = smart_commute_ebike but I don't see them displayed.
I then tried to regenerated with program = testme for double checking, and I still don't see them displayed

Screen.Recording.2023-03-22.at.9.01.41.AM.mov

Note that when I export QR codes, I do see them, but they are not visible in the UI.

$ ls -al nrelop_smart_commute_ebike_* | wc -l
     102

There is also no method to remove tokens (in case the admin wanted to change the format, etc).

swastis10 · 2023-03-22T17:57:15Z

@swastis10 See if the TOKEN_ENDPOINT in the config.py file is set correctly. You can also find more information about them in the updated version of the Readme file.

@AlirezaRa94 I think it is correct. Here are is my config.py, let me know if I need to make some other changes:
class CognitoConfig:
CLIENT_ID = {clientID}
CLIENT_SECRET = '' // this is blank
REDIRECT_URL = 'http://localhost:8050'
TOKEN_ENDPOINT = '{Cognito Domain Name}/oauth2/token'
USER_POOL_ID = {User pool ID}
REGION = 'us-west-2'
AUTH_URL = '{Cognito Domain Name}/login?response_type=code&client_id={clientID}&redirect_uri=http://localhost:8050'

In Cognito - Allowed callback URLs is set to http://localhost:8050

AlirezaRa94 · 2023-03-23T01:31:04Z

@shankari
The data table in the UI only displays 50 tokens per page and has pagination.

I couldn't find the method to remove tokens from the database (emission/storage/decorations/token_queries.py). Could you point me to this method?

shankari · 2023-03-23T15:47:01Z

@AlirezaRa94 I see the pagination now, thanks.
you are also right that there does not appear to be a method to delete tokens
when this code was originally written by @aGuttman, the assumption was that people would create the token list offline and then load it into the database when they were happy with it.

I can think of two possible approaches:

Go back to that model - allow people to create token lists at will but only save to the database when they have "finalized" it
Allow people to delete tokens that they are unhappy with; ideally this would be "bulk delete" so that if people created 50 tokens, they would not have to go through and select 50 entries to delete. This would indeed require a new method in the token_list decorations.

@AlirezaRa94 which is easier for you to implement? I have a mild preference for (1) because it mirrors the workflow for which we designed this, and seems to be easier to implement

We can always tweak later based on admin feedback.

AlirezaRa94 · 2023-03-24T06:36:14Z

@shankari
I prefer the first approach as well.

Another point to mention about the token page is that it displays all the tokens available in the database, including the new ones. It also exports all of them, which I believe is not the intended behavior.
We could either show only the new tokens or if we need to include the existing ones, we should separate them.

shankari · 2023-03-24T17:14:13Z

@AlirezaRa94 I think it would be good to show both but separate them
that way, if people forget to hit the "save" button, they are not like "why aren't the new tokens working"? They can see the loaded tokens and understand why. You might want to make the existing tokens page display in reverse cron, though so the newly added tokens will show up first instead of making people paginate all the way to the end.

shankari · 2023-03-25T03:21:46Z

@AlirezaRa94 production dockerfile does not work; container continually fails start with

Connecting to database URL db
emission/tests/data/real_examples/shankari_2015-07-22
Loading file emission/tests/data/real_examples/shankari_2015-07-22
After registration, test_july_22 -> 28828dff-a670-4c49-878e-96d9f6d95473
Finished loading 0 entries into the usercache and 1906 entries into the timeseries
Traceback (most recent call last):
  File "app_sidebar_collapsible.py", line 20, in <module>
    from utils.db_utils import query_uuids, query_confirmed_trips
ModuleNotFoundError: No module named 'utils'

That's why I ask you to indicate the testing done:
#7 (comment)

AlirezaRa94 · 2023-03-25T05:36:34Z

@shankari
I haven't yet tested the production docker-compose, and it is not ready for your testing. I will let you know when it's done.

swastis10 · 2023-04-05T21:53:05Z

@AlirezaRa94 , I have created a test App client on NREL AWS console with the following settings:

In Config.py, my configs are as follows:

class CognitoConfig:
    CLIENT_ID = '5e5c3qdvq6so59alca304f9tv3'
    CLIENT_SECRET = '8htj7mvrnuf34ik6khi7a74qsdvgj11uaja7pmkb4a8oheb1olt'

    REDIRECT_URL = 'http://localhost:8050'
    TOKEN_ENDPOINT = 'https://openpath-stage.auth.us-west-2.amazoncognito.com/oauth2/token'

    USER_POOL_ID = 'us-west-2_O184jYo10'
    REGION = 'us-west-2'
    AUTH_URL = 'https://openpath-stage.auth.us-west-2.amazoncognito.com/login?response_type=code&client_id=5e5c3qdvq6so59alca304f9tv3&redirect_uri=http://localhost:8050'

Even then, I am getting the following error.

shankari · 2023-04-05T22:33:21Z

@swastis10 are there any error logs? For that matter, where do you see logs in a dash app?

AlirezaRa94 · 2023-04-06T01:31:16Z

@swastis10 I can sign in without any problems with the config that you have provided. Can you check the config.py file inside the docker container and see if it is the same?
Is config.py at the root directory of your project?

shankari · 2023-04-17T15:25:38Z

Closing this for now since portmanteau issues are hard to debug and resolve

shankari closed this as completed Apr 17, 2023

shankari mentioned this issue Apr 17, 2023

Finalize production docker build #32

Open

shankari mentioned this issue Jul 11, 2023

QR codes don't show up on the token screen #47

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing op-admin-dashboard #21

Testing op-admin-dashboard #21

swastis10 commented Mar 16, 2023 •

edited

Loading

swastis10 commented Mar 16, 2023

swastis10 commented Mar 16, 2023 •

edited

Loading

swastis10 commented Mar 16, 2023 •

edited

Loading

swastis10 commented Mar 17, 2023 •

edited

Loading

shankari commented Mar 18, 2023 •

edited

Loading

shankari commented Mar 20, 2023

AlirezaRa94 commented Mar 20, 2023

shankari commented Mar 20, 2023

AlirezaRa94 commented Mar 20, 2023

shankari commented Mar 21, 2023

swastis10 commented Mar 21, 2023

swastis10 commented Mar 21, 2023

swastis10 commented Mar 21, 2023

swastis10 commented Mar 21, 2023

swastis10 commented Mar 21, 2023 •

edited

Loading

shankari commented Mar 21, 2023

swastis10 commented Mar 21, 2023

shankari commented Mar 21, 2023 •

edited

Loading

AlirezaRa94 commented Mar 22, 2023

swastis10 commented Mar 22, 2023 •

edited

Loading

AlirezaRa94 commented Mar 22, 2023

shankari commented Mar 22, 2023

swastis10 commented Mar 22, 2023 •

edited

Loading

AlirezaRa94 commented Mar 23, 2023

shankari commented Mar 23, 2023

AlirezaRa94 commented Mar 24, 2023 •

edited

Loading

shankari commented Mar 24, 2023

shankari commented Mar 25, 2023 •

edited

Loading

AlirezaRa94 commented Mar 25, 2023 •

edited

Loading

swastis10 commented Apr 5, 2023

shankari commented Apr 5, 2023

AlirezaRa94 commented Apr 6, 2023 •

edited

Loading

shankari commented Apr 17, 2023

Testing op-admin-dashboard #21

Testing op-admin-dashboard #21

Comments

swastis10 commented Mar 16, 2023 • edited Loading

swastis10 commented Mar 16, 2023

Testing Home Tab

swastis10 commented Mar 16, 2023 • edited Loading

Testing Data tab

Errors encountered:

Error resolution:

swastis10 commented Mar 16, 2023 • edited Loading

Testing Token tab

Error resolution:

swastis10 commented Mar 17, 2023 • edited Loading

Testing Maps Tab

shankari commented Mar 18, 2023 • edited Loading

shankari commented Mar 20, 2023

AlirezaRa94 commented Mar 20, 2023

shankari commented Mar 20, 2023

AlirezaRa94 commented Mar 20, 2023

shankari commented Mar 21, 2023

swastis10 commented Mar 21, 2023

swastis10 commented Mar 21, 2023

Testing Push notification tab

swastis10 commented Mar 21, 2023

Testing settings tab

swastis10 commented Mar 21, 2023

Test Dash basic Authentication

swastis10 commented Mar 21, 2023 • edited Loading

Testing with AWS Cognito

shankari commented Mar 21, 2023

swastis10 commented Mar 21, 2023

shankari commented Mar 21, 2023 • edited Loading

AlirezaRa94 commented Mar 22, 2023

swastis10 commented Mar 22, 2023 • edited Loading

AlirezaRa94 commented Mar 22, 2023

shankari commented Mar 22, 2023

swastis10 commented Mar 22, 2023 • edited Loading

AlirezaRa94 commented Mar 23, 2023

shankari commented Mar 23, 2023

AlirezaRa94 commented Mar 24, 2023 • edited Loading

shankari commented Mar 24, 2023

shankari commented Mar 25, 2023 • edited Loading

AlirezaRa94 commented Mar 25, 2023 • edited Loading

swastis10 commented Apr 5, 2023

shankari commented Apr 5, 2023

AlirezaRa94 commented Apr 6, 2023 • edited Loading

shankari commented Apr 17, 2023

swastis10 commented Mar 16, 2023 •

edited

Loading

swastis10 commented Mar 16, 2023 •

edited

Loading

swastis10 commented Mar 16, 2023 •

edited

Loading

swastis10 commented Mar 17, 2023 •

edited

Loading

shankari commented Mar 18, 2023 •

edited

Loading

swastis10 commented Mar 21, 2023 •

edited

Loading

shankari commented Mar 21, 2023 •

edited

Loading

swastis10 commented Mar 22, 2023 •

edited

Loading

swastis10 commented Mar 22, 2023 •

edited

Loading

AlirezaRa94 commented Mar 24, 2023 •

edited

Loading

shankari commented Mar 25, 2023 •

edited

Loading

AlirezaRa94 commented Mar 25, 2023 •

edited

Loading

AlirezaRa94 commented Apr 6, 2023 •

edited

Loading