Added documentation for exporting saved Metric Explorer charts with the XDMoD API #1907

aestoltm · 2024-08-22T21:30:08Z

Description

This documentation contains a Python script that uses the XDMoD API to authenticate with a local XDMoD account for the given credentials and site URL. Then the script will export the saved Metric Explorer charts into the current working directory. This documentation also describes how to edit certain parts of the script and where to find more information about the request schema for the request that exports the images.

Motivation and Context

We received a ticket asking about using the XDMoD API to export saved images for an Open XDMoD - Open OnDemand integration. This documentation will be used with the response and for future tickets that refer to issues about exporting images with the XDMoD API.

Tests performed

Tested script locally.

Checklist:

The pull request description is suitable for a Changelog entry
The milestone is set correctly on the pull request
The appropriate labels have been added to the pull request

connersaeli · 2024-08-22T22:24:43Z

docs/howto-api-image-export.md

+    if 'config' in chart:
+        chart_json = json.loads(chart['config'])
+        for attribute in chart_json:
+            if (isinstance(chart_json[attribute], dict)):


I suggest the chart_json[attribute] is stored in a variable for clarity/readability because of how often it is referenced here.

connersaeli · 2024-08-23T13:47:16Z

docs/howto-api-image-export.md

+
+username = os.getenv('XDMOD_USERNAME')
+password = os.getenv('XDMOD_PASSWORD')
+site_address = ""


Super minor, but I would suggest making this clearer that the site_address should be filled in. Either a note in the the top comment or site_address = "<XDMOD_URL_HERE>". I assume most people will copy-paste this script without reading it and then encounter an issue when this isn't populated.

ryanrath

Just need to add that the user needs to also pip install requests and it'll be good to go.

aaronweeden · 2024-08-23T17:07:08Z

docs/howto-api-image-export.md

@@ -0,0 +1,82 @@
+You can use the XDMoD API to image export your saved metric explorer charts. A local XDMoD account is **required** to authenticate through the API.


Suggested change

You can use the XDMoD API to image export your saved metric explorer charts. A local XDMoD account is **required** to authenticate through the API.

You can use the XDMoD REST API to export your saved Metric Explorer charts as images.

SSO can also be used for authenticating through the REST API. The local account is required to run the Python script, though. I moved this line into another suggestion below.

aaronweeden · 2024-08-23T17:10:34Z

docs/howto.md

@@ -8,3 +8,4 @@ The Open XDMoD HOWTOs are "how to" documents on specific subjects.
 - [Change Metric Explorer Colors](howto-colors.html)
 - [Enable Node Utilization Statistics](howto-node-utilization.html)
 - [Reconstruct Slurm Accounting Logs](howto-reconstruct-slurm.html)
+- [Export Saved Metric Explorer Charts Through the XDMOD API](howto-api-image-export.md)


Suggested change

- [Export Saved Metric Explorer Charts Through the XDMOD API](howto-api-image-export.md)

- [Export Saved Metric Explorer Charts Through the REST API](howto-api-image-export.md)

aaronweeden · 2024-09-05T17:54:26Z

docs/howto-api-image-export.md

+  'Token': auth_response['results']['token'],
+  'Authorization': auth_response['results']['token'],
+  'Content-Type': 'application/x-www-form-urlencoded'


Suggested change

'Token': auth_response['results']['token'],

'Authorization': auth_response['results']['token'],

'Content-Type': 'application/x-www-form-urlencoded'

'Token': auth_response['results']['token'],

'Authorization': auth_response['results']['token'],

'Content-Type': 'application/x-www-form-urlencoded'

For consistent indentation style.

aaronweeden · 2024-09-05T17:55:56Z

docs/howto-api-image-export.md

+            f.write(chart_response.content)
+```
+
+The default image format is `svg`, but `png` and `pdf` formats are also supported. Refer to the XDMoD [Metric Explorer Tab Controller API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the request body schema.


I would move this above the code so all the documentation is above the code.

aaronweeden · 2024-09-05T17:58:17Z

docs/howto-api-image-export.md

+        chart_json['width'] = 916
+        chart_json['height'] = 484


I would make these numbers global constants.

aaronweeden · 2024-09-05T17:58:29Z

docs/howto-api-image-export.md

+load_dotenv()
+
+username = os.getenv('XDMOD_USERNAME')
+password = os.getenv('XDMOD_PASSWORD')
+site_address = "<XDMOD_URL_HERE>"
+image_format = "svg"


Suggested change

load_dotenv()

username = os.getenv('XDMOD_USERNAME')

password = os.getenv('XDMOD_PASSWORD')

site_address = "<XDMOD_URL_HERE>"

image_format = "svg"

site_address = ""

image_format = "svg"

load_dotenv()

username = os.getenv('XDMOD_USERNAME')

password = os.getenv('XDMOD_PASSWORD')

I would suggest keeping the value of site_address blank so people don't think they're supposed to put the URL in between the angle brackets and so that they don't think they should replace the other all caps strings nearby with their usernames and passwords. I'd also suggest grouping the global constants together and moving them to the top so all the things the user would want to edit are in one place near the top.

aaronweeden · 2024-09-05T17:59:04Z

docs/howto-api-image-export.md

+The following Python script will export your saved metric explorer charts. The `dotenv` and `requests` libraries are used when authenticating through the XDMoD API. You can install these libraries through:
+
+```shell
+pip install python-dotenv
+pip install requests
+```
+
+Before running the script,
+
+1. Install the required dependencies listed above.
+1. Create a `.env` file with your local XDMoD account credentials in the same directory as the script.
+1. Update `site_address` within the script with site address associated with your XDMoD instance.
+1. Confirm the `image_format` within the script.
+
+The script will export your saved metric explorer charts to the current working directory. **\*Note:** Replace `<XDMOD_URL_HERE>` with your site address.


Suggested change

The following Python script will export your saved metric explorer charts. The `dotenv` and `requests` libraries are used when authenticating through the XDMoD API. You can install these libraries through:

```shell

pip install python-dotenv

pip install requests

```

Before running the script,

1. Install the required dependencies listed above.

1. Create a `.env` file with your local XDMoD account credentials in the same directory as the script.

1. Update `site_address` within the script with site address associated with your XDMoD instance.

1. Confirm the `image_format` within the script.

The script will export your saved metric explorer charts to the current working directory. **\*Note:** Replace `<XDMOD_URL_HERE>` with your site address.

The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.

Before running the script,

1. Install the required `python-dotenv` and `requests` dependencies.

1. Create a `.env` file in the same directory as the script that contains the following contents, replacing `<username>` with your XDMoD username and `<password>` with your XDMoD password — make sure to secure this file with read-only access.

```

XDMOD_USERNAME=<username>

XDMOD_PASSWORD=<password>

```

1. Update the value of `site_address` within the script with the URL associated with your XDMoD portal.

1. Confirm the `image_format` within the script.

I would also suggest making the path where the images are written a global constant.

docs/howto-api-image-export.md

aestoltm · 2024-09-05T19:02:22Z

@aaronweeden Updated per your feedback. You can now export either a single chart or all your saved charts to a specified path.

aaronweeden · 2024-09-05T19:05:10Z

docs/howto-api-image-export.md

@@ -0,0 +1,93 @@
+The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.


Suggested change

The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.

The following Python script can be used to export your saved Metric Explorer charts to image files using the XDMoD REST API. An XDMoD username and password are required to run the script.

aaronweeden · 2024-09-05T19:06:58Z

docs/howto-api-image-export.md

@@ -0,0 +1,93 @@
+The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.
+Before running the script,
+1. Install the required `python-dotenv` and `requests` dependencies.


Suggested change

1. Install the required `python-dotenv` and `requests` dependencies.

1. Install the required `python-dotenv` and `requests` Python dependencies (e.g., using `pip`).

aaronweeden · 2024-09-05T19:11:39Z

docs/howto-api-image-export.md

+1. Update the value of `site_address` within the script with the URL associated with your XDMoD portal.
+1. Update the value of `export_path` within the script with the desired path to export the images.
+1. Confirm the `image_format` within the script.
+
+The default image format is `svg`, but `png` and `pdf` formats are also supported. Refer to the XDMoD [Metric Explorer Tab Controller API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the request body schema.


Suggested change

1. Update the value of `site_address` within the script with the URL associated with your XDMoD portal.

1. Update the value of `export_path` within the script with the desired path to export the images.

1. Confirm the `image_format` within the script.

The default image format is `svg`, but `png` and `pdf` formats are also supported. Refer to the XDMoD [Metric Explorer Tab Controller API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the request body schema.

1. Update the value of `site_address` at the top of the script with the URL associated with your XDMoD portal.

1. Update the value of `export_path` at the top of the script with the desired directory path where the images will be written.

1. Confirm the desired values for `image_format`, `width`, and `height` at the top of the script. The default image format is `svg`, but `png` and `pdf` formats are also supported.

By default, the script will download all of your saved Metric Explorer charts. You can have it instead download a single chart by providing the `-n` or `--name` option followed by the name of the saved chart.

Refer to the XDMoD [Metric Explorer Tab Controller REST API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the REST request body schema.

aaronweeden · 2024-09-05T19:13:40Z

docs/howto-api-image-export.md

+password = os.getenv('XDMOD_PASSWORD')
+
+parser = argparse.ArgumentParser(description='Export XDMoD saved Metric Explorer charts with the REST API.')
+parser.add_argument('-n', '--name',type=str, default='', help='Specify the chart name of a saved chart to export.')


Suggested change

parser.add_argument('-n', '--name',type=str, default='', help='Specify the chart name of a saved chart to export.')

parser.add_argument('-n', '--name', help='Specify the chart name of a saved chart to export.')

The type will already be string, and None will be the default. For simplicity, I'd stick with these.

aaronweeden · 2024-09-05T19:19:57Z

docs/howto-api-image-export.md

+import json
+import urllib
+import argparse
+from dotenv import load_dotenv


Suggested change

from dotenv import load_dotenv

from dotenv import load_dotenv

import sys

aaronweeden · 2024-09-05T19:22:21Z

docs/howto-api-image-export.md

+    print('Specified chart not found.')
+    exit()


Suggested change

print('Specified chart not found.')

exit()

print('Specified chart not found.', file=sys.stderr)

sys.exit(1)

aaronweeden · 2024-09-05T19:25:03Z

docs/howto-api-image-export.md

+saved_charts = session.get(f'{site_address}/rest/v1/metrics/explorer/queries', headers=header)
+saved_charts_data = saved_charts.json()
+
+if args.name != '' and not any(chart_obj['name'] == args.name for chart_obj in saved_charts_data['data']):


Suggested change

if args.name != '' and not any(chart_obj['name'] == args.name for chart_obj in saved_charts_data['data']):

if args.name is not None and not any(chart_obj['name'] == args.name for chart_obj in saved_charts_data['data']):

aaronweeden · 2024-09-05T19:28:17Z

docs/howto-api-image-export.md

+    print('Specified chart not found.')
+    exit()
+
+for idx, chart in enumerate(saved_charts_data['data']):


Suggested change

for idx, chart in enumerate(saved_charts_data['data']):

for idx, chart in enumerate(saved_charts_data['data']):

if args.name is not None and args.name != chart['name']:

continue

and then you don't need to wrap the writing in an if/else or needlessly fetch each chart that isn't going to be written.

aaronweeden · 2024-09-05T19:42:51Z

docs/howto-api-image-export.md

+image_format = 'svg'
+image_width = 916
+image_height = 484
+


Suggested change

if site_address == '':

print('Please edit the script to specify a site_address.', file=sys.stderr)

sys.exit(1)

aaronweeden · 2024-09-05T19:48:51Z

docs/howto-api-image-export.md

+        if args.name != '' and args.name == chart['name']:
+            with open(export_path + chart_name, "wb") as f:
+                f.write(chart_response.content)
+                exit()
+        else:
+            with open(export_path + chart_name, "wb") as f:
+                f.write(chart_response.content)


Suggested change

if args.name != '' and args.name == chart['name']:

with open(export_path + chart_name, "wb") as f:

f.write(chart_response.content)

exit()

else:

with open(export_path + chart_name, "wb") as f:

f.write(chart_response.content)

with open(export_path + chart_name, "wb") as f:

f.write(chart_response.content)

See suggestion above with continue.

aestoltm added 6 commits August 22, 2024 15:05

First draft at XDMoD API image export documentation

b4f934a

Wordsmith

c1d7a8d

More wording adjustments

7c9951e

More editing

1b3ccf2

More edits

882dad4

Fix site address wording

a13948b

aestoltm added the documentation Documentation updates label Aug 22, 2024

aestoltm requested review from eiffel777, jpwhite4, ryanrath, aaronweeden and connersaeli August 22, 2024 21:30

aestoltm self-assigned this Aug 22, 2024

More updates

00b2b7c

connersaeli reviewed Aug 22, 2024

View reviewed changes

Address feedback

3cb6496

aestoltm requested a review from connersaeli August 23, 2024 13:19

connersaeli reviewed Aug 23, 2024

View reviewed changes

ryanrath requested changes Aug 23, 2024

View reviewed changes

Address feedback

38f05f2

aestoltm requested review from ryanrath and connersaeli August 23, 2024 14:16

ryanrath previously approved these changes Aug 23, 2024

View reviewed changes

aaronweeden requested changes Sep 5, 2024

View reviewed changes

Address feedback

dbfb4df

aestoltm dismissed ryanrath’s stale review via dbfb4df September 5, 2024 19:01

aestoltm requested a review from aaronweeden September 5, 2024 19:02

aaronweeden requested changes Sep 5, 2024

View reviewed changes

Address feedback ubccr#2

398000e

aestoltm added 2 commits September 5, 2024 16:53

Fix variable name

8aa2f52

Fix more variable names

7644d0f

aestoltm requested a review from aaronweeden September 6, 2024 18:12

jpwhite4 added this to the 11.0.0 milestone Sep 10, 2024

aestoltm modified the milestones: 11.0.0, 11.0.1, 11.0.2 Oct 2, 2024

aestoltm changed the base branch from main to xdmod11.0 November 22, 2024 18:59

aestoltm and others added 2 commits November 22, 2024 14:08

Default export directory should be working directory

f0b7976

Merge branch 'xdmod11.0' into rest_api_image_export

d9c4055

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added documentation for exporting saved Metric Explorer charts with the XDMoD API #1907

Added documentation for exporting saved Metric Explorer charts with the XDMoD API #1907

aestoltm commented Aug 22, 2024 •

edited

Loading

connersaeli Aug 22, 2024

connersaeli Aug 23, 2024

ryanrath left a comment

aaronweeden Aug 23, 2024

aaronweeden Aug 23, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aestoltm commented Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024

aaronweeden Sep 5, 2024 •

edited

Loading

aaronweeden Sep 5, 2024

		@@ -0,0 +1,82 @@
		You can use the XDMoD API to image export your saved metric explorer charts. A local XDMoD account is required to authenticate through the API.

	You can use the XDMoD API to image export your saved metric explorer charts. A local XDMoD account is required to authenticate through the API.
	You can use the XDMoD REST API to export your saved Metric Explorer charts as images.

	- [Export Saved Metric Explorer Charts Through the XDMOD API](howto-api-image-export.md)
	- [Export Saved Metric Explorer Charts Through the REST API](howto-api-image-export.md)

		@@ -0,0 +1,93 @@
		The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.

	The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.
	The following Python script can be used to export your saved Metric Explorer charts to image files using the XDMoD REST API. An XDMoD username and password are required to run the script.

	1. Install the required `python-dotenv` and `requests` dependencies.
	1. Install the required `python-dotenv` and `requests` Python dependencies (e.g., using `pip`).

-. Update the value of `site_address` within the script with the URL associated with your XDMoD portal.
-. Update the value of `export_path` within the script with the desired path to export the images.
-. Confirm the `image_format` within the script.
-The default image format is `svg`, but `png` and `pdf` formats are also supported. Refer to the XDMoD [Metric Explorer Tab Controller API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the request body schema.
+. Update the value of `site_address` at the top of the script with the URL associated with your XDMoD portal.
+. Update the value of `export_path` at the top of the script with the desired directory path where the images will be written.
+. Confirm the desired values for `image_format`, `width`, and `height` at the top of the script. The default image format is `svg`, but `png` and `pdf` formats are also supported.
+By default, the script will download all of your saved Metric Explorer charts. You can have it instead download a single chart by providing the `-n` or `--name` option followed by the name of the saved chart.
+Refer to the XDMoD [Metric Explorer Tab Controller REST API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the REST request body schema.

	parser.add_argument('-n', '--name',type=str, default='', help='Specify the chart name of a saved chart to export.')
	parser.add_argument('-n', '--name', help='Specify the chart name of a saved chart to export.')

	from dotenv import load_dotenv
	from dotenv import load_dotenv
	import sys

-    print('Specified chart not found.')
-    exit()
+    print('Specified chart not found.', file=sys.stderr)
+    sys.exit(1)

	if args.name != '' and not any(chart_obj['name'] == args.name for chart_obj in saved_charts_data['data']):
	if args.name is not None and not any(chart_obj['name'] == args.name for chart_obj in saved_charts_data['data']):

+if site_address == '':
+    print('Please edit the script to specify a site_address.', file=sys.stderr)
+    sys.exit(1)

Added documentation for exporting saved Metric Explorer charts with the XDMoD API #1907

Are you sure you want to change the base?

Added documentation for exporting saved Metric Explorer charts with the XDMoD API #1907

Conversation

aestoltm commented Aug 22, 2024 • edited Loading

Description

Motivation and Context

Tests performed

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanrath left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aestoltm commented Sep 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronweeden Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aestoltm commented Aug 22, 2024 •

edited

Loading

aaronweeden Sep 5, 2024 •

edited

Loading