Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added documentation for exporting saved Metric Explorer charts with the XDMoD API #1907

Open
wants to merge 15 commits into
base: xdmod11.0
Choose a base branch
from
82 changes: 82 additions & 0 deletions docs/howto-api-image-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
You can use the XDMoD API to image export your saved metric explorer charts. A local XDMoD account is **required** to authenticate through the API.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can use the XDMoD API to image export your saved metric explorer charts. A local XDMoD account is **required** to authenticate through the API.
You can use the XDMoD REST API to export your saved Metric Explorer charts as images.

SSO can also be used for authenticating through the REST API. The local account is required to run the Python script, though. I moved this line into another suggestion below.


The following Python script will export your saved metric explorer charts. The `dotenv` and `requests` libraries are used when authenticating through the XDMoD API. You can install these libraries through:

```shell
pip install python-dotenv
pip install requests
```

Before running the script,

1. Install the required dependencies listed above.
1. Create a `.env` file with your local XDMoD account credentials in the same directory as the script.
1. Update `site_address` within the script with site address associated with your XDMoD instance.
1. Confirm the `image_format` within the script.

The script will export your saved metric explorer charts to the current working directory. **\*Note:** Replace `<XDMOD_URL_HERE>` with your site address.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following Python script will export your saved metric explorer charts. The `dotenv` and `requests` libraries are used when authenticating through the XDMoD API. You can install these libraries through:
```shell
pip install python-dotenv
pip install requests
```
Before running the script,
1. Install the required dependencies listed above.
1. Create a `.env` file with your local XDMoD account credentials in the same directory as the script.
1. Update `site_address` within the script with site address associated with your XDMoD instance.
1. Confirm the `image_format` within the script.
The script will export your saved metric explorer charts to the current working directory. **\*Note:** Replace `<XDMOD_URL_HERE>` with your site address.
The following Python script will export your saved Metric Explorer charts. An XDMoD username and password is required to run the script.
Before running the script,
1. Install the required `python-dotenv` and `requests` dependencies.
1. Create a `.env` file in the same directory as the script that contains the following contents, replacing `<username>` with your XDMoD username and `<password>` with your XDMoD password — make sure to secure this file with read-only access.
```
XDMOD_USERNAME=<username>
XDMOD_PASSWORD=<password>
```
1. Update the value of `site_address` within the script with the URL associated with your XDMoD portal.
1. Confirm the `image_format` within the script.

I would also suggest making the path where the images are written a global constant.


```python
#!/usr/bin/env python3
import os
import requests
import json
import urllib
from dotenv import load_dotenv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from dotenv import load_dotenv
from dotenv import load_dotenv
import sys


load_dotenv()

username = os.getenv('XDMOD_USERNAME')
password = os.getenv('XDMOD_PASSWORD')
site_address = "<XDMOD_URL_HERE>"
image_format = "svg"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
load_dotenv()
username = os.getenv('XDMOD_USERNAME')
password = os.getenv('XDMOD_PASSWORD')
site_address = "<XDMOD_URL_HERE>"
image_format = "svg"
site_address = ""
image_format = "svg"
load_dotenv()
username = os.getenv('XDMOD_USERNAME')
password = os.getenv('XDMOD_PASSWORD')

I would suggest keeping the value of site_address blank so people don't think they're supposed to put the URL in between the angle brackets and so that they don't think they should replace the other all caps strings nearby with their usernames and passwords. I'd also suggest grouping the global constants together and moving them to the top so all the things the user would want to edit are in one place near the top.


session = requests.Session()

auth_response = session.post(f'{site_address}/rest/auth/login', auth=(username, password))

if auth_response.status_code != 200:
print('Authentication failed. Check provided credentials and check if you have a local XDMoD account')
quit()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
print('Authentication failed. Check provided credentials and check if you have a local XDMoD account')
quit()
print('Authentication failed. Check provided credentials.', file=sys.stderr)
sys.exit(1)


auth_response = auth_response.json()

header = {
'Token': auth_response['results']['token'],
'Authorization': auth_response['results']['token'],
'Content-Type': 'application/x-www-form-urlencoded'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'Token': auth_response['results']['token'],
'Authorization': auth_response['results']['token'],
'Content-Type': 'application/x-www-form-urlencoded'
'Token': auth_response['results']['token'],
'Authorization': auth_response['results']['token'],
'Content-Type': 'application/x-www-form-urlencoded'

For consistent indentation style.

}

saved_charts = session.get(f'{site_address}/rest/v1/metrics/explorer/queries', headers=header, cookies=session.cookies)
saved_charts_data = saved_charts.json()

for idx, chart in enumerate(saved_charts_data['data']):
aaronweeden marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for idx, chart in enumerate(saved_charts_data['data']):
for idx, chart in enumerate(saved_charts_data['data']):
if args.name is not None and args.name != chart['name']:
continue

and then you don't need to wrap the writing in an if/else or needlessly fetch each chart that isn't going to be written.

if 'config' in chart:
chart_json = json.loads(chart['config'])
for attribute in chart_json:
chart_parameter = chart_json[attribute]
if (isinstance(chart_parameter, dict)):
if 'data' in attribute:
encoded_str = urllib.parse.quote_plus(str(chart_parameter['data']))
else:
encoded_str = urllib.parse.quote_plus(str(chart_parameter))
encoded_str = encoded_str.replace('%27','%22').replace('False', 'false').replace('True', 'true').replace('None', 'null')
chart_json[attribute] = encoded_str
if chart_parameter in (True, False, None):
chart_json[attribute] = str(chart_parameter).replace('False', 'false').replace('True', 'true').replace('None', 'null')

chart_json['operation'] = "get_data"
chart_json['controller_module'] = "metric_explorer"
chart_json['show_title'] = "y"
chart_json['format'] = image_format
chart_json['width'] = 916
chart_json['height'] = 484
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make these numbers global constants.


chart_response = session.post(f'{site_address}/controllers/metric_explorer.php', data=chart_json, headers=header, cookies=session.cookies)
chart_name = f"{chart['name']}.{image_format}" if ('name' in chart) else f"xdmod_API_export_{idx}.{image_format}"

with open(chart_name, "wb") as f:
f.write(chart_response.content)
```

The default image format is `svg`, but `png` and `pdf` formats are also supported. Refer to the XDMoD [Metric Explorer Tab Controller API](rest.html#tag/Metric-Explorer/paths/~1controllers~1metric_explorer.php/post) `get_data` operation for more information on the request body schema.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this above the code so all the documentation is above the code.

1 change: 1 addition & 0 deletions docs/howto.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ The Open XDMoD HOWTOs are "how to" documents on specific subjects.
- [Change Metric Explorer Colors](howto-colors.html)
- [Enable Node Utilization Statistics](howto-node-utilization.html)
- [Reconstruct Slurm Accounting Logs](howto-reconstruct-slurm.html)
- [Export Saved Metric Explorer Charts Through the XDMOD API](howto-api-image-export.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [Export Saved Metric Explorer Charts Through the XDMOD API](howto-api-image-export.md)
- [Export Saved Metric Explorer Charts Through the REST API](howto-api-image-export.md)