Skip to content

Latest commit

 

History

History
42 lines (29 loc) · 1.13 KB

README.md

File metadata and controls

42 lines (29 loc) · 1.13 KB

Spark UI

You can use this Docker image to start the Apache Spark History Server (SHS) and view the Spark UI locally.

Pre-requisite

  • Install Docker

Build Docker image

You can either build this Docker image yourself, or use the public image here: ghcr.io/aws-samples/emr-serverless-spark-ui:latest

  1. Download the Dockerfile in the spark-ui directory from the GitHub repository.
  2. Login to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 755674844232.dkr.ecr.us-east-1.amazonaws.com
  1. Build the image
cd $SOURCE_ROOT/utilities/spark-ui
docker build -t emr/spark-ui .

Start the Spark History Server

You can use a pair of AWS access key and secret key, or temporary AWS credentials.

  1. Set your AWS access key and secret key, and optionally session token.
export AWS_ACCESS_KEY_ID="ASIAxxxxxxxxxxxx"
export AWS_SECRET_ACCESS_KEY="yyyyyyyyyyyyyyy"
export AWS_SESSION_TOKEN="zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
  1. Run start-ui.sh
./start-ui.sh <S3_BUCKET> <ApplicationID> <jobId>
  1. Access the Spark UI via http://localhost:18080