Skip to content

Latest commit

 

History

History
45 lines (32 loc) · 1.63 KB

README.md

File metadata and controls

45 lines (32 loc) · 1.63 KB

baikal-hdfs

This repository contains the Docker components necessary to deploy a local Hadoop and Node development environment. The current configuration includes 2-node HDFS and a node app.

This example shows how to use Node.js with Express to connect to HDFS via WebHDFS.

The example includes samples on how to:

  • Look up HDFS directory contents with a specified path (?path=/path/to/dir)
  • Upload a file with Express and Formidable into HDFS
  • View the contents of a file (as ascii with ?path=/path/to/file)

Instructions

Prerequisite for Docker on Windows

Due to an update in Hyper-V, some ports are now reserved and cannot be bound within a container. The following will need to be run, which will require system restarts.

At an elevated command prompt (see commands in code block below):

  1. Disable Hyper-V, restart
dism.exe /Online /Disable-Feature:Microsoft-Hyper-V
  1. Reserve ports, enable Hyper-V, restart (note, may take a few minutes for services to fully disable and allow ports to be excluded).
  • To view excluded ports: netsh int ipv4 show excludedportrange protocol=tcp
dism.exe /Online /Disable-Feature:Microsoft-Hyper-V
netsh int ipv4 add excludedportrange protocol=tcp startport=50070 numberofports=1
netsh int ipv4 add excludedportrange protocol=tcp startport=50090 numberofports=1
dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All

Startup:

  1. git clone https://github.com/ComputationalHealth/baikal-hdfs.git
  2. docker-compose up -d --build
  3. docker exec hadoop-namenode /bin/bash startup.sh
  4. Access UI from http://localhost:9090

Shutdown:

  1. docker-compose down -v