Intro handson

Introduction to NGS hands-on workshop

( This document is a work in progress )

This document includes talk points and workshop tasks for the "Introduction to NGS hands-on workshop".

Scratchpad section (skip this)

TODO:

• f(x) === x.f()
• define a function
• boolean expressions
• string concatenation / +
• methods
• string indexing
• string substitution "xx $yy zz"
• array indexing

Feedback

• Motivate first, show the value
• More practical outcome
• More "Ops" oriented examples

Why NGS

• Covers the modern ops domain-specific niche.
  • bash is not modern
  • python, ruby, go are not ops domain-specific
• Typical tasks:
  • Running external programs, pipes, redirects
  • Data manipulation
  • Both in one go: run external program and parse output

Intro to NGS

Online documentation is at https://ngs-lang.org/doc/0.2.5/index.html

Installing NGS

If you want to try NGS without installing it, issue the command: docker run --rm -it ngslang/ngs:v0.2.5 . It will take you to a shell prompt from which you can run the ngs binary.

TODO: installation, verification

Running and exiting NGS

Running and exiting NGS are important points in which NGS interacts with the environment. At these points we can see that NGS aims to integrate well with the environment.

• Running
  • Switches: -e, -p, -pl, -pj, -pjl, -pt
    • ngs -e 'echo("Hello!")' - evaluate
    • ngs -p '"Hello!"' - print
    • ngs -pl '[1,"aha",2]' - print lines
    • ngs -pl '0..5' - print lines
    • ngs -pj '["a", "b", "c"] * 2' - print JSON
    • ngs -pjl '["a", "b", "c"] * 2' - print JSON lines
    • ngs -pt '[{"first_col": 10, "second_col": 20}, {"first_col": 11, "third_col": 31}]' -- print table
  • Run a script: ngs MY_SCRIPT.ngs
  • Run a script: ./MY_SCRIPT.ngs (must have first line #!/usr/bin/env ngs)
• Exiting
  • Exit codes - all objects are converted to exit codes using ExitCode method.
    • ExitCode of a process is ... exit code of the process. In the following examples, the $(...) expression runs external program. Unlike in bash, the resulting output is not used as if it was literally written instead of the $(...) expression.
      • ngs -e '$(true)'; echo $?
      • ngs -e '$(false)'; echo $?
      • ngs -e '$(bash -c "exit 10")'; echo $?
      • ngs -e '$(ok: bash -c "exit 10")'; echo $?
    • ExitCode of a processes pipeline is the the exit code of a first process that has non-zero exit code. If there is no such process, the ExitCode will be zero.
      • ngs -e '$(echo aa | ok:[0,1] grep bb | wc -l)'
    • ExitCode of an NGS object is type-specific: each type has its own rules of converting objects of that type to exit codes.
      • ngs -e true; echo $?
      • ngs -e false; echo $?
      • ngs -e 10; echo $?
      • ngs -e '"abc" ~ /^a/'; echo $?
      • ngs -e '"abc" ~ /^x/'; echo $?
  • Exceptions when running external programs. Why does ngs -e '$(bash -c "exit 10")'; echo $? fail?
    • finished_ok
  • die() - has optional parameter for the exit code, defaults to 1
    • ngs -e 'die("mymessage")'; echo $?
    • ngs -e 'die("mymessage", 10)'; echo $?
  • exit() - has optional parameter for the exit code, defaults to 1
    • ngs -e 'exit()'; echo $?
    • ngs -e 'exit(10)'; echo $?
    • ngs -e 'exit("mymessage")'; echo $?
    • ngs -e 'exit("mymessage", 10)'; echo $?

Interactive: Please try some of the above.

Syntax basics

NGS has two syntaxes: commands syntax and expressions syntax.

The reasoning behind two syntaxes: common ops tasks deserve their own syntax. Common ops tasks include running external programs, with i/o redirection and pipes.

• The two syntaxes and switching between them
  • Commands syntax (in a file)
    • This is the top level syntax in a file, similar to other shells
    • ls
    • ls >ls.txt
    • Switching from commands syntax to expressions syntax
      • Code block: { "aaa".echo() }
      • RHS of assignment: =, +=, -=, *=, /=, .=
      • Function definition: F f(x) x*10
      • Function call: echo(...) (but not func.method(...) yet)
      • Namespace declaration: ns { ... }
      • Control structures: if, while, switch, for
      • echo ${1+2}
  • Expressions syntax
    • This is the syntax in:
      • (A) command line switches
      • (B) after switch from commands syntax via one of the syntax constructs above
    • All the things that worked in commands syntax work, except for running external commands
    • Things that are exclusive to expressions syntax
      • 1 + 2
      • [1,2,3].each(echo)
    • Switching to commands syntax
      • ngs -e '$(ls)'
      • ngs -e '$(top_level:: ls)'
      • ngs -p '$(ls)'
      • ngs -pi '$(ls)' (-pi is explained in next section)
      • ngs -pi '$(ls | wc -l)'

Interactive:

git clone [email protected]:ngs-lang/ngs.wiki.git
cd ngs.wiki/Intro-handson
cat 01-syntax.ngs
./check.ngs 01-syntax.ngs

You will see that first test succeeds and the second test fails. Near the end of the file, before TEST lines, using expressions syntax, add an array which contains elements "abc" and "def". Use .each(echo) on the array to output the elements of the array. The new section that you add to the file should look similar to the few lines that appear earlier in the 01-syntax.ngs file just below # switch to "expressions" syntax ... .

Inspecting NGS objects

• -pi switch
  • ngs -pi Str
  • ngs -pi Str.constructors
  • ngs -pi ExitCode
• inspect()
  • ngs -e '["a", "b", "c" ].inspect().echo()'
    • data.method() and method(data) syntax
  • ngs -e '["a", "b", "c" ].inspect().each(echo)'
    • each() - one of the basic components of functional programming

Interactive: Please run the commands above.

A bit about data manipulation and functional programming

• What is functional programming?
  • Functions that either take another functions as parameters (examples from shell: xargs, find) or return functions
• Why functional programming is good for data manipulation?
  • Separates traversing from the operations
  • Another level of abstraction, facilitates higher-order thinking
• Function definition in NGS
  • Named: ngs -p 'F f(x) x*2; f(10)'
  • Anonymous:
    • ngs -p 'f = F(x) x*2; f(10)'
    • ngs -p '(F(x) x*2)(10)'
• Basic higher order functions: each, map, filter, reject, Pred
  • ngs -e '[1,2,3].each(echo)'
  • ngs -e '[1,2,3].map(F(x) x*2).each(echo)'
  • ngs -e '[1,2,3,4].filter(F(x) x>2).each(echo)'
  • ngs -e '[1,2,3,4].reject(F(x) x>2).each(echo)'
  • ngs -e '[1,2,"abc","def"].filter(Int).each(echo)'
  • ngs -e '[1,2,"abc","def"].filter(Str).each(echo)'

Interactive:

• Please try some of the above.
• Implement mymap function in 02-functional.ngs . It should return in array where:
  • For each element that had original value above 10 return the element multiplied by 5.
  • For other elements there will be no output elements.
  • Hint: use filter and map
• Run test: ./check.ngs 02-functional.ngs

Accessing Environment Variables

• ENV.varname (ngs -p ENV.HOME)
• ENV.get('varname')
• ENV.get('varname', 'dflt')

Interactive:

Try the following commands:

ngs -p ENV.HOME
ngs -p ENV.HOMEx
ngs -p 'ENV.get("HOME")'
ngs -p 'ENV.get("HOMEx")'
ngs -p 'ENV.get("HOMEx", "dflt")'
ngs -p ENV
ngs -pj ENV
ngs -p 'typeof(ENV)'
ngs -p 'len(ENV)'
ngs -pi ENV

Running external programs

• $(my_command arg1 arg2 ...) syntax - run and get reference
• `my_command arg1 arg2 ...` syntax - capture output
• ``my_command arg1 arg2 ...`` syntax - capture and parse output

Interactive:

ngs -p '$(curl -s yahoo.com)'
ngs -pi '$(curl -s yahoo.com)'
ngs -pi '`curl -s yahoo.com`'
ngs -pi '``curl -s yahoo.com``'
ngs -pi '``curl -s "https://api.myip.com"``'
ngs -pi '``curl -s "https://api.myip.com"``.ip'

Operators and functions

• f(arg1, arg2, ...) syntax
• arg1.f(arg2, ...) syntax
• var .= f(arg2) syntax

Interactive:

ngs -p 'limit([10,20,30,40], 2)'
ngs -p '[10,20,30,40].limit(2)'
ngs -p 'x = [10,20,30,40]; x .= limit(2); x'

Fetch HN stories

Interactive:

• Write a program to fetch HackerNews stories
  • Step 1
    • Edit the file 03-fetch.ngs
    • Purpose: fetch and parse JSON from the API URL.
    • Use ``my_command arg1 arg2 ...`` syntax to run curl to fetch the stories IDs
    • curl argument should be BASE_API_URL variable suffixed with the string /newstories.json
      • Use "abc${VARIABLE}xyz" string interpolation syntax to prepare the full URL.
    • Store the result in newstories variable
  • Step 2
    • Limit the result to 10 stories
    • Use var .= my_function(arg2) with newstories variable, limit function and argument 10.