Make expression language compatible with Python

[andychu]

• Binary operators
  • Get rid of div, mod, xor -- these should be punctuation. I guess // is fine.
  • Restore ** for exponentiation
• Literals
  • %{name: val} should be {name: val} like JS
    • is this a problem for the tea language? It uses { } for blocks as well and will use pgen2. I don't think so but
      we'll see ... Core issue: Does tea have lexer modes? I think it needs some kind of contextual keywords and that's it? for override and init and stuff like that?
    • It's problem for lambdas within blocks ... |x| { f(x); f(y) } ? Maybe not in the Tea language?
  • blocks are ^(...) in expression mode (inconsistent with comand mode { }, but that's OK)
  • True / False / None for porting, but really true / false / null is better (only types should be capital)

Syntactic changes:

• 1, isn't allowed, it's tup(1) ?
• ++ is preferred for concatenation of strings, lists (and maybe dicts), but + is also accepted

Semantic changes

• I think we need a ?? operator that is like or, except it returns the first argument if it's non-null (None), not if len(x) == 0. This matches C semantics.
• I don't like in on lists (fixed too many accidentally quadratic bugs). It good for dicts, and maybe acceptable for strings ...

Note on enhancements:

• prettier
  • new syntax for container literal: %(bare words)
  • d->key as a shortcut for d['key']
  • Func[Int, Str => Int] instead of Func[[Int, Str], Int]
  • maybe optional sigils like push!(x) and even?(i)
  • Optional suffixes after numeric literals, like 100 MiB -- (is that just a function called suffix_MiB ?)
  • Tagged prefixes for strings. In addition to r'foo', maybe we want html'foo' or html"foo", sql'foo'
    • or we could copy JS strings which use backticks
  • multiline strings should strip leading space of closing delimiter (and probably disallow anything on first line?)

New functionality:

• => for method call, universal function call on first arg type
• interned string / symbol %mykey. Parsed but not implemented. So equality is fast.
  • note: hashing forces interning, but equality doesn't?
• x ~== y operator that does type conversion (and also for floats maybe)
• code point / character syntax
  • \u{123456} -- in C-style string literals, and stand alone!
  • #'a' and #'\n' and \n -- these evaluate to integers
• ~ !~ for regex/eggex,
• and ~~ !~~ for glob patterns
  • other option: tagged strings, like (x ~ g'*.py') vs (x ~~ '*.py')
    • sort of inspired by Perl's "taint" mode? That approximation might be useful
  • note: dot <=> ? and dot* <=> * -- is there a way to reconcile these? Or it's actually [^/]*
  • ~~ should NOT respect extended globs. Eggexes/regexes are preferred.
• maybe: overflow operators ... &+ and &* -- language semantics shouldn't depend on the compiler #824
  • note: Zig has modulus and remainder operators for unsigned; we don't have unsigned yet
• Can we have function expressions, like func(x Int) Int { var x = 3; set x = 5; return x+1 }
  • need the whole Tea gramma here

Lexer issues:

• () and [] omit newlines like Python
• {} doesn't, needs something more clever. Optional newlines are in the grammar, I think

[andychu]

Related to #477

[andychu]

And #636

[andychu]

{} vs %{} vs &{} issue:

• like Python, newlines are significant -- equivalent to ;
  • except when inside {} [] (). But then we can't use {} for blocks ... doh. So %{} still makes sense unfortunately ?

• Hacky solution: require func(x) %{ return x } or some such in expression mode ...

  • I guess if we have &{ ... } for blocks, then %{ } isn't horrible
  • but it would also mean if x == y %{ ...} hm yeah that seems unreasonable

• another solution:

  • the grammar of {} and [] could be changed to allow newlines in certain places? Replacing commas?
  • problem: function arguments, and signatures too: f(x, y)

or maybe only dicts need to change? If newlines are significant, you can still write:

x = {name: val}

# newline can substitute for trailing comma?
x = {
  name: val,    # optional
  k2: v2
  k3: v3  
}

# NOT ALLOWED

x = {
  key:                   # syntax error  due to significant newline
  value
}

What about:

data  Point { x Int, y Int }

enum Value {
  String { s Str },
}

Same rule: newline or comma? Then you can omit commas too

[andychu]

Moved to #865