Consider supporting character encodings other than utf-8

[drevell]

TL;DR

Currently actions like string_replace and regex_replace assume that the files they are operating on are utf-8 encoded. We could consider making this more flexible.

Detailed design

No response

Alternatives considered

No response

Additional information

No response

[sethvargo]

Devil's advocate: we could document that we only support utf-8 and explicitly check that inputs are utf8.ValidString. I don't see a use case where we'd want to support binary encodings, and I'm not sure how prevalent utf-16 and utf-32 strings will be for our use cases.

It's also possible to "force" a string to be utf-8:

v := make([]rune, 0, len(s))

for i, r := range s {
  if r == utf8.RuneError {
    if _, size := utf8.DecodeRuneInString(s[i:]); size == 1 {
      continue
    }
  }
  v = append(v, r)
}

s = string(v)