nomad: core refactor and setup

[noahehall]

C

• complete consul ticket before continuing
  • @see https://github.com/orgs/nirv-ai/projects/6/views/1?pane=issue&itemId=17710056
• the core stack now includes vault, haproxy and consul+envoy;
  • bff, ui and postgres are now in a separate stack
  • this separation of platform (core) and web (ui, bff, db) concerns helps drive faster iteration
• we need to refactor and integrate nomad for orchestration in validation
• core goals
  • take the output from dev as input to validation
  • validation: execute services on prod like infra
  • push artifacts to nexus for downstream envs

T

• nomad review: its been awhile
  • nomad notes
  • nomad docs
• refactor existing nomad logic with intelligence gained from consul ticket
  • directory hierarchy: i think it should be IaC now instead of a nomad dir
  • nomad.sh incorporate new utils dir
  • docker env file: incorporate new .env.auto logic
• save docker images as tar files so you can use artifact + load instead of running a registry
  • push this to the nexus ticket as that will determine which route we take
• take another swing at nomad pack it should reduce the amount of inhouse stuff we have to create
  • stay away from levant, no matter how sweet it is
• integrate nomad with core
• review nomad resource utilization and update defaults (we were way off in estimates)
• update aws AMIs to include nomad binary, cni plugins, and post install files
  • see https://developer.hashicorp.com/nomad/tutorials/get-started/get-started-install
• think through the interoperability between envs and devise a more efficient management process
  • the initial nomad integration is tiresome, it shouldnt be this way
  • albeit just awhole lotta copypasta and things, this highlights a need for automation/better architecture

A

• see Build on arm64 multani/docker-nomad#55
• https://github.com/multani/docker-nomad/blob/master/docker-compose.yml
• https://github.com/codefresh-contrib/nomad-sample-app
• https://manicminer.io/posts/getting-started-with-hashicorp-nomad/
• https://developer.hashicorp.com/nomad/tutorials/nomad-pack/nomad-pack-writing-packs

---

• perm issues
  • failing to allocate job in busybox environment hashicorp/nomad#6721
  • option to manage volume permissions hashicorp/nomad#8892

---

issue 1: perm
chown: /consul/data: Operation not permitted

we switched the container workdir from /consul to /opt/consul to align with consul web docs
however if you read the consul dockerhub docs it uses /consul and not /opt/consul
solution is to follow the docker hub docs rather than dealing with nomad perm issues at this juncture
a longer term solution is to deal with nomad volume perm issues which doesnt seem as straight forward

issue 2: perm
su-exec: setgroups(994): Operation not permitted

relates to issue 1
finding the root cause of nomad perm issues will likely solve this
and truly resolve issue 1
quick fix: remove `USER consul` from image