Error occurred when launching bundle server: ensureAlive failed

[SongWithoutWords]

I'm having trouble deploying a Yesod app with Keter on a Ubuntu 18.04 VM hosted by Google Cloud Platform, using the sqlite scaffold as a jumping off point. I compiled both the Yesod app and Keter on my own machine, running the same version of Ubuntu.

When I visit the IP of my site, I am greeted by:

Welcome to Keter
The hostname you have provided, 104.198.13.16, is not recognized.

It seems that the Yesod App has timed out after a minute, based on the contents of /opt/keter/log/keter/current.log:

2018-10-22 08:06:55.34: Unpacking bundle '/opt/keter/etc/../incoming/server.keter'
2018-10-22 08:06:55.66: Sanity checks passed
2018-10-22 08:06:55.66: Reserving hosts for app server: 104.198.13.16
2018-10-22 08:06:55.66: Process created: /opt/keter/temp/server-0/dist/bin/server
2018-10-22 08:08:26.66: Forgetting host reservations for app server: 104.198.13.16
2018-10-22 08:08:26.67: Error occured when launching bundle server: ensureAlive failed
CallStack (from HasCallStack):
  error, called at ./Keter/App.hs:331:14 in keter-1.4.3.2-4FXhcevLhnt270CSYCVtlD:Keter.App

The contents of `/opt/keter/log/app-server/current.log are as follows (it doesn't seem like the app is crashing):

Migrating: CREATE TABLE "user"("id" INTEGER PRIMARY KEY,"ident" VARCHAR NOT NULL,"password" VARCHAR NULL,CONSTRAINT "unique_user" UNIQUE ("ident"))
Migrating: CREATE TABLE "email"("id" INTEGER PRIMARY KEY,"email" VARCHAR NOT NULL,"user_id" INTEGER NULL REFERENCES "user","verkey" VARCHAR NULL,CONSTRAINT "unique_email" UNIQUE ("email"))
Migrating: CREATE TABLE "comment"("id" INTEGER PRIMARY KEY,"message" VARCHAR NOT NULL,"user_id" INTEGER NULL REFERENCES "user")
Migrating: CREATE TABLE "x_user"("id" INTEGER PRIMARY KEY,"email" VARCHAR NOT NULL,"password_hash" VARCHAR NOT NULL,"salt" VARCHAR NOT NULL,"data" VARCHAR NOT NULL,CONSTRAINT "user_email" UNIQUE ("email"))
Migrating: CREATE TABLE "doctor"("id" INTEGER PRIMARY KEY,"first_name" VARCHAR NOT NULL,"last_name" VARCHAR NOT NULL)
Migrating: CREATE TABLE "patient"("id" INTEGER PRIMARY KEY,"first_name" VARCHAR NOT NULL,"last_name" VARCHAR NOT NULL,"date_of_birth" DATE NULL)
Migrating: CREATE TABLE "doctor_patient_request"("id" INTEGER PRIMARY KEY,"doctor" INTEGER NOT NULL REFERENCES "doctor","patient" INTEGER NOT NULL REFERENCES "patient",CONSTRAINT "unique_request" UNIQUE ("patient","doctor"))
Migrating: CREATE TABLE "doctor_patient_relation"("id" INTEGER PRIMARY KEY,"doctor" INTEGER NOT NULL REFERENCES "doctor","patient" INTEGER NOT NULL REFERENCES "patient",CONSTRAINT "unique_relation" UNIQUE ("patient","doctor"))
Migrating: CREATE TABLE "prescription"("id" INTEGER PRIMARY KEY,"medication" VARCHAR NOT NULL)

The contents of the /opt/keter directory are :

./bin:
keter*
./etc:
keter-config.yaml  key.pem
./incoming:
server.keter

The contents of opt/keter/keter-config.yaml are (I've tried both with and without the port):

root: ..
listeners:
    - host: "*4" # Listen on all IPv4 hosts
      port: 80 # Could be used to modify port
setuid: root

The contents of config/keter.yml are:

user-edited: true
stanzas:
  - type: webapp
    exec: ../dist/bin/server
    args: []
    hosts:
      - 104.198.13.16
    requires-secure: false
  - type: static-files
    hosts:
      - 104.198.13.16
    root: ../static

Does anyone have any ideas?

[SongWithoutWords]

@snoyberg, @creichert, I'm fairly stumped but will try deploying an unaltered scaffold, and try deploying with a domain name (rather than a static IP) later today. Do either of you know what the issue might be?

[creichert]

I looked at this for a little bit, and I'm not 100% sure, sorry. Maybe @tolysz has an idea. This part of Keter has been quite stable for some time and your app looks like it's loaded correctly. I would try a few other options as you mentioned to sanity check a real domain name. Please report back.

[creichert]

Ah, did you check into the the app possible crashing after the migrations? Are you running the migrations using Keter or through your app?

[SongWithoutWords]

Thanks for following up @creichert - my Yesod app is using SQLite and does the migrations itself without Keter when running locally. The "migrations" above are just it creating the tables in the database for the first time. I don't have any reason to believe that it is crashing based on the output above, and it doesn't crash when run locally with Stack. Do you know of any way to find out for sure?

[SongWithoutWords]

So I got a domain name, changed keter.yml accordingly, redeployed, and the results were the same.

Next I'll try deploying the Yesod SQLite scaffold with this domain name and see if that helps.

[creichert]

I'm thinking... the domain name / "not found" might not be the core issue. Are you dynamically reading the `PORT` environment variable in your app? Can you add some logging for your application immediately after the migration run to sanity check what point it's getting to? This doesn't happen often, so I'm still leaning towards the app not communicating with Keter in some fashion. Even if your app isn't failing locally, it would help to have an idea of what it does after the migration.

[SongWithoutWords]

@creichert, in order to get away from the specifics of my app, I tried deploying the Yesod SQLite Scaffold, with similar results.

It seems that after two minutes the application gets killed by Warp's timeout reaper, but I don't know why that would be.

● keter.service - Keter
   Loaded: loaded (/etc/systemd/system/keter.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-10-23 00:22:06 UTC; 1min 29s ago
 Main PID: 28561 (keter)
    Tasks: 14 (limit: 4401)
   CGroup: /system.slice/keter.service
           ├─28561 /opt/keter/bin/keter /opt/keter/etc/keter-config.yaml
           ├─28576 /opt/keter/bin/keter /opt/keter/etc/keter-config.yaml
           ├─28579 sudo -E -u root -- /opt/keter/temp/yesod-sqlite-0/dist/bin/yesod-sqlite
           └─28583 /opt/keter/temp/yesod-sqlite-0/dist/bin/yesod-sqlite
Oct 23 00:22:06 instance systemd[1]: Started Keter.
Oct 23 00:22:06 instance sudo[28579]:     root : TTY=unknown ; PWD=/opt/keter/temp/yesod-sqlite-0 ; USER=root ; COMMAND=/o
Oct 23 00:22:06 instance sudo[28579]: pam_unix(sudo:session): session opened for user root by (uid=0)

status keter
● keter.service - Keter
   Loaded: loaded (/etc/systemd/system/keter.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-10-23 00:22:06 UTC; 2min 11s ago
 Main PID: 28561 (keter)
    Tasks: 7 (limit: 4401)
   CGroup: /system.slice/keter.service
           ├─28561 /opt/keter/bin/keter /opt/keter/etc/keter-config.yaml
           └─28576 /opt/keter/bin/keter /opt/keter/etc/keter-config.yaml

Oct 23 00:22:06 instance systemd[1]: Started Keter.
Oct 23 00:22:06 instance sudo[28579]:     root : TTY=unknown ; PWD=/opt/keter/temp/yesod-sqlite-0 ; USER=root ; COMMAND=/o
Oct 23 00:22:06 instance sudo[28579]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 23 00:23:36 instance sudo[28579]: pam_unix(sudo:session): session closed for user root
Oct 23 00:24:14 instance keter[28561]: keter: Thread killed by Warp's timeout reaper
Oct 23 00:24:14 instance keter[28561]: keter: Thread killed by Warp's timeout reaper

[SongWithoutWords]

The log files with the Yesod SQLite scaffold are the same as before, but with fewer tables created by the application.

[creichert]

Does your app read and use the `PORT` environment variable required by Keter? The yesod-scaffold appears to only read YESOD_PORT env var. Sorry you're having trouble getting everything setup. If we can figure out what the problem is, I'm sure we can find a way to ensure this is documented/fixed for future users.

[SongWithoutWords]

@creichert my app didn't make any changes in this regard: it uses the YESOD_PORT same as the scaffold

[SongWithoutWords]

I am going to try deploying after changing this to PORT as you've suggested

[SongWithoutWords]

@creichert, thank you so very much, it works!

[SongWithoutWords]

I think it might be worth changing the Yesod scaffold to work with Keter out of the box.

[creichert]

I agree and the Yesod scaffold used to do this by default. It was likely overlooked as the PORT env var is something that is frequently added by default without further thought. If you have time, it would be great if you could review the Keter README, and ensure your problem is at least somewhat mentioned. I'd definitely be happy to receive documentation pull requests. I also encourage you to open an issue with the Yesod Scaffold repo. I think the YESOD_PORT var is used by the yesod cli tools, so I wouldn't immediately create a PR.

[schoettl]

Thank you @SongWithoutWords for tracking down this problem! I overlooked your change in the README because I read the Bundle section and then only used yesod keter thinking this simply must work.