tomcat9 startup failing

[BrzIGH]

Hi folks,

2 weeks ago I ran the playbook just fine.
Starting last week I have had 5 consecutive fails at the exact same location in the playbook run...
When running the playbook with the "true" switch I always get:

TASK [Islandora-Devops.tomcat : start tomcat9] ******************************************************************************************************************************************
Monday 10 June 2024 10:47:55 -0700 (0:00:01.371) 0:31:33.950 ***********
fatal: [default]: FAILED! => {"changed": false, "msg": "Unable to start service tomcat9: Job for tomcat9.service failed because the control process exited with error code.\nSee "systemctl status tomcat9.service" and "journalctl -xeu tomcat9.service" for details.\n"}

When checking the tomcat status thereafter I get:-----------------------------------------------------------------

× tomcat9.service - Apache Tomcat 9 Web Application Server
Loaded: loaded (/lib/systemd/system/tomcat9.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/tomcat9.service.d
└─override.conf
Active: failed (Result: exit-code) since Tue 2024-06-11 12:34:02 PDT; 3min 55s ago
Docs: https://tomcat.apache.org/tomcat-9.0-doc/index.html
Process: 31220 ExecStartPre=/usr/libexec/tomcat9/tomcat-update-policy.sh (code=exited, status=226/NAMESPACE)
CPU: 5ms

Jun 11 12:34:02 islandora3 systemd[1]: Starting Apache Tomcat 9 Web Application Server...
Jun 11 12:34:02 islandora3 systemd[31220]: tomcat9.service: Failed to set up mount namespacing: /run/systemd/unit-root/opt/blazegraph/data: No such file or directory
Jun 11 12:34:02 islandora3 systemd[31220]: tomcat9.service: Failed at step NAMESPACE spawning /usr/libexec/tomcat9/tomcat-update-policy.sh: No such file or directory
Jun 11 12:34:02 islandora3 systemd[1]: tomcat9.service: Control process exited, code=exited, status=226/NAMESPACE
Jun 11 12:34:02 islandora3 systemd[1]: tomcat9.service: Failed with result 'exit-code'.
Jun 11 12:34:02 islandora3 systemd[1]: Failed to start Apache Tomcat 9 Web Application Server.

and-----------------------------------------------------------------------------------------------------------

journalctl -xeu tomcat9.service
Jun 11 12:34:02 islandora3 systemd[1]: Starting Apache Tomcat 9 Web Application Server...
░░ Subject: A start job for unit tomcat9.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit tomcat9.service has begun execution.
░░
░░ The job identifier is 4288.
Jun 11 12:34:02 islandora3 systemd[31220]: tomcat9.service: Failed to set up mount namespacing: /run/systemd/unit-root/opt/blazegraph/data: No such file or directory
Jun 11 12:34:02 islandora3 systemd[31220]: tomcat9.service: Failed at step NAMESPACE spawning /usr/libexec/tomcat9/tomcat-update-policy.sh: No such file or directory
░░ Subject: Process /usr/libexec/tomcat9/tomcat-update-policy.sh could not be executed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The process /usr/libexec/tomcat9/tomcat-update-policy.sh could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Jun 11 12:34:02 islandora3 systemd[1]: tomcat9.service: Control process exited, code=exited, status=226/NAMESPACE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStartPre= process belonging to unit tomcat9.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 226.
Jun 11 12:34:02 islandora3 systemd[1]: tomcat9.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit tomcat9.service has entered the 'failed' state with result 'exit-code'.
Jun 11 12:34:02 islandora3 systemd[1]: Failed to start Apache Tomcat 9 Web Application Server.
░░ Subject: A start job for unit tomcat9.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit tomcat9.service has finished with a failure.
░░
░░ The job identifier is 4288 and the job result is failed.

---

I posted this on the Islandora Google group but since the issue is always replicable I though I should post here too.

Thanks

[BrzIGH]

Hi folks,

unusual that no one has responded to this...Could it be because it cannot be replicated or because it is vacation time ?

Just curious... PS. just tried again and got the same error on a vanilla jammy installation.

Thanks

[BrzIGH]

I am posting the contents of the /opt directory.

Top part shows a good working installation from about a month ago.
Bottom part shows current setup with lost of items missing. Probably explains why Tomcat cannot start and the presence of the FITS and Mirador warning messages in system status.
Hope this helps.
Thanks

[rosiel]

Hi, I was unable to replicate this on Vagrant, with Ubuntu 22.04.2 LTS.

With ansible 2.15, I was able to run the playbook successfully (with ISLANDORA_BUILD_BASE=true) and Tomcat started ("It works!" at localhost:8080).
With ansible 2.17, there are some deprecation issues with ansible.builtin.include (that I'll be fixing in a separate pull request) but that kills ansible immediately and doesn't result in the errors you got.

It sounds like something failed much earlier on in the ansible playbook, before it tried to start tomcat. Is there other "red" in the playbook that doesn't say "Fail OK"?

Also for context, is there anything that's been updated since it worked last month, such as: Ubuntu versions, ansible versions, etc.?

[BrzIGH]

Don't think I updated anything....checked my ansible version - I am running 2.10.17.

Will try upgrading that on the installation server...

I believe there always was a "Failed OK" for the working installations... I never noticed any errors during the two phases of installation..will try again and taker a closer look.

[rosiel]

If you update ansible, you will need to try this branch: #281

[BrzIGH]

Yes, it looks like my version of ansible is not compatible any more...got the message below:

### Collection community-general does not support Ansible version 2.10.17

I presume your branch will support Ansible 2.17.x ?

Thanks

[rosiel]

I've been testing it on 2.16 and 2.17, so I hope so!

[BrzIGH]

Got it going but failed here:
(Yes, I did change the requirements.yml file to state 4.3.4)

TASK [geerlingguy.mysql : Check master replication status.] *****************************************************************************************************************************
fatal: [default -> {{ mysql_replication_master }}]: FAILED! => {"msg": "Empty hostname produced from delegate_to: "{{ mysql_replication_master }}""}

PLAY RECAP ******************************************************************************************************************************************************************************
default : ok=46 changed=12 unreachable=0 failed=1 skipped=20 rescued=0 ignored=0

[BrzIGH]

So after replacing ../geerlingguy.mysql/tasks/replication.yml with the version from branch 280 and applying all the changes from branch 281 I am back to where I started: Tomcat9 cannot start..

Weird..

[rosiel]

Oh yeah, I'm sorry I should have mentioned that the change to the version of any role in roles/external won't take effect until you remove the existing role - ansible will see there's a role there and not download the new version.

Are you on Slack? It might be easier to troubleshoot in realtime (I'm Rosie Le Faive there).