Give some sort of warning/error if a non-managed token production token was used in jobsub_submit

[shreyb]

This is a request originating from Fermilab ServiceNow incident INC000001158064, submitted by @hgreenlee. He writes:

If jobsub_submit could predict based on the kind of token it is given that the submission would fail, that would be good.

He requests that we have some sort of token-checking mechanism for these non-default role tokens (production, MARS, etc.), that sees if they are generated from a managed token or not. This should be possible by checking the subject and groups field.

Background

The background for this is that it appears that either HTCondor, or the CredmonOAuth plugin for scitokens doesn't know how to handle multiple tokens that map to the same condor user. When jobsub used proxy auth, the mapping from credential to user was handled external to condor, and so we had the possibility for users with the Production role to submit jobs from both their own UNIX account, as well as a shared account associated with that role for their experiment. For example, if I had role Production with experiment Hypot, and the associated shared account was called hypotpro, I could submit Production jobs both from my own username, as well as from hypotpro, that, on the grid, would map to hypotpro.

Unfortunately, this does not seem to be possible with token auth. It is entirely possible for a production user to obtain a production-role token with a subject that maps back to their username (the analog to the user-subject production proxy), but the subsequent submission fails if a managed token is already stored in the relevant schedd's credd. The error message that the user gets back is not descriptive at all (something like condor_submit failed with exit code 1).

[marcmengel]

The "condor_submit failed with exit code 1" and no other error was largely fixed in
https://github.com/fermitools/jobsub_lite/pull/479/files#diff-4247d9d2a5b7386fd2052176f8e7763eb3f0fb0164448a70e8c674c065ada182
where stderr of the condor_submit was getting swallowed.

So it would be interesting to see what if any error you get with jobsub_lite 1.5

[shreyb]

Maybe we do this as a first-order workaround:

1. Get a list from FERRY of capability sets that map to a username (cache it in /tmp)
2. If user is using non-standard role, and matches capability set, but is NOT the mapped user, we reject user

Roughly.