Prevent ClusterQueue with pending workloads to block borrowing in other flavors in the cohort

[alculquicondor]

What would you like to be added:

In #475 and #805, we guaranteed the following:
If a ClusterQueue has pending workloads that could fit in the nominalQuota if other workloads finish (regardless of whether preemption is enabled), we don't admit other workloads in the cohort that can fit by borrowing. This allows the pending workloads to be admitted as soon as the quota becomes available without needing to trigger extra preemptions (or waiting for more jobs to finish, if preemption is disabled).

However, the check is too coarse: if the borrowing is happening in a different flavor, we shouldn't block it.

A possible solution is to track not just which cohorts already had workloads admitted in a cycle, but also which combinations of (flavor, resource).

Current check per cohort:

kueue/pkg/scheduler/scheduler.go

Lines 145 to 160 in 913d536

	if cq.Cohort != nil {
	// Having more than one workloads from the same cohort admitted in the same scheduling cycle can lead
	// to over admission if:
	// 1. One of the workloads is borrowing, since during the nomination the usage of the other workloads
	// evaluated in the same cycle is not taken into account.
	// 2. An already admitted workload from a different cluster queue is borrowing, since all workloads
	// evaluated in the current cycle will compete for the resources that are not borrowed.
	if usedCohorts.Has(cq.Cohort.Name) && (e.assignment.Borrows() || cq.Cohort.HasBorrowingQueues()) {
	e.status = skipped
	e.inadmissibleMsg = "other workloads in the cohort were prioritized"
	continue
	}
	// Even if there was a failure, we shouldn't admit other workloads to this
	// cohort.
	usedCohorts.Insert(cq.Cohort.Name)
	}

Why is this needed:

As users start setting up complex ClusterQueues, with multiple flavors and multiple CQs per cohort, the current check doesn't match user expectations, specially if low priority pending workloads in a ClusterQueue could end up blocking other ClusterQueue.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

/assign @trasc