Distinguish between Preemption due to reclamation and fair sharing

[alculquicondor]

What would you like to be added:

An additional reason in the Preemption condition to indicate that the preemption was due to fair sharing and not regular reclamation.

The current reason for reclamation or fair sharing is InCohort

We could have: InCohortReclamation and InCohortFairSharing.

Why is this needed:

For better visibility

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[vladikkuzn]

/assign

[vladikkuzn]

@alculquicondor

kueue/pkg/scheduler/scheduler.go

Lines 247 to 251 in 6ecaa87

	if e.assignment.RepresentativeMode() != flavorassigner.Fit {
	if len(e.preemptionTargets) != 0 {
	// If preemptions are issued, the next attempt should try all the flavors.
	e.LastAssignment = nil
	preempted, err := s.preemptor.IssuePreemptions(ctx, &e.Info, e.preemptionTargets, cq)

If Assigment is borrowing then the reason should be InCohortFairSharing? Or should the decision be based on enableFairSharing of Preemptor?

kueue/pkg/scheduler/preemption/preemption.go

Lines 49 to 59 in 45eee46

	type Preemptor struct {
	client client.Client
	recorder record.EventRecorder
	workloadOrdering workload.Ordering
	enableFairSharing bool
	fsStrategies []fsStrategy
	// stubs
	applyPreemption func(context.Context, *kueue.Workload, string, string) error
	}

[alculquicondor]

I think the decision is based entirely on the preemptor. If the preemptor fits in the nominal quota of its ClusterQueue, then it's preempting others that are borrowing its quota.
Otherwise, it's past the nominal quota, meaning that it's using resources from the fair share.

Note that whether fair sharing is enabled is global.

/assign @vladikkuzn

[alculquicondor]

cc @gabesaba

[trasc]

I think the decision is based entirely on the preemptor. If the preemptor fits in the nominal quota of its ClusterQueue, then it's preempting others that are borrowing its quota.
Otherwise, it's past the nominal quota, meaning that it's using resources from the fair share.

Also, if the preemptor and preemptee are in the same cluster queue we can say it's a simple priority based eviction (no quota is reclaimed from a borrower)