Create a version 4.0 format which doesn't include the expectations section

[evansd]

The expectations configuration e.g.

expectations:
  population_size: 1000

is no longer used by ehrQL, and therefore we should get rid of it.

However it is currently required by version 3.0 of the project.yaml format:

pipeline/pipeline/features.py

Lines 6 to 10 in 8c706d6

	# The version of `project.yaml` where each feature was introduced
	FEATURE_FLAGS_BY_VERSION = {
	"UNIQUE_OUTPUT_PATH": 2,
	"EXPECTATIONS_POPULATION": 3,
	}

pipeline/pipeline/models.py

Lines 186 to 211 in 8c706d6

	def validate_expectations_per_version(cls, values: RawPipeline) -> RawPipeline:
	"""Ensure the expectations key exists for version 3 onwards"""
	try:
	version = float(values["version"])
	except (KeyError, TypeError, ValueError):
	# this is handled in the validate_version_exists and
	# validate_version_value validators
	return values
	feat = get_feature_flags_for_version(version)
	if not feat.EXPECTATIONS_POPULATION:
	# set the default here because pydantic doesn't seem to set it
	# otherwise
	values["expectations"] = {"population_size": 1000}
	return values
	if "expectations" not in values:
	raise ValueError("Project must include `expectations` section")
	if "population_size" not in values["expectations"]:
	raise ValueError(
	"Project `expectations` section must include `population_size` section",
	)
	return values

Version 4.0 should reject this section.

Once we have done this we should update the research-template accordingly.

See also:

• Replace cohortextractor with ehrql opensafely/research-template#110 (comment)

[lucyb]

After speaking with Peter, this doesn't sound urgent and could be done by the upcoming RAP Team rather than the current Pipeline Team. Please let me know if I've missed anything.

[inglesp]

When we introduce version 4, we'll also want to raise a validation error if there are any cohort-extractor actions in a project.yaml.

Additionally, we should:

• update the pipeline.yaml example in the research template: Merge Update project.yaml to version 4 template opensafely/research-template#141
• ensure that all examples at docs.opensafely.org use version 4 and don't include an expectations section: Merge Update project.yaml examples to pipeline v4 if the example only has ehrQL actions opensafely/documentation#1639 and Update example project.yaml in docs to version 4 template ehrql#2145

[alarthast]

The steps are:

• Use jobrunner.cli.local_run to verify that a project.yaml using v4 and no expectation sections does not produce unexpected errors
• Merge Create version 4.0 #341
• Create a new release PR following DEVELOPERS.md
• Merge the release PR Release 2024.10.08.124104 #343
• Update job-runner to use the new release of pipeline by merging Bump to pipeline 2024.10.08.124104 job-runner#754. The feat: flag in the commit message builds a new release.
• (Break backend-server and fix by) Updating job-runner-dependencies
• Close Raise a ValidationError for an empty action #344 (which includes merging Bump pipeline to v2024.10.11.170331 job-server#4666)
• Merge feat: Update job-runner to v2.75.1 opensafely-cli#281
• Use opensafely-cli on a project.yaml file with v4 and test that everything works again

then, do the tasks listed in Peter's comment