fix: Use CTE in Enrollment analytics queries [DHIS-16705]

[luciano-fiandesio]

Summary

This PR refactors the SQL query generation process for Enrollment analytics to enhance compatibility with Doris by utilizing Common Table Expressions (CTEs) instead of sub-queries.

Problem

Previously, SQL queries for Enrollment analytics leveraged sub-queries to fetch event values from analytics_event_* tables. While functional in Postgres, these queries were incompatible with Doris due to its limitations in handling nested sub-queries correlated with outer query layers.

Example of the original query:

select
    enrollment,
    ...
    ax."ou",
    (select
         "fyjPqlHE7Dn"
     from
	 analytics_event_M3xtLkYBlKI
     where
	 analytics_event_M3xtLkYBlKI.eventstatus != 'SCHEDULE'
	 and analytics_event_M3xtLkYBlKI.enrollment = ax.enrollment
	 and ps = 'CWaAcQYKVpq'
limit 1 ) as "CWaAcQYKVpq[-1].fyjPqlHE7Dn",
from
     analytics_enrollment_m3xtlkyblki as ax

Mitigation

This refactor converts sub-queries into Common Table Expressions (CTEs), which are supported by Doris. CTEs simplify the queries, making them more modular and readable, thereby improving performance and compatibility.

Example of the refactored query using CTE:

with ps_cwaacqykvpq_fyjpqlhe7dn as (
select
	distinct on
	(enrollment) enrollment,
	"fyjPqlHE7Dn" as value
from
	analytics_event_M3xtLkYBlKI
where
	eventstatus != 'SCHEDULE'
	and ps = 'CWaAcQYKVpq'
order by
	enrollment,
	occurreddate desc,
	created desc)
select
	ax.enrollment,
	...
	ps_cwaacqykvpq_fyjpqlhe7dn.value as "CWaAcQYKVpq.fyjPqlHE7Dn"
from
	analytics_enrollment_m3xtlkyblki as ax
left join ps_cwaacqykvpq_fyjpqlhe7dn on
	ax.enrollment = ps_cwaacqykvpq_fyjpqlhe7dn.enrollment
where
	(((enrollmentdate >= '2021-01-01'
	and enrollmentdate < '2022-01-01')))
	and (ax."uidlevel1" = 'ImspTQPwCqd')

Testing Strategy

The changes have been tested using the e2e project to ensure all tests pass successfully on both Postgres and Doris.

Types of Queries Affected

Both aggregated and non-aggregated Enrollment queries are impacted by this refactor.

Experimental Flag

To prevent potential platform issues, the refactored CTE-based queries are not enabled by default. They can be activated by setting the following system configurations to true:

• experimentalAnalyticsSqlEngineEnabled
• or by configuring DHIS2 analytics.database to doris.

Performance Comparison:

• Original query with sub-select
  
• Refactored query with CTEs
  

[maikelarabori]

Nice changes @luciano-fiandesio. 👍
I left a few comments/suggestions, please take a look, thx!

To fix the problem, we need to validate the offset value before performing the arithmetic operation. Specifically, we should ensure that the offset value is within a safe range to prevent overflow or underflow. We can achieve this by adding a guard clause that checks if the offset value is within the bounds of Integer.MIN_VALUE and Integer.MAX_VALUE.

• Add a validation check for the offset value in the computeRowNumberOffset method.
• If the offset value is outside the safe range, we can either throw an exception or clamp the value to the nearest safe value.

[sonarqubecloud]

Quality Gate failed

Failed conditions
2 New issues
2 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE