TAXII2 pagination incorrectly sets more to false if MAX_TAXII_PAGINATION > ES_DEFAULT_PAGINATION

[samczsun]

Description

We have app:data_sharing:taxii:max_pagination_result set to 10000. When requesting /objects with no limit parameter, it returns 5000 entries and more=false. However, there are more entries and requesting with the next parameter properly returns additional entries.

The value of more is determined by pageInfo.hasNextPage

opencti/opencti-platform/opencti-graphql/src/domain/taxii.js

Lines 134 to 143 in 18ce0c0

	export const restCollectionStix = async (context, user, collection, args) => {
	const { edges, pageInfo } = await collectionQuery(context, user, collection, args);
	const edgeIds = edges.map((e) => e.node.internal_id);
	const instances = await stixLoadByIds(context, user, edgeIds);
	return {
	more: pageInfo.hasNextPage,
	next: R.last(edges)?.cursor || '',
	objects: instances,
	};
	};

This value is true if instances.length === limit

opencti/opencti-platform/opencti-graphql/src/database/utils.js

Lines 223 to 234 in 18ce0c0

	export const buildPagination = (limit, searchAfter, instances, globalCount) => {
	const edges = R.pipe(
	R.mapObjIndexed((record) => {
	const { node, sort, types } = record;
	const cursor = sort ? offsetToCursor(sort) : '';
	return { node, cursor, types };
	}),
	R.values
	)(instances);
	// Because of stateless approach its difficult to know if its finish
	// this test could lead to an extra round trip sometimes
	const hasNextPage = instances.length === limit;

This conditional will evaluate to false if limit is not set as a query parameter and MAX_TAXII_PAGINATION > ES_DEFAULT_PAGINATION, because instances.length == ES_DEFAULT_PAGINATION but limit == MAX_TAXII_PAGINATION.

Environment

1. OS (where OpenCTI server runs): Ubuntu
2. OpenCTI version: 6.2.6
3. OpenCTI client: frontend
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create a TAXII2 collection which has more than ES_DEFAULT_PAGINATION elements
2. Set MAX_TAXII_PAGINATION to a value greater than ES_DEFAULT_PAGINATION
3. Request the /objects endpoint with no explicit limit parameter

Expected Output

more should be true, because there are more elements

Actual Output

more is false, because of the reasons outlined above

Additional information

Screenshots (optional)

[romain-filigran]

@richard-julien @SouadHadjiat @aHenryJard : Can we plan to look at this together ?

[richard-julien]

Thanks for the report @samczsun. There is a collision in term of option between the data_sharing:taxii:max_pagination_result and the elasticsearch:max_pagination_result that create this issue.
We be fixed in the next release.
However you need to be really careful about changing this kind of option. Too many results will impact the processing and the elasticsearch.
For now please use 5000 as the max configuration.