[BUG] Non published references are returned by DataSync when using include([Array of refs]) function and the UIDs are mixed up then.

[idmedia-kaiser]

Hi, we are facing an issue with DataSync when using the include() function to include only specific references.
https://contentstack.github.io/datasync-filesystem-sdk/global.html#include

We have a form content type with a reference field for fields
https://app.contentstack.com/#!/stack/blt81b4b787a1850697/content-type/form/de-de/entry/blt0e8af3662cf665b0/edit?branch=main

Some of those references are not published. If we check in the DEV Tool widget we correctly don't get this fields back that are not published.

https://cdn.contentstack.io/v3/content_types/form/entries/blt0e8af3662cf665b0?api_key=blt81b4b787a1850697&access_token=&environment=development&locale=de-de

{
...
  "fields": [
    {
      "uid": "blte52259e1308823f8",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "bltf45bb3e8d8c04dd1",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "bltb673a18f783fa5b6",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "blt9788f975762ad3e0",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "blt4984c9008d23711a",
      "_content_type_uid": "input_field"
    }
  ]
}

If we use the include(["fields"]) function then the not published entries get mixed up with entries that are actually published and we get duplicated content

[{
    "content_type_uid": "form",
    "locale": "de-at",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1"
                },
                {
                    "uid": "bltb673a18f783fa5b6"  // -> Not published should be "blta8fed94360b45377" or actually not present at all
                },
                {
                    "uid": "blt9788f975762ad3e0"  // -> Not published should be "blta22b07fa5fc5fce9" or actually not present at all
                },
                {
                    "uid": "bltb673a18f783fa5b6"
                },
                {
                    "uid": "blt9788f975762ad3e0"
                },
                {
                    "uid": "blt4984c9008d23711a"
                }
            ]
        }
    ]
}]

Composed query

let blogQuery = Stack.contentType("form");
blogQuery = blogQuery.equalTo("uid", "blt0e8af3662cf665b0");
blogQuery = blogQuery.include(["fields"]);
blogQuery.find().then((result: {}) => {
	res.json(result);
})

This works correctly if we use replace
blogQuery = blogQuery.include(["fields"]);
with
blogQuery = blogQuery.includeReferences();

but we don't want all references but only some of them.

[idmedia-kaiser]

@harshithad0703 As discussed I opened this as a new issue with the correct description of the error.
@aman19K @netrajpatel @cs-raj @shafeeqd959 FYI

[aman19K]

Hi @idmedia-kaiser ,
We’ve tested the include() function in DataSync, and it’s working as expected. To include the content of referenced entries in your response, you need to use the include[] parameter and provide the UID of the reference field as the value. It looks like you're currently using fields instead of UIDs, which is why it's not working correctly. Please try using UIDs and let us know how it goes?

For example:-
Stack.contentType('example').entries().include(['authors','categories']).find()
If we have three CTs (authors, categories, and blogs) referenced in example, and we just want to retrieve the authors and categories, then we can use include() function.

FYI
DataSync:-
https://github.com/contentstack/datasync-filesystem-sdk/blob/master/src/stack.ts#L800
https://github.com/contentstack/datasync-filesystem-sdk/blob/master/src/stack.ts#L1090
API:- https://www.contentstack.com/docs/developers/apis/content-delivery-api#include-reference

CC: @netrajpatel @harshithad0703

[idmedia-kaiser]

Hi @aman19K, as I wrote also in the ticket.If you check the demo
API:- https://www.contentstack.com/docs/developers/apis/content-delivery-api#include-reference
categories is the name of the reference field and the content type uid inside is category. And we do the same.
fields is the reference field that holds the content type uid input_field.
Therefore we pass fields and not input_field to the include[] same as in the demo we pass categories and not category

And the issue is not the its not working, the issue is with unpublished entries getting returned partially and mixing up with existing entries

"entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1", <---- unpublished entry which actuallay has UID blt983d6cdb2ec347cd see below
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "title": "PLZ (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }

same resposne without the include(["fields"]

 "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt983d6cdb2ec347cd", <-- this is the unpublished entry
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "_content_type_uid": "input_field"
                }
            ]
        }
    ]

expected is this

"entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "title": "PLZ (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }

[aman19K]

Yes @idmedia-kaiser , We are checking the demo. As soon as we can reproduce the problem, we'll let you know.
CC: @harshithad0703