LoadListDataAsStreamAsync() fails when a list has more than 5000 items and if the item count is multiple of ViewXml RowLimit

[NewGHUser4321]

Category

• Bug

Describe the bug

LoadListDataAsStreamAsync() fails when a list has more than 5000 items and if the item count is multiple of ViewXml <RowLimit>

Steps to reproduce

Try to get all the list items using spList.LoadListDataAsStreamAsync() from a list that has more than 5000 items. Make sure the list item count in the list is multiple of <RowLimit>.

Eg: If a list has 6000 list items use <RowLimit Paged="TRUE">1000</RowLimit>

I'm getting the list items as below. Make sure the viewXml has <RowLimit Paged="TRUE">1000</RowLimit>

while (paging)
{
    var output = await spList.LoadListDataAsStreamAsync(new RenderListDataOptions()
    {
        ViewXml = viewXml,
        RenderOptions = RenderListDataOptionsFlags.ListData,
        Paging = nextPage ?? null,
    }).ConfigureAwait(false);

    if (output.ContainsKey("NextHref"))
    {
        nextPage = output["NextHref"].ToString().Substring(1);
    }
    else
    {
        paging = false;
    }
}

I get this Error:

An item with the same key has already been added. Key: restId

And the stack trace is

Expected behavior

Should not throw error. It should get all the list items as expected.

Environment details (development & target environment)

• SDK version: 1.8.0

Thank you.

[jansenbe]

@NewGHUser4321 : interesting finding, let me try to reproduce that

[jansenbe]

@NewGHUser4321 : nice catch, in this case the 6000th row is returned to indicate paging is over but as that row was already there some code populating a dictionary broke with a duplicate key exception. I've fixed this, the next nightly build should work fine.

For my own reference, the test case use for this:

public async Task Bert()
{
    TestCommon.Instance.Mocking = false;
    using (var context2 = await TestCommon.Instance.GetContextAsync(TestCommon.TestSite))
    using (var context = await context2.CloneAsync(new Uri("https://bertonline.sharepoint.com/sites/prov-1")))
    {
        var list = context.Web.Lists.GetByTitle("Large2");
        if (list != null)
        {
            bool paging = true;
            string nextPage = null;
            while (paging)
            {
                var output = await list.LoadListDataAsStreamAsync(new RenderListDataOptions()
                {
                    ViewXml = "<View><ViewFields><FieldRef Name='Title' /></ViewFields><RowLimit Paged=\"TRUE\">1000</RowLimit></View>",
                    RenderOptions = RenderListDataOptionsFlags.ListData,
                    Paging = nextPage ?? null,
                }).ConfigureAwait(false);

                if (output.ContainsKey("NextHref"))
                {
                    nextPage = output["NextHref"].ToString().Substring(1);
                }
                else
                {
                    paging = false;
                }
            }

            Assert.IsTrue(list.Items.Length == 6000);
        }
    }
}

[NewGHUser4321]

@jansenbe thank you for fixing this.

[jansenbe]

@NewGHUser4321 : closing this one, let me know in case it's not working