support $batch operations

[aappddeevv]

Support $batch requests through the API.

[derekfinlinson]

Until David can get this implemented, feel free to try out my library: xrm-webapi.

[aappddeevv]

Saw that. Had catalogued all of the APIs as well. Thought you did a good job but I was not using typescript at the time. http://msdynamicscrmmeanderings.blogspot.com/2017/05/dynamics-crm-rest-api-choices.html

…

On Fri, Sep 29, 2017 at 7:33 PM Derek Finlinson ***@***.***> wrote: Until David can get this implemented, feel free to try out my library: xrm-webapi <https://github.com/derekfinlinson/xrm-webapi>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#47 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABIOaewvd-RPvaucb7HRLd1ishtlu6xBks5snX42gaJpZM4Pkw8A> .

[davidyack]

@aappddeevv is there a particular batch operation you are looking for ? We do support it for Create in the C#, I'm assuming you are looking for it in the JS?

@derekfinlinson how are you handling parsing the response?

[aappddeevv]

Yes, JS is correct. I need to issue a $batch in order to push a fetchxml query that is too long to go in the standard http package....it needs to go in the body. I hacked it up, so it would work: https://pastebin.com/BudCssGe I eyeballed' Derek's and others lib/examples to make this work.

…

On Sat, Sep 30, 2017 at 9:48 AM David Yack ***@***.***> wrote: @aappddeevv <https://github.com/aappddeevv> is there a particular batch operation you are looking for ? We do support it for Create in the C#, I'm assuming you are looking for it in the JS? @derekfinlinson <https://github.com/derekfinlinson> how are you handling parsing the response? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#47 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABIOaagYF6MngkdrdE5QYNxAwxZQZG5Wks5snkbFgaJpZM4Pkw8A> .

[derekfinlinson]

@davidyack I'm not parsing the response at all. Just returning it as JSON.

@aappddeevv That's exactly the reason I added it to my library in the first place :)

[aappddeevv]

I built a scala.js based CLI tool for Dynamics (swiss army knife type of thing) and with that client I had to restructure the client so that one could create "requests" independently of the actual call e.g. make a create request as a separate function that is called inside the actual client.Create(...) method. Then backout a hacked parser to read each section. The batch calls are hard to hack well quickly. If a lib does not expose request making and body parsing then doing batch request/response processing is really hard.

[davidyack]

So for your Fetch need, couldn't we just dynamically switch to Post for long FetchXML queries?

[aappddeevv]

I don't think you can issue a Post using the entitySet url and the real issue is URL length. FetchXml can be quite large and exceed the server's 2,000 char limit on the URL. If the fetchxml is in a GET inside a batch (with a POST) then the URL limit is not hit.

[NaveenGaneshe]

@davidyack , we are using your CRMWebApi for our project and i really appreciate your approach.As we are trying to implement $batch request for the large fetchxml to get records by post request in Dotnet Core but unable to get the expected result. Please let us know the right approach to achieve the correct result.
Here is the sample code:
`internal async Task GetList(string fetchXML)
{
try
{
HttpClient httpClient = new HttpClient();

            var OrganizationAPI = _configuration.GetSection("appSettings").GetSection("OrganizationWebApi").Value;
            var OrganizationUrl = _configuration.GetSection("appSettings").GetSection("OrganizationUrl").Value;
            var AccessToken = _cache.Get<string>("AccessToken");

            httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", AccessToken);

            httpClient.DefaultRequestHeaders.Add("OData-MaxVersion", "4.0");
            httpClient.DefaultRequestHeaders.Add("OData-Version", "4.0");

            var body = "--batch_accountFetch" + Environment.NewLine;
            body += "Content-Type: application/http" + Environment.NewLine;
            body += "Content-Transfer-Encoding: binary" + Environment.NewLine;
            body += Environment.NewLine;
            body += "GET " + OrganizationUrl + OrganizationAPI + "accounts?fetchXml=" + fetchXML + " HTTP/1.1" + Environment.NewLine;
            body += "Content-Type: application/json" + Environment.NewLine;
            body += "OData-Version: 4.0" + Environment.NewLine;
            body += "OData-MaxVersion: 4.0" + Environment.NewLine;
            body += Environment.NewLine;
            body += "--batch_accountFetch--";

            var myContent = JsonConvert.SerializeObject(body);
            var buffer = Encoding.UTF8.GetBytes(myContent);
            var byteContent = new ByteArrayContent(buffer);
            byteContent.Headers.TryAddWithoutValidation("Content-Type", "multipart/mixed;boundary=batch_accountFetch");

            var response = await httpClient.PostAsync(OrganizationUrl + OrganizationAPI + "$batch", byteContent);

            string result = await response.Content.ReadAsStringAsync();

            return result;
        }
        catch (Exception Ex)
        {
            throw new Exception(Ex.Message);
        }
    }

`

[davidyack]

I think your issue is your trying to json convert the body which needs to be just posted as data. look at the https://pastebin.com/BudCssGe that @aappddeevv posted before - it's JS but should give you some insight. I do still plan on adding support for transparently switching to post on the Fetch if it's too big - just haven't had the time to implement it. I also think in many cases I've seen people should simplify their fetches :)

[NaveenGaneshe]

Thank you for your reply @davidyack . You are right it was json parsing issue. Below code is the implementation of executing large fetch in Dotnet core app:

`HttpClient httpClient = new HttpClient();

            var OrganizationAPI = _configuration.GetSection("appSettings").GetSection("OrganizationWebApi").Value;
            var OrganizationUrl = _configuration.GetSection("appSettings").GetSection("OrganizationUrl").Value;
            var AccessToken = _cache.Get<string>("AccessToken");

            httpClient.BaseAddress = new Uri(OrganizationUrl + OrganizationAPI + "$batch");

            httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", AccessToken);

            httpClient.DefaultRequestHeaders.Add("OData-MaxVersion", "4.0");
            httpClient.DefaultRequestHeaders.Add("OData-Version", "4.0");

            var body = "--batch_accountFetch" + Environment.NewLine;
            body += "Content-Type: application/http" + Environment.NewLine;
            body += "Content-Transfer-Encoding: binary" + Environment.NewLine;
            body += Environment.NewLine;
            body += "GET " + OrganizationUrl + OrganizationAPI + "accounts?fetchXml=" + QueryOptions.FetchXml + " HTTP/1.1" + Environment.NewLine;
            body += "Prefer:odata.include-annotations=OData.Community.Display.V1.FormattedValue" + Environment.NewLine;
            body += "Content-Type: application/json" + Environment.NewLine;
            body += "OData-Version: 4.0" + Environment.NewLine;
            body += "OData-MaxVersion: 4.0" + Environment.NewLine;
            body += Environment.NewLine;
            body += "--batch_accountFetch--";

            var multipartContent = new MultipartContent("mixed");
            multipartContent.Headers.Remove("Content-Type");
            multipartContent.Headers.TryAddWithoutValidation("Content-Type", "multipart/mixed; boundary=batch_accountFetch");

            var myContent = JsonConvert.SerializeObject(body);
            var stringContent = new StringContent(body, Encoding.UTF8, "application/json");
            multipartContent.Add(stringContent);

            var response = await httpClient.PostAsync(OrganizationUrl + OrganizationAPI + "$batch", multipartContent);

            response.EnsureSuccessStatusCode();

            var data = await response.Content.ReadAsStringAsync();
            CRMGetListResult<ExpandoObject> resultList = new CRMGetListResult<ExpandoObject>();
            resultList.List = new List<ExpandoObject>();

            var values = JObject.Parse(data.Substring(data.IndexOf('{'), data.LastIndexOf('}') - data.IndexOf('{') + 1));
           
            var valueList = values["value"].ToList();

            foreach (var value in valueList)
            {
                FormatResultProperties((JObject)value);
                resultList.List.Add(value.ToObject<ExpandoObject>());
            }
            var deltaLink = values["@odata.deltaLink"];
            if (deltaLink != null)
                resultList.TrackChangesLink = deltaLink.ToString();
            var recordCount = values["@odata.count"];
            if (recordCount != null)
                resultList.Count = int.Parse(recordCount.ToString());
            
           return resultList;`

Alternatively we can also use the belo code:
`public async Task GetRecordsInBatch(string entity, string fetchXML)
{
string returnVal = "";
try
{
//Connect to crm
var accessToken = await _crm.GetApiToken();

            var httpClient = new HttpClient();

            httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
            httpClient.DefaultRequestHeaders.Add("OData-MaxVersion", "4.0");
            httpClient.DefaultRequestHeaders.Add("OData-Version", "4.0");
            var batchid = "batch_" + Guid.NewGuid().ToString();

            MultipartContent batchContent = new MultipartContent("mixed", batchid);
            var changesetID = "changeset_" + Guid.NewGuid().ToString();
           //MultipartContent changeSetContent = new MultipartContent("mixed", changesetID);

            HttpRequestMessage req = new HttpRequestMessage(HttpMethod.Get, _organizationWebApi + entity + "?fetchXml=" + fetchXML);
            req.Version = new Version(1, 1);

            //req.Content = new StringContent(JsonConvert.SerializeObject(data), Encoding.UTF8, "application/json");
            HttpMessageContent content = new HttpMessageContent(req);
            content.Headers.Remove("Content-Type");
            content.Headers.TryAddWithoutValidation("Content-Type", "application/http");
            content.Headers.TryAddWithoutValidation("Content-Transfer-Encoding", "binary");
            content.Headers.TryAddWithoutValidation("Content-ID","1");

            //changeSetContent.Add(content);

            batchContent.Add(content);

            HttpRequestMessage batchRequest = new HttpRequestMessage(HttpMethod.Post, _organizationWebApi + "$batch");

            batchRequest.Content = batchContent;

            var batchstring = await batchRequest.Content.ReadAsStringAsync();

            var response = await httpClient.SendAsync(batchRequest);
            var responseString = response.Content.ReadAsStringAsync();
            MultipartMemoryStreamProvider batchStream = await response.Content.ReadAsMultipartAsync();
            var changesetStream = batchStream.Contents.FirstOrDefault();

        }
        catch (Exception ex)
        {
            return ex.Message;
        }

        return returnVal;
    }

`

[AshV]

Hello all,

Associate on create is not working with $batch for me, I have raised below issue in the product documentation repo with request/response details.
https://github.com/MicrosoftDocs/dynamics-365-customer-engagement/issues/887

I saw another question about this in stack overflow https://stackoverflow.com/questions/32761311/create-annotation-to-a-contact-entity-in-microsoft-dynamics-crm-by-api

Has anyone tried this with $batch? Is that a product issue or something wrong with my request format.

I am adding this here because this thread has a very productive discussion about $batch.

Please have a look into given links & help me out.

Thanks,
AshV