Implementing cached dataset querying #96

Shalelol · 2017-07-26T01:00:22Z

Changes

Implementing a way to query the new cached datasets API.
Adding the user of a Snake Case Contract Resolver for NewtonSoft.Json because the use of the JsonPropertyAttribute can get a bit unwieldy once the data structures get larger and more complex, as they can with the cached datasets.

Comments

I am moving towards strongly typed models, leveraging the convenience of the contract resolver, and trying to work with JToken as infrequently as possible, please let me know if anyone has any issues with this.
I'm trying to adhere to the naming and code styling convention as much as possible, please do let me know if I stray from that too much.
I'm quite inexperienced with unit testing, particularly with NUnit, so would love to hear any feedback or advice regarding that.
I plan to finish work on this pull request mid to late August this year (2017).

Issues addressed

Delete Cached Dataset #92 - Delete Cached Dataset
List Cached Dataset Definitions #91 - List Cached Dataset Definitions
Retrieve Cached Dataset Results #90 - Retrieve Cached Dataset Results
Get Cached Dataset Definition #89 - Get Cached Dataset Definition
Create Cached Dataset #88 - Create Cached Dataset

masojus · 2017-07-26T01:06:24Z

Thanks @Shalelol I will take a look! Meanwhile could you specify which Issues this resolves? Thanks!

Shalelol · 2017-07-26T01:09:16Z

Hey @masojus I've updated my initial comment to include the issues addressed so far. I also may look into addressing more of the issues related to cached datasets as my requirements grow.

masojus · 2017-07-26T01:17:06Z

Thank you @Shalelol! And don't worry about the AppVeyor failure. I imagine it's an expected failure I need to deal with but all the tests look to have passed. I presume manual testing went well? Or are there any major corner cases to watch out for?

Shalelol · 2017-07-26T01:24:55Z

@masojus So far all manual testing is working perfectly. I will inform you if any corner cases pop up.

I'm also still yet to write a unit test for GetDatasetDefinition but will be doing so shortly.

coveralls · 2017-07-26T03:59:33Z

Coverage decreased (-2.3%) to 55.106% when pulling 6f3a5b4 on clickviewapp:cached-datasets into 7f20112 on keenlabs:master.

masojus · 2017-07-26T06:15:58Z

Keen/ContractResolvers/SnakeCaseContractResolver.cs

+     * 
+     * Credit to Chris Allen https://github.com/crallen 
+     */
+    public class SnakeCaseContractResolver : DefaultContractResolver


Is this necessary because the version of Json.NET we're using doesn't yet have the SnakeCaseNamingStrategy or is it the case that even that strategy does not satisfy your needs? How does this SnakeCaseContractResolver handle camelCase or PascalCase or other casing like kebab-case, and how does that compare to Json.NET's own SnakeCaseNamingStrategy? How does this treat names that have a JsonPropertyAttribute explicitly specifying the name?

http://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_Serialization_SnakeCaseNamingStrategy.htm
http://www.phrohdoh.com/blog/2016/09/02/newtonsoft-json-custom-property-names/

I've used the Contract Resolver rather than Json.NET because all the attributes defined on all the existing models are using the JsonPropertyAttribute from Newtonsoft.Json. Which will only be respected when using the JsonConvert.SerializeObject() and JsonConvert.DeserializeObject() functions.

The SnakeCaseContractResolver doesnt handle camelCase, PascalCase or kebab-case but custom resolvers may be defined for those. The resolver is defined at serialization/deserialization time by passing it as an option to JsonConvert like so, rather than in a global setting:

JsonConvert.DeserializeObject<DatasetDefinition>(responseString, new JsonSerializerSettings { ContractResolver = new SnakeCaseContractResolver()});

This, and indeed any ContractResolver extending Newtonsoft.Json's DefaultContractResolver will respect all JsonPropertyAttributes

masojus · 2017-07-26T06:21:00Z

Keen/Dataset/Datasets.cs

+    internal class Datasets : IDataset
+    {
+        private readonly IKeenHttpClient _keenHttpClient;
+        private readonly string _cachedDatasetReleativeUrl;


typo: _cachedDatasetReleativeUrl->_cachedDatasetRelativeUrl

masojus · 2017-07-26T06:22:06Z

Keen/KeenClient.cs

@@ -970,5 +975,39 @@ public JObject Query(string queryName, Dictionary<string, string> parms)
                throw ex.TryUnwrap();
            }
        }
+
+        public async Task<JToken> QueryDatasetAsync(string datasetName, string indexBy, string timeframe)


We'll want doc comments on these public methods.

masojus · 2017-07-26T06:24:18Z

Keen/Dataset/Datasets.cs

+    using Newtonsoft.Json.Linq;
+    using Query;
+
+    public interface IDataset


I would tend to prefer interfaces in their own file, and public API surface (types, fields, methods, etc.) should have doc comments.

masojus · 2017-07-26T06:35:24Z

Keen/Dataset/Dataset.cs

+
+namespace Keen.Core.Dataset
+{
+    using Query;


Reminds me we need to start creating some guidelines around things like using inside and outside of namespaces and configure some stylecop rulesets: https://github.com/DotNetAnalyzers/StyleCopAnalyzers

I'll file an Issue.

Shalelol · 2017-07-26T06:39:56Z

Hey @masojus, I just want to apologize for the messy state of some of the code. I just realized that I didn't really communicate how WIP this Pull Request is. I had/have every intention of coming back and cleaning everything up, things such as xml comments, moving files out and scoping usings to match the rest of the project. This is still very much an in progress project at the moment as we move all of our querying to cached querying.

I was just putting the WIP PR up nice and early so that it's on your radar =]

Thanks for your comments so far, I will make sure to clean everything up.

masojus · 2017-07-26T06:40:43Z

Keen/Dataset/Datasets.cs

+                                        responseMessage.StatusCode);
+            }
+
+            return JsonConvert.DeserializeObject<DatasetDefinition>(responseString, new JsonSerializerSettings { ContractResolver = new SnakeCaseContractResolver()});


I'll have to think about this and ponder the answers to the questions in my comment above about SnakeCaseContractResolver and what it means to always use this resolver for all Dataset definitions.

masojus · 2017-07-26T06:42:05Z

@Shalelol No worries. I'm just skimming it anyway, so I'll take a closer look at it later when you've tidied up a bit and think it's ready for prime time. 👍 Thanks!

tbarn · 2017-08-04T16:30:23Z

@Shalelol just wanted to say, thanks for working on this! We really appreciate it.

Always using outside of namespace.

We don't need to expose this in the client.

coveralls · 2017-08-11T07:29:22Z

Coverage decreased (-2.3%) to 55.106% when pulling e0a6ebb on clickviewapp:cached-datasets into 7f20112 on keenlabs:master.

masojus · 2017-08-11T07:31:11Z

We don't have the Travis config merged into master yet, so I wouldn't worry too much about that one.

Shalelol · 2017-08-16T01:59:38Z

Just updated the opening comment with issues. I plan to implement #88, #91 and #92 for completeness.

coveralls · 2017-08-17T06:19:29Z

Coverage decreased (-6.3%) to 51.138% when pulling 891b1ae on clickviewapp:cached-datasets into 7f20112 on keenlabs:master.

coveralls · 2017-08-17T06:24:23Z

Coverage decreased (-6.2%) to 51.301% when pulling c2602e9 on clickviewapp:cached-datasets into 7f20112 on keenlabs:master.

coveralls · 2017-08-17T06:57:52Z

Coverage decreased (-5.9%) to 51.539% when pulling 3fcb789 on clickviewapp:cached-datasets into 7f20112 on keenlabs:master.

Because the sync ones end up calling the async ones. This way all the dataset methods on KeenClient are covered.

Shalelol · 2017-08-17T10:26:21Z

Coveralls likes to point out when you've been bad, but not when you're trying to improve the coverage 😞.

baumatron · 2017-08-17T19:17:14Z

I think It should hand out gold stars and/or free beers for increased coverage ⭐️ 🍻.

coveralls · 2017-08-18T00:53:27Z

Coverage decreased (-1.1%) to 56.346% when pulling ff25092 on clickviewapp:cached-datasets into 72457ff on keenlabs:master.

Also refactored the tests to be a bit neater

coveralls · 2017-08-21T05:25:44Z

Coverage increased (+2.3%) to 59.724% when pulling 7674dc4 on clickviewapp:cached-datasets into 72457ff on keenlabs:master.

coveralls · 2017-08-21T07:18:46Z

Coverage increased (+3.9%) to 61.314% when pulling f90189c on clickviewapp:cached-datasets into 72457ff on keenlabs:master.

coveralls · 2017-08-21T07:31:47Z

Coverage increased (+3.9%) to 61.314% when pulling 16b6f51 on clickviewapp:cached-datasets into 72457ff on keenlabs:master.

Shalelol · 2017-08-21T07:34:08Z

Hey @masojus. I've finished implementing all the Cached Dataset issues, and I've brought the coverage of Keen.Core.Dataset up to 91%. I've also gone through and added XML comments to everything that is public facing.

Could you please give this another review when you have time?

Thanks 😃.

masojus · 2017-08-23T22:21:18Z

Thank you! We'll get some eyes on this as soon as possible.

masojus · 2017-09-20T17:24:58Z

Sorry for the delay here. We're going to be merging the dotnetsummer branch shortly and will address this after that so as to also port it to .NET Standard 2.0 while we're at it.

masojus · 2017-10-11T17:54:46Z

Hello @Shalelol, we've now merged the big chunk of work we wanted to get into master before taking this PR on. Could you merge from master and update the PR? If you need help with that we can pitch in, but then we could work on getting this PR in. Thanks!

masojus · 2017-10-26T10:24:44Z

I created branch jm_UpdateCachedDatasetsBranch to get this up to date with master and continue the conversation. I'll probably close this PR and we can finish polishing that off in the new PR.

masojus · 2017-10-26T10:32:35Z

See PR #128 for a continuation of this PR.

Sha added 6 commits July 26, 2017 10:22

Add SnakeCaseContractResolved for deserializing responses to CLR types

415c417

Add Datasets and relevant models

6b553e1

Add a query definition model

9e59776

Add functions in KeenClient to query Datasets and their definitions

4fc9a8b

Add unit test for QueryDatasetAsync

5b9643f

Merge remote-tracking branch 'keenlabs/master' into cached-datasets

6f3a5b4

masojus reviewed Jul 26, 2017

View reviewed changes

Sha added 6 commits August 11, 2017 15:23

Fix typo on private field: _cachedDatasetRelativeUrl

fec9c16

Provide XML comments for all new public facing methods

7c98001

Provide XML comments for the dataset classes exposed via KeenClient

4a5b56f

Move IDataset into it's own file

0394b1d

Clean up scope of using statements

fd294c4

Always using outside of namespace.

Fix scope of SnakeCasecontractResolver

e0a6ebb

We don't need to expose this in the client.

Rename Dataset.cs and provide serialization and validation utility

ac0e91f

Remove models which are not used

c2602e9

Sha added 2 commits August 17, 2017 16:45

Fix minor typo in DeleteDataset

5c2c2ac

Add some simple tests for the new dataset methods

3fcb789

Sha added 5 commits August 17, 2017 16:59

Update dataset tests to test sync methods instead of async ones

7b98a80

Because the sync ones end up calling the async ones. This way all the dataset methods on KeenClient are covered.

Use shared GetSerializerSettings function for getdatasetdefinition

306cdaa

Add testing for the custom serialization logic

2e69245

Added teast for serializing datasetdefinitioncollection

16e3fc8

Merge branch 'master' into cached-datasets

883f1eb

Fix bug with ListDatasetDefinition that popped up in unit tests

ff25092

Include testing for Create and Delete datasets

7674dc4

Also refactored the tests to be a bit neater

Add tests for dataset error modes

f90189c

Shalelol changed the title ~~[WIP] Implementing cached dataset querying~~ Implementing cached dataset querying Aug 21, 2017

Clean up scope of imports

16b6f51

masojus mentioned this pull request Oct 26, 2017

Datasets #128

Merged

masojus closed this Oct 26, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing cached dataset querying #96

Implementing cached dataset querying #96

Shalelol commented Jul 26, 2017 •

edited

Loading

masojus commented Jul 26, 2017

Shalelol commented Jul 26, 2017 •

edited

Loading

masojus commented Jul 26, 2017

Shalelol commented Jul 26, 2017

coveralls commented Jul 26, 2017

masojus Jul 26, 2017

Shalelol Jul 26, 2017

masojus Jul 26, 2017

masojus Jul 26, 2017

masojus Jul 26, 2017

masojus Jul 26, 2017 •

edited

Loading

Shalelol commented Jul 26, 2017 •

edited

Loading

masojus Jul 26, 2017

masojus commented Jul 26, 2017

tbarn commented Aug 4, 2017

coveralls commented Aug 11, 2017

masojus commented Aug 11, 2017

Shalelol commented Aug 16, 2017

coveralls commented Aug 17, 2017

coveralls commented Aug 17, 2017

coveralls commented Aug 17, 2017

Shalelol commented Aug 17, 2017

baumatron commented Aug 17, 2017

coveralls commented Aug 18, 2017

coveralls commented Aug 21, 2017

coveralls commented Aug 21, 2017

coveralls commented Aug 21, 2017

Shalelol commented Aug 21, 2017

masojus commented Aug 23, 2017

masojus commented Sep 20, 2017

masojus commented Oct 11, 2017

masojus commented Oct 26, 2017

masojus commented Oct 26, 2017

Implementing cached dataset querying #96

Implementing cached dataset querying #96

Conversation

Shalelol commented Jul 26, 2017 • edited Loading

Changes

Comments

Issues addressed

masojus commented Jul 26, 2017

Shalelol commented Jul 26, 2017 • edited Loading

masojus commented Jul 26, 2017

Shalelol commented Jul 26, 2017

coveralls commented Jul 26, 2017

masojus Jul 26, 2017

Choose a reason for hiding this comment

Shalelol Jul 26, 2017

Choose a reason for hiding this comment

masojus Jul 26, 2017

Choose a reason for hiding this comment

masojus Jul 26, 2017

Choose a reason for hiding this comment

masojus Jul 26, 2017

Choose a reason for hiding this comment

masojus Jul 26, 2017 • edited Loading

Choose a reason for hiding this comment

Shalelol commented Jul 26, 2017 • edited Loading

masojus Jul 26, 2017

Choose a reason for hiding this comment

masojus commented Jul 26, 2017

tbarn commented Aug 4, 2017

coveralls commented Aug 11, 2017

masojus commented Aug 11, 2017

Shalelol commented Aug 16, 2017

coveralls commented Aug 17, 2017

coveralls commented Aug 17, 2017

coveralls commented Aug 17, 2017

Shalelol commented Aug 17, 2017

baumatron commented Aug 17, 2017

coveralls commented Aug 18, 2017

coveralls commented Aug 21, 2017

coveralls commented Aug 21, 2017

coveralls commented Aug 21, 2017

Shalelol commented Aug 21, 2017

masojus commented Aug 23, 2017

masojus commented Sep 20, 2017

masojus commented Oct 11, 2017

masojus commented Oct 26, 2017

masojus commented Oct 26, 2017

Shalelol commented Jul 26, 2017 •

edited

Loading

Shalelol commented Jul 26, 2017 •

edited

Loading

masojus Jul 26, 2017 •

edited

Loading

Shalelol commented Jul 26, 2017 •

edited

Loading