Support for .NET Standard 2.0 #5

mhowlett · 2018-03-11T23:53:31Z

System.CodeDom is what we needed to be able to target netstandard2.0 (for .net core compatibility). It's been available for 7 months, but I only just realized it's here.

The tests use NUnit 2.6.4, not the lates since NUnit 3.x is a complete rewrite, and it's quite an effort to convert all the tests over. For future reference, to run tests, I used this: https://github.com/nunit/nunit-console/releases/tag/3.8

Unfortunately, there are 11 test failures, so we shouldn't merge until we get to the bottom of them. This is unexpected - there were no code changes required to get the library compiling, and it should have just worked.

mhowlett · 2018-03-15T23:11:29Z

Some outstanding issues:

The avrogen.exe tool build is failing with the following on mac os and linux (however, this is just a build problem, the executable works fine on all platforms):

/usr/local/share/dotnet/sdk/2.1.101/Microsoft.Common.CurrentVersion.targets(2991,5): error MSB3552: Resource file "**/*.resx" cannot be found. [/git/avro/lang/csharp/src/apache/codegen/Avro.codegen.csproj]

Some of the Confluent.Kafka.Avro.Integration tests are failing with errors like the below (a type loading error). However, the AvroSpecific and AvroGeneric examples work as expected (and test the same functionality).

Failed   Confluent.Kafka.Avro.IntegrationTests.Tests.ProduceConsume(bootstrapServers: "10.200.7.144:9092", schemaRegistryServers: "10.200.7.144:8081")
Error Message:
 System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
   at Avro.Specific.ObjectCreator.FindType(String name, Boolean throwError)
   at Avro.Specific.ObjectCreator.GetType(String name, Type schemaType)
   at Avro.Specific.ObjectCreator.New(String name, Type schemaType)
   at Avro.Specific.SpecificDefaultReader.ReadRecord(Object reuse, RecordSchema writerSchema, Schema readerSchema, Decoder dec)
   at Avro.Generic.DefaultReader.Read(Object reuse, Schema writerSchema, Schema readerSchema, Decoder d)
   at Avro.Generic.DefaultReader.Read[T](T reuse, Decoder decoder)
   at Avro.Specific.SpecificReader`1.Read(T reuse, Decoder dec)
   at Confluent.Kafka.Serialization.SpecificDeserializerImpl`1.Deserialize(String topic, Byte[] array) in /git/confluent-kafka-dotnet/src/Confluent.Kafka.Avro/SpecificDeserializerImpl.cs:line 128
   at Confluent.Kafka.Serialization.AvroDeserializer`1.Deserialize(String topic, Byte[] data) in /git/confluent-kafka-dotnet/src/Confluent.Kafka.Avro/AvroDeserializer.cs:line 106
   at Confluent.Kafka.Serialization.MessageExtensions.Deserialize[TKey,TValue](Message message, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer) in /git/confluent-kafka-dotnet/src/Confluent.Kafka/Serialization/Extensions/Message.cs:line 58
Expected: True
Actual:   False
Stack Trace:
   at Confluent.Kafka.Avro.IntegrationTests.Tests.<>c.<ProduceConsume>b__3_2(Object o, Message e) in /git/confluent-kafka-dotnet/test/Confluent.Kafka.Avro.IntegrationTests/Tests/ProduceConsume.cs:line 88
   at Confluent.Kafka.Consumer`2.Consume(Message`2& message, Int32 millisecondsTimeout) in /git/confluent-kafka-dotnet/src/Confluent.Kafka/Consumer.cs:line 167
   at Confluent.Kafka.Consumer`2.Consume(Message`2& message, TimeSpan timeout) in /git/confluent-kafka-dotnet/src/Confluent.Kafka/Consumer.cs:line 179
   at Confluent.Kafka.Consumer`2.Poll(TimeSpan timeout) in /git/confluent-kafka-dotnet/src/Confluent.Kafka/Consumer.cs:line 219
   at Confluent.Kafka.Avro.IntegrationTests.Tests.ProduceConsume(String bootstrapServers, String schemaRegistryServers) in /git/confluent-kafka-dotnet/test/Confluent.Kafka.Avro.IntegrationTests/Tests/ProduceConsume.cs:line 98
[xUnit.net 00:00:48.0957330]     Confluent.Kafka.Avro.IntegrationTests.Tests.SharedSchemaRegistryClient(bootstrapServers: "10.200.7.144:9092", schemaRegistryServers: "10.200.7.144:8081") [FAIL]

mhowlett · 2018-03-19T23:06:16Z

lang/csharp/src/apache/codegen/Avro.codegen.csproj

-  <PropertyGroup>
-    <AssemblyOriginatorKeyFile>..\..\..\Avro.snk</AssemblyOriginatorKeyFile>
+    <VersionPrefix>1.7.7.4</VersionPrefix>
+    <TargetFramework>netcoreapp2.0</TargetFramework>


when 2.1 comes out of preview, we'll target 2.1 and PackAsTool == true.

edenhill

Hard to review csproj files and get an overall picture, but I tried to pick at low hanging fruit.

Removing licenses from files seems questionable.

Some references to the dotnet client.

Otherwise LGTM

edenhill · 2018-03-20T05:47:59Z

lang/csharp/src/apache/codegen/Avro.codegen.csproj

@@ -1,166 +1,24 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!--


why are the license headers removed in this file and others?

A mistake, glad you caught that one.

edenhill · 2018-03-20T05:48:56Z

lang/csharp/src/apache/ipc/Avro.ipc.csproj

+    <Description>Fork of Apache.Avro.ipc maintained by Confluent</Description>
+    <Copyright>Apache Foundation</Copyright>
+    <PackageIconUrl>https://raw.githubusercontent.com/confluentinc/confluent-kafka-dotnet/master/confluent_logo.png</PackageIconUrl>
+    <PackageReleaseNotes>https://github.com/confluentinc/confluent-kafka-dotnet/releases</PackageReleaseNotes>


should probably point to avro repo, not dotnet client?

whatever logic I was using to make that decision was not sound.

edenhill · 2018-03-20T05:49:32Z

lang/csharp/src/apache/main/Avro.main.csproj

+    <Description>Fork of Apache.Avro maintained by Confluent</Description>
+    <Copyright>Apache Foundation</Copyright>
+    <PackageIconUrl>https://raw.githubusercontent.com/confluentinc/confluent-kafka-dotnet/master/confluent_logo.png</PackageIconUrl>
+    <PackageReleaseNotes>https://github.com/confluentinc/confluent-kafka-dotnet/releases</PackageReleaseNotes>


edenhill · 2018-03-20T05:50:01Z

lang/csharp/src/apache/main/CodeGen/CodeGen.cs

@@ -840,7 +842,7 @@ public virtual void WriteTypes(string outputdir)
                for (int j = 0; j < types.Count; j++)
                {
                    var ctd = types[j];
-                    string file = dir + "\\" + CodeGenUtil.Instance.UnMangle(ctd.Name) + ".cs";
+                    string file = Path.Combine(dir, CodeGenUtil.Instance.UnMangle(ctd.Name) + ".cs");


edenhill · 2018-03-20T05:50:45Z

lang/csharp/src/apache/msbuild/Avro.msbuild.csproj

+    <Description>Fork of Apache.Avro.msbuild maintained by Confluent</Description>
+    <Copyright>Apache Foundation</Copyright>
+    <PackageIconUrl>https://raw.githubusercontent.com/confluentinc/confluent-kafka-dotnet/master/confluent_logo.png</PackageIconUrl>
+    <PackageReleaseNotes>https://github.com/confluentinc/confluent-kafka-dotnet/releases</PackageReleaseNotes>


edenhill · 2018-03-20T05:50:58Z

lang/csharp/src/apache/perf/Avro.perf.csproj

+    <Description>Fork of Apache.Avro.perf maintained by Confluent</Description>
+    <Copyright>Apache Foundation</Copyright>
+    <PackageIconUrl>https://raw.githubusercontent.com/confluentinc/confluent-kafka-dotnet/master/confluent_logo.png</PackageIconUrl>
+    <PackageReleaseNotes>https://github.com/confluentinc/confluent-kafka-dotnet/releases</PackageReleaseNotes>


edenhill · 2018-03-20T05:51:10Z

lang/csharp/src/apache/test/Avro.test.csproj

+    <Description>Fork of Apache.Avro.test maintained by Confluent</Description>
+    <Copyright>Apache Foundation</Copyright>
+    <PackageIconUrl>https://raw.githubusercontent.com/confluentinc/confluent-kafka-dotnet/master/confluent_logo.png</PackageIconUrl>
+    <PackageReleaseNotes>https://github.com/confluentinc/confluent-kafka-dotnet/releases</PackageReleaseNotes>


edenhill · 2018-03-20T05:51:37Z

lang/csharp/src/apache/test/Schema/SchemaNormalizationTests.cs

@@ -50,15 +50,15 @@ public void FingerprintTest(string input, string expectedOutput)

        private static List<object[]> ProvideFingerprintTestCases()
        {
-            using (StreamReader reader = new StreamReader("../../../../../share/test/data/schema-tests.txt"))
+            using (StreamReader reader = new StreamReader("../../../../../../../../share/test/data/schema-tests.txt"))


So many dots on the dance floor!

TBH this could just be an artifact of how I'm running the tests, but the tests all need to be upgraded to work well with new tooling anyway, so I didn't spend much time thinking about it.

Matt Howlett and others added 6 commits March 9, 2018 16:12

-> standard2.0 / netcoreapp2.0+net452

5772787

changes to test project

7a2ce64

make tests pass

c219daf

updated project descriptions

7f37b82

target net452;netstandard2.0, make netstandard deps conditional

5b5d95b

update avrogen project to create 2.1 tool package

3860020

Matt Howlett added 2 commits March 15, 2018 16:58

fixed cross-platform path creation issue in CodeGen.

a59dd1b

shorten avrogen assembly name

d2f5cc4

This was referenced Mar 16, 2018

Avro serde support for .net standard 2.0 confluentinc/confluent-kafka-dotnet#465

Merged

avro and schema registry support confluentinc/confluent-kafka-dotnet#357

Merged

2.1 -> 2.0 remove PackAsTool

95e5285

mhowlett commented Mar 19, 2018

View reviewed changes

mhowlett requested a review from edenhill March 19, 2018 23:10

edenhill suggested changes Mar 20, 2018

View reviewed changes

Matt Howlett added 2 commits March 20, 2018 09:10

added back apache license comments to project files

06fe46e

package release notes -> avro not confluent-kafka-dotnet

b9bb9fb

mhowlett merged commit 07b4341 into confluentinc:confluent-fork Mar 20, 2018

mhowlett mentioned this pull request Aug 19, 2019

Are you interested in using the official Apache.Avro NuGet package? confluentinc/confluent-kafka-dotnet#1033

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for .NET Standard 2.0 #5

Support for .NET Standard 2.0 #5

mhowlett commented Mar 11, 2018

mhowlett commented Mar 15, 2018 •

edited

Loading

mhowlett Mar 19, 2018

edenhill left a comment

edenhill Mar 20, 2018

mhowlett Mar 20, 2018

edenhill Mar 20, 2018

mhowlett Mar 20, 2018

edenhill Mar 20, 2018

edenhill Mar 20, 2018

edenhill Mar 20, 2018

edenhill Mar 20, 2018

edenhill Mar 20, 2018

edenhill Mar 20, 2018

mhowlett Mar 20, 2018

		@@ -1,166 +1,24 @@
		<?xml version="1.0" encoding="utf-8"?>
		<!--

Support for .NET Standard 2.0 #5

Support for .NET Standard 2.0 #5

Conversation

mhowlett commented Mar 11, 2018

mhowlett commented Mar 15, 2018 • edited Loading

Choose a reason for hiding this comment

edenhill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhowlett commented Mar 15, 2018 •

edited

Loading