How can we reduce the boilerplate code for repetitive class signatures and method override?

[Nefcanto]

I'm writing an ETL solution. It has hundreds of small ETL classes.

AttributeImporter.cs
BrandSeoImporter.cs
CategorySeoImporter.cs
CategoryImporter.cs
ProductAttributeImporter.cs
PageImporter.cs
ArticleImporter.cs
ArticleContentImporter.cs
BrandImporter.cs
ProductImporter.cs
ProductCategoryImporter.cs
BrandContentImporter.cs
ProductImageImporter.cs
...

And basically, the content of each file follows this formula:

public XImporter : Importer
{
    public override void Import()
    {
        // ETL code for this X
    }
}

Is there a way for us to just write the // ETL code for this X and commit those 6 lines of boilerplate? Something similar to the top-level statements, but for all of these files.

[Nefcanto]

This is a real example:

public class AttributeImporter : Importer
{
    public void Import()
    {
        var productModelDetails = GetSourceList(@$"
            select distinct ProductDetailsName
            from ProductModelDetails
        ");
        RunParallel(productModelDetails, productModelDetail =>
        {
            var translation = new InfraTranslator().Translate(productModelDetail.ProductDetailsName);
            var key = KeyHelper.GetCleanKey(translation);
            var slug = SlugHelper.GetCleanSlug(translation);
            var query = @$"
                insert into Attributes.Attributes
                (
                    `Key`,
                    Slug,
                    Module,
                    EntityType,
                    Locale,
                    DataType,
                    Title
                )
                values
                (
                    ""{key}"",
                    ""{slug}"",
                    ""products"",
                    ""product"",
                    ""fa"",
                    ""{DataType.Text.ToString().Camelize()}"",
                    ""{productModelDetail.ProductDetailsName}""

                )
                on duplicate key update
                    `Key` = values(`Key`),
                    Slug = values(Slug),
                    Module = values(Module),
                    EntityType = values(EntityType),
                    Locale = values(Locale),
                    Title = values(Title)
            ";
            RunOnTarget(query);
        });
    }
}

Is it possible for us to convert it to:

var productModelDetails = GetSourceList(@$"
    select distinct ProductDetailsName
    from ProductModelDetails
");
RunParallel(productModelDetails, productModelDetail =>
{
    var translation = new InfraTranslator().Translate(productModelDetail.ProductDetailsName);
    var key = KeyHelper.GetCleanKey(translation);
    var slug = SlugHelper.GetCleanSlug(translation);
    var query = @$"
        insert into Attributes.Attributes
        (
            `Key`,
            Slug,
            Module,
            EntityType,
            Locale,
            DataType,
            Title
        )
        values
        (
            ""{key}"",
            ""{slug}"",
            ""products"",
            ""product"",
            ""fa"",
            ""{DataType.Text.ToString().Camelize()}"",
            ""{productModelDetail.ProductDetailsName}""

        )
        on duplicate key update
            `Key` = values(`Key`),
            Slug = values(Slug),
            Module = values(Module),
            EntityType = values(EntityType),
            Locale = values(Locale),
            Title = values(Title)
    ";
    RunOnTarget(query);
});

[colejohnson66]

An abstract class with a virtual method?

[En3Tho]

First of all, would saving these few characters really help you there? This real example of yours is just showing that the boilerplate is 1/15 of code written. Just make a template then ctrl-c ctrl-v and you win.

How do you declare what type it is in a file? Top level file implicitly assumes it is a Program.Main. You can't just implicitly assume something like this for an arbitrary class

What if there is a second method? What if method is changed to be generic?

[Nefcanto]

@En3Tho, those what if statements are all valid. Yet for this very specific case, we know that we won't get there. We have written a lot of ETL solutions and for each over a couple of iterations we have come to this design. A simple Import method with something near 20 up to 50 lines of code in it. That's why we ask.

It's not just "1/15". It's also a lot of indentations. That's another point.

[HaloFour]

then could you please tell me why the C# team removed the extra indentation of the namespace and the extra indentation of the using blocks?

Because the team deemed it to be worth it. However, they have expressed that they do not feel that flattening a class declaration within a file to be worth it.

If you want less levels of indentation, change the formatter to not require it. The language isn't going to change its syntax because you have a particular stylistic preference. Notably, method-scoped using still requires a declaration.

[chrisoverzero]

public class AttributeImporter : Importer
{
    public void Import()
    {
        var productModelDetails = GetSourceList("""
select distinct ProductDetailsName
from ProductModelDetails
""");
        RunParallel(productModelDetails, productModelDetail =>
        {
            var translation = new InfraTranslator().Translate(productModelDetail.ProductDetailsName);
            var key = KeyHelper.GetCleanKey(translation);
            var slug = SlugHelper.GetCleanSlug(translation);
            var query = $"""
insert into Attributes.Attributes
(
  `Key`,
   Slug,
   Module,
   EntityType,
   Locale,
   DataType,
   Title
)
values
(
  "{key}",
  "{slug}",
  "products",
  "product",
  "fa",
  "{DataType.Text.ToString().Camelize()}",
  "{productModelDetail.ProductDetailsName}"
)
on duplicate key update
  `Key` = values(`Key`),
  Slug = values(Slug),
  Module = values(Module),
  EntityType = values(EntityType),
  Locale = values(Locale),
  Title = values(Title)
""";
            RunOnTarget(query);
        });
    }
}

A lot of indentation can be omitted without sacrificing organizational culture.

[Nefcanto]

@HaloFour, could you please give me a link to where the C# team expressed they don't seem flattening a class worth it?

[Nefcanto]

@chrisoverzero, that code is not readable. Honestly that's not a good solution.

[HaloFour]

#4722 (comment)

[Nefcanto]

@En3Tho, I thought of CSharp scripts. But I don't know how to reuse methods there. Maybe we can define some global methods. Instead of inheritance, we can have a lot of global static methods.

[colejohnson66]

You can have an ImporterHelpers class (possibly internal) that implementations defer to:

public void Import()
{
    ImporterHelpers.Import(<some args to handle variations>);
}

[Nefcanto]

@colejohnson66 I did not understand. Can you please elaborate?

[iam3yal]

I understand what you're trying to achieve, but based on the code you shared, there might be several alternative solutions that don’t necessarily involve C#. Sometimes insisting on a C# solution can limit the possibilities. In my opinion exploring other approaches could potentially offer better ways to solve the problem, e.g., Create a PowerShell module that exports some of the infrastructure methods to run the queries or whatever and make an 'PowerShellImporter' that basically loads the PowerShell environment and execute these files, then you would be dealing only with the variations.

[Nefcanto]

@iam3yal, that sounds like a possible solution. Though I'm working on Linux and I don't know Powershell. But I assume that what you tell me is to load the simple script myself, dynamically wrap it inside boilerplate codes, dynamically add required assemblies to it and after dynamically compiling it, run it.

Am I right?

[iam3yal]

No. You don't need to dynamically load assemblies or dynamically compile anything, you have to change these variations to PowerShell scripts and then have a single Importer class that based on the name of the file or however you want to store it can execute the given importer. PowerShell supports all major OS.

[Nefcanto]

@iam3yal, can you please explain more? Or give me a simple pseudo-code?

[iam3yal]

@Nefcanto Sorry for the late reply! You can read about PowerShell binary modules here, then you can add the the necessary references or whatever as you would normally to the assembly, then to interoperate you can host PowerShell see here, then what's left is just to load the module into the hosted environment and execute whatever script you like., just for clarification the module should export your infrastructure methods into the PowerShell (PS) environment so you can use it inside the scripts, it's possible to call .NET methods directly too and avoid create a PS module but I feel it would be much more cleaner with a module and that's exactly what you're trying to do. Good luck! I hope it helps. :)

[Nefcanto]

Another part of our application that is exactly like this one, is our Migration engine. Basically, our developers write code like this:

public class Flows1DropSmsColumns : Migrator
{
    public void Apply()
    {
        DropColumn("Flows", "Flows", "SmsParametersProviderMethod");
        DropColumn("Flows", "Stages", "HasNotification");
        DropColumn("Flows", "Stages", "SmsKey");
        DropColumn("Flows", "Stages", "SmsParametersProviderMethod");
    }
}

So, a migratory code is just a single class, inside a single file, with a single method called Apply and inside it, we have a bunch of statements. And based on our current codebase (more than 200K LoC) we never go beyond a single method and never more than 30 lines.

So wouldn't it be nicer if we could simply write the body without the boilerplate?

DropColumn("Flows", "Flows", "SmsParametersProviderMethod");
DropColumn("Flows", "Stages", "HasNotification");
DropColumn("Flows", "Stages", "SmsKey");
DropColumn("Flows", "Stages", "SmsParametersProviderMethod");

[chrisoverzero]

When you: […] Then you should be able to omit those things from the source code altogether.

Is the class public, internal, &c? Is it sealed, static, partial, &c? Is it a class at all, or is it a struct, interface, &c? Is it the record kind? (And what about primary constructors?) What does it inherit or implement? Does it have attributes applied?

When I press F2 in my editor to rename a class, would you expect the IDE to change the source file’s name? Turned the other way, what if the filename contains characters which can’t be used in a class name? (And if the answer is a compilation error, on what line of what file is the error reported?)

Are you sure you would want directory structure to determine namespace? It seems to me that it would be far too easy accidentally to change the public API of a library.

[Nefcanto]

@chrisoverzero, you are absolutely right. All those cases apply. I'm not saying that we should easily forget all those extra parameters.

What I say though, is that when all those parameters are clear, and they are predetermined and won't change, then there should be a way to reduce the boilerplate.

I should be able to somehow (outside each file, once, in a central place) tell the C# compiler that:

1. Hey, everything in this directory is similar
2. They are all classes
3. They all inherit from the same base class
4. They all have one and only one method that they override from that base class
5. The name of the file equals the name of the class
6. And file names are all only alphanumeric ASCII

We are repeating those 6 criteria, for a lot of projects and in a lot of files. I again invite you to remember the reason behind global using. We could specify using System; in thousands of files. But since it's used almost in all files, then it's better to have a way to write it only once.

We have ETL and Migration code. Both are abundant. Both follow strict rules. They matured over years, maybe a decade, so they won't change. We just want to be able to reduce 4 lines of boilerplate for all and reduce 2-level of indentations for all lines of code. That's a huge improvement because we have thousands of these files. That's why I asked for it.

We create tailored software for our customers. Hundreds of customers. For each, we should write an ETL to transfer their data to our platform. Each would end up with tens of ETL files. And for all, we have migration code to update databases. Hundreds of migration files. It's literally thousands of files.

[chrisoverzero]

This sounds like a source generator to me.

[Nefcanto]

@chrisoverzero, I thought of that. But I could not figure it out. Could you please help? I should have a partial class, called GeneralImporter that inherits from Importer:

public partial class GeneralImporter : Importer
{
    public override void Import()
    {
    }
}

Then I write my code in my file the way I want: For example in the BrandImporter.cs file I write:

var sourceBrands = GetSourceList(@$"
    select *
    from Brands
");
RunParallel(sourceBrands, sourceBrand =>
{
    // importing code here
});

Then I create a source generator to read this file, embed it in a file called GeneralImporterGenerated.cs or something like that? I'm not sure where to go from here.

[iam3yal]

@Nefcanto This repo isn't the right place for these sorts of asks, it's much more appropriate to look for "help" on Discord or SO.

[zms9110750]

I think blazor's grammar is good. Let's introduce a template file again.

@template Importer.Import()

 // ETL code for this X