Architecture discussion

[kasbah]

Moved here from #6, @AbdulrhmnGhanem said:

why do we need git-hosting?
I don't know whether the current version of the site supports versioning for projects or not. Even if we want to support versioning it will be way simpler than git; roughly it will be the content of version X of some board will be Y, the content of version X.1 of some board will be Y.2, and so forth. This can be done as a database table, we don't need fully-fledged version control.

• For users experienced with GitHub, we can make a GitHub app or a plug and play GitHub actions, to make the experience similar to Netlify. I would prefer using a third-party GitHub application than signing up for a new service.
• For other users, we will make them drop the design files and mark each one with its role. This will require some validation to handle all the crazy security issues that come with rendering user content.

Note: I'm pushing back to get more clarity and if possible reach better architecture, not to oppose you, and not because I can't build the project properly!

[kasbah]

• The current version of the site does not support versioning, we always build the latest version. In the future we want to support this though.
• If we do our own versioning, even if it's simpler, we are re-inventing version control. Git is an established standard and protocol for versioning that works pretty well for PCB files. Electronics services like CADLab.io and InventHub.io support it too. Our current user-base are all using Git
• Using a full-fledged version control will let us make more use of it down the line
• Github integration ala Netlify is a good idea, but we can do that and use Gitea as a back-end

For other users, we will make them drop the design files and mark each one with its role. This will require some validation to handle all the crazy security issues that come with rendering user content.

• I don't understand what you mean by this last point. You mean we'll have to really think about security if we implement our own backend from scratch?

On the whole, what it came down to for me is: if we want versioning and we want user accounts we are going to re-implement something Gitea/Gitlab like (I evaluated Gitlab too but prefer Gitea). Re-using it we get tons of developers who are adding features and fixing security issues.

[AbdulrhmnGhanem]

• If we do our own versioning, even if it's simpler, we are re-inventing version control. Git is an established standard and protocol for versioning that works pretty well for PCB files. Electronics services like CADLab.io and InventHub.io support it too. Our current user-base are all using Git.

Whether we use gitea or anything else, users will need to use git(those who know how to use GitHub) or we will do the versioning on their behalf(those who don't know how to use GitHub). Gitea will replace GitHub not git.

• Using a full-fledged version control will let us make more use of it down the line.

There is a software principle called 'YAGNI: You Ain't Gonna Need It'. If we need more features in the future, we worry about it in the future. We don't know how the requirements are going to evolve.

• Github integration ala Netlify is a good idea, but we can do that and use Gitea as a back-end.

Doing this ourselves will is orders of magnitude harder than profiles management; it will need tons of infrastructure and some skilled DevOps engineers.

• I don't understand what you mean by this last point. You mean we'll have to really think about security if we implement our own backend from scratch?

No, we will be vulnerable whether we use gitea or custom backend. Letting users upload files and using these files to build the UI will make us vulnerable to XSS attacks.

Unless you really want to migrate from GitHub, you can make a vote on twitter or something and see the users will prefer sticking to their GitHub accounts or create new accounts on a different platform. User experience plays a major part here not just the technical aspect of it.

[kasbah]

Gitea will replace Github but it will also act as the Git client for users that don't want to use Git by using the Gitea API. It will still let us mirror/sync repositories for users that just want to use Github (or Gitlab, or self-host something).

Current Kitspace is very much built with a YAGNI principle. This re-write and the chosen architecture is a reflection of the features I think we will need to expand the user base and add interesting features. Bringing versioning into the UI is one of those features.

Security concerns are very similar to those of the current architecture I believe, though with user authentication etc. we do have to be more careful. We already process user generated content. The main mitigation right now is filtering the contents of the readme to only safe HTML tags. It's always worth considering if we need to do more though.

I don't really understand the alternative you are proposing and do feel a bit like you are picking holes in this before taking the time to understand it fully.

[AbdulrhmnGhanem]

Import Projects Architecture Proposal

• Gitea supports rendering custom file through external binaries, see here. So it's just a matter of figuring out how to expose the rendered file to NextJS. Though I guess it only outputs HTML
• If we managed to set rendering binaries for all the assets needed by the frontend there will be no need to make a hook on project sync
  • This will also eliminate the need for caching; It generates the assets for newly imported projects, all old projects are already generated

Tasks to generate assets from v1

• page
  • makePage
    • Is this responsible for rendering a project page? If so then there's no need for it in the new architecture
  • makeReadme
    • Probably it renders the project markdown/rst to html.
    • Gitea supports rendering custom file through external binaries, see here. So it's just a matter of figuring out how to expose the rendered file to NextJS
  • plot_kicad_grebers
    • It's a simple python script with a external dependency kicad
  • processBOM
    • It depends on the 1-click-bom . There's a go library for generating the bom. Pleas, check it to see whether it provides the functionality required to fulfill this task
  • ProcessGrebers
    • It depends on Inkscape and probably it also compresses the output. Mostly it's external command. I guess it's easy to be ported to go
• global
  • makeBoardsJson
    • The frontend already can access all the repos on gitea.
    • It depends on parseProject which is plain logic which can be ported to go.
  • makeBOMBuilder
    • The page is already done in the v2
  • makeIndex
    • Currently, the index can access all the repos
    • On each project import, we can generate the boardInfo.json for that project and commit it with user files.
    • boardInfo.json will contain the thumbnail, URL, and description; the info required to make the make board card.
    • We can have a reserved repo, that contain all the boardInfo.json for all boards, similar to boards.json
  • makeNewsletter
  • makeOneClickBOM
    • The page is already done in the v2
  • makeSearchIndex
    • Use gitea search if it turned out not that good we can typesense
  • makeSubmit
    • There's /projects/new route in v2

[kasbah]

Thanks for the detailed write up. I think it's time to make a proof of concept using a custom renderer. One problem I foresee is that we don't only have a straight-forward 1-file to 1-file mapping: we need to transform multiple Gerber files into the front and back SVGs.

processBOM

It depends on the 1-click-bom . There's a go library for generating the bom. Pleas, check it to see whether it provides the functionality required to fulfill this task

1-click-bom is very specific to Kitspace and handles more than just KiCad, it's used in every tool, it defines our interchange format for BOMs. Better to just keep using it.

On each project import, we can generate the boardInfo.json for that project and commit it with user files.

I think it's a mistake to generate anything just on "import", a user can upload new files directly or through syncing with a mirrored repo. We need to get this info from the repo in its current state.

This is complicated by the fact that we allow a single git repo to be the source of multiple project pages, how should we handle this?

[AbdulrhmnGhanem]

One problem I foresee is that we don't only have a straight-forward 1-file to 1-file mapping: we need to transform multiple Gerber files into the front and back SVGs.

I don't understand this!

1-click-bom is very specific to Kitspace and handles more than just KiCad, it's used in every tool, it defines our interchange format for BOMs. Better to just keep using it.

Yeah, I took a look at it. We can interface it with go. In the future, I think it should be re-written in go if javascript turned out to be a bottleneck.

I think it's a mistake to generate anything just on "import", a user can upload new files directly or through syncing with a mirrored repo. We need to get this info from the repo in its current state.

I expressed this badly what I meant is any change in the repo. We can't depend on getting the current state as this means on each request to a project page, it has to generate all the assets. Instead, it should generate it ahead-of-time, on any file change. Maybe later, we can add a manual trigger option.

This is complicated by the fact that we allow a single git repo to be the source of multiple project pages, how should we handle this?

How the multi-part projects are currently handled? can't we use the same approach?

[kasbah]

One problem I foresee is that we don't only have a straight-forward 1-file to 1-file mapping: we need to transform multiple Gerber files into the front and back SVGs.

I don't understand this!

The renderer in Gitea assumes you want to render a single file as something else, e.g. a .aciidoc as HTML. We actually need to read in multiple Gerber files and turn them into a single SVG so we don't perfectly fit into the model as it stands.

it should generate it ahead-of-time, on any file change. Maybe later, we can add a manual trigger option.

I agree with the rest of it, but I disagree with having a manual trigger. We should be able to generate assets for every Git version and trigger a rebuild whenever the repo changes. Not sure what a manual trigger would add.

How the multi-part projects are currently handled? can't we use the same approach?

We basically split them at some point. The issue is, with v2 we don't want to split them on import, because that will break the ability to use Git to sync. So I guess we need to split them when reading from our own Git. It does complicate things a bit, but it should be possible.

[AbdulrhmnGhanem]

How are these assets going to be stored? Committing them to the same repo won't work as this will break the repo mirror case.

[kasbah]

Where are the outputs of the Gitea custom renders stored? I suspect for the initial version we just use the file system. Then later to reduce costs we can store them on AWS S3.

[AbdulrhmnGhanem]

Where are the outputs of the Gitea custom renders stored? I suspect for the initial version we just use the file system. Then later to reduce costs we can store them on AWS S3.

I don't see gitea fronted gets files directly so I guess there's some templating is happening behind the scene though I haven't looked at any source code for custom rendering.

Thinking about how to update files made me realize that we need a kitspace user account which will mostly be a coating on top of the gitea account. One of the accounts base use cases will be displaying his current project and adding the option for updating them.

[kasbah]

Like I said before, I think it should be seemless. No one should ever need to manually update their project.

[AbdulrhmnGhanem]

Then if a user updated the project files and tries to push these updates to Kitspace how will this be handled? How to differentiate between updating an exiting project and creating a new one?

[kasbah]

What you said before makes me think you are envisioning some kind of button: the user has to push to the repository and then press the "update button". I am saying don't have a button, always update.

The way I like to think of it: the Kitspace page is a rendering of repository. Including the latest push of course. A user git pushes, the page updates.

Not sure what updating vs creating has to do with it.

[AbdulrhmnGhanem]

I have uploaded some files(not mirror), I want to update these files, how should I do it? Taking into account I know nothing about version control. In mirror case, it's rendering the latest commit form the default branch

…

On Sat, Nov 7, 2020, 12:50 AM Kaspar Emanuel ***@***.***> wrote: What you said before makes me think you are envisioning some kind of button: the user has to push to the repository and then press the "update button". I am saying don't have a button, always update. The way I like to think of it: the Kitspace page is a rendering of repository. Including the latest push of course. A user git pushes, the page updates. Not sure what updating vs creating has to do with it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3OMSN4SFGTZNLZUYMTMWLSOR4UHANCNFSM4PKXUGSQ> .

[kasbah]

Oh right, I think we should have a "edit project" page where you can edit, update files and add new files to a project. We can hopefully jump to this "edit project" page, from the /projects/new page as soon as we have created a project for the user, i.e. created project and added the first file.