Roadmap progress thread

[haxscramper]

Thread is created in order to make it easier to keep track of what is being worked on, and what has already been implemented. This is a high-level overview of the current state of the progress - most of the relevant discussions and elaborations should still go in the respective PRs.

Roadmap log should only include things that are actually being worked on - this is not a todo list, but rather an ongoing log that (ideally) should help people realize where exactly we are now in the roadmap progress.

Current state of the near-term plans is show on the graph. It shows the actual state of the development at the moment, and due to the complexity of the overall refactoring might seem overly convoluted - that is not intentional, but is a direct consequence of the difficulty of the overall task.

• Yellow nodes show tasks that are currently in the design stage - we need to have a better understanding of the involved refactoring and/or think up a better API.
• Green nodes represent things that are done
• Orange nodes show entries that are being worked on right now

Related discussions that go into more details of the particular steps

• {.debug.} define and compiler debugging Compiler debugging #197
• `nimpretty` #113
• Collection of ideas for `nim doc` improvements (low priority) #75
• RFC: A builder library to power koch #123
• Collection of ideas for VM improvements #176
• Data Oriented Design for the AST #139

NOTE: the graph is created in SVG format, so you can open it in a separate tab and zoom in/out if you need to read a full description of the element.

The roadmap in the readme represents a long-term strategy that we will follow to the best of our ability, but along the way more problems are discovered that might temporarily change the importance of different tasks, or add new minor dependencies. These changes will be reflected on this graph instead. Graph source

[haxscramper]

Finished and approved code of conduct

• Merged pull request
• Code of conduct text

[haxscramper]

Updated contribution, moderation and coding guidelines

• RP1 and PR2
• Updated documents for the contribution, moderation and style guide

[haxscramper]

[@haxscramper] Implemented structured report system

• Main pull request

  Full rework of the compiler message handling pipeline. Remove old-style message generation that was based purely on strings that were formatted in-place, and instead implement structured logging where main compiler code only instantiates objects with required information.

[haxscramper]

[help wanted] Working on - language specification

Implemented basic sections of the language specification, added toplevel documentation and reasoning for the language specification.

• spec reasoning
• ongoing project
  - Specification tests PR1 #28
  - Specification tests PR2 #43
  - Specification tests PR3 #59

Language specification implementation progress can be split into many smaller contributions, and it will be a long-standing effort to create a comprehensive test suite that can actually ensure the language correctness.

[haxscramper]

[@alaviss] Working on - builder library for koch

• RFC RFC: A builder library to power koch #123
• Implementation PR koch: introduce builder #133

[haxscramper]

[@saem] Working on - testament rework

• Implementation PR Testament: test process batching and internals rework #135
• RFC/ideas discussion Collection of Ideas for Testament #60

[haxscramper]

[@shayanhabibi] Implemented --retry and fixed --failing flags for testament

• Testament - implement --retry and fix --failing #96

Store information about failing tests in order to avoid re-running test that are already passing and speed up iteration time.

[haxscramper]

[@alaviss] Implemented nightlies build

• project
• niminst: add target suffix to binary archive #171
• koch.py: add download bootstrap source command #172
• ci: add macOS into the test matrix #168
• koch: generate zstd archives by default #165
• niminst: generate archive manifest after build #219
• niminst: remove koch detection from build scripts #220
• niminst: set default path for binary to bin/ #221
• test and generate release binaries #226
• verify and publish binaries generated by CI #229

CI releases are in https://github.com/nim-works/nimskull/releases

[haxscramper]

[@haxscramper] implemented - minor testament UX improvements

• add colored text handling library #169
• std: generic diff and levenshtein algorithms #166
• Testament diff rework - unstructured, structured #206

[haxscramper]

After working on the testament UX there are some more fundamental improvements that are discussed in #60 - this will be elevated to a project a bit later.

[haxscramper]

[@haxscramper] working on streamlining testament-compiler interactions and command-line processing

• Refactor .cfg file processing #159
• refactor compiler CLI parser #210
• Make testament actually understand the test specification #211
• Move away from the string for handling executed commands #213

[haxscramper]

[@zerbina] implemented refactoring of the VM data representation and performed associated code cleanup

• 🔥 vm: store data as raw contiguous bytes #242
• Rework stack-frame handling in the VM #255 and related specification Add language specification for finally clause #256
• internal: don't use VM callback for addFloatRoundtrip #308
• Replace PCtx parameters with TCtx in VM code #252
• Improve internal reporting of errors in vm/vmgen #268
• Remove mNSetType magic and associated VM opcode #269
• vm: move ExpandToAst logic into own instruction #281
• vm: improve aliasing of slots under ORC #297
• vm: remove unused opcSubStr #299
• internal: don't use VM callback for addFloatRoundtrip #308
• Collection of ideas for VM improvements #176

Data is now stored in the form of the raw bytes that are interpreted by the VM using information stored in VmType object. This change unblocks further progress for the Data-Oriented design for the AST, because semantic analysis and other AST-centric parts of the compiler are no longer tied with compile-time code evaluation.

[haxscramper]

[@haxscramper] working on parser refactoring

• Decouple parser from PNode with DOD approach #423

  decouple the parser from PNode
  improve parser data types to have a more data oriented approach and get it working

• decouple parser from PNode #425
• parser refactor follow-up code style fixes #428

[haxscramper]

[@saem] is working on semantic analysis refactoring

• partially reworked semProcAux #422
• Sketch a TType replacement for all judgements #420
• internal: add missing reports and clean-up #421
• pragma: remove the deprecated unroll pragma #406
• dialect: remove old case object define #399
• macros: remove deprecated TNimSym/TypeKinds #397
• internal: remove sons from leaf nodes empty/none #392
• cpp: remove this pragma, breaks codegen #398
• pragmas: type pragmas must follow generic params #400
• internal: rewrite semTemplateDef in case style #415

[haxscramper]

[@zerbina] working on - refactoring of the compiler backend

• WIP: a re-implementation of the compiler backend #424

  The overall goal is to design and implement a simple, modular, and data-oriented framework for developing the various back-ends/target with.

[haxscramper]

Closing in favor of https://github.com/nim-works/nimskull/milestones -- it is more automated, can be easily edited. Does not provide so detailed overview of the task dependencies, but I'm not exactly sure if this even necessary.