Support for org files

[rafamadriz]

How difficult would it be to add support for org files ? Like reading the metadata, for example: #+TITLE: This is a title, #+TAGS: :org:, #+ID: note-id, #+DATE:, etc

How does zk searches for those patterns in a markdown file with the yaml frontamatter ? does it uses regex or some other method to retrieve that data ?

[mickael-menu]

I tried to keep zk independent from Markdown, so I think it wouldn't be super difficult.

• There's a NoteContentParser interface to implement to support additional formats:
  https://github.com/mickael-menu/zk/blob/63dee0ce3f09c7a786fc5d158f411a785bfe1a29/internal/core/note_parse.go#L22-L41
• Everything related to Markdown is under internal/adapter/markdown.
  • It implements NoteContentParser and uses full third-party parsers for Markdown and YAML, not much regexes.
• The NoteContentParser to use is setup in internal/cli/container:
  https://github.com/mickael-menu/zk/blob/63dee0ce3f09c7a786fc5d158f411a785bfe1a29/internal/cli/container.go#L90-L99
• The config related to Markdown is under [format.markdown], so more could be added.

The LSP server might need more work as it is more coupled to Markdown (especially for the auto-completion).

I don't use Org Mode and I'm not familiar at all with its syntax, so I don't think I would implement this myself. But I'm open for external contributions related to this and I'm happy to talk about architecture.

There are many Go libraries parsing Org Mode which could be used: https://github.com/search?q=org+mode+parser+go

[mickael-menu]

By the way I've been toying with the idea of adding a JavaScript plugin architecture in zk. This would be an interesting use case for it.

[rafamadriz]

What do you think about letting people configure how zk searches for the metadata fields that it can use ?

zk doesn't need to parse (or know) the entire syntax of org, markdown or asciidoc, etc. In theory it just needs to recognize whenever it finds any of the metadata that zk can use. This would allow people to use any text format they want, and then in the config file there could be an option to change how zk is going to search for metadata, for example:

[metadata]
title-prefix = "#+TITLE:"

This would let zk know that in order to find a valid title, it needs to have the prefix that was defined in the config file. Or even better, we could allow people to directly pass a regex that is going to be use as a prefix for searching a valid field of metadata.

This would make zk extremely flexible and truly agnostic to use with any file format. The most important benefits I can see are:

1. No longer having to "officially" support any file format.
2. It would work with anything that people want to use. They just need to define their regexes to be use.

The metadata that zk supports is title, date, tags, keywords and aliases. So we could let people define a specific pattern or regex on how zk is going to find valid metadata for each one of these.

By the way the plugin idea is awesome. That could take zk to the next level, may I ask why are you considering javascript as a language for the plugin architecture ? What about lua ? It has proven to be an excellent scripting language for other programs, and it is very small compared to javascript.

Let me know how I can help, my experience with go is none and I'm still learning programming, but with some guidance I could be helpful, or maybe I can help with testing. I truly like zk, it has been a pleasure to use and I'm willing to invest my time to help maintain and make this better.

[mickael-menu]

Prefixes or regexes might be too fragile, as you also need to validate the context, for example that you are inside a YAML frontmatter. It gets even more complicated with lists (e.g. tags). This would also be pretty limited to give a proper support in the LSP server.

So I'd rather add an extension endpoint and let people decide on the best strategy for each format.

may I ask why are you considering javascript as a language for the plugin architecture ? What about lua ? It has proven to be an excellent scripting language for other programs, and it is very small compared to javascript.

I'm open to alternatives, but my reasoning was:

• Much more people know and are comfortable with JavaScript. Lua doesn't even appear on the recent StackOverflow surveys, the last time seems to be in 2018 where it appeared in the most dreaded languages.
• JSON might be the interchange format and this has first-class support in JavaScript.

This is not super high on my roadmap though, there are other stuff I'd like to add to zk like renaming or deleting tags/notes.

Let me know how I can help, my experience with go is none and I'm still learning programming, but with some guidance I could be helpful, or maybe I can help with testing. I truly like zk, it has been a pleasure to use and I'm willing to invest my time to help maintain and make this better.

Thanks! Discussing ideas is always useful. Go is quite easy to learn as it is pretty limited, if you're interested. I actually learned Go for this project. Are there other areas except from the note format that you wish were extensible?

[rafamadriz]

Prefixes or regexes might be too fragile, as you also need to validate the context, for example that you are inside a YAML frontmatter. It gets even more complicated with lists (e.g. tags). This would also be pretty limited to give a proper support in the LSP server.

So I'd rather add an extension endpoint and let people decide on the best strategy for each format.

Yes, you're right, the concern I had with my ideas is that you need to validate some context or set some rules for the prefix/regex so it doesn't match actual body text by mistake. And making any fileformat work with LSP sounds very complicated.

I'm open to alternatives, but my reasoning was:

• Much more people know and are comfortable with JavaScript. Lua doesn't even appear on the recent StackOverflow surveys, the last time seems to be in 2018 where it appeared in the most dreaded languages.
• JSON might be the interchange format and this has first-class support in JavaScript.

Oh that makes sense, in that case Lua could limit the amount of potential plugin creators. Didn't know Lua was that dreaded, I wonder why.

this is not super high on my roadmap though, there are other stuff I'd like to add to zk like renaming or deleting tags/notes.

Oh yes renaming and deleting is definitely needed, but I'd also add moving notes between folders. I have folders in my notebook for specific categories, like cheatsheets, howto, etc. Whenever I move notes I have to do the whole filtering and pipe it into mv, rm, etc.

Thanks! Discussing ideas is always useful. Go is quite easy to learn as it is pretty limited, if you're interested. I actually learned Go for this project. Are there other areas except from the note format that you wish were extensible?

The only reason I asked for the possibility of org support is because the only decent plugin in neovim for managing projects with TODOs, agenda/calendar view, deadlines, schedule, etc. Is nvim-orgmode/orgmode, Which as it names suggests, uses org fileformat. I want all my TODOs, agendas, etc. to be indexed by zk so I can filter and access them in the command file and not only within nvim, emacs.

With the plugin architecture, other people might be able to add the whole TODO, TASK, AGENDA managing. But I'd argue that is much better to just add support for org files, the format has existed for almost 20 years and has already figured out the moving tasks around and semantic meaning between all of it.

The other thing I think would be cool to add is adding non text files to a notebook and being able to link those files to other notes, like images and documents. In my case, I have images that I link to notes, mostly for giving a visual representation of a concept. If zk indexes those, I would be able to do zk list --link-to image.png and see all notes that link to an image. For example kb allows adding non text files.

One final thing I want to ask. I have my notes in version control and I didn't add notebook.db (tho I added my zk config), but when I cloned my notes in another folder for testing, zk was able to figured out the exact same ID's that the notes already had in the other notebook, so I can do zk edit note-id by seeing the ID in the title, I was worried that zk would assign new ID's when creating a new notebook, which would create the confusion of the actual note-id being different from what you see in the title. How does zk assigns the exact same ID you had in another database ? Does it look at the title where the ID is ?

[mickael-menu]

Oh that makes sense, in that case Lua could limit the amount of potential plugin creators. Didn't know Lua was that dreaded, I wonder why.

Maybe the unorthodox syntax or quirks like 1-indexed arrays?

Oh yes renaming and deleting is definitely needed, but I'd also add moving notes between folders. I have folders in my notebook for specific categories, like cheatsheets, howto, etc. Whenever I move notes I have to do the whole filtering and pipe it into mv, rm, etc.

Yes, also an important feature 👍

The other thing I think would be cool to add is adding non text files to a notebook and being able to link those files to other notes, like images and documents. In my case, I have images that I link to notes, mostly for giving a visual representation of a concept. If zk indexes those, I would be able to do zk list --link-to image.png and see all notes that link to an image. For example kb allows adding non text files.

Not the first time this was raised, also a very useful feature, as well as the auto-completion of media links.

One final thing I want to ask. I have my notes in version control and I didn't add notebook.db (tho I added my zk config), but when I cloned my notes in another folder for testing, zk was able to figured out the exact same ID's that the notes already had in the other notebook, so I can do zk edit note-id by seeing the ID in the title, I was worried that zk would assign new ID's when creating a new notebook, which would create the confusion of the actual note-id being different from what you see in the title. How does zk assigns the exact same ID you had in another database ? Does it look at the title where the ID is ?

Ignoring notebook.db in the repository is a good idea. The "source of truth" of a notebook should always be the note themselves and zk rebuilds the database automatically when missing. The DB is currently only used to perform complex queries over the notes.

There's no real concept of "note ID" with zk¹, the note path (relative to the root) is used to identify uniquely a note instead.

When you use zk edit <note-id>, zk will find the note with the path prefix you provide, for example if you have "0123 A note.md", you can open it with zk edit 0123. See this page in the doc for more infos: https://github.com/mickael-menu/zk/blob/main/docs/note-id.md

Footnotes

1. Although zk allows to generate IDs in a template. ↩

[gregoryhugaerts]

I wonder if leveraging pandoc to convert formats on the fly as a preprocess step would be an idea? It'd give the option to use a lot of common formats

[mickael-menu]

It could be a workaround for the parser/indexation. It wouldn't work with the LSP server though.