Developing v0.9.0 (Rewriting the app in ratatui)

[Builditluc]

This issue tracks the development and discussion of v0.9.0. For discussing/submitting bugs in the development version, please see #234.

I'm planning to get the version released before the end of december. Currently, the most work is needed in bug-testing (and subsequent fixing) and writing the documentation for the new version (changelog, migration guide from 0.8.x, documenting the changes, etc.).

The documentation (in its current state) for v0.9.0-pre can be viewed here: https://wiki-tui.net/develop/

So please feel free to try out the dev version and report any bugs you find.

[Redhawk18]

Why the switch? I found your project through the cursive's showcase section. I'm starting my own tui now so I'd like to know.

[Builditluc]

Mainly because of the lack of configuration cursive gives us to programmatically modify the look of certain UI components. For example, we cannot change the color of the border around views and changing predefined keybindings across the app is tedious.

If you're unsure about what library to use, I can recommend reading the comparison page from the cursive wiki, because the libraries focus on different key aspects.

If you want to have a more complex ui with deeply nested view trees and popup windows, and are fine with the sane defaults cursive sets, I would recommend cursive over ratatui (formerly tui-rs).

[0323pin]

@Builditluc Long time, no see 😃

I've just came by to check on the progress and saw the recent commits. Are you getting closer now? Looks like it will be loaded with new features, configurable looks ... Looking forward to take it for a spin.

Regards

[Builditluc]

Hey @0323pin!

Yes I'm getting closer. The app should now have all of the features it had before (and a few new ones). Originally, I wanted to rewrite the app and release a new version before working on new things, but now I'm developing a few new features before releasing.

I didn't have a lot of time generally for programming the last few months (did my finals, etc.) so most of work you see was done in the last few weeks, as I now have a lot more time.

If you want, I'll notify you when the version is almost done (when all of the features are there and the polishing begins), so you can try it out! I would appreciate any feedback from your side!

fyi, a bit more about the UI

I'm currently working on improving the UI for searches. The plan is to have it working like the search on the fandom.com site (wikipedia works also that way, but for the example I'm using fandom).

For example, when searching initially it recommends articles based on your query and immediately opens the page when your query matches something. (like this)

If it didn't find anything or you want to see all results, it switches to a more detailed search page where you can see summaries, more context of individual search results (revision id, when was it last edited, etc.) and pagination (I don't like the current approach of just appending more search results when you want to continue).

[0323pin]

If you want, I'll notify you when the version is almost done (when all of the features are there and the polishing begins), so you can try it out! I would appreciate any feedback from your side!

Sure, bring it on. I'm all in for testing.
Specially now, that with ratatui, I most probably, won't need a dependency on ncurses 😄

[0323pin]

@Builditluc How is it going? Hopefully getting there.

Just to let you know that the latest published release fails to build with Rust > 1.79 because of the time crate.
This need to be at least v0.3.36 else builds fail with:

   |
83 |     let items = format_items
   |         ^^^^^
...
86 |     Ok(items.into())
   |              ---- type must be known at this point
   |
help: consider giving `items` an explicit type, where the placeholders `_` are specified
   |
83 |     let items: Box<_> = format_items
   |              ++++++++

Regards

[Builditluc]

Hey @0323pin, please excuse my delayed response.
Yes I'm getting there. Thank you for the notice, fortunately, our current (dev-version) dependencies don't seem to use the time crate, so it should build just fine with rust 1.80.

[0323pin]

Hey @0323pin, please excuse my delayed response.

you are forgiven 🤣

Yes I'm getting there. Thank you for the notice, fortunately, our current (dev-version) dependencies don't seem to use the time crate, so it should build just fine with rust 1.80.

That's good but, it doesn't fix the broken builds.
Now, stable packages for 2024-Q3 are fine, as that shipped with Rust-1.79

I'm hopeful that a new release will happen before end of December, cause than, stable 2024-Q4 will also be fine.
I can ignore the breakage in between.

Regards

[0323pin]

@Builditluc Trying the new 0.9.0-pre 😃

How to scroll down? pagedown doesn't seem to be working.

[Builditluc]

Hey @0323pin, please excuse my delayed response.

you are forgiven 🤣

Yes I'm getting there. Thank you for the notice, fortunately, our current (dev-version) dependencies don't seem to use the time crate, so it should build just fine with rust 1.80.

That's good but, it doesn't fix the broken builds. Now, stable packages for 2024-Q3 are fine, as that shipped with Rust-1.79

I'm hopeful that a new release will happen before end of December, cause than, stable 2024-Q4 will also be fine. I can ignore the breakage in between.

Regards

I will try my best to release a new version before the end of December! The rewrite should already be feature complete (except for the configuration, I still haven't tackled how the configuration show look, structure and feature wise. Maybe you have some suggestions?).

The part where the most work is needed is for bug-testing/fixing and the documentation (changelog, website updates, etc.). I would be very grateful for any bug-reporting for things I've missed re-implementing (like pagedown/up) or that are broken (I have written my own event system and since I've never done that before, there are bound to be some bugs in there).

[Builditluc]

@Builditluc Trying the new 0.9.0-pre 😃

How to scroll down? pagedown doesn't seem to be working.

The basic vim keys (h/j/k/l and ctrl+d/ctrl+u, aswell as g and G) are implemented. It seems I just forgot to add pagedown to the list. My bad!

EDIT: Also home and end don't work (yet)

[Builditluc]

@0323pin bound the keys to their scrolling action (50a34cc)

[0323pin]

bound the keys to their scrolling action (50a34cc)

Thank you! I should have looked for it and fix that.

I will try my best to release a new version before the end of December!

I can package an alpha or, beta if, that's an option. Don't want to force a release before it's ready.

I still haven't tackled how the configuration show look, structure and feature wise. Maybe you have some suggestions?

Actually, I was pretty happy with the available configuration options on the old 0.8.x release series.
Maybe now, that it's using ratatui the options are larger. But, once you have users suggestion will come naturally.
I'm sure, I can start to find things when using it. I think the most important is a configuration example. As long as I can configure the language fonts and colors, I'm happy.

The part where the most work is needed is for bug-testing/fixing and the documentation (changelog, website updates, etc.). I would be very grateful for any bug-reporting for things I've missed re-implementing (like pagedown/up) or that are broken (I have written my own event system and since I've never done that before, there are bound to be some bugs in there).

I can rebuild from HEAD tomorrow and test it. Will report issues as they come-up.

[Builditluc]

I can package an alpha or, beta if, that's an option. Don't want to force a release before it's ready.

Yes you're right.

Maybe now, that it's using ratatui the options are larger

ratatui uses a builder-like structure for creating most of the views, meaning we can basically configure everything. Color-wise, I've already implemented a theming system (its not documented, yet, but the loading code is found here). It provides a few options for changing the global colors (which can be expanded pretty easily)

I can rebuild from HEAD tomorrow and test it. Will report issues as they come-up.

Thank you, I'm currently working on documenting all of the things I've changed/added

[0323pin]

@Builditluc Let me know if you would prefer to start a new issue to discuss found bugs/issues on a new thread.

New build from 50a34cc

Starting the TUI without a theme file shows the following:

which, is fine. The problem is, if one presses ESC to Dismiss the information, the search field becomes inactive and searches are not possible. I've tried to with TAB to change focus but nothing happen.

If I search first and dismiss afterwards, things work as expected.

pagedown still doesn't work.

The basic vim keys (h/j/k/l and ctrl+d/ctrl+u, aswell as g and G) are implemented.

h: seems to do nothing
j and k: work as expected
l: opens another field showing log info

ctrl+d and ctrl+u: work as expected
g and G: work as expected.

I've added a package to our Work-In-Progress repo so that I can easily rebuild when you push changes by simply change the commit tag and regen the checksums. This, of course, makes it possible for anyone with access to build and install a package. Let me know if you don't want this and I can move it away to a private repo.

[0323pin]

Finally, minor thing that doesn't really need to be addressed. Distance between displayed text a TUI info currently looks like this,

Perhaps change the color for the TUI info text or, slightly increase distance/padding so it is clear what is what.

But, as said, this is really minor.

[Builditluc]

@Builditluc Let me know if you would prefer to start a new issue to discuss found bugs/issues on a new thread.

Yes that makes more sense. I'll create the new issue for bugs/issues in a moment (EDIT: #234).

which, is fine. The problem is, if one presses ESC to Dismiss the information, the search field becomes inactive and searches are not possible. I've tried to with TAB to change focus but nothing happen.

That's weird, I checked the code and found a little bug with the ordering of handling the key_events recieved. Basically, the issue was here:

wiki-tui/src/app.rs

Lines 91 to 100 in 50a34cc

	if self.search_bar.is_focussed {
	return self.search_bar.handle_key_events(key);
	}
	if let Some(ref mut popup) = self.popups.last_mut() {
	let result = popup.handle_key_events(key);
	if result.is_consumed() {
	return result;
	}
	}

The key events of any active popups should be handled first (not second) (EDIT: should be fixed in 6c5d0b0).

You can enter the search-bar with i and exit it with esc (honestly, when developing it made sense to me, but now that you mention it I don't know why I changed it). The issue with tab to select the searchbar is that it collides with selecting the toc (I would have to implement my own rolling focus for this, since we don't have the cursive event system handling that for us).

What do you think about adding little key-hints to the bottom of the borders (not only for the searchbar, but for example also for the toc)?

l: opens another field showing log info

Yes thats the logger, you can change the logging level by setting the WIKI_TUI_LOG env variable to whatever level you want (or use the --level cli arg).

For movement inside of an article j/k is for scrolling (you're right, h doesn't do anything). Selecting links is done with the arrow keys.

I've added a package to our Work-In-Progress repo so that I can easily rebuild when you push changes by simply change the commit tag and regen the checksums

That's fine by me, thanks for the information tho!

[Builditluc]

Perhaps change the color for the TUI info

Yeah you're right, it does get confusing if you're reading. We can change the background color for the info (maybe create a new theme item, called status_bar_bg)

[0323pin]

Thanks for 6c5d0b0, it does fix the issue I was having.

you can change the logging level by setting the WIKI_TUI_LOG env variable to whatever level you want (or use the --level cli arg).

Yeah, I remember this, always set it to error, so it wouldn't generate too much noise.
Right now, I'm just using default setting, as I want to catch weird behavior that could be masked by personal settings.

Quoting myself,

pagedown still doesn't work.

I was laying 🤣
It does work, I was just trying to do it with the down arrow. On my laptop, 'fn + down arrow' = 'pagedown', so it does work.

But, I guess this setting can be override on a config.toml as I would actually like to use arrow down instead.

[0323pin]

@Builditluc I couldn't resist 😄

# config.toml

page.zen_mode.include = "SEARCH_BAR | SCROLLBAR | STATUS_BAR"
page.padding          = 2

All looks good.

[Builditluc]

Pushed d713a9d which adds size constraints for the content in zen-mode. Documentation for the feature is also live at: https://wiki-tui.net/develop/configuration/page/#changing-the-size-of-the-content-in-zen-mode

[0323pin]

Everything working great 👍

[Builditluc]

Pushed 0c11c2d implementing configuration options for the global keybindings. Please excuse me taking so long, it was a journey trying to get the configuration schema the way I want.

Documentation for the global keybindings is also live at: https://wiki-tui.net/develop/configuration/keybindings/

[0323pin]

🥳

# config.toml

page.zen_mode.include = "SEARCH_BAR | SCROLLBAR | STATUS_BAR"
page.padding          = 2

[bindings.global]
scroll_down           = "down"
scroll_up             = "up"

Thank you!

Please excuse me taking so long, it was a journey trying to get the configuration schema the way I want.

No worries, I'm glad you got it.

Minor thing on the docs, there are two "scroll down" 😉

Getting close to a release now 👍

[Builditluc]

With the new commits, all the keybindings should be configurable now (and documented). Now the only thing still keeping us from release is an update of the website (redoing pictures, removing the notice of the rewrite, etc.)

I also need to re-implement the logging and api configuration options (maybe I can implement some more).

[0323pin]

[Builditluc]

Looking good!
What I'm noticing, you cannot configure the colors the renderer assigns to the page elements (links, headers, etc.)

What do you think, should configuration options for that also be in this release or should we do this in the next one?

[0323pin]

Personally, I'm happy but you're the boss here. Maybe leave it for 0.9.1? The changes are already quite large.

If I would have to pick one it would be to be able to configure the border of the search bar. Or, is this already possible and I've just missed it?

[Builditluc]

No, it's currently not possible. Yes, I also think it makes more sense to leave these options for the next version

[0323pin]

One thing, if possible. I would like us (NetBSD) to ship a working binary on the next stable branch, which is currently not the case as, 0.8.2 doesn't build with Rust > 1.79

Is it reasonable to believe this is possible? I mean, a release before Christmas? If not, I understand it, no rush.

[Builditluc]

I'm currently working on implementing api configuration options and then we just need to update the website.

I'm aiming for a new release either this or next week (then we have enough time to test and fix bug reports before the NetBSD release cycle)

[0323pin]

Thanks!

[0323pin]

Just for info: wiki-tui was marked BROKEN yesterday.

[Builditluc]

With 10b6243 all of the api configuration settings are implemented.

I'll now go ahead and update the rest of the website (and readme) for the new version and take one more look over the code to see if I spot anything obvious

[0323pin]

@Builditluc Just updated my local copy to the latest commit. Let the testing begin :)

[Builditluc]

There is still the issue with scrolling. At least I can now reproduce the issue (on the page Pindar try scrolling down to the bottom, at the very end you'll jump back to the top).

I'll debug scrolling with gdb and try to find the underlying cause for the issue.

Regarding release @0323pin, I believe there's "only" this scrolling issue left (if you or I don't find anything more), so if I find a fix for that today, we should be able to create a release in the evening (CET).

About the release notes, I'd just use the big list in the CHANGELOG

[Builditluc]

If we find anything more in the following days, we can always push another version (or pre-release version) with these fixes

[0323pin]

Agreed on the release notes and to address potential new bugs with point releases. I'll try to check the srolling issue at my end but, it will need to wait a few hours. Building Rust-1.83 at the moment.

[Builditluc]

No worries, if you look at my comment in #234, I believe I have found the issue.

EDIT: Just pushed 0f07f3c which fixes the issue on my end. Would appreciate you also checking it out before we release

[Builditluc]

@0323pin wiki-tui v0.9.0 is now finally online! release notes

[0323pin]

This is great. I'll try to get it merged tonight.

I'll keep my git-HEAD package a few weeks, just in case there will point releases and you want someone to test.

[Builditluc]

What I've found interesting is that we need to specify --locked for cargo install to work correctly (see here)

[Builditluc]

@0323pin maybe this also affects you. When trying to package the new version for nixpkgs, I needed to symlink wiki-api/data to data for the proc-macro to find the languages.json file (EDIT: see the diff here)

[Builditluc]

Closing this as v0.9.0 has been released

[0323pin]

@0323pin maybe this also affects you. When trying to package the new version for nixpkgs, I needed to symlink wiki-api/data to data for the proc-macro to find the languages.json file (EDIT: see the diff here)

Yes, it does.

error: proc macro panicked
 --> wiki-api/src/languages.rs:3:1
  |
3 | parse_languages!("./data/languages.json");
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: message: called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }

error[E0432]: unresolved import `super::languages::Language`
  --> wiki-api/src/page.rs:15:5
   |
15 | use super::languages::Language;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `Language` in `languages`

error[E0432]: unresolved import `crate::languages::Language`
  --> wiki-api/src/page.rs:18:17
   |
18 |     use crate::{languages::Language, search::Namespace, Endpoint};
   |                 ^^^^^^^^^^^^^^^^^^^ no `Language` in `languages`
   |
   = help: consider importing this unresolved item through its public re-export instead:
           crate::page::Language

error[E0432]: unresolved import `crate::languages::Language`
 --> wiki-api/src/parser.rs:9:5
  |
9 |     languages::Language,
  |     ^^^^^^^^^^^^^^^^^^^ no `Language` in `languages`

error[E0432]: unresolved import `crate::languages::Language`
  --> wiki-api/src/search.rs:14:5
   |
14 | use crate::languages::Language;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^ no `Language` in `languages`

   Compiling human-panic v1.2.2
For more information about this error, try `rustc --explain E0432`.
error: could not compile `wiki-api` (lib) due to 5 previous errors
warning: build failed, waiting for other jobs to finish...
*** Error code 101

Stop.

Sorry but, this is a deal breaker 😞
I cannot merge a package update as is. Can this be fixed and point release made?

[Builditluc]

Alright, but I'm going to move this into its own issue (EDIT: #236)