Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework and consolidate the page parser #5324

Closed
bep opened this issue Oct 16, 2018 · 2 comments
Closed

Rework and consolidate the page parser #5324

bep opened this issue Oct 16, 2018 · 2 comments
Assignees
@bep
Labels
Enhancement Outdated
Milestone
v0.50

Comments

@bep
Copy link
Member

bep commented Oct 16, 2018 •
edited
Loading

Working on #5284 I hit some hurdles when I wanted to show proper file/line context for errors in content files. The main problem is that the current parser splits front matter and the rest. There are currently some line number calculations in there, but it seems to be impossible to get exactly right in every situation.

The current "front matter" parser is also a little hard to read, and while considered stable and working, It is easy to read from the Git history that there have been corner cases that is simpler to avoid with a proper lexer.

So I suggest that:

  • Move the shortcode parser to the parser package
  • Extend that parser to handle front matter tokens
  • Return that AST to the client with the full and original content byte slice (possibly with some shortcut methods to get the front matter etc.)

There is also an obvious potential performance benefit in the "do it once" approach above.
@bep bep added this to the v0.50 milestone Oct 16, 2018
@bep bep self-assigned this Oct 16, 2018
@bep bep changed the title Rework the page parser Rework and consolidate the page parser Oct 16, 2018
bep added a commit to bep/hugo that referenced this issue Oct 16, 2018
@bep
commands: Show server error info in browser
144e9cf 
The main item in this commit is showing of errors with a file context when running `hugo server`.

This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).

But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.

Fixes gohugoio#5284
Fixes gohugoio#5290
See gohugoio#5325
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 16, 2018
@bep
commands: Show server error info in browser
09ca1d2 
The main item in this commit is showing of errors with a file context when running `hugo server`.

This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).

But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.

Fixes gohugoio#5284
Fixes gohugoio#5290
See gohugoio#5325
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 16, 2018
@bep
commands: Show server error info in browser
68d56bf 
The main item in this commit is showing of errors with a file context when running `hugo server`.

This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).

But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.

Fixes gohugoio#5284
Fixes gohugoio#5290
See gohugoio#5325
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 16, 2018
@bep
commands: Show server error info in browser
3f3bf7c 
The main item in this commit is showing of errors with a file context when running `hugo server`.

This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).

But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.

Fixes gohugoio#5284
Fixes gohugoio#5290
See gohugoio#5325
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 16, 2018
@bep
commands: Show server error info in browser
8a887ab 
The main item in this commit is showing of errors with a file context when running `hugo server`.

This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).

But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.

Fixes gohugoio#5284
Fixes gohugoio#5290
See gohugoio#5325
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 16, 2018
@bep
commands: Show server error info in browser
35fbfb1 
The main item in this commit is showing of errors with a file context when running `hugo server`.

This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).

But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.

Fixes gohugoio#5284
Fixes gohugoio#5290
See gohugoio#5325
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
Move the shortcode parser to the new pageparser package
8b088e1 
See gohugoio#5324
@bep bep mentioned this issue Oct 17, 2018
Merged
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: File renames and splitting
420198d 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: File renames and splitting
2d54c5f 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: Add front matter etc. support
ee0ce98 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: Add front matter etc. support
3eaefed 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: Add front matter etc. support
d4207e8 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: Add front matter etc. support
bcec9e5 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 17, 2018
@bep
parser/pageparser: Add front matter etc. support
8f769f1 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
parser/pageparser: Add front matter etc. support
cf49b38 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
parser/pageparser: Add front matter etc. support
2073f51 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
parser/pageparser: Use []byte in page lexer
ffbee7d 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Use []byte in shortcode parsing
057b16b 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
a333cfd 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
7074568 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
7fd241a 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
83f470b 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
8ad35a8 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
b7a6b95 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 18, 2018
@bep
hugolib: Integrate new page parser
716f891 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
parser/metadecoders: Consolidate the metadata decoders
8436c16 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
Convert the rest to new page parser code paths
0d870ed 
And remove some now unused code.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Remove the now superflous Source struct
11d979c 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Continue the file context/line number errors work
24ce085 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Improve errors in /data handlling
1be8882 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Improve errors in /i18n handlling
31edb7f 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Integrate new page parser
1e3e340 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Redo the summary delimiter logic
44da60d 
Now that we have a proper page parse tree, this can be greatly simplified.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
parser/metadecoders: Consolidate the metadata decoders
129c27e 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
Convert the rest to new page parser code paths
eb038cf 
And remove some now unused code.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Remove the now superflous Source struct
7930d21 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Continue the file context/line number errors work
d1661b8 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Improve errors in /data handlling
9f74dc2 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 22, 2018
@bep
hugolib: Improve errors in /i18n handlling
2bf686e 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
herrors: Improve handling of JSON errors
89b9658 
`*json.UnmarshalTypeError` and `*json.SyntaxError` has a byte `Offset`, so use that.

This commit also reworks/simplifies the errror line matching logic. This also makes the file reading unbuffered, but that should be fine in this error case.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
herrors: Improve handling of JSON errors
d97d280 
`*json.UnmarshalTypeError` and `*json.SyntaxError` has a byte `Offset`, so use that.

This commit also reworks/simplifies the errror line matching logic. This also makes the file reading unbuffered, but that should be fine in this error case.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
herrors: Improve handling of JSON errors
ecfa500 
`*json.UnmarshalTypeError` and `*json.SyntaxError` has a byte `Offset`, so use that.

This commit also reworks/simplifies the errror line matching logic. This also makes the file reading unbuffered, but that should be fine in this error case.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
herrors: Improve handling of JSON errors
f669ef6 
`*json.UnmarshalTypeError` and `*json.SyntaxError` has a byte `Offset`, so use that.

This commit also reworks/simplifies the errror line matching logic. This also makes the file reading unbuffered, but that should be fine in this error case.

See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
Resolve error handling/parser related TODOs
c422ad2 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
Resolve error handling/parser related TODOs
b296872 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
Resolve error handling/parser related TODOs
45a44a4 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
Resolve error handling/parser related TODOs
864931d 
See gohugoio#5324
bep added a commit to bep/hugo that referenced this issue Oct 23, 2018
@bep
Resolve error handling/parser related TODOs
6636cf1 
See gohugoio#5324
@bep
Copy link
Member Author

bep commented Oct 24, 2018

This is now done.

@bep bep closed this as completed Oct 24, 2018
@dependencies dependencies bot mentioned this issue Oct 29, 2018
Closed
@pjw91 pjw91 mentioned this issue Dec 29, 2018
Closed
@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bep bep

Labels
Enhancement Outdated
Projects
None yet
Milestone
v0.50
Development

No branches or pull requests
1 participant
@bep