v0.47.0

[gardener/docforge]

⚠️ Breaking Changes

• [USER] The --markdownfmt flag was removed. Adjust scripts using it. by @g-pavlov [#65]
• [USER] The change is not backwards compatible and previous manifests need to update to the new API by @g-pavlov [#54]

📰 Noteworthy

• [USER] Added configuration file as new option for providing Git authentication. Default file location is ~/.docforge/config and could be overridden with DOCFORGECONFIG env var. File format is yaml. by @swilen-iwanow [#178]
• [USER] improve error handling with additional context by @Kristian-ZH [#255]
• [USER] Using nodesSelector now it is possible to specify frontmatter properties as criteria to include or exclude nodes when the selector is resolved. by @g-pavlov [#160]
• [USER] Failures to parse a GitHub URL are now reported as errors. Invalid links on the documents will now fail the build. by @g-pavlov [#50]
• [USER] Removes duplicate blogs at https://gardener.cloud/blog/ by @dimitar-kostadinov [#246]
• [USER] A warning is printed when the link target does not exist by @dimitar-kostadinov [#203]
• [USER] Manifest files can now be designed as Go templates to make use of variables iterators and Go as a templating language. Manifests are resolved as templates before the forging process and supplied with variables provided as "string-name=string-value" map (with ',' delimiter) argument to the new --variables flag. The supported variables values are strings for the moment. Example use in template: {{ .gardener-version }} will render to v1.12.1 with the flag --variables="gardener-version=v1.12.1" by @g-pavlov [#79]
• [USER] Golang version was upgraded to 1.17 by @Kristian-ZH [#224]
• [USER] The default regex for generated names of referenced resources now includes <resource-name>_<abs_url>.<etx> by @Kristian-ZH [#188]
• [USER] The manifest API changed to a more intuitive structure and adds new capabilities. See the pull request for summary of changes. by @g-pavlov [#54]
• [USER] --hugo now uses a root-relative path for links between the node structure and downloaded resources. use the new flag --hugo-base-url to change the root-relative path by @swilen-iwanow [#176]
• [USER] New properties path, sha and shaalias added to gitinfo generated for GitHub resources. The path property is the resource path in GitHub after resource type and its sha. The shaaliasproperty is the human readable tag/branch-name assigned to a sha (e.g.master`). by @g-pavlov [#171]
• [USER] The manifest configuration for rewriting links and downloading resources is no longer needed, as it all happens automatically.
  Also, 'template' and 'contentSelectors' have been removed due to lack of use cases, instead a 'multiSource' has been introduced to cover simple content concatenation scenario. by @dimitar-kostadinov [#261]
• [USER] Fs handler is removed because it is not expected it's covering use case to be part of the scope of the project by @Kostov6 [#247]
• [USER] Default branches and number of versions can now be configured in the docforge config file by @Kostov6 [#237]
• [USER] With verbosity flag >= 6 enabled (-v=6), now logs contain also HTTP logs. The logs are simple and consist of the request method, URL and response status. by @g-pavlov [#129]
• [USER] --use-git now uses Git for retrieving source info data (--github-info-data) as well instead of the GitHub APIs. by @swilen-iwanow [#181]
• [USER] Docforge release includes docforge binaries for arm64 architecture on Linux and Darwin now.
  Docker images are published with multi-arch support for linux/amd64 and linux/arm64 now. by @oliver-goetz [#286]
• [USER] Rewriting absolute links to raw content references applies only to images markdown by @g-pavlov [#51]
• [USER] - Node names need not end with .md suffix to have the file written as.md. If there is no .md suffix, it will be appended to the file name prior to writing it down. by @g-pavlov [#40]
• [USER] The default value for --resources-download-path flag now is __resources by @g-pavlov [#62]
• [USER] Fixes an issue with cache invalidation as new content in the subfolders was not displayed by @dimitar-kostadinov [#275]
• [USER] Documents structures support "folder"-type nodes that have no contentSelector specified and result in a file folder named by the name of the node. by @g-pavlov [#40]
• [USER] Adaptive client requests throttling is implemented to respect the GitHub API rate-limits SLA. A slow down in performance may be experienced due to that. It will increase with the docforge executions per hour and the amount of content requests to GitHub. The feature is enabled with the --github-throttling boolean flag. by @g-pavlov [#143]
• [USER] Fixed a bug that was causing links of downloaded resources to be prepended with an extra ../ by @swilen-iwanow [#149]
• [USER] GitHub URLs are now supported as manifest path flag (-f/--manifest) arguments, in addition to (local) file system file paths. The --github-oauth-token-map and --github-oauth-token flags arguments are used to apply authentication token to the request when fetching manifest from a GitHub URL. by @g-pavlov [#120]
• [USER] Docforge is now built using go version 1.17.12. by @dimityrmirchev [#281]
• [USER] introducing local mappings for better local development. by @swilen-iwanow [#181]
• [USER] Hugo support
  The following features are available when the --hugo=true option flag is enabled:
  • Frontmatter that should be written to a document is specified in the properties map of the corresponding documentation structure node with key "frontmatter"
  • Documents frontmatter in yaml (only) is supported
  • Documents with frontmatter only (no other content) and without line ending on the last line are supported.
  • Documents with frontmatter preceded by any amount of whitespace are supported. Whitespace before or after the --- start/end marks is also supported.
  • Documents that start with --- (potentially preceded by any whitespace) are considered to specify frontmatter. An error is thrown if there is no closing --- and the content in between is not valid yaml
  • Frontmatter specified as node property and frontmatter available in the node document that will be written are merged. Duplicates are not reconciled.
  • A minimal frontmatter title: <node-name-title-case> is written when no other has been specified as node property or in the document)
  • Container nodes that specify a "frontmatter" structure in their properties will be written as folders with autogenerated _index.md file that contains the specified frontmatter.
  • Document nodes that need to be written as hugo section files are identified by an index": true entry in their Properties. Such nodes are written as _index.md files regardless of their name.
  • A section file node can be inferred automatically too. A document structure node that has content selectors, and with no other peer nodes named _index or _index_md, will be written as _index.md file, provided that its name matches one of the entries specified in the --hugo-section-files flag or its default list: readme, read.me, index (case insensitive). To disable this automation, set the --hugo-section-files flag value to empty string.
  • Multiple files that will potentially be written as _index.md in the same folder are detected and reported as error. by @g-pavlov [#40]
• [USER] Fixed the reasons for warning about not being able to read from tree object when nodeSelectors resolve to empty nodes referencing folders. by @g-pavlov [#52]
• [USER] Docforge main command usage is now shown only on flags/cmd/args error. by @g-pavlov [#104]
• [USER] A new proprty weburl added to files gitinfo. It is a URL that references the repository hosting the file for which the gitinfo has been created. by @g-pavlov [#166]
• [USER] Fixes a synchronization problem that was observed when having multiple errors during document download. by @swilen-iwanow [#145]
• [USER] Added API validations for the linked manifest by @Kristian-ZH [#253]
• [USER] Circular dependencies are prevented with an error. by @swilen-iwanow [#108]
• [USER] Docforge can run even without an OAuth token
  In this configuration the rate limit is only 60 by @Kostov6 [#250]
• [USER] Adds support for relative paths in manifests by @dimitar-kostadinov [#264]
• [USER] The bug where the last letter of the node names is sometimes truncated is fixed by @Kristian-ZH [#276]
• [USER] Logs for download failures now include more context to trace them back to the reference link and fix it if that is the reason. by @g-pavlov [#127]
• [USER] Fixes a runtime error when multiple links match a rewrite rule to be removed completely from a document. by @g-pavlov [#159]
• [USER] Docforge now uses distroless as a base container image. by @dimityrmirchev [#281]
• [USER] Fixes an occasional panic: send on closed channel while finishing a build by @g-pavlov [#59]
• [USER] Fixes link processing when the link is in round brackets and ends with a dot; e.g. "(https://foo.bar)." by @dimitar-kostadinov [#260]
• [USER] Hugo baseURL, pretty URLs, can be configured inside the Docforge config file by @swilen-iwanow [#182]
• [USER] Support for modules is introduced to minimize the complexity and maximize the agility for Docforge's manifests. Users can now use nodesSelector path property to specify a manifest file. The referenced structure nodes and links configuration will be then merged into the referencing node. by @swilen-iwanow [#87]
• [USER] Adds processing of links with fragments, that refers pages of the website by @dimitar-kostadinov [#269]
• [USER] The title front-matter property is generated smarter now. For index nodes it takes the name of the parent node. Name delimiters _ and - are replaced with space, .md extension is removed and finally the string is converted to title case.
  Example 1: A node with name abc_def-1.md gets title: Abc Def 1
  Example 2: A node with name README.md and parent node name concepts gets title: Concepts, assuming that README.md has been specified as index file name via the command line flag --hugo-section-files by @g-pavlov [#162]
• [USER] Markdown link processing takes advantage of the CommonMark compliant Goldmark parser and issues with reference links are resolved. by @dimitar-kostadinov [#258]
• [USER] If both document and a node referencing it as source define front-matter, they will be merged and where there are conflicting keys the node front-matter has priority. by @swilen-iwanow [#117]
• [USER] Fixes occasional panic: Trying to add to a not started workqueue by @g-pavlov [#60]
• [USER] Trying to completely remove a link in a link substitute with destination:"" and text:"" wasn't working correctly. This is fixed now. by @g-pavlov [#53]
• [USER] The issue where the Git handler breaks at times when there are unstaged changes is fixed by @Kostov6 [#240]
• [DEVELOPER] ResourceHandler interface is extended with GetClient() method that returns an HTTP client for accessing resources by @dimitar-kostadinov [#203]
• [DEVELOPER] Added script for consistency checking by @Kostov6 [#279]
• [DEVELOPER] The vendor directory was removed in favor of the go mod cache and go mod tidy. by @LucaBernstein [#321]
• [DEVELOPER] Documentation for refactored flags updated by @Kostov6 [#284]
• [DEVELOPER] License and copyright information is now specified in REUSE format. by @msohn [#73]
• [DEVELOPER] To avoid duplication when adding a api.Node during api.Documentation manifest resolving, merge is used instead of append by @dimitar-kostadinov [#216]
• [DEVELOPER] Introduces new resource handler with persistent cache by @dimitar-kostadinov [#269]
• [DEVELOPER] Added flag descriptions in README by @Kostov6 [#282]
• [OPERATOR] Git handler will now reuse the logic from the Github one by @Kristian-ZH [#199]
• [OPERATOR] The release notes will be posted in gardener-documentation channel by @Kristian-ZH [#196]

🏃 Others

• [USER] Introduced a new --resolve flag. It prints to the standard output the resolved documentation structure, expanding nodeSelector elements to node hierarchies. When resolve is used with nodeSelector it is useful for quickly scafolding structures from existing file hierarchies. by @g-pavlov [#48]
• [USER] Fixed an issue with text or title of markdown links not being updated silently. by @g-pavlov [#85]
• [USER] An optional --github-info-destination flag is available now to specify a destination and require the git commits history for the content sources of the structure's document nodes to be written to it. Applies only to document nodes from the structure. by @g-pavlov [#62]
• [USER] Using docforge with --github-info-destination flag failed with nil pointer reference panic when all commits for a file referenced by a node source in the structure were internal. This is fixed now. by @g-pavlov [#115]
• [USER] Fix wrong additions of a version segment in url paths that do not have such type of segment, like pulls/wiki etc. by @g-pavlov [#134]
• [USER] "Multiple peer nodes with property index: true detected in ..." is no longer thrown when there is 1 or less nodes declaring index:true property and building with --hugo flag. by @g-pavlov [#110]
• [USER] Fixed an issue with failure to process documents with markdown links, which contain src= or href= by @g-pavlov [#82]
• [USER] Absolute links to page embedded objects (images) in documents are now rewritten to match GitHub's format for raw content links consistently and display embedded objects correctly. The behavior is controlled with the --rewrite-embedded-to-raw flag, which is true by default. by @g-pavlov [#50]
• [USER] Fixed the list of error messages, which used to include only the last error. Now the list includes all recorded errors. Applies to fault-tolerant forge runs. by @g-pavlov [#127]
• [USER] Links in HTML tag href and src attributes are processed for Hugo too now. by @g-pavlov [#44]
• [USER] Fixes failures to forge document bundles with links to GitHub commit(s) resources. by @g-pavlov [#91]
• [USER] Rewriting links now does not change formatting of the document. by @g-pavlov [#65]
• [USER] Fixes a runtime error when using manifests that reference GitHub repositories with submodules. by @g-pavlov [#174]
• [USER] New gen-cmd-docs --destination --format command generating docforge commands reference documentation. Supported formats are Markdown (default, -f md) and Man pages (-f man). Details how to use are available with docforge gen-cmd-docs -h. by @g-pavlov [#70]
• [USER] Document nodes now support content assignment with Template. by @g-pavlov [#64]
• [USER] Fixed an issue with simple manifests with a single top-level NodeSelector, which had their links always absolute even when referencing resolved and downloaded documents from the structure. by @g-pavlov [#151]
• [USER] Configuration capabilities are improved by @Kostov6 [#266]
• [USER] Git info is supported for document nodes with local file system content assignment. The author/contributor details are limited to what git log can offer - email and name. Requires git to be available and sufficient privileges for docforge to run git log. by @g-pavlov [#64]
• [USER] New completion [bash|zsh|fish|powershell] command generating a completion script for the supported shells (bash|zsh|fish|powershell). Type dockforge completion -h for details on how to use in each shell. by @g-pavlov [#69]
• [USER] Document nodes now support content assignment from local file system sources. NodeSelectors with local file path are supported but the optional filters of the selectors are not implemented yet. by @g-pavlov [#64]
• [USER] Fixed reported errors from command execution. Previously many were logged but the few were reported as errors summary report at the end of the command execution. by @g-pavlov [#131]
• [USER] Changed the --dry-run flag behavior. When running docforge with --dry-run it will do a full scale build as instructed, but without writing down the result. Instead, it will print to the standard output the projected file/folder structure with statistics for the processing of each document in the structure and total processing time. by @g-pavlov [#48]
• [USER] Logs had entries like -> abc, which were result from text, incorrectly parsed text as links. Such incorrectly parsed links will no longer be processed as links. by @g-pavlov [#131]
• [USER] Fixed a problem with HTML links not getting updated to absolute or to downloaded resource references in cases when they needed to. by @g-pavlov [#119]
• [USER] It is possible now to replace document links before they are handed over for analysis and transformation to docforge, including to remove them completely. This is achievable with the linkSubstitutes map in a localityDomain. It maps a set of regular expressions matching absolute form of links in a document to a replacement link for positively matching targets. An empty replacement results in link removal leaving only its text in the document and complete image removal in case of images. The empty string replacement is implemented only for markdown links and images and not for HTML tags. by @g-pavlov [#45]
• [USER] Fixed an issue with silent download fail in the specific case of GitHub resource with correct path and wrong resource name. by @g-pavlov [#89]
• [USER] Now it is possible to influence the name pattern used to generate a downloaded resource name, including to replace it completely delegating the responsibility for uniqueness to an external mechanism. by @g-pavlov [#45]
• [USER] Locality domain now can be defined also on (document) nodes. Node-level definitions override global ones. A node-level definition is valid only for the node on which it is defined. by @g-pavlov [#45]
• [USER] docforge now respects standards go flags such as "v", which allows to control the output of the program. Use --v=2 for quiet builds with minimal output, --v=4 for warnings and important output, and --v=6 for a lot of insight and debugging purposes. See the complete list of flags and their default values with docforge -h by @g-pavlov [#43]
• [USER] Fixes an issue with failing parse of HTML links when dockforge is run with --hugo flag by @g-pavlov [#102]
• [USER] Fixes a bug when using the --hugo-section-files that did not result in rename of matching argument file names as expected. by @g-pavlov [#154]
• [USER] Documentation structures can now use GitHub wiki pages for contentSource source. The wiki page url has to end with .md. Example: https://github.com/gardener/documentation/wiki/Architecture.md by @g-pavlov [#49]
• [USER] Fixed an issue with failing updates of text and destination components of markdown links when they are longer than the original ones. Updated text in these cases used to overlap the next component(s) because they were not offset with the changed length. by @g-pavlov [#85]
• [USER] Locality domain now features exclude and include lists of regular expressions that are matched to the absolute forms of the documents links to define whether a link belongs to the domain or not. The evaluation starts with excludes and proceeds with includes. No assumption is made about the collective correctness of the matching rules. They are executed in order. Exclusions do not apply to documents referenced by structure nodes contentSelectors. They are considered in locality domain no matter what. by @g-pavlov [#45]
• [DEVELOPER] Vars and flags refactor by @Kostov6 [#283]
• [DEVELOPER] Added git and github resource hander tests with coverage ~80% by @Kostov6 [#262]
• [DEVELOPER] The git node selector's missing argument processing has been added by @Kostov6 [#259]
• [DEVELOPER] Git package refactoring by @Kostov6 [#248]
• [DEVELOPER] GitHub resource handler cache now supports read/write lock and helps avoid redundant API calls by @dimitar-kostadinov [#203]
• [DEVELOPER] Todo cleanup by @Kostov6 [#285]
• [DEVELOPER] Fixes an issue in release 0.19 where github token is overriden with an empty string by @Kostov6 [#251]
• [DEVELOPER] gomock is now part of the project and is going to be used for generating mocks by @swilen-iwanow [#186]
• [DEVELOPER] Version feature tests are converted to Ginkgo tests
  Code generation tool adds license headers to generated code files by @Kostov6 [#242]
• [DEVELOPER] Removes inefficient usage of the resource_locator_cache in git_resource_handle.go by @Kostov6 [#263]
• [DEVELOPER] Fix error in hugo when parsing links ending with ) by @Kostov6 [#287]
• [DEVELOPER] Minor config changes by @Kostov6 [#267]
• [DEVELOPER] Collision check is moved to be in parser instead of reactor by @Kostov6 [#278]
• [DEVELOPER] Fix a bug when nodeSelector overrides node list by @Kostov6 [#257]
• [DEVELOPER] Change in parser configuration by @Kostov6 [#268]
• [OPERATOR] When collisions with node names collide, a warning is displayed. by @vlvasilev [#189]
• [OPERATOR] Performance improvements and code clean up by @Kostov6 [#315]

Docker Images

• docforge: europe-docker.pkg.dev/gardener-project/releases/docforge:v0.47.0