diff --git a/content/en/about/features.md b/content/en/about/features.md index 022dfffa9a..ff7e9ce138 100644 --- a/content/en/about/features.md +++ b/content/en/about/features.md @@ -109,7 +109,7 @@ toc: true [CommonMark]: https://spec.commonmark.org/current/ [Content adapters]: /content-management/content-adapters/ [Content formats]: /content-management/formats/ -[Data]: /templates/data-templates/ +[Data]: /content-management/data-sources/ [Diagrams]: /content-management/diagrams/ [GDPR]: https://en.wikipedia.org/wiki/General_Data_Protection_Regulation [GitHub Flavored Markdown]: https://github.github.com/gfm/ diff --git a/content/en/content-management/content-adapters.md b/content/en/content-management/content-adapters.md index 5c24501e89..11257b8956 100644 --- a/content/en/content-management/content-adapters.md +++ b/content/en/content-management/content-adapters.md @@ -6,8 +6,8 @@ keywords: [] menu: docs: parent: content-management - weight: 280 -weight: 280 + weight: 290 +weight: 290 toc: true --- diff --git a/content/en/content-management/data-sources.md b/content/en/content-management/data-sources.md new file mode 100644 index 0000000000..9d1ec66ca7 --- /dev/null +++ b/content/en/content-management/data-sources.md @@ -0,0 +1,122 @@ +--- +title: Data sources +description: Use local and remote data sources to augment or create content. +categories: [content management] +keywords: [data,json,toml,yaml,xml] +menu: + docs: + parent: content-management + weight: 280 +weight: 280 +toc: true +aliases: [/extras/datafiles/,/extras/datadrivencontent/,/doc/datafiles/,/templates/data-templates/] +--- + +Hugo can access and [unmarshal] local and remote data sources including CSV, JSON, TOML, YAML, and XML. Use this data to augment existing content or to create new content. + +[unmarshal]: /getting-started/glossary/#unmarshal + +A data source might be a file in the data directory, a [global resource], a [page resource], or a [remote resource]. + +[global resource]: /getting-started/glossary/#global-resource +[page resource]: /getting-started/glossary/#page-resource +[remote resource]: /getting-started/glossary/#remote-resource + +## Data directory + +The data directory in the root of your project may contain one or more data files, in either a flat or nested tree. Hugo merges the data files to create a single data structure, accessible with the `Data` method on a `Site` object. + +Hugo also merges data directories from themes and modules into this single data structure, where the data directory in the root of your project takes precedence. + +Theme and module authors may wish to namespace their data files to prevent collisions. For example: + +```text +project/ +└── data/ + └── mytheme/ + └── foo.json +``` + +{{% note %}} +Do not place CSV files in the data directory. Access CSV files as page, global, or remote resources. +{{% /note %}} + +See the documentation for the [`Data`] method on `Page` object for details and examples. + +[`Data`]: /methods/site/data/ + +## Global resources + +Use the `resources.Get` and `transform.Unmarshal` functions to access data files that exist as global resources. + +See the [`transform.Unmarshal`](/functions/transform/unmarshal/#global-resource) documentation for details and examples. + +## Page resources + +Use the `Resources.Get` method on a `Page` object combined with the `transform.Unmarshal` function to access data files that exist as page resources. + +See the [`transform.Unmarshal`](/functions/transform/unmarshal/#page-resource) documentation for details and examples. + +## Remote resources + +Use the `resources.GetRemote` and `transform.Unmarshal` functions to access remote data. + +See the [`transform.Unmarshal`](/functions/transform/unmarshal/#remote-resource) documentation for details and examples. + +## Augment existing content + +Use data sources to augment existing content. For example, create a shortcode to render an HTML table from a global CSV resource. + +{{< code file=assets/pets.csv >}} +"name","type","breed","age" +"Spot","dog","Collie","3" +"Felix","cat","Malicious","7" +{{< /code >}} + +{{< code file=content/example.md lang=text >}} +{{}} +{{< /code >}} + +{{< code file=layouts/shortcodes/csv-to-table.html >}} +{{ with $file := .Get 0 }} + {{ with resources.Get $file }} + {{ with . | transform.Unmarshal }} + + + + {{ range index . 0 }} + + {{ end }} + + + + {{ range after 1 . }} + + {{ range . }} + + {{ end }} + + {{ end }} + +
{{ . }}
{{ . }}
+ {{ end }} + {{ else }} + {{ errorf "The %q shortcode was unable to find %s. See %s" $.Name $file $.Position }} + {{ end }} +{{ else }} + {{ errorf "The %q shortcode requires one positional parameter, the path to the CSV file relative to the assets directory. See %s" .Name .Position }} +{{ end }} +{{< /code >}} + +Hugo renders this to: + +name|type|breed|age +:--|:--|:--|:-- +Spot|dog|Collie|3 +Felix|cat|Malicious|7 + +## Create new content + +Use [content adapters] to create new content. + +[content adapters]: /content-management/content-adapters/ diff --git a/content/en/getting-started/directory-structure.md b/content/en/getting-started/directory-structure.md index ca37d328ac..da2dde6b20 100644 --- a/content/en/getting-started/directory-structure.md +++ b/content/en/getting-started/directory-structure.md @@ -91,7 +91,7 @@ content : The `content` directory contains the markup files (typically Markdown) and page resources that comprise the content of your site. See [details](/content-management/organization/). data -: The `data` directory contains data files (JSON, TOML, YAML, or XML) that augment content, configuration, localization, and navigation. See [details](/templates/data-templates/). +: The `data` directory contains data files (JSON, TOML, YAML, or XML) that augment content, configuration, localization, and navigation. See [details](/content-management/data-sources/). i18n : The `i18n` directory contains translation tables for multilingual sites. See [details](/content-management/multilingual/). diff --git a/content/en/methods/site/Data.md b/content/en/methods/site/Data.md index 91c606beb2..65cdadd014 100644 --- a/content/en/methods/site/Data.md +++ b/content/en/methods/site/Data.md @@ -101,7 +101,13 @@ To find a fiction book by ISBN: {{ end }} ``` -In the template examples above, each of the keys is a valid identifier. For example, none of the keys contains a hyphen. To access a key that is not a valid identifier, use the [`index`] function: +In the template examples above, each of the keys is a valid [identifier]. For example, none of the keys contains a hyphen. To access a key that is not a valid identifier, use the [`index`] function. For example: + +[identifier]: /getting-started/glossary/#identifier + +```go-html-template +{{ index .Site.Data.books "historical-fiction" }} +``` [`index`]: /functions/collections/indexfunction/ [chaining]: /getting-started/glossary/#chain diff --git a/content/en/templates/data-templates.md b/content/en/templates/data-templates.md deleted file mode 100644 index 45e7c7bd7c..0000000000 --- a/content/en/templates/data-templates.md +++ /dev/null @@ -1,178 +0,0 @@ ---- -title: Data templates -description: In addition to Hugo's built-in variables, you can specify your own custom data in templates or shortcodes that pull from both local and dynamic sources. -categories: [templates] -keywords: [data,dynamic,csv,json,toml,yaml,xml] -menu: - docs: - parent: templates - weight: 150 -weight: 150 -toc: true -aliases: [/extras/datafiles/,/extras/datadrivencontent/,/doc/datafiles/] ---- - -Hugo supports loading data from YAML, JSON, XML, and TOML files located in the `data` directory at the root of your Hugo project. - -{{< youtube FyPgSuwIMWQ >}} - -## The data directory - -The `data` directory should store additional data for Hugo to use when generating your site. - -Data files are not for generating standalone pages. They should supplement content files by: - -- Extending the content when the front matter fields grow out of control, or -- Showing a larger dataset in a template (see the example below). - -In both cases, it's a good idea to outsource the data in their (own) files. - -These files must be YAML, JSON, XML, or TOML files (using the `.yml`, `.yaml`, `.json`, `.xml`, or `.toml` extension). The data will be accessible as a `map` in the `.Site.Data` variable. - -To access the data using the `site.Data.filename` notation, the file name must begin with an underscore or a Unicode letter, followed by zero or more underscores, Unicode letters, or Unicode digits. For example: - -- `123.json` - Invalid -- `x123.json` - Valid -- `_123.json` - Valid - -To access the data using the [`index`](/functions/collections/indexfunction) function, the file name is irrelevant. For example: - -Data file|Template code -:--|:-- -`123.json`|`{{ index .Site.Data "123" }}` -`x123.json`|`{{ index .Site.Data "x123" }}` -`_123.json`|`{{ index .Site.Data "_123" }}` -`x-123.json`|`{{ index .Site.Data "x-123" }}` - -## Data files in themes - -Data Files can also be used in themes. - -However, note that the theme data files are merged with the project directory taking precedence. That is, Given two files with the same name and relative path, the data in the file in the root project `data` directory will override the data from the file in the `themes//data` directory *for keys that are duplicated*). - -Therefore, theme authors should be careful not to include data files that could be easily overwritten by a user who decides to [customize a theme][customize]. For theme-specific data items that shouldn't be overridden, it can be wise to prefix the folder structure with a namespace; e.g. `mytheme/data//somekey/...`. To check if any such duplicate exists, run hugo with the `-v` flag. - -The keys in the map created with data templates from data files will be a dot-chained set of `path`, `filename`, and `key` in the file (if applicable). - -This is best explained with an example: - -## Examples - -### Jaco Pastorius' Solo Discography - -[Jaco Pastorius](https://en.wikipedia.org/wiki/Jaco_Pastorius_discography) was a great bass player, but his solo discography is short enough to use as an example. [John Patitucci](https://en.wikipedia.org/wiki/John_Patitucci) is another bass giant. - -The example below is a bit contrived, but it illustrates the flexibility of data Files. This example uses TOML as its file format with the two following data files: - -* `data/jazz/bass/jacopastorius.toml` -* `data/jazz/bass/johnpatitucci.toml` - -`jacopastorius.toml` contains the content below. `johnpatitucci.toml` contains a similar list: - -{{< code-toggle file=data/jazz/bass/jacopastorius >}} -discography = [ -"1974 - Modern American Music … Period! The Criteria Sessions", -"1974 - Jaco", -"1976 - Jaco Pastorius", -"1981 - Word of Mouth", -"1981 - The Birthday Concert (released in 1995)", -"1982 - Twins I & II (released in 1999)", -"1983 - Invitation", -"1986 - Broadway Blues (released in 1998)", -"1986 - Honestly Solo Live (released in 1990)", -"1986 - Live In Italy (released in 1991)", -"1986 - Heavy'n Jazz (released in 1992)", -"1991 - Live In New York City, Volumes 1-7.", -"1999 - Rare Collection (compilation)", -"2003 - Punk Jazz: The Jaco Pastorius Anthology (compilation)", -"2007 - The Essential Jaco Pastorius (compilation)" -] -{{< /code-toggle >}} - -The list of bass players can be accessed via `.Site.Data.jazz.bass`, a single bass player by adding the file name without the suffix, e.g. `.Site.Data.jazz.bass.jacopastorius`. - -You can now render the list of recordings for all the bass players in a template: - -```go-html-template -{{ range $.Site.Data.jazz.bass }} - {{ partial "artist.html" . }} -{{ end }} -``` - -And then in the `partials/artist.html`: - -```go-html-template - -``` - -Discover a new favorite bass player? Just add another `.toml` file in the same directory. - -### Accessing named values in a data file - -Assume you have the following data structure in your `user0123` data file located directly in `data/`: - -{{< code-toggle file=data/user0123 >}} -Name: User0123 -"Short Description": "He is a **jolly good** fellow." -Achievements: - - "Can create a Key, Value list from Data File" - - "Learns Hugo" - - "Reads documentation" -{{}} - -You can use the following code to render the `Short Description` in your layout: - -```go-html-template -
Short Description of {{ .Site.Data.user0123.Name }}:

{{ index .Site.Data.user0123 "Short Description" | markdownify }}

-``` - -Note the use of the [`markdownify`] function. This will send the description through the Markdown rendering engine. - -## Remote data - -Retrieve remote data using these template functions: - -- [`resources.GetRemote`](/functions/resources/getremote) (recommended) -- [`data.GetCSV`](/functions/data/getcsv) -- [`data.GetJSON`](/functions/data/getjson) - -## LiveReload with data files - -There is no chance to trigger a [LiveReload] when the content of a URL changes. However, when a *local* file changes (i.e., `data/*` and `themes//data/*`), a LiveReload will be triggered. Note too that because downloading data takes a while, Hugo stops processing your Markdown files until the data download has been completed. - -{{% note %}} -If you change any local file and the LiveReload is triggered, Hugo will read the data-driven (URL) content from the cache. If you have disabled the cache (i.e., by running the server with `hugo server --ignoreCache`), Hugo will re-download the content every time LiveReload triggers. This can create *huge* traffic. You may reach API limits quickly. -{{% /note %}} - -## Examples of data-driven content - -- Photo gallery JSON powered: [https://github.com/pcdummy/hugo-lightslider-example](https://github.com/pcdummy/hugo-lightslider-example). -- GitHub Starred Repositories [in a post](https://github.com/SchumacherFM/blog-cs/blob/master/content%2Fposts%2Fgithub-starred.md) using data-driven content in a [custom short code](https://github.com/SchumacherFM/blog-cs/blob/master/layouts%2Fshortcodes%2FghStarred.html). -- Importing exported social media data from popular services using [https://github.com/ttybitnik/diego](https://github.com/ttybitnik/diego). - -## Specs for data formats - -* [TOML Spec][toml] -* [YAML Spec][yaml] -* [JSON Spec][json] -* [CSV Spec][csv] -* [XML Spec][xml] - -[config]: /getting-started/configuration/ -[csv]: https://tools.ietf.org/html/rfc4180 -[customize]: /hugo-modules/theme-components/ -[json]: https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf -[LiveReload]: /getting-started/usage/#livereload -[lookup]: /templates/lookup-order/ -[`markdownify`]: /functions/transform/markdownify/ -[OAuth]: https://en.wikipedia.org/wiki/OAuth -[partials]: /templates/partials/ -[toml]: https://toml.io/en/latest -[variadic]: https://en.wikipedia.org/wiki/Variadic_function -[vars]: /methods/ -[yaml]: https://yaml.org/spec/ -[xml]: https://www.w3.org/XML/