diff --git a/.github/styles/config/vocabularies/fabric-vocab/accept.txt b/.github/styles/config/vocabularies/fabric-vocab/accept.txt index 2318a24e..75d5e680 100644 --- a/.github/styles/config/vocabularies/fabric-vocab/accept.txt +++ b/.github/styles/config/vocabularies/fabric-vocab/accept.txt @@ -1,3 +1,4 @@ GitHub Fabric boolean +namespace diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 870b55f2..00000000 --- a/docs/plugins.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Plugins -type: docs -weight: 5 ---- - -# Plugins - -[TBD] diff --git a/docs/plugins/_index.md b/docs/plugins/_index.md new file mode 100644 index 00000000..939aafab --- /dev/null +++ b/docs/plugins/_index.md @@ -0,0 +1,16 @@ +--- +title: Plugins +type: docs +weight: 5 +--- + +# Plugins + +Fabric relies on plugins to implement data sources and content providers. To utilise a plugin's data sources and content providers, it must be installed by Fabric. The global configuration should specify all required plugins (see [Global configuration]({{< ref "../language/configs.md/#global-configuration" >}}) for the details). Additionally, some data sources and content providers require configuration (for example, API keys, URLs, credentials, etc). + +A plugin name consists of a namespace (a name of a plugin vendor) and a short name. For example, `blackstork/elasticsearch` plugin implements Elasticsearch client data source and is released by [BlackStork](https://blackstork.io). + +## Where to get the plugins + +Plugins are released and distributed independently from Fabric, with their own release cycle and version. +You can find a list of plugins released by BlackStork at the [Releases page](https://github.com/blackstork-io/fabric/releases) in Fabric GitHub. diff --git a/docs/plugins/builtin.md b/docs/plugins/builtin.md new file mode 100644 index 00000000..5c3317e2 --- /dev/null +++ b/docs/plugins/builtin.md @@ -0,0 +1,163 @@ +--- +title: Built-in +weight: 10 +type: docs +--- + +# Built-in data sources and content providers + +`fabric` binary includes a set of built-in data sources and content providers, available out-of-the-box. + +## Data sources + +### `csv` + +#### Configuration + +The data source supports the following configuration parameters: + +```hcl +config data csv { + delimiter = # optional +} +``` + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data csv { + path = # required +} +``` + +### `inline` + +#### Configuration + +The data source doesn't support configuration. + +#### Usage + +The data source doesn't define any parameters in the `data` block. + +### `json` + +#### Configuration + +The data source doesn't support configuration. + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data json { + glob = # required +} +``` + +### `txt` + +#### Configuration + +The data source doesn't support configuration. + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data txt { + path = # required +} +``` + +## Content providers + +### `frontmatter` + +#### Configuration + +The content provider doesn't support configuration. + +#### Usage + +The content source supports the following parameters in the content blocks: + +```hcl +content frontmatter { + content = # optional + format = # optional +} +``` + +### `image` + +#### Configuration + +The content provider doesn't support configuration. + +#### Usage + +The content source supports the following parameters in the content blocks: + +```hcl +content image { + alt = # optional + src = # required +} +``` + +### `list` + +#### Configuration + +The content provider doesn't support configuration. + +#### Usage + +The content source supports the following parameters in the content blocks: + +```hcl +content list { + format = # optional + item_template = # required +} +``` + +### `table` + +#### Configuration + +The content provider doesn't support configuration. + +#### Usage + +The content source supports the following parameters in the content blocks: + +```hcl +content table { + columns = # required +} +``` + +### `text` + +#### Configuration + +The content provider doesn't support configuration. + +#### Usage + +The content source supports the following parameters in the content blocks: + +```hcl +content text { + absolute_title_size = # optional + code_language = # optional + format_as = # optional + text = # required +} +``` diff --git a/docs/plugins/elasticsearch.md b/docs/plugins/elasticsearch.md new file mode 100644 index 00000000..4992f4f0 --- /dev/null +++ b/docs/plugins/elasticsearch.md @@ -0,0 +1,56 @@ +--- +title: blackstork/elasticsearch +weight: 20 +type: docs +--- + +# `blackstork/elasticsearch` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/elasticsearch" = "=> v0.0.0-dev" + } +} +``` + +## Data sources + +The plugin has the following data sources available: + +### `elasticsearch` + +#### Configuration + +The data source supports the following configuration parameters: + +```hcl +config data elasticsearch { + api_key = # optional + api_key_str = # optional + base_url = # required + basic_auth_password = # optional + basic_auth_username = # optional + bearer_auth = # optional + ca_certs = # optional + cloud_id = # optional +} +``` + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data elasticsearch { + fields = # optional + id = # optional + index = # required + query = # optional + query_string = # optional +} +``` \ No newline at end of file diff --git a/docs/plugins/github.md b/docs/plugins/github.md new file mode 100644 index 00000000..223de247 --- /dev/null +++ b/docs/plugins/github.md @@ -0,0 +1,55 @@ +--- +title: blackstork/github +weight: 20 +type: docs +--- + +# `blackstork/github` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/github" = "=> v0.0.0-dev" + } +} +``` + +## Data sources + +The plugin has the following data sources available: + +### `github_issues` + +#### Configuration + +The data source supports the following configuration parameters: + +```hcl +config data github_issues { + github_token = # required +} +``` + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data github_issues { + assignee = # optional + creator = # optional + direction = # optional + labels = # optional + limit = # optional + mentioned = # optional + milestone = # optional + repository = # required + since = # optional + sort = # optional + state = # optional +} +``` \ No newline at end of file diff --git a/docs/plugins/graphql.md b/docs/plugins/graphql.md new file mode 100644 index 00000000..07338056 --- /dev/null +++ b/docs/plugins/graphql.md @@ -0,0 +1,46 @@ +--- +title: blackstork/graphql +weight: 20 +type: docs +--- + +# `blackstork/graphql` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/graphql" = "=> v0.0.0-dev" + } +} +``` + +## Data sources + +The plugin has the following data sources available: + +### `graphql` + +#### Configuration + +The data source supports the following configuration parameters: + +```hcl +config data graphql { + auth_token = # optional + url = # required +} +``` + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data graphql { + query = # required +} +``` \ No newline at end of file diff --git a/docs/plugins/openai.md b/docs/plugins/openai.md new file mode 100644 index 00000000..771e204d --- /dev/null +++ b/docs/plugins/openai.md @@ -0,0 +1,49 @@ +--- +title: blackstork/openai +weight: 20 +type: docs +--- + +# `blackstork/openai` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/openai" = "=> v0.0.0-dev" + } +} +``` + + + +## Content providers +The plugin has the following content providers available: + +### `openai_text` + +#### Configuration + +The content provider supports the following configuration parameters: + +```hcl +config content openai_text { + api_key = # required + organization_id = # optional + system_prompt = # optional +} +``` + +#### Usage + +The content source supports the following parameters in the content blocks: + +```hcl +content openai_text { + model = # optional + prompt = # required +} +``` diff --git a/docs/plugins/postgresql.md b/docs/plugins/postgresql.md new file mode 100644 index 00000000..2da82be1 --- /dev/null +++ b/docs/plugins/postgresql.md @@ -0,0 +1,46 @@ +--- +title: blackstork/postgresql +weight: 20 +type: docs +--- + +# `blackstork/postgresql` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/postgresql" = "=> v0.0.0-dev" + } +} +``` + +## Data sources + +The plugin has the following data sources available: + +### `postgresql` + +#### Configuration + +The data source supports the following configuration parameters: + +```hcl +config data postgresql { + database_url = # required +} +``` + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data postgresql { + sql_args = # optional + sql_query = # required +} +``` \ No newline at end of file diff --git a/docs/plugins/sqlite.md b/docs/plugins/sqlite.md new file mode 100644 index 00000000..9de43b68 --- /dev/null +++ b/docs/plugins/sqlite.md @@ -0,0 +1,46 @@ +--- +title: blackstork/sqlite +weight: 20 +type: docs +--- + +# `blackstork/sqlite` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/sqlite" = "=> v0.0.0-dev" + } +} +``` + +## Data sources + +The plugin has the following data sources available: + +### `sqlite` + +#### Configuration + +The data source supports the following configuration parameters: + +```hcl +config data sqlite { + database_uri = # required +} +``` + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data sqlite { + sql_args = # optional + sql_query = # required +} +``` \ No newline at end of file diff --git a/docs/plugins/terraform.md b/docs/plugins/terraform.md new file mode 100644 index 00000000..08bfcb81 --- /dev/null +++ b/docs/plugins/terraform.md @@ -0,0 +1,39 @@ +--- +title: blackstork/terraform +weight: 20 +type: docs +--- + +# `blackstork/terraform` plugin + +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{< ref "configs.md#global-configuration" >}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "blackstork/terraform" = "=> v0.0.0-dev" + } +} +``` + +## Data sources + +The plugin has the following data sources available: + +### `terraform_state_local` + +#### Configuration + +The data source doesn't support configuration. + +#### Usage + +The data source supports the following parameters in the data blocks: + +```hcl +data terraform_state_local { + path = # required +} +``` \ No newline at end of file diff --git a/go.mod b/go.mod index 38118e2d..a5468c3e 100644 --- a/go.mod +++ b/go.mod @@ -20,6 +20,7 @@ require ( github.com/pelletier/go-toml/v2 v2.1.1 github.com/sanity-io/litter v1.5.5 github.com/spf13/cobra v1.8.0 + github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.8.4 github.com/testcontainers/testcontainers-go v0.27.0 github.com/testcontainers/testcontainers-go/modules/elasticsearch v0.27.0 @@ -84,7 +85,6 @@ require ( github.com/shirou/gopsutil/v3 v3.23.11 // indirect github.com/shoenig/go-m1cpu v0.1.6 // indirect github.com/sirupsen/logrus v1.9.3 // indirect - github.com/spf13/pflag v1.0.5 // indirect github.com/stretchr/objx v0.5.0 // indirect github.com/tklauser/go-sysconf v0.3.12 // indirect github.com/tklauser/numcpus v0.6.1 // indirect diff --git a/internal/builtin/plugin.go b/internal/builtin/plugin.go index 52cc622b..593e4e57 100644 --- a/internal/builtin/plugin.go +++ b/internal/builtin/plugin.go @@ -6,7 +6,7 @@ import ( func Plugin(version string) *plugin.Schema { return &plugin.Schema{ - Name: "builtin", + Name: "blackstork/builtin", Version: version, DataSources: plugin.DataSources{ "csv": makeCSVDataSource(), diff --git a/internal/builtin/plugin_test.go b/internal/builtin/plugin_test.go index d735d4a5..f58423ac 100644 --- a/internal/builtin/plugin_test.go +++ b/internal/builtin/plugin_test.go @@ -8,7 +8,7 @@ import ( func TestPluginSchema(t *testing.T) { schema := Plugin("1.2.3") - assert.Equal(t, "builtin", schema.Name) + assert.Equal(t, "blackstork/builtin", schema.Name) assert.Equal(t, "1.2.3", schema.Version) assert.NotNil(t, schema.DataSources["csv"]) assert.NotNil(t, schema.DataSources["txt"]) diff --git a/justfile b/justfile index bf5fcd0e..ce51832f 100644 --- a/justfile +++ b/justfile @@ -27,4 +27,7 @@ test-e2e: go test -timeout 5m -race -v ./test/e2e/... generate: - go generate ./... \ No newline at end of file + go generate ./... + +generate-docs: + go run ./tools/docgen --version v0.0.0-dev --output ./docs/plugins/ diff --git a/tools/docgen/main.go b/tools/docgen/main.go new file mode 100644 index 00000000..fafdddbf --- /dev/null +++ b/tools/docgen/main.go @@ -0,0 +1,95 @@ +package main + +import ( + _ "embed" + "fmt" + "os" + "path/filepath" + "strings" + "text/template" + + "github.com/hashicorp/hcl/v2/hcldec" + "github.com/spf13/pflag" + + "github.com/blackstork-io/fabric/internal/builtin" + "github.com/blackstork-io/fabric/internal/elasticsearch" + "github.com/blackstork-io/fabric/internal/github" + "github.com/blackstork-io/fabric/internal/graphql" + "github.com/blackstork-io/fabric/internal/openai" + "github.com/blackstork-io/fabric/internal/postgresql" + "github.com/blackstork-io/fabric/internal/sqlite" + "github.com/blackstork-io/fabric/internal/terraform" + "github.com/blackstork-io/fabric/plugin" +) + +var ( + version string + outputDir string +) + +//go:embed markdown.gotempl +var markdownTempl string + +var templ *template.Template + +func main() { + // parse flags + flags := pflag.NewFlagSet("docgen", pflag.ExitOnError) + flags.StringVar(&version, "version", "v0.0.0-dev", "version of the build") + flags.StringVar(&outputDir, "output", "./dist/docs", "output directory") + flags.Parse(os.Args[1:]) + // ensure output directory exists + if err := os.MkdirAll(outputDir, 0o755); err != nil { + panic(err) + } + // load all plugins + plugins := []*plugin.Schema{ + builtin.Plugin(version), + elasticsearch.Plugin(version), + github.Plugin(version, nil), + graphql.Plugin(version), + openai.Plugin(version, nil), + postgresql.Plugin(version), + sqlite.Plugin(version), + terraform.Plugin(version), + } + // generate markdown for each plugin + for _, p := range plugins { + fp := filepath.Join(outputDir, fmt.Sprintf("%s.md", shortname(p.Name))) + fmt.Printf("Generating '%s': '%s'\n", p.Name, fp) + if err := generate(p, fp); err != nil { + panic(err) + } + } +} + +func generate(schema *plugin.Schema, fp string) error { + f, err := os.Create(fp) + if err != nil { + return err + } + defer f.Close() + return templ.Execute(f, schema) +} + +func shortname(name string) string { + parts := strings.SplitN(name, "/", 2) + if len(parts) == 2 { + return parts[1] + } + return name +} + +func init() { + templ = template.Must(template.New("markdown").Funcs(template.FuncMap{ + "shortname": shortname, + "attrType": func(val hcldec.Spec) string { + switch v := val.(type) { + case *hcldec.AttrSpec: + return v.Type.FriendlyName() + default: + return "unknown" + } + }, + }).Parse(markdownTempl)) +} diff --git a/tools/docgen/markdown.gotempl b/tools/docgen/markdown.gotempl new file mode 100644 index 00000000..990ee825 --- /dev/null +++ b/tools/docgen/markdown.gotempl @@ -0,0 +1,116 @@ +--- +title: {{ if eq .Name "blackstork/builtin" -}}Built-in{{- else -}}{{ .Name }}{{- end }} +{{ if eq .Name "blackstork/builtin" -}}weight: 10{{- else -}}weight: 20{{- end }} +type: docs +--- + +{{ if eq .Name "blackstork/builtin" -}} +# Built-in data sources and content providers +{{- else -}} +# `{{ .Name }}` plugin +{{- end }} + +{{ if eq .Name "blackstork/builtin" -}} +`fabric` binary includes a set of built-in data sources and content providers, available out-of-the-box. +{{ else -}} +## Installation + +To install the plugin, add it to `plugin_versions` map in the Fabric global configuration block (see [Global configuration]({{"{{"}}< ref "configs.md#global-configuration" >{{"}}"}}) for more details), with a version constraint restricting which available versions of the plugin the codebase is compatible with: + +```hcl +fabric { + plugin_versions = { + "{{ .Name }}" = "=> {{ .Version }}" + } +} +``` +{{ end }} +{{ $plugin := . -}} + +{{ with .DataSources -}} +## Data sources +{{- if eq $plugin.Name "blackstork/builtin" -}} +{{ else }} + +The plugin has the following data sources available: +{{- end -}} +{{ range $name, $ds := . }} + +### `{{ $name }}` + +#### Configuration + +{{ with $ds.Config -}} +The data source supports the following configuration parameters: + +```hcl +config data {{ $name }} { +{{- range $key, $value := . }} + {{ $key }} = <{{ $value | attrType }}> # {{if $value.Required}}required{{else}}optional{{end}} +{{- end }} +} +``` +{{- else -}} +The data source doesn't support configuration. +{{- end }} + +#### Usage + +{{with $ds.Args -}} +The data source supports the following parameters in the data blocks: + +```hcl +data {{ $name }} { +{{- range $key, $value := . }} + {{ $key }} = <{{ $value | attrType }}> # {{if $value.Required}}required{{else}}optional{{end}} +{{- end }} +} +``` +{{- else -}} +The data source doesn't define any parameters in the `data` block. +{{- end -}} +{{ end -}} +{{ end }} + +{{- with .ContentProviders }} + +## Content providers +{{ if eq $plugin.Name "blackstork/builtin" -}} +{{ else -}} +The plugin has the following content providers available: +{{ end -}} +{{ range $name, $provider := . }} +### `{{ $name }}` + +#### Configuration + +{{with $provider.Config -}} +The content provider supports the following configuration parameters: + +```hcl +config content {{ $name }} { +{{- range $key, $value := . }} + {{ $key }} = <{{ $value | attrType }}> # {{if $value.Required}}required{{else}}optional{{end}} +{{- end }} +} +``` +{{ else -}} +The content provider doesn't support configuration. +{{ end }} +#### Usage + +{{with $provider.Args -}} +The content source supports the following parameters in the content blocks: + +```hcl +content {{ $name }} { +{{- range $key, $value := . }} + {{ $key }} = <{{ $value | attrType }}> # {{if $value.Required}}required{{else}}optional{{end}} +{{- end }} +} +``` +{{ else }} +The content provider doesn't define any parameters in the `content` block. +{{ end }} +{{- end -}} +{{- end -}}