Skip to content

Commit

Permalink
Go all in
Browse files Browse the repository at this point in the history
Signed-off-by: Jo Stichbury <[email protected]>
  • Loading branch information
stichbury committed Apr 3, 2023
1 parent 17dd1d5 commit 45b5a16
Show file tree
Hide file tree
Showing 25 changed files with 208 additions and 141 deletions.
10 changes: 10 additions & 0 deletions docs/source/configuration/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Configuration

```{toctree}
:hidden:
configuration_basics
credentials
parameters
advanced_configuration
```
12 changes: 12 additions & 0 deletions docs/source/contribution/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Contribute to Kedro

```{toctree}
:hidden:
contribute_to_kedro
developer_contributor_guidelines
backwards_compatibility
documentation_contributor_guidelines
technical_steering_committee
development_for_databricks
```
8 changes: 8 additions & 0 deletions docs/source/data/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Data Catalog

```{toctree}
:hidden:
data_catalog
kedro_io
```
17 changes: 17 additions & 0 deletions docs/source/deployment/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Deployment

```{toctree}
:hidden:
deployment_guide
single_machine
distributed
argo
prefect
kubeflow
aws_batch
aws_sagemaker
aws_step_functions
airflow_astronomer
dask
```
12 changes: 12 additions & 0 deletions docs/source/development/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Development

```{toctree}
:hidden:
set_up_vscode
set_up_pycharm
commands_reference
debugging
automated_testing
linting
```
9 changes: 9 additions & 0 deletions docs/source/extend_kedro/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Extend Kedro

```{toctree}
:hidden:
common_use_cases
custom_datasets
plugins
```
9 changes: 0 additions & 9 deletions docs/source/get_started/get_started.md

This file was deleted.

9 changes: 9 additions & 0 deletions docs/source/get_started/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# First steps

```{toctree}
:hidden:
install
new_project
kedro_concepts
```
9 changes: 9 additions & 0 deletions docs/source/hooks/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Hooks

```{toctree}
:hidden:
introduction
common_use_cases
examples
```
108 changes: 28 additions & 80 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -54,142 +54,90 @@ Welcome to Kedro's documentation!
.. toctree::
:maxdepth: 2

introduction/introduction_index.md
introduction/index.md

.. toctree::
:maxdepth: 2

get_started/get_started.md
get_started/index.md

.. toctree::
:maxdepth: 2

tutorial/tutorial.md
tutorial/index.md

.. toctree::
:maxdepth: 2

visualisation/visualisation.md
visualisation/index.md

.. toctree::
:maxdepth: 2
:caption: Notebooks & IPython users

notebooks_and_ipython/kedro_and_notebooks
notebooks_and_ipython/kedro_as_a_data_registry
notebooks_and_ipython/index.md

.. toctree::
:maxdepth: 2
:caption: Kedro project setup

kedro_project_setup/starters
kedro_project_setup/dependencies
kedro_project_setup/session
kedro_project_setup/settings

kedro_project_setup/index.md

.. toctree::
:maxdepth: 2
:caption: Configuration

configuration/configuration_basics
configuration/credentials
configuration/parameters
configuration/advanced_configuration

configuration/index.md

.. toctree::
:maxdepth: 2
:caption: Data Catalog

data/data_catalog
data/kedro_io

data/index.md

.. toctree::
:maxdepth: 2
:caption: Nodes and pipelines

nodes_and_pipelines/nodes
nodes_and_pipelines/pipeline_introduction
nodes_and_pipelines/modular_pipelines
nodes_and_pipelines/pipeline_registry
nodes_and_pipelines/micro_packaging
nodes_and_pipelines/run_a_pipeline
nodes_and_pipelines/slice_a_pipeline

nodes_and_pipelines/index.md

.. toctree::
:maxdepth: 2
:caption: Extend Kedro

extend_kedro/common_use_cases
extend_kedro/custom_datasets
extend_kedro/plugins

extend_kedro/index.md

.. toctree::
:maxdepth: 2
:caption: Hooks

hooks/introduction
hooks/common_use_cases
hooks/examples

hooks/index.md


.. toctree::
:maxdepth: 2
:caption: Logging

logging/logging
logging/index.md

.. toctree::
:maxdepth: 2
:caption: Integrations

integrations/databricks.rst
integrations/pyspark.rst

integrations/databricks.md
integrations/pyspark.md

.. toctree::
:maxdepth: 2
:caption: Development

development/set_up_vscode
development/set_up_pycharm
development/commands_reference
development/debugging
development/automated_testing
development/linting

development/index.md

.. toctree::
:maxdepth: 2
:caption: Deployment

deployment/deployment_guide
deployment/single_machine
deployment/distributed
deployment/argo
deployment/prefect
deployment/kubeflow
deployment/aws_batch
deployment/aws_sagemaker
deployment/aws_step_functions
deployment/airflow_astronomer
deployment/dask

deployment/index.md

.. toctree::
:maxdepth: 2
:caption: Resources

faq/faq
resources/glossary

resources/index.md


.. toctree::
:maxdepth: 2
:caption: Contribute to Kedro

contribution/contribute_to_kedro
contribution/developer_contributor_guidelines
contribution/backwards_compatibility
contribution/documentation_contributor_guidelines
contribution/technical_steering_committee
contribution/development_for_databricks
contribution/index.md

API documentation
=================
Expand Down
3 changes: 1 addition & 2 deletions docs/source/integrations/databricks.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# Databricks integration

```{toctree}
:caption: Databricks
:maxdepth: 2
:hidden:
databricks_workspace.md
databricks_visualisation.md
Expand Down
3 changes: 1 addition & 2 deletions docs/source/integrations/pyspark.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# PySpark integration

```{toctree}
:caption: PySpark
:maxdepth: 2
:hidden:
pyspark_integration.md
```
11 changes: 11 additions & 0 deletions docs/source/introduction/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Introduction to Kedro

```{toctree}
:hidden:
introduction
```

Kedro is an open-source Python framework to create reproducible, maintainable, and modular data science code. It uses software engineering best practices to help you build production-ready data science pipelines.

Kedro is hosted by the [LF AI & Data Foundation](https://lfaidata.foundation/), and you can find the [Kedro source code on GitHub](https://github.com/kedro-org/kedro).

8 changes: 1 addition & 7 deletions docs/source/introduction/introduction.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
# What is Kedro?

Kedro is an open-source Python framework to create reproducible, maintainable, and modular data science code. It uses software engineering best practices to help you build production-ready data science pipelines.

Kedro is hosted by the [LF AI & Data Foundation](https://lfaidata.foundation/), and you can find the [Kedro source code on GitHub](https://github.com/kedro-org/kedro).

## Learn how to use Kedro
# Learn how to use Kedro

In the following chapters, you will learn [how to set up Kedro](../get_started/install.md) and discover the [key Kedro concepts](../get_started/kedro_concepts.md). You can then review the [spaceflights tutorial](../tutorial/tutorial_template.md) to get hands-on experience with a Kedro project.

Expand Down
7 changes: 0 additions & 7 deletions docs/source/introduction/introduction_index.md

This file was deleted.

10 changes: 10 additions & 0 deletions docs/source/kedro_project_setup/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Kedro project setup

```{toctree}
:hidden:
starters
dependencies
session
settings
```
17 changes: 17 additions & 0 deletions docs/source/logging/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Logging

```{toctree}
:hidden:
logging
```

Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in two places:
1. [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). This cannot be altered.
2. Your project-side logging configuration. Every project generated using Kedro's CLI `kedro new` command includes a file `conf/base/logging.yml`. You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md).

```{note}
Providing project-side logging configuration is entirely optional. You can delete the `conf/base/logging.yml` file and Kedro will run using the framework's built in configuration.
```

Framework-side and project-side logging configuration are loaded through subsequent calls to [`logging.config.dictConfig`](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig). This means that, when it is provided, the project-side logging configuration typically _fully overwrites_ the framework-side logging configuration. [Incremental configuration](https://docs.python.org/3/library/logging.config.html#incremental-configuration) is also possible if the `incremental` key is explicitly set to `True` in your project-side logging configuration.
13 changes: 1 addition & 12 deletions docs/source/logging/logging.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,5 @@
# Logging

Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in two places:
1. [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). This cannot be altered.
2. Your project-side logging configuration. Every project generated using Kedro's CLI `kedro new` command includes a file `conf/base/logging.yml`. You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md).

```{note}
Providing project-side logging configuration is entirely optional. You can delete the `conf/base/logging.yml` file and Kedro will run using the framework's built in configuration.
```

Framework-side and project-side logging configuration are loaded through subsequent calls to [`logging.config.dictConfig`](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig). This means that, when it is provided, the project-side logging configuration typically _fully overwrites_ the framework-side logging configuration. [Incremental configuration](https://docs.python.org/3/library/logging.config.html#incremental-configuration) is also possible if the `incremental` key is explicitly set to `True` in your project-side logging configuration.

## Default framework-side logging configuration
# Default framework-side logging configuration

Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the [Rich logging handler](https://rich.readthedocs.io/en/stable/logging.html) to format messages. We also use the [Rich traceback handler](https://rich.readthedocs.io/en/stable/traceback.html) to render exceptions.

Expand Down
13 changes: 13 additions & 0 deletions docs/source/nodes_and_pipelines/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Nodes and pipelines

```{toctree}
:hidden:
nodes
pipeline_introduction
modular_pipelines
pipeline_registry
micro_packaging
run_a_pipeline
slice_a_pipeline
```
8 changes: 8 additions & 0 deletions docs/source/notebooks_and_ipython/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Notebooks & iPython users

```{toctree}
:hidden:
kedro_and_notebooks
kedro_as_a_data_registry
```
Loading

1 comment on commit 45b5a16

@astrojuanlu
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go all in

🔥

Please sign in to comment.