diff --git a/docs/api.rst b/docs/api.rst index c0a6a0b..997daf2 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -9,14 +9,17 @@ etlhelper etlhelper.row_factories -^^^^^^^^^^^^^^^^^^^^^^^ +----------------------- .. automodule:: etlhelper.row_factories :members: -etlhelper.db_helpers -^^^^^^^^^^^^^^^^^^^^^^^ + +.. _db_helpers: + +DB Helpers +^^^^^^^^^^^ .. automodule:: etlhelper.db_helpers - :members: + diff --git a/docs/conf.py b/docs/conf.py index 5c329a0..52bc5d4 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -40,6 +40,7 @@ # ones. extensions = [ 'sphinx.ext.autodoc', + 'sphinx.ext.viewcode' ] # Add any paths that contain templates here, relative to this directory. @@ -84,6 +85,9 @@ html_theme_options = { 'logo': None, 'sidebar_collapse': True, + 'show_relbar_top': False, + 'show_relbar_bottom': True, + 'show_related': True } html_show_sourcelink = True diff --git a/docs/connecting_to_databases.rst b/docs/connecting_to_databases.rst index 634f8b6..16db870 100644 --- a/docs/connecting_to_databases.rst +++ b/docs/connecting_to_databases.rst @@ -1,6 +1,6 @@ .. _connecting_to_databases: -Connecting to Databases +Connecting to databases ======================= @@ -10,6 +10,8 @@ Users are free to create their own connections directly or to supply them from c Alternatively, ETL Helper's :func:`DbParams ` class can be used. +.. _db_params: + DbParams ^^^^^^^^ diff --git a/docs/etl_functions.rst b/docs/etl_functions.rst index 7d4e7b0..32e20e3 100644 --- a/docs/etl_functions.rst +++ b/docs/etl_functions.rst @@ -1,4 +1,4 @@ -ETL Functions +ETL functions ============= Extract diff --git a/docs/index.rst b/docs/index.rst index 6e07a8b..82a3681 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -28,7 +28,7 @@ ETL Helper is a Python ETL library to simplify data transfer into and out of dat dependency list to avoid disruption and watch the project on GitHub for notification of new releases. -``etlhelper`` makes it easy to run SQL queries via Python and return the +ETL Helper makes it easy to run SQL queries via Python and return the results. It is built upon the `DBAPI2 specification `__ and takes care of cursor management, importing drivers and formatting connection strings, @@ -36,9 +36,12 @@ while providing memory-efficient functions to read, write and transform data. This reduces the amount of boilerplate code required to manipulate data within relational databases with Python. +Features +^^^^^^^^ + - ``fetchall``, ``iter_rows``, ``fetchone`` functions for querying databases -- ``execute``, ``executemany``, and ``load`` functions to insert data +- ``execute``, ``executemany``, and ``load`` functions to insert or update data - ``copy_rows`` and ``copy_table_rows`` to transfer data between databases - Data transfer uses memory-efficient generators (``iter_chunks`` and ``iter_rows``) - User-defined transform functions transform data in-flight @@ -51,49 +54,52 @@ databases with Python. These tools can create easy-to-understand, lightweight, versionable and testable Extract-Transform-Load (ETL) workflows. -This documentation site explains how the main features are used. -See the individual function docstrings for full details of parameters and -options. ETL Helper components ^^^^^^^^^^^^^^^^^^^^^ ETL Helper has three components. -The *ETL functions* are used to extract, transform and load rows of data from relational databases. +The :doc:`etl_functions` are used to extract, transform and load rows of data from relational databases. They can be used with any DB API 2.0-compliant database connections. Logging and helpful error messages are provided out-of-the-box. -The *DbParams* class provides a convenient way to define database connections. +A :ref:`DbParams ` class provides a convenient way to define database connections. For any given database system, it identifies the correct driver, the required parameters and defines connection strings. It provides convenience methods for checking databases are reachable over a network and for connecting to them. -The *DbHelper* classes work behind the scenes to smooth out inconsistencies between different database systems. +The :ref:`DbHelper ` classes work behind the scenes to smooth out inconsistencies between different database systems. They also apply database-specific optimisations e.g., using the faster ``executebatch`` function for PostgreSQL connections instead of ``executemany``. In normal use, users do not interact with the DbHelper classes. -Example -^^^^^^^^ +Quickstart examples +^^^^^^^^^^^^^^^^^^^ + +Loading data +------------ -The following script shows how to create a database table and load data into it. +The following script uses the ``execute``, ``load`` and ``fetchall`` functions to +create a database table and populate it with data. .. literalinclude:: demo_load.py :language: python -The output is +The output is: .. code:: bash {'id': 1, 'name': 'basalt', 'grain_size': 'fine'} {'id': 2, 'name': 'granite', 'grain_size': 'coarse'} +Copying data +------------ This script copies the data to another database, with transformation and logging. .. literalinclude:: demo_copy.py :language: python -The output is +The output is: .. code:: bash @@ -105,4 +111,4 @@ The output is # 2024-05-08 14:57:42,057 executemany: 2 rows processed in total {'id': 1, 'name': 'basalt', 'category': 'igneous', 'last_update': '2024-05-08 14:59:54.878726'} - {'id': 2, 'name': 'granite', 'category': 'igneous', 'last_update': '2024-05-08 14:59:54.879034'} \ No newline at end of file + {'id': 2, 'name': 'granite', 'category': 'igneous', 'last_update': '2024-05-08 14:59:54.879034'} diff --git a/etlhelper/db_helpers/__init__.py b/etlhelper/db_helpers/__init__.py index 0e5e985..9ab2e4d 100644 --- a/etlhelper/db_helpers/__init__.py +++ b/etlhelper/db_helpers/__init__.py @@ -1,10 +1,19 @@ """ DbHelper classes are used behind-the-scenes for customising the behaviour of different database drivers. +They are not normally called directly. For more details, see the source code in each module. + +.. autoclass:: DbHelper +.. autoclass:: SQLiteDbHelper +.. autoclass:: PostgresDbHelper +.. autoclass:: OracleDbHelper +.. autoclass:: MSSQLDbHelper + """ # flake8: noqa +from etlhelper.db_helpers.db_helper import DbHelper from etlhelper.db_helpers.oracle import OracleDbHelper from etlhelper.db_helpers.postgres import PostgresDbHelper from etlhelper.db_helpers.mssql import MSSQLDbHelper