Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Initial implementation of HTTP connector #1649

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
002c4f8
Initial implementation of HTTP connector
edgarrmondragon Apr 28, 2023
9ea4189
Merge branch 'main' into http-connector
May 24, 2023
86ce0a1
Merge branch 'main' into http-connector
edgarrmondragon May 24, 2023
375bfa3
Merge branch 'main' into http-connector
edgarrmondragon Jul 11, 2023
a408431
Feedback: Address private method and attribute access for connector API
edgarrmondragon Jul 11, 2023
79c4f56
Fix warning line
edgarrmondragon Jul 11, 2023
9045268
Test deprecations
edgarrmondragon Jul 12, 2023
536bb2f
Fix session type annotation
edgarrmondragon Jul 12, 2023
8927a0e
Merge branch 'main' into http-connector
edgarrmondragon Jul 12, 2023
f0619cf
Merge branch 'main' into http-connector
edgarrmondragon Jul 14, 2023
e72e193
Merge branch 'main' into http-connector
edgarrmondragon Jul 14, 2023
f2c442a
Merge branch 'main' into http-connector
edgarrmondragon Jul 17, 2023
557b121
Merge branch 'main' into http-connector
edgarrmondragon Jul 18, 2023
fe06ab2
Merge branch 'main' into http-connector
edgarrmondragon Jul 18, 2023
493ba22
Merge branch 'main' into http-connector
edgarrmondragon Jul 27, 2023
138efbe
Rename type var
edgarrmondragon Jul 27, 2023
7fec03e
Fix types
edgarrmondragon Jul 27, 2023
1b5b753
Merge branch 'main' into http-connector
edgarrmondragon Jul 27, 2023
4f3f2d5
Merge branch 'main' into http-connector
edgarrmondragon Aug 2, 2023
dcf83f3
Merge branch 'main' into http-connector
edgarrmondragon Dec 7, 2023
5a53ce7
Merge branch 'main' into http-connector
edgarrmondragon Jan 11, 2024
e5a7c99
Merge branch 'main' into http-connector
edgarrmondragon Jan 19, 2024
ecb522a
Merge branch 'main' into http-connector
edgarrmondragon Jan 22, 2024
ec574c8
Merge branch 'main' into http-connector
edgarrmondragon Jan 24, 2024
87f94a5
Merge branch 'main' into http-connector
edgarrmondragon Jan 30, 2024
495d744
Merge branch 'main' into http-connector
edgarrmondragon Jan 31, 2024
1c3b6f0
Merge branch 'main' into http-connector
edgarrmondragon May 29, 2024
0b35bde
chore: Run `poetry lock` to install the latest transitive dependencies
edgarrmondragon May 29, 2024
eedc63f
Make Ruff happy
edgarrmondragon May 29, 2024
911da30
Merge branch 'main' into http-connector
edgarrmondragon Aug 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/classes/singer_sdk.connectors.BaseConnector.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
singer_sdk.connectors.BaseConnector
===================================

.. currentmodule:: singer_sdk.connectors

.. autoclass:: BaseConnector
:members:
:special-members: __init__, __call__
32 changes: 32 additions & 0 deletions docs/guides/custom-connector.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Using a custom connector class

The Singer SDK has a few built-in connector classes that are designed to work with a variety of sources:

* [`SQLConnector`](../../classes/singer_sdk.SQLConnector) for SQL databases

If you need to connect to a source that is not supported by one of these built-in connectors, you can create your own connector class. This guide will walk you through the process of creating a custom connector class.

## Subclass `BaseConnector`

The first step is to create a subclass of [`BaseConnector`](../../classes/singer_sdk.connectors.BaseConnector). This class is responsible for creating streams and handling the connection to the source.

```python
from singer_sdk.connectors import BaseConnector


class MyConnector(BaseConnector):
pass
```

## Implement `get_connection`

The [`get_connection`](http://127.0.0.1:5500/build/classes/singer_sdk.connectors.BaseConnector.html#singer_sdk.connectors.BaseConnector.get_connection) method is responsible for creating a connection to the source. It should return an object that implements the [context manager protocol](https://docs.python.org/3/reference/datamodel.html#with-statement-context-managers), e.g. it has `__enter__` and `__exit__` methods.

```python
from singer_sdk.connectors import BaseConnector


class MyConnector(BaseConnector):
def get_connection(self):
return MyConnection()
```
1 change: 1 addition & 0 deletions docs/guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ The following pages contain useful information for developers building on top of

porting
pagination-classes
custom-connector
```
9 changes: 9 additions & 0 deletions docs/reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,12 @@ Batch

batch.BaseBatcher
batch.JSONLinesBatcher

Abstract Connector Classes
--------------------------

.. autosummary::
:toctree: classes
:template: class.rst

connectors.BaseConnector
3 changes: 2 additions & 1 deletion noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@
test_dependencies = [
"coverage[toml]",
"pytest",
"pytest-snapshot",
"pytest-durations",
"pytest-httpserver",
"pytest-snapshot",
"freezegun",
"pandas",
"pyarrow",
Expand Down
103 changes: 68 additions & 35 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ types-simplejson = "^3.18.0"
types-PyYAML = "^6.0.12"
coverage = {extras = ["toml"], version = "^7.2"}
pyarrow = ">=11,<13"
pytest-httpserver = "^1.0.6"
pytest-snapshot = "^0.9.0"

# Cookiecutter tests
Expand Down
Loading