openlineage, postgres: add OpenLineage support for Postgres #31617
Conversation
boring-cyborg
bot
added
area:dev-tools
area:providers
provider:common-sql
provider:DBs
provider:openlineage
JDarDagran
force-pushed
the
aip-53-postgresql
branch
from
31ef61c
to
5cfb9fe
Compare
JDarDagran
requested review from
potiuk,
ashb,
jedcunningham and
eladkal
as code owners
JDarDagran
force-pushed
the
aip-53-postgresql
branch
2 times, most recently
from
10c4792
to
da3f29c
Compare
| Returns default schema specific to database. | ||
| See: :class:`~providers.openlineage.utils.sqlparser.SQLParser` | ||
| """ | ||
| return self.__schema or "public" |
There was a problem hiding this comment.
If self.__schema is coming from the connection, it will map to the Postgres database, due to the legacy implementation:
https://github.com/apache/airflow/blob/main/airflow/providers/postgres/hooks/postgres.py#L84-L102
Until recently, the Postgres connection didn't allow users to specify the schema. Psycopg would allow us to do this using:
psycopg2.connect(host="localhost",
port="5432",
user="postgres",
password="password",
database="database",
options="-c search_path=dbo,public")
https://stackoverflow.com/questions/59298580/how-to-specify-schema-in-psycopg2-connection-method
But until recently, this wasn't exposed.
We could look into allowing users to define the default schema per database using a Airflow configs, similar to what was done in:
https://astro-sdk-python.readthedocs.io/en/stable/configurations.html#configuring-the-database-default-schema
There was a problem hiding this comment.
I've added support for both database and schema.
JDarDagran
force-pushed
the
aip-53-postgresql
branch
3 times, most recently
from
e9293a4
to
c2c3403
Compare
|
Need to fix the errors |
@uranusjr I'm on it. Spotted that PostgresOperator has the capability to connect to Redshift as well so I'm converting to draft for a bit to provide support for that too. |
JDarDagran
force-pushed
the
aip-53-postgresql
branch
from
c2c3403
to
f514041
Compare
JDarDagran
force-pushed
the
aip-53-postgresql
branch
2 times, most recently
from
b2b130f
to
31dbf3d
Compare
Signed-off-by: Jakub Dardzinski <[email protected]>
Signed-off-by: Jakub Dardzinski <[email protected]>
Signed-off-by: Jakub Dardzinski <[email protected]> Add default port for Postgres connection. Signed-off-by: Jakub Dardzinski <[email protected]>
JDarDagran
force-pushed
the
aip-53-postgresql
branch
from
31dbf3d
to
2b65e35
Compare
This PR adds OpenLineage support for PostgresOperator.
depends on: #31398