feat: supprt empty ibis.spark.connect()

[lostmygithubaccount]

Is your feature request related to a problem?

you can do:

import ibis

con = ibis.connect("pyspark://")

(as long as you have PySpark and whatnot installed)

but not:

import ibis

con = ibis.pyspark.connect()

you need to pass in the session object in the latter case; in the former, it seems to be automagically created on your behalf

of course, feel free to close if this is intentional or technical reasons for this behavior or whanot

[ins] In [1]: import ibis
c
[ins] In [2]: con = ibis.connect("pyspark://")
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/02/28 19:07:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/02/28 19:07:34 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
c
[ins] In [3]: con = ibis.pyspark.connect()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 con = ibis.pyspark.connect()

File ~/repos/ibis/ibis/__init__.py:107, in __getattr__.<locals>.connect(*args, **kwargs)
    106 def connect(*args, **kwargs):
--> 107     return backend.connect(*args, **kwargs)

File ~/repos/ibis/ibis/backends/__init__.py:764, in BaseBackend.connect(self, *args, **kwargs)
    740 """Connect to the database.
    741
    742 Parameters
   (...)
    761
    762 """
    763 new_backend = self.__class__(*args, **kwargs)
--> 764 new_backend.reconnect()
    765 return new_backend

File ~/repos/ibis/ibis/backends/__init__.py:778, in BaseBackend.reconnect(self)
    776 def reconnect(self) -> None:
    777     """Reconnect to the database already configured with connect."""
--> 778     self.do_connect(*self._con_args, **self._con_kwargs)

TypeError: Backend.do_connect() missing 1 required positional argument: 'session'

Describe the solution you'd like

the ibis.pyspark.connect() just works

What version of ibis are you running?

main

What backend(s) are you using, if any?

pyspark

Code of Conduct

• I agree to follow this project's Code of Conduct