Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InterfaceError: Error binding parameter 1 - probably unsupported type. #1138

Closed
felipejinli opened this issue Apr 3, 2021 · 6 comments
Closed
Labels
bug Something isn't working not our bug The problem was elsewhere
Milestone

Comments

@felipejinli
Copy link

felipejinli commented Apr 3, 2021

Describe the bug
When running pytest with the pytest-cov package (includes coveragepy), an INTERNALERROR is thrown. This looks similar to #1010 @nedbat , but it seems like the bug is still not fixed as my project is using v5.5.0 of coverage.

To Reproduce
Run command pytest --cov-report=term-missing --ignore=./test/functional/ --cov=onionbalance
NB. if the above command is ran without the cov option, no internal error is thrown, BUT the coverage report is not generated still

platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/***/Desktop/onionbalance, configfile: setup.cfg
plugins: cov-2.11.1, mock-3.5.1
collected 46 items                                                              

test/v2/test_consensus.py ....                                            [  8%]
test/v2/test_descriptor.py .....                                          [ 19%]
test/v2/test_settings.py ..                                               [ 23%]
test/v2/test_util.py ............................                         [ 84%]
test/v3/test_v3_descriptor.py .                                           [ 86%]
test/v3/test_v3_hashring.py .                                             [ 89%]
test/v3/test_v3_keys.py .                                                 [ 91%]
test/v3/test_v3_onionbalance.py ..                                        [ 95%]
test/v3/test_v3_status.py ..                                              [100%]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 230, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 659, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 727, in get_data
INTERNALERROR>     if self._collector and self._collector.flush_data():
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/collector.py", line 442, in flush_data
INTERNALERROR>     self.covdata.add_lines(self.mapped_file_dict(self.data))
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 438, in add_lines
INTERNALERROR>     self._choose_lines_or_arcs(lines=True)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 495, in _choose_lines_or_arcs
INTERNALERROR>     with self._connect() as con:
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 300, in _connect
INTERNALERROR>     self._create_db()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 252, in _create_db
INTERNALERROR>     db.executemany(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 1106, in executemany
INTERNALERROR>     return self.con.executemany(sql, data)
INTERNALERROR> sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.

Expected behavior
I expected a coverage report to be generated without throwing an InternalError.

@felipejinli felipejinli added the bug Something isn't working label Apr 3, 2021
@nedbat
Copy link
Owner

nedbat commented Apr 4, 2021

Thanks for the report. Is this reproducible for you? It looks like the "fix" for #1010 needs to be applied in another place, though you are not using PyPy, which was a key element in #1010.

Can you try this patch?

diff --git a/coverage/sqldata.py b/coverage/sqldata.py
index a150fdfd..f58486ac 100644
--- a/coverage/sqldata.py
+++ b/coverage/sqldata.py
@@ -1103,7 +1103,14 @@ def executemany(self, sql, data):
         if self.debug:
             data = list(data)
             self.debug.write("Executing many {!r} with {} rows".format(sql, len(data)))
-        return self.con.executemany(sql, data)
+        try:
+            return self.con.executemany(sql, data)
+        except Exception:
+            # In some cases, an error might happen that isn't really an
+            # error.  Try again immediately.
+            # https://github.com/nedbat/coveragepy/issues/1010
+            # https://github.com/nedbat/coveragepy/issues/1138
+            return self.con.executemany(sql, data)

     def executescript(self, script):
         """Same as :meth:`python:sqlite3.Connection.executescript`."""

Or you can try installing the fix from my branch:

pip install git+https://github.com/nedbat/coveragepy.git@nedbat/bug1138#egg=coverage

@nedbat nedbat added this to the 5.next milestone Apr 4, 2021
@felipejinli
Copy link
Author

@nedbat Thanks for the quick response. However, I still get an INTERNALERROR. What I did? I tried patching it by first pip uninstalling coverage and then installed coverage from your branch ( 5.5.1a0 ).

Have you got any other insights as to how this can be fixed?

See the logs:

pytest --cov-report=term-missing --ignore=./test/functional/ --cov=onionbalance
============================== test session starts ==============================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/***/Desktop/onionbalance, configfile: setup.cfg
plugins: cov-2.11.1, mock-3.5.1
collected 46 items                                                              

test/v2/test_consensus.py ....                                            [  8%]
test/v2/test_descriptor.py .....                                          [ 19%]
test/v2/test_settings.py ..                                               [ 23%]
test/v2/test_util.py ............................                         [ 84%]
test/v3/test_v3_descriptor.py .                                           [ 86%]
test/v3/test_v3_hashring.py .                                             [ 89%]
test/v3/test_v3_keys.py .                                                 [ 91%]
test/v3/test_v3_onionbalance.py ..                                        [ 95%]
test/v3/test_v3_status.py ..                                              [100%]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 1107, in executemany
INTERNALERROR>     return self.con.executemany(sql, data)
INTERNALERROR> sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 230, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 659, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 727, in get_data
INTERNALERROR>     if self._collector and self._collector.flush_data():
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/collector.py", line 442, in flush_data
INTERNALERROR>     self.covdata.add_lines(self.mapped_file_dict(self.data))
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 438, in add_lines
INTERNALERROR>     self._choose_lines_or_arcs(lines=True)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 495, in _choose_lines_or_arcs
INTERNALERROR>     with self._connect() as con:
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 300, in _connect
INTERNALERROR>     self._create_db()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 252, in _create_db
INTERNALERROR>     db.executemany(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 1113, in executemany
INTERNALERROR>     return self.con.executemany(sql, data)
INTERNALERROR> sqlite3.IntegrityError: UNIQUE constraint failed: meta.key

@nedbat
Copy link
Owner

nedbat commented Apr 4, 2021

It seems like you have an unusual situation. Can you give me a way to reproduce it?

@nedbat
Copy link
Owner

nedbat commented Apr 4, 2021

Though: the error you report here could just be that it's too simplistic to retry all of the statements in an executemany, since some will have succeeded by the time one of them fails, and retrying all of them will cause the constraint failure we're seeing.

Either way, it would be helpful to have a way to reproduce it.

@felipejinli
Copy link
Author

felipejinli commented Apr 4, 2021

@nedbat Yes, absolutely. To reproduce, run the following commands for this dev branch https://github.com/felipejinli/onionbalance/tree/dev.
0. should be on root project directory onionbalance

  1. pip install -r requirements.txt
  2. pip install -r test-requirements.txt
  3. pytest --cov-report=term-missing --ignore=./test/functional/ --cov=onionbalance
  4. pip uninstall coverage
  5. pip install git+https://github.com/nedbat/coveragepy.git@nedbat/bug1138#egg=coverage
    PS. I'm unsure as to how I should link your branch's coverage patch into requirements.txt (say in a situation where the patch fully fixed it to avoid steps 4 and 5). Have you got any suggestions?
    Regarding reproducibility, lmk if you have any issues. Thanks!

@nedbat
Copy link
Owner

nedbat commented Apr 4, 2021

Thanks for the reproducible case.

The problem is your mock of datetime.datetime in test_outdated_consensus. You never undo that mock, so datetime.datetime is a mock object for the rest of the test run. Coverage.py later tries to get the current time to put in its database, and gets a mock object instead of a timestamp. SQLite can't write the mock object to the database.

The best thing to do would be to use mock.patch as a context manager instead. It will automatically clean up the mock at the end of the test:

    def test_outdated_consensus(self):
        current_time = datetime.datetime.fromtimestamp(10101010101)

        consensus = DummyConsensus()

        consensus.consensus = mock.Mock()
        with mock.patch("datetime.datetime") as mock_datetime:
            consensus.consensus.valid_after = current_time
            # valid_until is 3 hours in the future
            consensus.consensus.valid_until = current_time + datetime.timedelta(seconds=3600*3)

            # Test some legitimate cases
            mock_datetime.utcnow.return_value = current_time
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time + datetime.timedelta(seconds=3600*11)
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time + datetime.timedelta(seconds=3600*12)
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time + datetime.timedelta(seconds=3600*24)
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time - datetime.timedelta(seconds=3600*24)
            self.assertTrue(consensus.is_live())

            # Now test some bad cases. The is_live() function is lenient up to 24
            # hours after the valid_until, or 24 hours before the valid_after
            mock_datetime.utcnow.return_value = consensus.consensus.valid_until + datetime.timedelta(seconds=3600*24+1)
            self.assertFalse(consensus.is_live())

            mock_datetime.utcnow.return_value = consensus.consensus.valid_after - datetime.timedelta(seconds=3600*24+1)
            self.assertFalse(consensus.is_live())

As another tip: it's better to mock where you use an object, rather than where it is defined. That will limit the scope to just the code that needs the mock. That might be harder in this case, it looks like a few different places use utcnow.

@nedbat nedbat closed this as completed Apr 4, 2021
@nedbat nedbat added the not our bug The problem was elsewhere label Apr 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working not our bug The problem was elsewhere
Projects
None yet
Development

No branches or pull requests

2 participants