Coverage >= 5 cannot handle Cython public or readonly members in .pxd file

[nodakai]

Describe the bug
After upgrading to coverage >= 5,
coverage run outputs this error message for a Cython pxd file with certain conditions I describe below:

Can't add file tracer data for unmeasured file '/home/kai/tst/cy/kai.pxd'

Then coverage report fails with this error:

Couldn't parse '/home/kai/tst/cy/kai.pyx' as Python source: 'invalid syntax' at line 1

To Reproduce

1. What version of coverage.py are you using? The output of coverage debug sys is helpful.

-- sys -------------------------------------------------------
                        version: 5.1
                       coverage: /home/kai/tst/lib64/python3.7/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: Cython.Coverage.Plugin
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                                 setup.cfg
                   configs_read: /home/kai/tst/setup.cfg
                    config_file: /home/kai/tst/setup.cfg
                config_contents: '[coverage:run]\n# http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html\nplugins = Cython.Coverage\n'
                      data_file: -none-
                         python: 3.7.6 (default, Feb 26 2020, 20:54:15) [GCC 7.3.1 20180712 (Red Hat 7.3.1-6)]
                       platform: Linux-4.19.76-linuxkit-x86_64-with-glibc2.2.5
                 implementation: CPython
                     executable: /home/kai/tst/bin/python3
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 9474
                            cwd: /home/kai/tst
                           path: /home/kai/tst/bin
                                 /usr/lib64/python37.zip
                                 /usr/lib64/python3.7
                                 /usr/lib64/python3.7/lib-dynload
                                 /home/kai/tst/lib64/python3.7/site-packages
                                 /home/kai/tst/lib/python3.7/site-packages
                    environment: -none-
                   command_line: /home/kai/tst/bin/coverage debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.7.17
             sqlite3_temp_store: 0
        sqlite3_compile_options: DISABLE_DIRSYNC
                                 ENABLE_COLUMN_METADATA
                                 ENABLE_FTS3
                                 ENABLE_RTREE
                                 ENABLE_UNLOCK_NOTIFY
                                 SECURE_DELETE
                                 TEMP_STORE=1
                                 THREADSAFE=1

1. What versions of what packages do you have installed? The output of pip freeze is helpful.

coverage==5.1
Cython==0.29.16

Tested with coverage 4.5.4 (OK), 5.0.4 (fail), 5.1 (fail)

1. What code are you running? Give us a specific commit of a specific repo that we can check out.
   cy/__init__.pxd (empty file)

cy/kai.pxd

cdef class Kai:
    cdef readonly bint x

    cpdef bint get_x(self)

cy/kai.pyx

cdef class Kai:
    def __init__(self):
        self.x = False

    cpdef bint get_x(self):
        return self.x

cy/setup.py

from Cython.Build import cythonize
from distutils.core import setup
from distutils.extension import Extension

define_macros = [('CYTHON_TRACE', '1')]

compiler_directives = {'always_allow_keywords': True, 'profile': True, 'linetrace': True}
extensions = [
    Extension('cy.kai', ['cy/kai.pyx'], define_macros=define_macros),
    ]
setup(ext_modules=cythonize(extensions, nthreads=4, annotate=True, language_level="3", compiler_directives=compiler_directives))

runner.py

from cy import kai

k = kai.Kai()
print(k.get_x())
print(k.x)

1. What commands did you run?

$ python3 cy/setup.py build_ext --inplace
$ coverage run runner.py
$ coverage report

Expected behavior
Coverage statistics for pyx and pxd files

Additional context
The problem manifests itself only when

1. coverage >= 5
2. public or readonly members are in .pxd file, separate from .pyx file, and
3. runner.py actually accesses the members directly

My understanding is readonly and public modifiers make Cython define wrapper functions for them transparently. coverage >= 5 seem to play badly with them.

[nedbat]

Thanks, I see the error from "coverage report", but I don't see the message you show for "coverage run". Do you have any other context I need?

[nedbat]

I got it from the "debug sys" output you showed: you have a .coveragerc file with this:

[coverage:run]
# http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html
plugins = Cython.Coverage

Now I see the same results you do.

[jbrockmendel]

xref cython/cython#3515 tentative workaround is to add a dummy inline function in the affected .pxd file

[saullocastro]

The best workaround I could find for this was to edit the .coveragerc file adding the relative path of all Cython .pxd files in the list of files to be ignored by coverage. More details on https://github.com/saullocastro/composites:

[run]
plugins = Cython.Coverage
source = composites
omit =
    composites/core.pxd

[matusvalo]

I was able to bisect the commit of the issue: 106828c. The root cause is following: reproducer described in this issue is yielding following data in parameter d:

coveragepy/coverage/collector.py

Line 417 in 5ec587c

def mapped_file_dict(self, d):

{
    '/Users/matus/dev/cython/test/coverage/runner.py': {1: None, 3: None, 4: None, 5: None}, 
    '/Users/matus/dev/cython/test/coverage/cy/kai.pyx': {5: None, 1: None, 16: None, 11: None, 3: None, 6: None}, 
    '/Users/matus/dev/cython/test/coverage/cy/kai.pxd': {}
}

It can be seen that kai.pxd file is mapped to empty dictionary. This is obviously filtered in following line:

coveragepy/coverage/collector.py

Line 434 in 5ec587c

return {self.cached_mapped_file(k): v for k, v in items if v}

This is causing that kai.pxd is not inserted in database here:

coveragepy/coverage/sqldata.py

Lines 383 to 394 in 5ec587c

	def _file_id(self, filename, add=False):
	"""Get the file id for `filename`.
	If filename is not in the database yet, add it if `add` is True.
	If `add` is not True, return None.
	"""
	if filename not in self._file_map:
	if add:
	with self._connect() as con:
	cur = con.execute("insert or replace into file (path) values (?)", (filename,))
	self._file_map[filename] = cur.lastrowid
	return self._file_map.get(filename)

When in the later stage the data needs to be fetched from database, the final error is raised due missing record in DB:

coveragepy/coverage/sqldata.py

Lines 549 to 553 in 5ec587c

	file_id = self._file_id(filename)
	if file_id is None:
	raise DataError(
	f"Can't add file tracer data for unmeasured file '{filename}'"
	)

As a proof of concept I have created a PR which seems to fix the issue. I am not familiar with coverage codebase so I cannot tell if this is proper fix or the issue needs to be fixed somewhere else.

[nedbat]

This is now released as part of coverage 6.4.3.