Question about column-level linting

[mattfishburn]

Hello,

I'm trying to understand if there is a technical reason that column-level linting does not exist, or if it could exist but doesn't for any reason (no one has asked for it, no one has implemented it, it's outside the bounds of pandera, etc.). For the code below the line validated_df.identifier + 'abc' could have a similar mypy error to the line 1 + 'abc', but it does not. Is there a technical reason that this is impossible? Or is there another reason that pandera doesn't support it? I'm using pandera version 0.16.

Thanks,
Matt

P.S. pandera is great, thank you for creating it!

import pandera as pa
import pandas as pd
from pandera.typing import DataFrame as TypedDf


class __IntStrSchema(pa.DataFrameModel):
    identifier: int
    message: str


IntStrSchema = TypedDf[__IntStrSchema]


def transform(df: IntStrSchema) -> IntStrSchema:
    # do stuff
    df_temp = IntStrSchema(df.copy())
    df_temp.identifier += 1
    return df_temp


data = [{'identifier': 1, 'message': 'foo'}, {'identifier':2, 'message': 'bar'}]

pandas_df = pd.DataFrame(data)
validated_df = IntStrSchema(data)

# mypy error: incompatible type
print(transform(pandas_df))

# this line passes the linter, and also gives run-time validation
print(transform(validated_df))

try:
    # this line also passes the linter, but would fail in an ideal world
    print(validated_df.identifier + 'abc')
except Exception:
    print('Exception when adding identifier and abc')

try:
    # mypy error: Unsupported operand types for +
    print(1 + 'abc')
except Exception:
    print('Exception when adding int and abc')

[cosmicBboy]

Great questions @mattfishburn! short answer: no one has built a working prototype of this functionality yet.

As far as I know the challenges around this is that IntStrSchema is effectively a pandas DataFrame type. What we'd need to do is expose the attributes and their types of the underlying pandera schema __IntStrSchema and having mypy recognize valid/invalid operations on it.

This related to the fact that, unlike say pydantic Models, which are effectively python objects, pandera Models are simply classes containing metadata about a dataframe, not dataframes themselves.

If you (or a future reader of this post) can create a working prototype of this in pandera that would be 🔥, would definitely support this effort.