max-model-input-length argument not working in prepare input, lead to python kernel crash

[JasonRichard]

eland/eland/ml/pytorch/transformers.py

max-model-input-length argument not working in prepare input,
max_length uses max_length from model weight/config.json
leads to python kernel crash when max_length is too large

class _TraceableTextEmbeddingModel(_TransformerTraceableModel):
    def _prepare_inputs(self) -> transformers.BatchEncoding:
        return self._tokenizer(
            "This is an example sentence.",
            padding="max_length", #may lead to python kernel crash
            return_tensors="pt",
        )

maybe change it to 'longest' ?

[davidkyle]

Hi @JasonRichard

Which model were you importing when you saw the error and please can you share the command you do use so I can reproduce the error. And which version of Eland are you using?