Inference params support

[serena-ruan]

Related Issues/PRs

Inference params are parameters that are passed to the model at inference time. These parameters do not need to be specified when training the model, but could be useful for inference. In some cases, especially popular LLMs, the same model may require different parameter configurations for different samples at inference time.

With params support, you can now specify a dictionary of inference params during model inference, providing a broader utility and improved control over the generated inference results, particularly for LLM use cases. By passing different params such as temperature, max_length, etc. to the model at inference time, you can easily control the output of the model.

What changes are proposed in this pull request?

(Please fill in changes proposed in this fix)

How is this patch tested?

• Existing unit/integration tests
• New unit/integration tests
• Manual tests (describe details, including test results, below)

Does this PR change the documentation?

• No. You can skip the rest of this section.
• Yes. Make sure the changed pages / sections render correctly in the documentation preview.

Release Notes

Is this a user-facing change?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release notes for MLflow users.

Support extra params for model inference.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/gateway: AI Gateway service, Gateway client APIs, third-party Gateway integrations
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• rn/feature - A new user-facing feature worth mentioning in the release notes
• rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• rn/documentation - A user-facing documentation change worth mentioning in the release notes

[mlflow-automation]

Documentation preview for 01eeba2 will be available here when this CircleCI job completes successfully.

More info

• Ignore this comment if this PR does not change the documentation.
• It takes a few minutes for the preview to be available.
• The preview is updated when a new commit is pushed to this PR.
• This comment was created by https://github.com/mlflow/mlflow/actions/runs/5688711952.

[dbczumar]

LGTM based on extensive bug bashing and the fact that all of the individual PRs that went into this feature branch were reviewed, tested, and approved. Thanks so much, @serena-ruan !

[liangz1]

@serena-ruan Does it mean the "shape" is only supposed to be used with tensor values?
If "shape" also applies to lists, then for a list like [101, 102], shouldn't the "shape" of it be (2,)?
Maybe this can be clarified by providing an example of a value whose shape is (2, 3), so that people understand the regular case.

[serena-ruan]

shape is supposed to only be used for list/array values, and currently params only support scalar value or 1D array value, so shape should either be None or (-1,). And for list like in this case [101, 102] it's a 1-dimensional array so the shape is (-1,), actually for whatever 1D array shape (-1,) always work.

[liangz1]

What if I also save the model with inference_config? Is it not allowed? If that is allowed, if I run inference without params, would the inference_config or default params be used?

[serena-ruan]

That's a great point!! I should add a test case for this. It is allowed, and if you run inference without params, then both inference_config and default params in the ModelSignature are applied, if there're overlaps params takes priority.