v1.0.4-spark3.5

chore: Adding Spark35 support

v1.0.4-spark3.3

chore: downgrade to spark3.3

SynapseML v1.0.4

v1.0.4

Bug Fixes 🐞

• companionModelClassName no longer returns generic type variable (#2195)
• Fix tag for pyCodeGenImpl (#2194)

Build 🏭

• bump azure/login from 1 to 2 (#2176)
• bump azure/CLI from 1 to 2 (#2178)

Maintenance 🔧

• Bump version to 1.0.4 (#2200)
• fix flaky HyperOpt NB (#2198)
• Bump python version to 3.11 (#2193)
• exclude non-executable docs from automated tests (#2197)
• add retry logic to build steps (#2192)
• update openai api version to 2024 (#2190)
• fix 1.0.1 version tags (#2189)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.

SynapseML v1.0.3

[v1.0.3]

Bug Fixes 🐞

• repair failing speech tests (#2179)
• update openai completion (#2142)

Build 🏭

• bump github/codeql-action from 2 to 3 (#2148)
• bump actions/setup-python from 4 to 5 (#2146)
• bump peter-evans/create-or-update-comment from 3 to 4 (#2162)
• bump actions/upload-artifact from 3.1.3 to 4.3.1 (#2165)

Features 🌈

• OpenAI embeddings with GPU based KNN (#2157)
• Synthetic difference in differences (#2095)

Maintenance 🔧

• Update to version 1.0.3 (#2183)
• update build system service principals (#2181)
• raise error with documentation link - find_secret (#2180)
• check Fabric Tenant (#2175)
• rotate outdates SAS url in speech tests (#2173)
• Support Token Provider Mode (#2160)
• Bump isolation forest to 3.0.4 (#2168)
• Add script to generate pypi mfa qr (#2150)
• Add Unified Logging Base Class for Python (#2159)
• fix failing tests (#2153)

v1.0.3-spark3.5

chore: Adding Spark35 support

v1.0.3-spark3.3

chore: downgrade to spark3.3

SynapseML v0.10.2-spark3.3

v1.0.2-spark3.3

chore: downgrade to spark3.3

SynapseML v1.0.2

[SynapseML v1.0.2]

Bug Fixes 🐞

• Add the error handling for Langchain transformer (#2137)
• use java class loader (#2135)
• Support to Bool input for Onnx models (#2130)

Build 🏭

• bump amannn/action-semantic-pull-request from 5.3.0 to 5.4.0 (#2125)

Doc

• update find_secret on Fabric and doc (#2132)

Documentation 📘

• update CONTRIBUTING.md (#2138)
• fix install instructions (#2136)
• fix readme install
• add audiobook paper to README
• add analyze text document (#2127)
• use the new AnalyzeText API in docs(#2126)
• removing spark 3.2 instructions

Maintenance 🔧

• bump to v1.0.2 (#2140)
• change udf vec2array to pyspark.ml.functions.vector_to_array (#2131)
• fix failing notebooks (#2134)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• 522661a chore: bump to v1.0.2 (#2140)
• 2a01c8e doc: update find_secret on Fabric and doc (#2132)
• 23222c0 fix: Add the error handling for Langchain transformer (#2137)
• f3ae146 fix: use java class loader (#2135)
• fc3a999 docs: update CONTRIBUTING.md (#2138)
• 9b20829 docs: fix install instructions (#2136)
• c10f46e docs: fix readme install
• 28cd6db chore: change udf vec2array to pyspark.ml.functions.vector_to_array (#2131)
• 46a1ef8 docs: add audiobook paper to README
• 5e9bae1 build: bump amannn/action-semantic-pull-request from 5.3.0 to 5.4.0 (#2125)

See More

• 241062f docs: add analyze text document (#2127)
• 4623219 review docs (#2128)
• 9195dee fix: Support to Bool input for Onnx models (#2130)
• 4c4fc8a chore: fix failing notebooks (#2134)
• 90ded80 docs: use the new AnalyzeText API in docs(#2126)
• 5cd78c9 Improve LightGBM Network logs (#2124)
• a187cd0 docs: removing spark 3.2 instructions

This list of changes was auto generated.

SynapseML v1.0.1

v1.0.1

Documentation 📘

• pointing cognitive apis to azure ai (#2119)
• bump readme to spark 3.4

Maintenance 🔧

• bump to v1.0.1 (#2123)
• add back in exclusions (#2122)

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of SynapseML.\n

Changes:

• cb4fd82 chore: bump to v1.0.1 (#2123)
• 91e8c85 chore: add back in exclusions (#2122)
• d240cbb docs: pointing cognitive apis to azure ai (#2119)
• 77be641 docs: bump readme to spark 3.4
• ef435a2 chore: bump to v1.0.0 (#2120)
• c2fdb05 chore: Adding Spark34 support (#2052) (#2116)
• 903dc6b docs: move cognitive namespace to services namespace (#2118)
• fd00b87 chore: refactor cognitive package to services (#2117)
• b0caf2e build: bump @babel/traverse from 7.18.9 to 7.23.2 in /website (#2098)
• c12afc5 chore: bump speech sdk version (#2107)

See More

• 1af71ed docs: update anomaly detector docs (#2103)
• 377df2f build: bump ossf/scorecard-action from 2.3.0 to 2.3.1 (#2108)
• cd43ee7 fix: unit test break in TranslatorSuite (#2111)
• cc77eda chore: removing gpt-review (#2113)
• 70dc523 fix: gpt-review action (#2112)

This list of changes was auto generated.

SynapseML v1.0.0

We are excited to announce the release and general availability of SynapseML v1.0 following seven years of continuous development. SynapseML is an open-source library that aims to streamline the development of massively scalable machine learning pipelines. It unifies several existing ML Frameworks and new Microsoft algorithms in a single, scalable API that is usable across Python, R, Scala, and Java. SynapseML is usable from any Apache Spark platform and is now generally available with enterprise support on Microsoft Fabric.

Highlights

Distributed Langchain	Vector Search Indices	Semantic Link
Deploy your LLM apps on millions of documents	Quickly create semantic and multi-modal search engines	Work with PowerBI datasets natively from Microsoft Fabric
View Notebook	Try an Example	Learn More

Keyless AI Services	Orthogonal Forests
Use built-in AI services without keys in Microsoft Fabric	Discover and measure heterogeneous causal effects
Learn More	Try an Example

New Features

General ✨

• Add support for spark 3.4.1 (#2052) (#2116)
• Enterprise support on Microsoft Fabric

Open AI and Langchain 🦜

• Add the LangchainTransformer for orchestrating LLMs at scale (#1925, #2036)
• Add ChatGPT through the OpenAIChatCompletion transformer (#1887)
• Add Langchain notebook (#2002, #2013)
• Add OpenAI document Q+A notebook (#2029, #2033)
• Add custom chatbot creation to form recognition demo (#1888)

Azure AI Services 🧠

• Add Support for Azure Cognitive Search Vector Indices (#2041)
• Add keyless Azure AI services on Microsoft Fabric (#2070, #1859)
• Support new form recognizer APIs (#1882)
• Support streaming multivariate anomaly detection (#1893)
• Add prerequisites page for setting up OpenAI and Azure AI services (#2008)

Deep Learning 🕸

• ONNX models support variable size inputs (#1851)
• Add distributed training overview (#1879)

Causal Learning 📈

• Add OrthogonalForestDML for causal learning with heterogeneous effects (#1873)
• Add Heterogeneous Effect Quickstart
• Support custom reference distribution in DistributionBalanceMeasures to detect data drift (#1885)
• Add statistical significance reporting for causal learners using getPValue (#1863)

LightGBM 🌳

• LightGBM streaming mode is now default (#2088)
• LightGBM supports passing Reference datasets to speed repeated execution (#1977)
• Add LightGBM streaming mode docs (#1992)

Additional Updates

Bug Fixes 🐞

• Improve LGBM exception and logging (#2037)
• AI Services and other HTTP Clients no longer retry 4XX codes other than 429 (#2005)
• Make geospatial services robust to 404s thrown by the service (#2007)
• Fix bug #1869, where DoubleML .setFitIntercept should default to true (#1876)
• Fix Multivariate Anomaly error handling (#1991)
• Fix import error when using AI services on Azure Machine Learning clusters (#1951)
• Fix default values of aadToken & url on Fabric (#1918)
• Fix ONNX model shape inference on batches with shape [-1] (#1906)
• Add getPValue to python API of DoubleML (#1909)
• Add diagnosticsInfo in Multivariate Anomaly detection response (#1892)
• Fix Double ML timeout on large datasets (#1903)
• Retry OnnxHub calls to improve test reliability (#1889)
• Remove case matching for erased generic types (#1880)
• Remove extraneous Foo type from Python codegen (#1867)
• Update OpenAIEmbedding Schema to account for internalServiceType
• Update Maven package to include correct GitHub path (#2073)

Documentation 📘

• Automatically create Azure docs from notebooks (#1976, #1911,#2023, #2021, #2043)
• Improve OpenAI Docs (#1938, #1937, #1999)
• Improved LightGBM docs (#2003)
• Improve Vowpal Wabbit Docs (#1971, #1972, #1970, #1969, #1968, #2072)
• General notebook quality improvements (#1979, #1932)
• Improve Causal learning docs (#1905)
• Remove old notebooks and demos (#1934)
• Fix R-setup.md docs (#1946)
• Fix broken links across website and repo (#2079, #2076, #2042, #2032, #2027, #2026, #2025,#2022, #1864, #2035, #2049)
• Fix Recommenders repo URL (#2086)
• Fix website developer API link (#1877)
• Migrate from Cognitive Services to Azure AI services (#2119, #2118)
• Update anomaly detect...