Update logos in index.md #5

…sary store configs (#837)

Add tests that read JSON files that describe flows and expected results. The goal is to run these tests for each genkit language, to make sure all implementations behave the same. There is currently a single test, whose expected trace came from writing the flow in ts and calling `genkit flow:run`. This test found a couple of bugs and discrepancies in the Go code, which are also fixed here. For #354.

Propagates the trace sampled state to the correct field in GCP logs

Update Devtools page and add some images and gifs --------- Co-authored-by: Kevin Cheung <[email protected]>

Add a guide for writing a plugin that exports telemetry data. --------- Co-authored-by: Konstantin Mandrika <[email protected]> Co-authored-by: Kevin Cheung <[email protected]>

Add a link to the API dashboard which lists all APIs enabled for a project.

Updated Genkit's getting started with firebase guide to include an "npm init -y" command in the project setup.

* More visual get-started * fix link

Add missing import to example.

…run (#13)

* Rewrite Firebase deployment docs * put cors step in the right place * indentation * prettier

* fix cmd formatting get-started.md (#868) * Fix issue with Firebase plugin --------- Co-authored-by: Kevin Cheung <[email protected]>

Co-authored-by: Chris Chestnut <[email protected]> Co-authored-by: Cleo Schneider <[email protected]> Co-authored-by: Samuel Bushi <[email protected]>

1. Prompt sample 2. Flow and tool sample 3. Chat sample 4. RAG sample 5. Multimodal and chaining sample

* Remove js/third_party from .gitignore * Update formatting for ragas metrics

Add the first Go sample, which implements the "basic" flow from samples/flow-sample1. Attempting to run this with `genkit start -a ...` flushed out a few bugs. This PR fixes enough so that the flow is listed in the left nav of the UI. The next step requires adding JSON schemas to the actions, which is a large piece, so this is a good place to stop.

This is a simple Pinecone client that provides just the features that we will want to use for Genkit. This simple client just uses the REST API. There is a Pinecone Go package at github.com/pinecone-io/go-pinecone, but 1) it is unstable; 2) for some reason it recommends building with "just" which is not ordinary practice for Go modules.

The intent is that the actual implementations of embedders and retrievers will implement these interfaces. They will be wrapped in actions for invocation from the genkit tool. A retriever will use the embedder interface to embed a Document. We may want to have a way for the retriever to invoke an embedder action, so that tracing works nicely.

* Changing eval sample to dog facts. * Changing rag sample topic to cat and dog facts

…filename (#38)

Just to be more consistent with the other action types.

Add a test that corresponds to the basic flow sample. Provide a way to add comments to the test JSON. Fix a bug in creating a unique step name that the test uncovered.

* fix cmd formatting get-started.md (#868) * rebase to main * continued wip; refactor * working code! * Clean up * Making sure evals can still be saved * wip with awesome new shape * WIP: Handle failure cases - we still aren't properly setting the status on the span. * Wrap runInNewSpan in try/catch to attempt to force an error in the inner span * Change metrics to throw errors instead of returning error responses. * Add a mutex so that the localFileStore isn't reading an empty file and then overwriting itself * Remove erroneous code and add logging. * discovery * Try to make things serial * Successfully adding trace and span ids into refactored eval state * Renaming in response to feedback * Make error spans show output * remove sampleIndex where it's not being used * add continue to failure case * Refactor evaluation types * Use Eval instead of Evaluator * Fix UI tests * feedback * Remove unused mocks and deps * Rename to TestCase --------- Co-authored-by: Kevin Cheung <[email protected]> Co-authored-by: Chris Chestnut <[email protected]> Co-authored-by: Cleo Schneider <[email protected]>

* [Go] make pinecone test more reliable The pinecone database is only eventually consistent. Wait for the results to be inserted before doing a query. I tried a few different approaches before finding one that seems to pass, and added the Stats method while working on this. Also, delete the test vectors when the test completes. * address reviewer comments

* [Go] add fake embedder implementation for testing * move files up one level

Will serve as a base for prompt-specific runner features that we will add.

* Revise Cloud Run docs * oops * rephrase * auth json not needed * start command * no openai

* Update overview page * link * prettier * DevUI run demo * no openai * feedback

Now calling pinecone.Init will register genkit actions.

… shown, we'll still see the previous message. (#67)

Set the input and output JSON schemas on an action by determining them from the Go types. Also make other minor changes needed to make the genkit UI work. With this PR, you can use the genkit UI to run a flow.

Change jsonschemagen to write types into more than one package. (Also, have it report unused config, as an extra check.) Move the AI types into their own package.

)

Use different inputs to test prompt runner UI behavior.

* Adding initial maliciousness eval * Fix eval-run command * Add additional example and tweak the prompt to get better output. * Turn off safety filters for maliciousness eval * Fix formatting. * Add a sample with the gut check dataset and instructions for running. * Fixing broken build and removing unused dependencies

Also removes references to OpenAI from UI.

* Also removed the process.exist from reflectionApi hook since that was killing the process before the open telemetry SDK could flush everything.

* Update faithfulness to v0.1.7 * Update METADATA

* Resolve app process kill right away. * Update init.ts * Clean up promise.

* Resolve app process kill right away. * Update init.ts * Clean up promise. * Clean up tracing before shutting down Reflection API. * Better implementation. * Fix await. * Update tracing.ts * format --------- Co-authored-by: Pavel Jbanov <[email protected]>

Co-authored-by: Chris Chestnut <[email protected]> Co-authored-by: Michael Doyle <[email protected]>

* Add answer relevancy to the gut check sample and tweak the prompt to penalize not answering the question * Add a preamble to the gut check sample

* Add faithfulness gut check dataset. * Replace Satyagraha test case with Silas B. Cobb test case

* [Go] add dotprompt parsing code * Apply review comments. Also gofmt the files.

…nent (#125)

Fixes google/genkit#130. This is more of a stop-gap fix, going to explore refactoring these components to utilize Angular signals to eliminate this class of error entirely.

* Add evaluators to vertexAI plugin * Remove commented code * Remove unused schemas * Replace fluency with groundedness * Throw error if context is an empty array * Format evaluation.ts * Address PR comments * Address PR comments

Any Action can stream its results back incrementally instead of returning them all at the end. The same is true for Flows. Modify Action and Flow to take a callback to be used for streaming. The dev server runAction route supports a boolean "stream" query parameter, which if true sets the streaming callback to write lines of JSON to the ResponseWriter. This matches the js implementation. For now, assume embedders, retrievers and indexers do not stream. Generators can stream, and they should, but this PR just lays the groundwork.

* Support writing results to csv * Responding to feedback * Switch execution order to write to store first.

* Remove optional metric_spec and hardcoded version * Keep metric_spec, which is actually required

* Update to use fuzzy matching * typo * Use Json5 instead of dirty-json * Make all parsing fuzzy * Update extractJson doc Co-authored-by: Michael Bleigh <[email protected]> --------- Co-authored-by: Michael Bleigh <[email protected]>

These types are only used in the UI, so moving out of the `genkit-tools/src` folder.

Also, update dubious maliciousness examples

* Remove outdated comment

Co-authored-by: Chris Chestnut <[email protected]>

Contains all common code/types that are shared amongst different tools clients (e.g. CLI, UI, etc.).

…ugin` (#165) Removes circular dependency.

* Update sample * Add evaluation tools plugin * feedback * typos * Merge and update step selectors * Update deps * Update order

Fixes ESM warning.

* Update and rename _book.yaml to _guides.yaml For move to Firebase docs * Update _guides.yaml Fixing the formatting * Update _guides.yaml * Update .prettierignore

* Move evaluators out of ragas and into genkit-evaluators * Rename genkit-evaluators to evaluators * Simularity -> similarity

)

…ators (#157) * Add support for defining display name and metric definition for evaluators * Fix import * Remove context_precision from local file eval store test * Replace context precision with faithfulness * Rename METRICS to METRICS_METADATA * Move shared types to type.ts * Only pass specific EvalMetric to eval-metrics-chip component * Address more PR comments

Add the RunFlow function, which runs a flow in the context of another flow. Also add the StreamFlow function, which runs a flow with streaming in the context of another flow.

* WIP * Add span.status.message to error callout * check for error first * Use span error first * Use flow error message as title if available * Type fix * Update condition * condition change again * Simplify * Missing ?

…#192)

The types files were moved in 06e5d49fbbf07793e92c3aefb9f4ecca0e3d9ffe.

Add a plugin for installing Google Cloud Platform telemetry. This PR implements tracing. Future PRs will add metrics and logging.

* Prefer reference to ground_truth * Add BLEU and ROUGE Vertex AI evaluators

* [Go] update generated types for TypeScript type changes * update for changes in 90298872c

Add support for exporting metrics to GCP. Calling `googlecloud.Init` will register a global MetricProvider, which can be used to create counters and other metric instruments. We use a global MeterProvider instead of having one per registry, to keep things simple. Define metric instruments for number of actions executed and their latency, and record those metrics when actions are run.

NoStream was defined in a way that made it impossible to write the corresponding Flow[I, O, S] type outside the package.

Fixes #207.

* [Go] define LookupGeneratorAction * [Go] define LookupAction * [Go] add TODO

* Updating first instance of Genkit to Firebase Genkit`< * prettier updates to make the linter happy

* Update sample * Add evaluation tools plugin * feedback * typos * new lock * WIP * WIP * WIP * WIP * Working, moved to common/ * With tests * fixes * Feedback

* Adding an example for writing custom evaluators * Addressing PR feedback and breakage after rebase * Fix dataset formatting * Missed z.infer reference

… in prod (#180)

Co-authored-by: Kevin Cheung <[email protected]>

…ations (#208)

…229)

Co-authored-by: Kevin Cheung <[email protected]>

) * Align Go and TS metric names. Change the separator to '/' from'.'.

* Update docs on custom extractors * Update evaluation.md * Kevin fix Co-authored-by: Kevin Cheung <[email protected]> * Kevin fix Co-authored-by: Kevin Cheung <[email protected]> * Update docs/evaluation.md Co-authored-by: Kevin Cheung <[email protected]> * Update docs/evaluation.md Co-authored-by: Kevin Cheung <[email protected]> * Update docs/evaluation.md Co-authored-by: Kevin Cheung <[email protected]> * Update docs/evaluation.md Co-authored-by: Kevin Cheung <[email protected]> --------- Co-authored-by: Pavel Jbanov <[email protected]> Co-authored-by: Kevin Cheung <[email protected]>

Ran these commands: go get -u google.golang.org/api go mod tidy Fixes #232

Change the name of the action latency histogram from "genkit.action.action_latency" to "genkit.action.latency" to avoid stutter.

)

…244)

…243) Co-authored-by: Pavel Jbanov <[email protected]>

* Added Go support to `genkit start`. * Added graceful shutdown of dev server. * Formatting. * Update utils.ts * Update init.ts

…gin name (#249)

…s follow the convention {pluginName}/{actionName} (#253)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update logos in index.md #5

Update logos in index.md #5

Commits on May 1, 2024