Skip to content

Commit

Permalink
Merge pull request #1 from om-ai-lab/add_doc/v0.2.1
Browse files Browse the repository at this point in the history
Add doc/v0.2.1
  • Loading branch information
panregedit authored Dec 4, 2024
2 parents ebfacbe + 18f8d51 commit 4826503
Show file tree
Hide file tree
Showing 92 changed files with 1,617 additions and 79 deletions.
3 changes: 0 additions & 3 deletions docs/en/Core Concepts/Memory/intro.md

This file was deleted.

3 changes: 0 additions & 3 deletions docs/en/Core Concepts/Tool/intro.md

This file was deleted.

3 changes: 0 additions & 3 deletions docs/en/Getting Started/1_quick_start.md

This file was deleted.

3 changes: 0 additions & 3 deletions docs/en/Getting Started/3_visual.md

This file was deleted.

112 changes: 112 additions & 0 deletions docs/en/Guides/outfit_with_loop.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Outfit Recommendation with Loop Example

This example demonstrates how to use the framework for outfit recommendation tasks with loop functionality. The example code can be found in the `examples/step3_outfit_with_loop` directory.
```bash
cd examples/step3_outfit_with_loop
```

## Overview

This example implements an interactive outfit recommendation workflow that uses a loop-based approach to refine recommendations based on user feedback. The workflow consists of the following key components:

1. **Initial Image Input**
- OutfitImageInput: Handles the upload and processing of the initial clothing item image
- Serves as the starting point for the recommendation process

2. **Interactive QA Loop with Weather Integration**
- OutfitQA: Conducts an interactive Q&A session to gather context and preferences
- Uses web search tool to fetch real-time weather data for the specified location
- OutfitDecider: Evaluates if sufficient information has been collected based on:
- User preferences
- Current weather conditions
- Uses DoWhileTask to continue the loop until adequate information is gathered
- Loop terminates when OutfitDecider returns decision=true

3. **Final Recommendation**
- OutfitRecommendation: Generates the final outfit suggestions based on:
- The initial uploaded image
- Information collected during the Q&A loop
- Current weather conditions from web search
- Other context (occasion, preferences, etc.)

4. **Workflow Flow**
```
Start -> Image Input -> OutfitQA Loop (QA + Weather Search + Decision) -> Final Recommendation -> End
```

The workflow leverages Redis for state management and the Conductor server for workflow orchestration. This architecture enables:
- Image-based outfit recommendations
- Weather-aware outfit suggestions using real-time data
- Interactive refinement through structured Q&A
- Context-aware suggestions incorporating multiple factors
- Persistent state management across the workflow


## Prerequisites

- Python 3.10+
- Required packages installed (see requirements.txt)
- Access to OpenAI API or compatible endpoint
- Access to Bing API key for web search functionality to search real-time weather information for outfit recommendations (see configs/tools/websearch.yml)
- Redis server running locally or remotely
- Conductor server running locally or remotely

## Configuration

The container.yaml file is a configuration file that manages dependencies and settings for different components of the system, including Conductor connections, Redis connections, and other service configurations. To set up your configuration:

1. Generate the container.yaml file:
```bash
python compile_container.py
```
This will create a container.yaml file with default settings under `examples/step3_outfit_with_loop`.

2. Configure your LLM settings in `configs/llms/gpt.yml` and `configs/llms/text_res.yml`:
- Set your OpenAI API key or compatible endpoint through environment variable or by directly modifying the yml file
```bash
export custom_openai_key="your_openai_api_key"
export custom_openai_endpoint="your_openai_endpoint"
```
- Configure other model settings like temperature as needed through environment variable or by directly modifying the yml file

3. Configure your Bing Search API key in `configs/tools/websearch.yml`:
- Set your Bing API key through environment variable or by directly modifying the yml file
```bash
export bing_api_key="your_bing_api_key"
```

4. Update settings in the generated `container.yaml`:
- Modify Redis connection settings:
- Set the host, port and credentials for your Redis instance
- Configure both `redis_stream_client` and `redis_stm_client` sections
- Update the Conductor server URL under conductor_config section
- Adjust any other component settings as needed

## Running the Example

1. Run the outfit recommendation workflow:

For terminal/CLI usage:
```bash
python run_cli.py
```

For app/GUI usage:
```bash
python run_app.py
```


## Troubleshooting

If you encounter issues:
- Verify Redis is running and accessible
- Check your OpenAI API key and Bing API key are valid
- Ensure all dependencies are installed correctly
- Review logs for any error messages
- Confirm Conductor server is running and accessible
- Check Redis Stream client and Redis STM client configuration

## Building the Example

Coming soon! This section will provide detailed instructions for building the step3_outfit_with_loop example step by step.
149 changes: 149 additions & 0 deletions docs/en/Guides/outfit_with_ltm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# Outfit Recommendation with Long-Term Memory Example

This example demonstrates how to use the framework for outfit recommendation tasks with long-term memory functionality. The example code can be found in the `examples/step4_outfit_with_ltm` directory.
```bash
cd examples/step4_outfit_with_ltm
```


## Overview

This example implements an outfit recommendation system with long-term memory capabilities through two main workflows:

1. **Image Storage Workflow**
- ImageIndexListener: Monitors and captures new clothing images
- OutfitImagePreprocessor: Processes and prepares images for storage
- Stores processed images in Milvus long-term memory (LTM) for future retrieval
- Workflow sequence: Image Listening -> Preprocessing -> LTM Storage

2. **Outfit Recommendation Workflow**
- OutfitQA: Conducts interactive Q&A to understand user preferences
- OutfitDecider: Determines if sufficient information is collected
- Uses DoWhileTask for iterative refinement until decision is positive
- OutfitGeneration: Generates outfit recommendations using stored image data
- OutfitConclusion: Presents final recommendations with explanations

The system leverages both short-term memory (Redis STM) and long-term memory (Milvus LTM) for:
- Efficient image storage and retrieval
- Persistent clothing item database
- Context-aware outfit recommendations
- Interactive preference refinement
- Stateful conversation management

3. **Workflow Architecture**
```
Image Storage: Listen -> Preprocess -> Store in LTM
Recommendation: QA Loop (QA + Decision) -> Generation -> Conclusion
```

The system uses Redis for state management, Milvus for long-term image storage, and Conductor for workflow orchestration. This architecture enables:
- Scalable image database management
- Intelligent outfit recommendations based on stored items
- Interactive preference gathering
- Persistent clothing knowledge base
- Efficient retrieval of relevant items

## Prerequisites

- Python 3.10+
- Required packages installed (see requirements.txt)
- Access to OpenAI API or compatible endpoint (see configs/llms/gpt.yml)
- Access to Bing API key for web search functionality to search real-time weather information for outfit recommendations (see configs/tools/websearch.yml)
- Redis server running locally or remotely
- Conductor server running locally or remotely
- Milvus vector database (will be started automatically when workflow runs)
- Sufficient storage space for image database
- Install Git LFS by `git lfs intall`, then pull sample images by `git lfs pull`

## Configuration

The container.yaml file is a configuration file that manages dependencies and settings for different components of the system, including Conductor connections, Redis connections, Milvus connections and other service configurations. To set up your configuration:

1. Generate the container.yaml files:
```bash
# For image storage workflow
python image_storage/compile_container.py

# For outfit recommendation workflow
python outfit_from_storage/compile_container.py
```
This will create two container.yaml files with default settings under `image_storage` and `outfit_from_storage` directories:
- `image_storage/container.yaml`: Configuration for the image storage workflow
- `outfit_from_storage/container.yaml`: Configuration for the outfit recommendation workflow

2. Configure your LLM settings in `configs/llms/gpt.yml` and `configs/llms/text_res.yml` in the two workflow directories:
- Set your OpenAI API key or compatible endpoint through environment variable or by directly modifying the yml file
```bash
export custom_openai_key="your_openai_api_key"
export custom_openai_endpoint="your_openai_endpoint"
```
- Configure other model settings like temperature as needed through environment variable or by directly modifying the yml file

3. Configure your Bing Search API key in `configs/tools/websearch.yml` in the two workflow directories:
- Set your Bing API key through environment variable or by directly modifying the yml file
```bash
export bing_api_key="your_bing_api_key"
```
4. Configure your text encoder settings in `configs/llms/text_encoder.yml` in the two workflow directories:
- Set your OpenAI text encoder endpoint and API key through environment variable or by directly modifying the yml file
```bash
export custom_openai_text_encoder_key="openai_text_encoder_key"
export custom_openai_text_encoder_endpoint="your_openai_endpoint"
```
- The default text encoder configuration uses OpenAI text embedding v3 with 3072 dimensions, make sure you change the dim value of `MilvusLTM` in `container.yaml`
- Adjust the embedding dimension and other settings as needed through environment variable or by directly modifying the yml file

4. Update settings in the generated `container.yaml`:
- Modify Redis connection settings:
- Set the host, port and credentials for your Redis instance
- Configure both `redis_stream_client` and `redis_stm_client` sections
- Update the Conductor server URL under conductor_config section
- Configure MilvusLTM in `components` section:
- Set the `storage_name` and `dim` for MilvusLTM
- Adjust other settings as needed
- Adjust any other component settings as needed

## Running the Example

1. Run the image storage workflow first:

For terminal/CLI usage:
```bash
python image_storage/run_image_storage_cli.py
```
For app usage:
```bash
python image_storage/run_image_storage_app.py
```

This workflow will store outfit images in the Milvus database.

2. Run the outfit recommendation workflow in a separate terminal:

For terminal/CLI usage:
```bash
python outfit_from_storage/run_outfit_recommendation_cli.py
```

For app/GUI usage:
```bash
python outfit_from_storage/run_outfit_recommendation_app.py
```

This workflow will retrieve outfit recommendations from the stored images.


## Troubleshooting

If you encounter issues:
- Verify Redis is running and accessible
- Check your OpenAI API key and Bing API key are valid
- Ensure all dependencies are installed correctly
- Review logs for any error messages
- Confirm Conductor server is running and accessible
- Check Redis Stream client and Redis STM client configuration

## Building the Example

Coming soon! This section will provide detailed instructions for building the step4_outfit_with_ltm example step by step.

112 changes: 112 additions & 0 deletions docs/en/Guides/outfit_with_switch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Outfit Recommendation with Switch Example

This example demonstrates how to use the framework for outfit recommendation tasks with switch_case functionality. The example code can be found in the `examples/step2_outfit_with_switch` directory.

```bash
cd examples/step2_outfit_with_switch
```

## Overview

This example implements an outfit recommendation workflow that uses switch-case functionality to conditionally include weather information in the recommendation process. The workflow consists of the following key components:

1. **Input Interface**
- Handles user input containing clothing requests and image data
- Processes and caches any uploaded images
- Extracts the user's outfit request instructions

2. **Weather Decision Logic**
- WeatherDecider: Analyzes the user's request to determine if weather information is needed
- Makes a binary decision (0 or 1) based on context in the user's request
- Controls whether weather data should be fetched

3. **Conditional Weather Search**
- WeatherSearcher: Only executes if WeatherDecider returns 0 (weather info needed)
- Uses web search functionality to fetch current weather conditions
- Integrates weather data into the recommendation context

4. **Outfit Recommendation**
- Generates final clothing suggestions based on:
- User's original request
- Weather information (if available)
- Any provided image context
- Provides complete outfit recommendations

The workflow follows this sequence:


## Prerequisites

- Python 3.10+
- Required packages installed (see requirements.txt)
- Access to OpenAI API or compatible endpoint (see configs/llms/gpt.yml)
- Access to Bing API key for web search functionality to search real-time weather information for outfit recommendations (see configs/tools/websearch.yml)
- Redis server running locally or remotely
- Conductor server running locally or remotely

## Configuration

The container.yaml file is a configuration file that manages dependencies and settings for different components of the system, including Conductor connections, Redis connections, and other service configurations. To set up your configuration:

1. Generate the container.yaml file:
```bash
python compile_container.py
```
This will create a container.yaml file with default settings under `examples/step2_outfit_with_switch`.



2. Configure your LLM settings in `configs/llms/gpt.yml` and `configs/llms/text_res.yml`:

- Set your OpenAI API key or compatible endpoint through environment variable or by directly modifying the yml file
```bash
export custom_openai_key="your_openai_api_key"
export custom_openai_endpoint="your_openai_endpoint"
```
- Configure other model settings like temperature as needed through environment variable or by directly modifying the yml file

3. Configure your Bing Search API key in `configs/tools/websearch.yml`:
- Set your Bing API key through environment variable or by directly modifying the yml file
```bash
export bing_api_key="your_bing_api_key"
```

4. Update settings in the generated `container.yaml`:
- Modify Redis connection settings:
- Set the host, port and credentials for your Redis instance
- Configure both `redis_stream_client` and `redis_stm_client` sections
- Update the Conductor server URL under conductor_config section
- Adjust any other component settings as needed

## Running the Example

3. Run the outfit recommendation with switch example:

For terminal/CLI usage:
```bash
python run_cli.py
```

For app/GUI usage:
```bash
python run_app.py
```



## Troubleshooting

If you encounter issues:

- Verify Conductor and Redis are running and accessible
- Check your OpenAI API key and Bing API key are valid
- Check Redis Stream client and Redis STM client configuration

- Ensure all dependencies are installed correctly
- Review logs for any error messages


## Building the Example

Coming soon! This section will provide detailed instructions for building the step2_outfit_with_switch example step by step.

Loading

0 comments on commit 4826503

Please sign in to comment.