Merge pull request #48 from SolaceDev/ap/add-websearch-component

Ap/add websearch component

docs/components/index.md

@@ -20,8 +20,13 @@
 | [message_filter](message_filter.md) | A filtering component. This will apply a user configurable expression. If the expression evaluates to True, the message will be passed on. If the expression evaluates to False, the message will be discarded. If the message is discarded, any previous components that require an acknowledgement will be acknowledged. |
 | [openai_chat_model](openai_chat_model.md) | OpenAI chat model component |
 | [openai_chat_model_with_history](openai_chat_model_with_history.md) | OpenAI chat model component with conversation history |
+| [parser](parser.md) | Parse a JSON string and extract data fields. |
 | [pass_through](pass_through.md) | What goes in comes out |
 | [stdin_input](stdin_input.md) | STDIN input component. The component will prompt for input, which will then be placed in the message payload using the output schema below. The component will wait for its output message to be acknowledged before prompting for the next input. |
 | [stdout_output](stdout_output.md) | STDOUT output component |
 | [timer_input](timer_input.md) | An input that will generate a message at a specified interval. |
 | [user_processor](user_processor.md) | A component that allows the processing stage to be defined in the configuration file. |
+| [web_scraper](web_scraper.md) | Scrape javascript based websites. |
+| [websearch_bing](websearch_bing.md) | Perform a search query on Bing. |
+| [websearch_duckduckgo](websearch_duckduckgo.md) | Perform a search query on DuckDuckGo. |
+| [websearch_google](websearch_google.md) | Perform a search query on Google. |

docs/components/parser.md

@@ -0,0 +1,17 @@
+# Parser
+
+Parse a JSON string and extract data fields.
+
+## Configuration Parameters
+
+```yaml
+component_name: <user-supplied-name>
+component_module: parser
+component_config:
+  input_format: <string>
+```
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| input_format | True |  | The input format of the data. Options: 'json', 'yaml'. |
+

docs/components/web_scraper.md

@@ -0,0 +1,31 @@
+# WebScraper
+
+Scrape javascript based websites.
+
+## Configuration Parameters
+
+```yaml
+component_name: <user-supplied-name>
+component_module: web_scraper
+component_config:
+```
+
+No configuration parameters
+
+
+## Component Input Schema
+
+```
+{
+  <freeform-object>
+}
+```
+
+
+## Component Output Schema
+
+```
+{
+  <freeform-object>
+}
+```

docs/components/websearch_bing.md

@@ -0,0 +1,38 @@
+# WebSearchBing
+
+Perform a search query on Bing.
+
+## Configuration Parameters
+
+```yaml
+component_name: <user-supplied-name>
+component_module: websearch_bing
+component_config:
+  api_key: <string>
+  count: <string>
+  safesearch: <string>
+```
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| api_key | True |  | Bing API Key. |
+| count | False | 10 | Number of search results to return. |
+| safesearch | False | Moderate | Safe search setting: Off, Moderate, or Strict. |
+
+
+## Component Input Schema
+
+```
+{
+  <freeform-object>
+}
+```
+
+
+## Component Output Schema
+
+```
+{
+  <freeform-object>
+}
+```

docs/components/websearch_duckduckgo.md

@@ -0,0 +1,40 @@
+# WebSearchDuckDuckGo
+
+Perform a search query on DuckDuckGo.
+
+## Configuration Parameters
+
+```yaml
+component_name: <user-supplied-name>
+component_module: websearch_duckduckgo
+component_config:
+  pretty: <string>
+  no_html: <string>
+  skip_disambig: <string>
+  detail: <string>
+```
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| pretty | False | 1 | Beautify the search output. |
+| no_html | False | 1 | The number of output pages. |
+| skip_disambig | False | 1 | Skip disambiguation. |
+| detail | False | False | Return the detail. |
+
+
+## Component Input Schema
+
+```
+{
+  <freeform-object>
+}
+```
+
+
+## Component Output Schema
+
+```
+{
+  <freeform-object>
+}
+```

docs/components/websearch_google.md

@@ -0,0 +1,38 @@
+# WebSearchGoogle
+
+Perform a search query on Google.
+
+## Configuration Parameters
+
+```yaml
+component_name: <user-supplied-name>
+component_module: websearch_google
+component_config:
+  api_key: <string>
+  search_engine_id: <string>
+  detail: <string>
+```
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| api_key | True |  | Google API Key. |
+| search_engine_id | False | 1 | The custom search engine id. |
+| detail | False | False | Return the detail. |
+
+
+## Component Input Schema
+
+```
+{
+  <freeform-object>
+}
+```
+
+
+## Component Output Schema
+
+```
+{
+  <freeform-object>
+}
+```

examples/custom_components/web_search.py

@@ -0,0 +1,67 @@
+# A simple pass-through component - what goes in comes out
+import sys
+import requests
+
+sys.path.append("src")
+
+from solace_ai_connector.components.component_base import ComponentBase
+
+
+info = {
+    "class_name": "WebSearchCustomComponent",
+    "description": "Search web using APIs.",
+    "config_parameters": [
+        {
+            "name": "engine",
+            "description": "The search engine.",
+            "type": "string",
+        },
+        {
+            "name": "output_format",
+            "description": "Output format in json or html.",
+            "type": "string",
+            "default": "json"
+        }
+    ],
+    "input_schema": {
+        "type": "object",
+        "properties": {},
+    },
+    "output_schema": {
+        "type": "object",
+        "properties": {},
+    },
+}
+
+
+class WebSearchCustomComponent(ComponentBase):
+    def __init__(self, **kwargs):
+        super().__init__(info, **kwargs)
+        self.engine = self.get_config("engine")
+        self.format = self.get_config("format")
+
+    def invoke(self, message, data):
+        query = data["text"]
+        print(query)
+        url = None
+        if self.engine == "DuckDuckGo":
+            url = "http://api.duckduckgo.com/"
+            params = {
+                "q": query,            # User query
+                "format": self.format, # Response format (json by default)
+                "pretty": 1,           # Beautify the output
+                "no_html": 3,          # Remove HTML from the response
+                "skip_disambig": 1     # Skip disambiguation
+            }
+
+        if url != None:
+            response = requests.get(url, params=params)
+            if response.status_code == 200:
+                if params["format"] == 'json':
+                    print(response)
+                    return response.json()  # Return JSON response if the format is JSON
+                else:
+                    return response  # Return raw response if not JSON format
+            else:
+                # Handle errors if the request fails
+                return f"Error: {response.status_code}"

examples/custom_components/web_search.yaml

@@ -0,0 +1,25 @@
+log:
+  stdout_log_level: INFO
+  log_file_level: INFO
+  log_file: solace_ai_connector.log
+
+flows:
+  - name: web_search_flow
+    components:
+      # Input from a standard in
+      - component_name: stdin
+        component_module: stdin_input
+
+    # Using Custom component
+      - component_name: web_search_component
+        component_base_path: .
+        component_module: web_search
+        component_config:
+          engine: DuckDuckGo
+          format: json
+        input_selection:
+          source_expression: previous
+
+     # Output to a standard out
+      - component_name: stdout
+        component_module: stdout_output

examples/llm/langchain_openai_with_history_chat.yaml

@@ -65,6 +65,7 @@ flows:
             base_url: ${OPENAI_API_ENDPOINT}
             model: ${MODEL_NAME}
             temperature: 0.01
+          llm_response_format: text # options: text, json, or yaml
           history_module: langchain_core.chat_history
           history_class: InMemoryChatMessageHistory
           history_max_turns: 20

examples/llm/openai_chat.yaml

@@ -63,6 +63,7 @@ flows:
           base_url: ${OPENAI_API_ENDPOINT}
           model: ${OPENAI_MODEL_NAME}
           temperature: 0.01
+          response_format: text # options: text, json_object, or json_schema
         input_transforms:
           - type: copy
             source_expression: |

examples/parser.yaml

@@ -0,0 +1,28 @@
+# This is a simple parser workflow
+# The input payload is:
+# <text>
+#
+
+log:
+  stdout_log_level: INFO
+  log_file_level: INFO
+  log_file: solace_ai_connector.log
+
+flows:
+  - name: parser
+    components:
+      # Input from a standard in
+      - component_name: stdin
+        component_module: stdin_input
+
+    # Using Custom component
+      - component_name: parser_component
+        component_module: parser
+        component_config:
+          input_format: yaml
+        input_selection:
+          source_expression: previous
+
+     # Output to a standard out
+      - component_name: stdout
+        component_module: stdout_output

examples/websearch/bing_web_search.yaml

@@ -0,0 +1,33 @@
+# This is a Bing search engine workflow
+# The input payload is:
+# <text>
+#
+# Required ENV variables:
+# - BING_API_KEY
+
+log:
+  stdout_log_level: INFO
+  log_file_level: INFO
+  log_file: solace_ai_connector.log
+
+flows:
+  - name: google_web_search_flow
+    components:
+      # Input from a standard in
+      - component_name: stdin
+        component_module: stdin_input
+
+    # Using Custom component
+      - component_name: web_search_component
+        component_module: websearch_bing
+        component_config:
+          api_key: ${BING_API_KEY}
+          safesearch: Moderate
+          count: 2
+          detail: false
+        input_selection:
+          source_expression: previous
+
+     # Output to a standard out
+      - component_name: stdout
+        component_module: stdout_output

examples/websearch/duckduckgo_web_search.yaml

@@ -0,0 +1,27 @@
+# This is a Duck Duck Go search engine workflow
+# The input payload is:
+# <text>
+
+log:
+  stdout_log_level: INFO
+  log_file_level: INFO
+  log_file: solace_ai_connector.log
+
+flows:
+  - name: duckduckgo_web_search_flow
+    components:
+      # Input from a standard in
+      - component_name: stdin
+        component_module: stdin_input
+
+    # Using Custom component
+      - component_name: web_search_component
+        component_module: websearch_duckduckgo
+        component_config:
+          detail: false
+        input_selection:
+          source_expression: previous
+
+     # Output to a standard out
+      - component_name: stdout
+        component_module: stdout_output