Skip to content

Commit

Permalink
deploy: b26ea1e
Browse files Browse the repository at this point in the history
  • Loading branch information
dhrubo-os committed Jan 6, 2025
1 parent 19dadef commit 4d34397
Show file tree
Hide file tree
Showing 184 changed files with 797 additions and 800 deletions.
2 changes: 1 addition & 1 deletion .buildinfo
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: b2e1a37e0c128dba184508885b405652
config: fc7813a011b7f8e129f28bf084179ffc
tags: 645f666f9bcd5a90fca523b33c5a78b7
4 changes: 2 additions & 2 deletions _modules/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Overview: module code &mdash; Opensearch-py-ml 1.1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=80d5e7a1" />
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=e59714d7" />
<link rel="stylesheet" type="text/css" href="../_static/plot_directive.css" />

Expand Down Expand Up @@ -85,7 +85,7 @@ <h1>All modules for which code is available</h1>
<hr/>

<div role="contentinfo">
<p>&#169; Copyright 2024, Opensearch.</p>
<p>&#169; Copyright 2025, Opensearch.</p>
</div>

Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
Expand Down
207 changes: 105 additions & 102 deletions _modules/opensearch_py_ml/dataframe.html

Large diffs are not rendered by default.

35 changes: 18 additions & 17 deletions _modules/opensearch_py_ml/etl.html
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>opensearch_py_ml.etl &mdash; Opensearch-py-ml 1.1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=80d5e7a1" />
<link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=e59714d7" />
<link rel="stylesheet" type="text/css" href="../../_static/plot_directive.css" />

Expand Down Expand Up @@ -97,27 +97,27 @@ <h1>Source code for opensearch_py_ml.etl</h1><div class="highlight"><pre>
<span class="c1"># specific language governing permissions and limitations</span>
<span class="c1"># under the License.</span>

<span class="kn">import</span> <span class="nn">csv</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">deque</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">Generator</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Mapping</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Tuple</span><span class="p">,</span> <span class="n">Union</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">csv</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">collections</span><span class="w"> </span><span class="kn">import</span> <span class="n">deque</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">Generator</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Mapping</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Tuple</span><span class="p">,</span> <span class="n">Union</span>

<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span> <span class="c1"># type: ignore</span>
<span class="kn">from</span> <span class="nn">opensearchpy</span> <span class="kn">import</span> <span class="n">OpenSearch</span>
<span class="kn">from</span> <span class="nn">opensearchpy.helpers</span> <span class="kn">import</span> <span class="n">parallel_bulk</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">pandas</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pd</span> <span class="c1"># type: ignore</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">opensearchpy</span><span class="w"> </span><span class="kn">import</span> <span class="n">OpenSearch</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">opensearchpy.helpers</span><span class="w"> </span><span class="kn">import</span> <span class="n">parallel_bulk</span>

<span class="kn">from</span> <span class="nn">opensearch_py_ml</span> <span class="kn">import</span> <span class="n">DataFrame</span>
<span class="kn">from</span> <span class="nn">opensearch_py_ml.common</span> <span class="kn">import</span> <span class="n">DEFAULT_CHUNK_SIZE</span><span class="p">,</span> <span class="n">PANDAS_VERSION</span>
<span class="kn">from</span> <span class="nn">opensearch_py_ml.field_mappings</span> <span class="kn">import</span> <span class="n">FieldMappings</span><span class="p">,</span> <span class="n">verify_mapping_compatibility</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">opensearch_py_ml</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataFrame</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">opensearch_py_ml.common</span><span class="w"> </span><span class="kn">import</span> <span class="n">DEFAULT_CHUNK_SIZE</span><span class="p">,</span> <span class="n">PANDAS_VERSION</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">opensearch_py_ml.field_mappings</span><span class="w"> </span><span class="kn">import</span> <span class="n">FieldMappings</span><span class="p">,</span> <span class="n">verify_mapping_compatibility</span>

<span class="k">try</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">pandas.io.parsers</span> <span class="kn">import</span> <span class="n">_c_parser_defaults</span> <span class="c1"># type: ignore</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pandas.io.parsers</span><span class="w"> </span><span class="kn">import</span> <span class="n">_c_parser_defaults</span> <span class="c1"># type: ignore</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">pandas.io.parsers.readers</span> <span class="kn">import</span> <span class="n">_c_parser_defaults</span> <span class="c1"># type: ignore</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pandas.io.parsers.readers</span><span class="w"> </span><span class="kn">import</span> <span class="n">_c_parser_defaults</span> <span class="c1"># type: ignore</span>

<span class="n">_DEFAULT_LOW_MEMORY</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="n">_c_parser_defaults</span><span class="p">[</span><span class="s2">&quot;low_memory&quot;</span><span class="p">]</span>


<span class="k">def</span> <span class="nf">pandas_to_opensearch</span><span class="p">(</span>
<span class="k">def</span><span class="w"> </span><span class="nf">pandas_to_opensearch</span><span class="p">(</span>
<span class="n">pd_df</span><span class="p">:</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">,</span>
<span class="n">os_client</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">OpenSearch</span><span class="p">],</span>
<span class="n">os_dest_index</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
Expand Down Expand Up @@ -182,6 +182,7 @@ <h1>Source code for opensearch_py_ml.etl</h1><div class="highlight"><pre>
<span class="sd"> ... &#39;G&#39;: [1, 2, 3],</span>
<span class="sd"> ... &#39;H&#39;: &#39;Long text - to be indexed as os type text&#39;},</span>
<span class="sd"> ... index=[&#39;0&#39;, &#39;1&#39;, &#39;2&#39;])</span>
<span class="sd"> &gt;&gt;&gt; pd_df[&#39;D&#39;] = pd_df[&#39;D&#39;].astype(&#39;datetime64[ns]&#39;)</span>
<span class="sd"> &gt;&gt;&gt; type(pd_df)</span>
<span class="sd"> &lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;</span>
<span class="sd"> &gt;&gt;&gt; pd_df</span>
Expand Down Expand Up @@ -273,7 +274,7 @@ <h1>Source code for opensearch_py_ml.etl</h1><div class="highlight"><pre>
<span class="n">index</span><span class="o">=</span><span class="n">os_dest_index</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;mappings&quot;</span><span class="p">:</span> <span class="n">mapping</span><span class="p">[</span><span class="s2">&quot;mappings&quot;</span><span class="p">]}</span>
<span class="p">)</span>

<span class="k">def</span> <span class="nf">action_generator</span><span class="p">(</span>
<span class="k">def</span><span class="w"> </span><span class="nf">action_generator</span><span class="p">(</span>
<span class="n">pd_df</span><span class="p">:</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">,</span>
<span class="n">os_dropna</span><span class="p">:</span> <span class="nb">bool</span><span class="p">,</span>
<span class="n">use_pandas_index_for_os_ids</span><span class="p">:</span> <span class="nb">bool</span><span class="p">,</span>
Expand Down Expand Up @@ -315,7 +316,7 @@ <h1>Source code for opensearch_py_ml.etl</h1><div class="highlight"><pre>
<span class="k">return</span> <span class="n">DataFrame</span><span class="p">(</span><span class="n">os_client</span><span class="p">,</span> <span class="n">os_dest_index</span><span class="p">)</span>


<span class="k">def</span> <span class="nf">opensearch_to_pandas</span><span class="p">(</span>
<span class="k">def</span><span class="w"> </span><span class="nf">opensearch_to_pandas</span><span class="p">(</span>
<span class="n">oml_df</span><span class="p">:</span> <span class="n">DataFrame</span><span class="p">,</span> <span class="n">show_progress</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
Expand Down Expand Up @@ -383,7 +384,7 @@ <h1>Source code for opensearch_py_ml.etl</h1><div class="highlight"><pre>

<div class="viewcode-block" id="csv_to_opensearch">
<a class="viewcode-back" href="../../reference/api/csv_to_opensearch.html#opensearch_py_ml.etl.csv_to_opensearch">[docs]</a>
<span class="k">def</span> <span class="nf">csv_to_opensearch</span><span class="p">(</span> <span class="c1"># type: ignore</span>
<span class="k">def</span><span class="w"> </span><span class="nf">csv_to_opensearch</span><span class="p">(</span> <span class="c1"># type: ignore</span>
<span class="n">filepath_or_buffer</span><span class="p">,</span>
<span class="n">os_client</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">OpenSearch</span><span class="p">],</span>
<span class="n">os_dest_index</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
Expand Down Expand Up @@ -640,7 +641,7 @@ <h1>Source code for opensearch_py_ml.etl</h1><div class="highlight"><pre>
<hr/>

<div role="contentinfo">
<p>&#169; Copyright 2024, Opensearch.</p>
<p>&#169; Copyright 2025, Opensearch.</p>
</div>

Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
Expand Down
Loading

0 comments on commit 4d34397

Please sign in to comment.