Prepare for v0.2.14 (#238)

* Update doc link Signed-off-by: Yhz <[email protected]> * update example Signed-off-by: Yhz <[email protected]> * Start to develop v0.2.11 Signed-off-by: Yhz <[email protected]> * update for hybrid Signed-off-by: Yhz <[email protected]> * add hybrid api Signed-off-by: Yhz <[email protected]> * update hybrid insert Signed-off-by: yhz <[email protected]> * add hybrid search Signed-off-by: yhz <[email protected]> * add search hybrid Signed-off-by: Yhz <[email protected]> * update hybrid searach example Signed-off-by: Yhz <[email protected]> * fix error when rpc timeout Signed-off-by: yhz <[email protected]> * Update hybrid api Signed-off-by: yhz <[email protected]> * add new api search_by_ids Signed-off-by: Yhz <[email protected]> * update example Signed-off-by: Yhz <[email protected]> * fix typo Signed-off-by: Yhz <[email protected]> * fix search by ids error Signed-off-by: Yhz <[email protected]> * add search_by_ids test case Signed-off-by: Yhz <[email protected]> * use connection pool Signed-off-by: yhz <[email protected]> * async api pass Signed-off-by: Yhz <[email protected]> * add async examples Signed-off-by: Yhz <[email protected]> * Fix TypeError raised in inserting vectors with async set true (#200) Signed-off-by: Yhz <[email protected]> * add timeout for async APIs Signed-off-by: Yhz <[email protected]> * add API close Signed-off-by: Yhz <[email protected]> * optimzie code Signed-off-by: Yhz <[email protected]> * remove terminate() in pool Signed-off-by: Yhz <[email protected]> * deprated connect Signed-off-by: Yhz <[email protected]> * get collection info failed when info is null (fix #201) Signed-off-by: yhz <[email protected]> * fix Signed-off-by: yhz <[email protected]> * update http handler Signed-off-by: yhz <[email protected]> * .. Signed-off-by: yhz <[email protected]> * filter -1 result Signed-off-by: Yhz <[email protected]> * Update APIs names Signed-off-by: yhz <[email protected]> * update future done not raise exception Signed-off-by: Yhz <[email protected]> * update Signed-off-by: yhz <[email protected]> * Fix excepted exceptions if async invoke with timeout (#207) Signed-off-by: yhz <[email protected]> * remove api compacity Signed-off-by: yhz <[email protected]> * add more params in pool Signed-off-by: yhz <[email protected]> * add pool try_connect and pre_ping params Signed-off-by: yhz <[email protected]> * remove parms recycle Signed-off-by: yhz <[email protected]> * update for http handler Signed-off-by: Yhz <[email protected]> * update example Signed-off-by: yhz <[email protected]> * update readme Signed-off-by: yhz <[email protected]> * update Signed-off-by: yhz <[email protected]> * rename project name Signed-off-by: yhz <[email protected]> * Prepare for v0.2.11 Signed-off-by: yhz <[email protected]> * update async example Signed-off-by: yhz <[email protected]> * Ready to develop v0.2.12 Signed-off-by: yhz <[email protected]> * add attribute name in stub Signed-off-by: yhz <[email protected]> * Prepare version 0.2.11 (#210) * add new api search_by_ids Signed-off-by: Yhz <[email protected]> * update example Signed-off-by: Yhz <[email protected]> * fix typo Signed-off-by: Yhz <[email protected]> * fix search by ids error Signed-off-by: Yhz <[email protected]> * add search_by_ids test case Signed-off-by: Yhz <[email protected]> * use connection pool Signed-off-by: yhz <[email protected]> * async api pass Signed-off-by: Yhz <[email protected]> * add async examples Signed-off-by: Yhz <[email protected]> * Fix TypeError raised in inserting vectors with async set true (#200) Signed-off-by: Yhz <[email protected]> * add timeout for async APIs Signed-off-by: Yhz <[email protected]> * add API close Signed-off-by: Yhz <[email protected]> * optimzie code Signed-off-by: Yhz <[email protected]> * remove terminate() in pool Signed-off-by: Yhz <[email protected]> * deprated connect Signed-off-by: Yhz <[email protected]> * get collection info failed when info is null (fix #201) Signed-off-by: yhz <[email protected]> * fix Signed-off-by: yhz <[email protected]> * update http handler Signed-off-by: yhz <[email protected]> * .. Signed-off-by: yhz <[email protected]> * filter -1 result Signed-off-by: Yhz <[email protected]> * Update APIs names Signed-off-by: yhz <[email protected]> * update future done not raise exception Signed-off-by: Yhz <[email protected]> * update Signed-off-by: yhz <[email protected]> * Fix excepted exceptions if async invoke with timeout (#207) Signed-off-by: yhz <[email protected]> * remove api compacity Signed-off-by: yhz <[email protected]> * add more params in pool Signed-off-by: yhz <[email protected]> * add pool try_connect and pre_ping params Signed-off-by: yhz <[email protected]> * remove parms recycle Signed-off-by: yhz <[email protected]> * update for http handler Signed-off-by: Yhz <[email protected]> * update example Signed-off-by: yhz <[email protected]> * update readme Signed-off-by: yhz <[email protected]> * update Signed-off-by: yhz <[email protected]> * rename project name Signed-off-by: yhz <[email protected]> * Prepare for v0.2.11 Signed-off-by: yhz <[email protected]> * update async example Signed-off-by: yhz <[email protected]> * update exxample Signed-off-by: yhz <[email protected]> * add retry Signed-off-by: yhz <[email protected]> * . Signed-off-by: yhz <[email protected]> * update grpc codegen for hybrid usage Signed-off-by: Yhz <[email protected]> * update hybrid api Signed-off-by: Yhz <[email protected]> * examples Signed-off-by: yhz <[email protected]> * Add new API called 'reload_segments' temporarily Signed-off-by: yhz <[email protected]> * add Signed-off-by: yhz <[email protected]> * Ready to release version 0.2.12 Signed-off-by: Yhz <[email protected]> * Add sdk statistics badges (fix #220) Signed-off-by: Yhz <[email protected]> * Merge Signed-off-by: yhz <[email protected]> * Return 2-D empty result if response is empty Signed-off-by: yhz <[email protected]> * Fix http get entity by id issue Signed-off-by: yhz <[email protected]> * Add single connection pool Signed-off-by: yhz <[email protected]> * Add SingleScopedConn Signed-off-by: yhz <[email protected]> * .. Signed-off-by: yhz <[email protected]> * enlarge timeout to 30 Signed-off-by: yhz <[email protected]> * Set default pool is sightenthread Signed-off-by: yhz <[email protected]> * Add keepalive time Signed-off-by: yhz <[email protected]> * P test V0.3.33 Signed-off-by: yhz <[email protected]> * Fix has_partition bug in http handler Signed-off-by: yhz <[email protected]> * Prepare for v0.2.14 Signed-off-by: yhz <[email protected]> * Prepare for v0.2.14 Signed-off-by: yhz <[email protected]> * Remove hybrid example Signed-off-by: yhz <[email protected]>
milvus-io · Jul 20, 2020 · 24582e3 · 24582e3
1 parent d6a986e
commit 24582e3
Show file tree

Hide file tree

Showing 12 changed files with 114 additions and 22 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,9 @@
+# pymilvus 0.2.14(2020-07-20)
+
+## Bug
+- \#237 - Fix wrong result on 'has_partition' with http handler
+
+
 # pymilvus 0.2.13(2020-06-13)
 
 ## New Feature

diff --git a/README.md b/README.md
@@ -9,7 +9,7 @@
 
 Python SDK for [Milvus](https://github.com/milvus-io/milvus). To contribute code to this project, please read our [contribution guidelines](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md) first.
 
-For detailed SDK documentation, refer to [API Documentation](https://milvus-io.github.io/milvus-sdk-python/pythondoc/v0.2.13/index.html).
+For detailed SDK documentation, refer to [API Documentation](https://milvus-io.github.io/milvus-sdk-python/pythondoc/v0.2.14/index.html).
 
 
 <!-- TOC -->
@@ -76,12 +76,13 @@ The following collection shows Milvus versions and recommended pymilvus versions
 | 0.9.0 | 0.2.11 |
 | 0.9.1 | 0.2.12 |
 | 0.10.0 | 0.2.13 |
+| 0.10.1 | 0.2.14 |
 
 
 You can install a specific version of pymilvus by:
 
 ```shell
-$ pip install pymilvus==0.2.13
+$ pip install pymilvus==0.2.14
 ```
 
 You can upgrade `pymilvus` to the latest version by:

diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -22,7 +22,7 @@
 author = 'Milvus'
 
 # The full version, including alpha/beta/rc tags
-release = '0.2.13'
+release = '0.2.14'
 
 
 # -- General configuration ---------------------------------------------------

diff --git a/examples/example.py b/examples/example.py
@@ -15,7 +15,7 @@
 # _PORT = '19121'  # default http value
 
 # Vector parameters
-_DIM = 128  # dimension of vector
+_DIM = 8  # dimension of vector
 
 _INDEX_FILE_SIZE = 32  # max file size of stored index
 

diff --git a/examples/example_insert_fail.py b/examples/example_insert_fail.py
@@ -0,0 +1,28 @@
+import random
+import sys
+import time
+
+from milvus import *
+
+if __name__ == '__main__':
+    dimension = 128
+    collection_name = "insert_fail"
+
+    insert_vectors = [[random.random() for _ in range(dimension)] for _ in range(100)]
+
+    client = Milvus()
+
+    client.create_collection({"collection_name": collection_name, "dimension": dimension})
+    start_time = time.time()
+    i = 0
+    while time.time() < start_time + 2 * 24 * 3600:
+        i = i + 1
+        # logger.debug(i)
+        # logger.debug("Row count: %d" % milvus_instance.count())
+        _, count = client.count_entities(collection_name)
+        print("Row count: {}".format(count))
+        status, res = client.insert(collection_name, insert_vectors)
+        if not status.OK():
+            print("Insert failed. {}".format(status))
+            sys.exit(1)
+        time.sleep(0.1)
diff --git a/examples/example_search_after_delete.py b/examples/example_search_after_delete.py
@@ -0,0 +1,52 @@
+import random
+import sys
+
+from milvus import *
+
+if __name__ == '__main__':
+    collection_name_ = "test_collection_search_after_delete_11"
+
+    client = Milvus(host="192.168.1.57", port=19531)
+    client.create_collection({"collection_name": collection_name_, "dimension": 128})
+
+    del_vecs = []
+    del_ids = []
+    for i in range(5):
+        print(f"Insert round {i} ...")
+        vectors = [[random.random() for _ in range(128)] for _ in range(100000)]
+        del_vecs.append(vectors[0])
+        vec_ids = [i * 100000 + j for j in range(100000)]
+        del_ids.append(vec_ids[0])
+        status, ids = client.insert(collection_name_, vectors, ids=vec_ids)
+        if not status.OK():
+            print(f"Insert failed: {status}", file=sys.stderr)
+            sys.exit(1)
+
+    client.flush([collection_name_])
+    print("Create index ... ")
+    status = client.create_index(collection_name_, IndexType.IVF_FLAT, params={"nlist": 1024})
+    if not status.OK():
+        print(f"Create index failed: {status}", file=sys.stderr)
+        sys.exit(1)
+
+    print("Search before delete ....")
+    status, results = client.search(collection_name_, 5, del_vecs, params={"nprobe": 10})
+
+    print("Delete entity ... ")
+    status = client.delete_entity_by_id(collection_name_, id_array=del_ids)
+    if not status.OK():
+        print(f"Create index failed: {status}", file=sys.stderr)
+        sys.exit(1)
+
+    client.flush([collection_name_])
+    print("Search after delete ... ")
+    status, results = client.search(collection_name_, 5, del_vecs, params={"nprobe": 10})
+    if not status.OK():
+        print(f"Search failed: {status}", file=sys.stderr)
+        sys.exit(1)
+
+    for del_id, result in zip(del_ids, results):
+        if result[0].id == del_id:
+            print(f"Unexcepted search result. {result[0]}")
+
+    client.drop_collection(collection_name_)
diff --git a/milvus/client/__init__.py b/milvus/client/__init__.py
@@ -1,3 +1,3 @@
 """client module"""
 
-__version__ = "0.2.13"
+__version__ = "0.2.14"
diff --git a/milvus/client/abstract.py b/milvus/client/abstract.py
@@ -147,14 +147,18 @@ def _unpack(self, _raw):
         if len(id_list) != len(dis_list):
             raise ParamError("number of id not match distance ")
 
-        col = len(id_list) // self._nq if self._nq > 0 else 0
+        len_ = len(id_list)
+        col = len_ // self._nq if self._nq > 0 else 0
 
         if col == 0:
             return
 
-        for si, i in enumerate(range(0, len(id_list), col)):
-            k = i + col
-            for j in range(i, i + col):
+        if len_ % col != 0:
+            raise ValueError("Search result is not aligned. (row_num={}, len of ids={})".format(_raw.row_num, len_))
+
+        for si, i in enumerate(range(0, len_, col)):
+            k = min(i + col, len_)
+            for j in range(i, k):
                 if id_list[j] == -1:
                     k = j
                     break
@@ -342,7 +346,8 @@ def __getattr__(self, item):
         if index < len(self._entity):
             return self._entity[index]
         if index == len(self._entity):
-            return list(self._vector.float_data) if len(self._vector.float_data) > 0 else bytes(self._vector.binary_data)
+            return list(self._vector.float_data) if len(self._vector.float_data) > 0 else bytes(
+                self._vector.binary_data)
 
         raise ValueError("Out of range ... ")
 
@@ -386,7 +391,8 @@ def __getitem__(self, item):
 
         item_entities = [e[item] for e in self._entities]
 
-        return HybridEntityResult(self._field_names, self._ids[item], item_entities, self._vectors[item], self._distances[item])
+        return HybridEntityResult(self._field_names, self._ids[item], item_entities, self._vectors[item],
+                                  self._distances[item])
         # if item in self._field_names:
         #     index = self._field_names[item]
 
@@ -417,7 +423,8 @@ def __getitem__(self, item):
 
         entities = self._raw.entity
 
-        slice_entity = lambda e, start, end: e.int_value[start: end] if len(e.int_value) > 0 else e.double_value[start: end]
+        slice_entity = lambda e, start, end: e.int_value[start: end] if len(e.int_value) > 0 else e.double_value[
+                                                                                                  start: end]
         seg_ids = self._raw.entity.entity_id[seg_start: seg_end]
         seg_entities = [slice_entity(e, seg_start, seg_end) for e in entities.attr_data]
         seg_vectors = self._raw.entity.vector_data[0].value[seg_start: seg_end]
@@ -477,6 +484,7 @@ def __getitem__(self, item):
     def field_names(self):
         return list(self._raw.field_names)
 
+
 class IndexParam:
     """
     Index Param

diff --git a/milvus/client/grpc_handler.py b/milvus/client/grpc_handler.py
@@ -46,7 +46,6 @@ def handler(self, *args, **kwargs):
                 return status if not rargs else tuple([status]) + rargs
             except grpc.RpcError as e:
                 record_dict["RPC error"] = str(datetime.datetime.now())
-
                 LOGGER.error("\nAddr [{}] {}\nRpc error: {}\n\t{}".format(self.server_address, func.__name__, e, record_dict))
                 status = Status(e.code(), message='Error occurred. {}'.format(e.details()))
                 return status if not rargs else tuple([status]) + rargs

diff --git a/milvus/client/http_handler.py b/milvus/client/http_handler.py
@@ -563,20 +563,15 @@ def show_partitions(self, table_name, timeout):
         return Status(Status(js["code"], js["message"])), []
 
     @handle_error(returns=(False,))
-    def has_partition(self, collection_name, tag):
+    def has_partition(self, collection_name, tag, timeout=30):
         url = self._uri + "/collections/{}/partitions".format(collection_name)
-        request = {
-            "filter": {
-                "partition_tag": tag
-            }
-        }
-        payload = ujson.dumps(request)
 
-        response = rq.get(url, data=payload)
+        response = rq.get(url, timeout=timeout)
         if response.status_code == 200:
             result = response.json()
             if result["count"] > 0:
-                return Status(), True
+                partitions = [p["partition_tag"] for p in list(result["partitions"])]
+                return Status(), tag in partitions
             return Status(), False
 
         js = response.json()

diff --git a/milvus/client/pool.py b/milvus/client/pool.py
@@ -85,6 +85,8 @@ def __init__(self, uri, pool_size=10, wait_timeout=30, try_connect=True, **kwarg
         self._uri = uri
         self._pool_size = pool_size
         self._wait_timeout = wait_timeout
+        # if try_connect:
+        #     self._max_retry = kwargs.get("max_retry", 3)
 
         # Record used connection number.
         self._used_conn = 0

diff --git a/tests/test_thread.py b/tests/test_thread.py
@@ -63,6 +63,7 @@ def test_run(gcon):
 
 def test_mult_insert(gcon):
     def multi_thread_opr(client, collection_name, utid):
+
         collection_param = {
             'collection_name': collection_name,
             'dimension': 64