[Bug]: [null & default] insert failed when insert "None" list to the array field and array field nullable is true

[qixuan0212]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20240916-dcd904d2-amd64
- Deployment mode(standalone or cluster): both
- MQ type(rocksmq, pulsar or kafka):  all  
- SDK version(e.g. pymilvus v2.0.0rc2): 2.5.0rc78
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

pymilvus.exceptions.MilvusException: <MilvusException: (code=1100, message=the num_rows (0) of field (array_string) is not equal to passed num_rows (100): invalid parameter[expected=100][actual=0])>

Expected Behavior

When insert "None" list to the array field and array field nullable is true, insert success

Steps To Reproduce

import array

import numpy as np
from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility
import json
import random

connections.connect(uri="http://10.102.9.5:19530")

dim, nb = 128, 100
int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
double_field = FieldSchema(name="nullableFid", dtype=DataType.DOUBLE, nullable=True, default_value=10.0, is_primary=False)
int32_field = FieldSchema(name="int32", dtype=DataType.INT64, default_value=10)
string_field = FieldSchema(name="string", dtype=DataType.VARCHAR, max_length=1000, nullable=True)
float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim, mmap_enabled=False)
array_string_field = FieldSchema(name="array_string", dtype=DataType.ARRAY, element_type=DataType.VARCHAR, max_length=nb,
                                 max_capacity=nb, nullable=True)
# json_field = FieldSchema(name="json", dtype=DataType.JSON, nullable=False)
schema = CollectionSchema(fields=[int64_field, double_field, int32_field, string_field, array_string_field, float_vector], enable_dynamic_field=True)
utility.drop_collection("test")
collection = Collection("test", schema=schema)
res = collection.schema
print(res)
index = {"index_type": "DISKANN", "metric_type": "L2", "params": {}}
vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
array_string_field = [None for _ in range(nb)]
# json_field = [None for _ in range(nb)]
data = [[i for i in range(nb)], [None for _ in range(nb)], [None for _ in range(nb)], [None for _ in range(nb)],
        array_string_field, vectors]

collection.insert(data=data)
collection.create_index("float_vector", index, index_name="index_name_1")
collection.load()
collection.flush()
res = collection.num_entities
print(res)
default_search_params = {"metric_type": "L2", "params": {}}
limit = 10
nq = 1
import time
start = time.time()
res1 = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 >= 0", output_fields=["nullableFid", "int32", "string", "json"])
end = time.time() - start
print(res1)
print("search successfully in %f s" % end)
# start = time.time()
# res = collection.query("int64>=0", output_fields=["nullableFid","int32", "string"])
# end = time.time() - start
# print(res)
# print("query successfully in %f s" % end)

Milvus Log

https://grafana-4am.zilliz.cc/explore?orgId=1&panes=%7B%22XDE%22:%7B%22datasource%22:%22vhI6Vw67k%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcluster%3D%5C%22devops%5C%22,namespace%3D%5C%22chaos-testing%5C%22,pod%3D~%5C%22default-none-qx-pgffh.%2A%5C%22%7D%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22vhI6Vw67k%22%7D%7D%5D,%22range%22:%7B%22from%22:%221726717703434%22,%22to%22:%221726717719437%22%7D%7D%7D&schemaVersion=1

Anything else?

NA

[binbinlv]

/assign @smellthemoon
/unassign @yanliang567

[smellthemoon]

related with #36030

[smellthemoon]

pr merged, could you help to verify? @qixuan0212

[binbinlv]

/assign @qixuan0212

[qixuan0212]

Verified and fixed.
pymilvus: 2.5.0rc78
milvus: master-20240924-52cce4de-amd64

result:
{'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'nullableFid', 'description': '', 'type': <DataType.DOUBLE: 11>, 'default_value': double_data: 10 , 'nullable': True}, {'name': 'int32', 'description': '', 'type': <DataType.INT64: 5>, 'default_value': long_data: 10 }, {'name': 'string', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 1000}, 'nullable': True}, {'name': 'array_string', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 100, 'max_capacity': 100}, 'nullable': True, 'element_type': <DataType.VARCHAR: 21>}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'mmap_enabled': False, 'dim': 128}}], 'enable_dynamic_field': True} 100 data: ['["id: 0, distance: 0.0, entity: {\'nullableFid\': 10.0, \'int32\': 10, \'string\': None, \'array_string\': None}", "id: 39, distance: 14.37320327758789, entity: {\'nullableFid\': 10.0, \'int32\': 10, \'string\': None, \'array_string\': None}"]'] search successfully in 0.022805 s