Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: [GoSDK] When inserting rows without primary key data, all the returned IDs are zero #33460

Open
1 task done
ThreadDao opened this issue May 29, 2024 · 7 comments
Open
1 task done
Assignees
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone

Comments

@ThreadDao
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20240528-b138ae74-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): go-sdk v2
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

  1. create collection with autoID true, int64(pk) anf floatVec field
  2. insert rows without int64 data gets ids: [0, ....., 0]
  • case:
// test insert rows enable or disable dynamic field
func TestInsertDefaultRows(t *testing.T) {
	t.Parallel()
	ctx := hp.CreateContext(t, time.Second*common.DefaultTimeout)
	mc := createDefaultMilvusClient(ctx, t)

	for _, autoId := range []bool{true} {
		cp := hp.NewCreateCollectionParams(hp.Int64Vec)
		_, schema := hp.CollPrepare.CreateCollection(ctx, t, mc, cp, hp.TNewFieldsOption(), hp.TNewSchemaOption().TWithAutoID(autoId))
		log.Info("fields", zap.Any("FieldNames", schema.Fields))

		// insert rows
		rows := hp.GenInt64VecRows(common.DefaultNb, false, autoId, *hp.TNewDataOption())
		log.Info("rows data", zap.Any("rows[8]", rows[8]))
		ids, err := mc.Insert(ctx, clientv2.NewRowBasedInsertOption(schema.CollectionName, rows...))
		common.CheckErr(t, err, true)

		int64Values := make([]int64, 0, common.DefaultNb)
		for i := 0; i < common.DefaultNb; i++ {
			int64Values = append(int64Values, int64(i))
		}
		common.CheckInsertResult(t, ids.IDs, column.NewColumnInt64(common.DefaultInt64FieldName, int64Values))

		// flush and check row count
		flushTask, errFlush := mc.Flush(ctx, clientv2.NewFlushOption(schema.CollectionName))
		common.CheckErr(t, errFlush, true)
		errFlush = flushTask.Await(ctx)
		common.CheckErr(t, errFlush, true)
	}
}
  • test log:
[2024/05/29 18:27:11.263 +08:00] [INFO] [base/milvus_client.go:37] [Request] [method=Connect] [reqs="{\"client_info\":{\"sdk_type\":\"GoMilvusClient\",\"sdk_version\":\"2.4.0-dev\",\"local_time\":\"2024-05-29 18:27:11.263334804 +0800 CST m=+0.032951711\",\"host\":\"nausicca\"}}"]
[2024/05/29 18:27:11.286 +08:00] [INFO] [base/milvus_client.go:56] [Response] [method=Connect] [resp="{\"status\":{},\"server_info\":{\"build_tags\":\"b138ae7\",\"build_time\":\"Tue May 28 07:34:23 UTC 2024\",\"git_commit\":\"b138ae7\",\"go_version\":\"go version go1.21.10 linux/amd64\",\"deploy_mode\":\"DISTRIBUTED\"},\"identifier\":450095593882648580}"]
[2024/05/29 18:27:11.286 +08:00] [DEBUG] [base/milvus_client.go:57] [Cost] [method=Connect] [cost=22.737308ms]
[2024/05/29 18:27:11.286 +08:00] [INFO] [helper/field_helper.go:280] [GenFieldsForCollection] [GenFieldsOption="{\"AutoID\":false,\"Dim\":128,\"IsDynamic\":false,\"MaxLength\":100,\"MaxCapacity\":100,\"IsPartitionKey\":false,\"ElementType\":0}"]
[2024/05/29 18:27:11.286 +08:00] [INFO] [base/milvus_client.go:37] [Request] [method=CreateCollection] [reqs="{\"collection_name\":\"pre_rWHYWt\",\"schema\":\"CgpwcmVfcldIWVd0GAEiCxIFaW50NjQYASgFIhgSCGZsb2F0VmVjKGUyCgoDZGltEgMxMjg=\",\"shards_num\":1,\"consistency_level\":2}"]
[2024/05/29 18:27:11.313 +08:00] [INFO] [base/milvus_client.go:56] [Response] [method=CreateCollection] [resp={}]
[2024/05/29 18:27:11.313 +08:00] [DEBUG] [base/milvus_client.go:57] [Cost] [method=CreateCollection] [cost=26.366242ms]
[2024/05/29 18:27:11.313 +08:00] [INFO] [testcases/insert_test.go:46] [fields] [FieldNames="[{\"ID\":0,\"Name\":\"int64\",\"PrimaryKey\":true,\"AutoID\":false,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0},{\"ID\":0,\"Name\":\"floatVec\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"128\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"ElementType\":0}]"]
[2024/05/29 18:27:11.321 +08:00] [INFO] [testcases/insert_test.go:50] ["rows data"] ["rows[8]"="{\"floatVec\":[0.6178241,0.21087281,0.44867408,0.7325401,0.16834813,0.93801135,0.7424177,0.45748225,0.31714168,0.52351075,0.23468941,0.20924239,0.39932206,0.27672815,0.94168675,0.15122142,0.67313886,0.33818346,0.490186,0.37304404,0.8647382,0.984301,0.86432034,0.6576149,0.034932584,0.11069491,0.9180387,0.27059072,0.7769772,0.4492993,0.9091361,0.49744037,0.79990107,0.947381,0.79302114,0.9386312,0.08769191,0.08857079,0.6660659,0.63789123,0.8110448,0.6232179,0.06447095,0.7867628,0.011526427,0.23668459,0.31532583,0.930082,0.5526945,0.35251418,0.7554627,0.6498429,0.8466115,0.15021151,0.6578219,0.6485041,0.36646286,0.022119252,0.6632878,0.90742284,0.024551086,0.69591314,0.041073326,0.6479019,0.0280014,0.5167739,0.76998764,0.96678704,0.3818616,0.7679683,0.58915144,0.75805616,0.8849203,0.3163661,0.6954391,0.29520997,0.012191617,0.817875,0.84755325,0.89885163,0.9787547,0.7072435,0.14403586,0.124327034,0.67088,0.24942382,0.40041453,0.6156071,0.57584834,0.0073799007,0.52532995,0.416126,0.18008909,0.14110237,0.99673915,0.13406962,0.33921134,0.41489682,0.442895,0.5816908,0.80970913,0.6648036,0.26837,0.5643006,0.53988045,0.02402428,0.16577701,0.91346854,0.61975455,0.2552258,0.6679491,0.01729772,0.6181179,0.4244343,0.31080252,0.36212626,0.89749485,0.5521082,0.40784967,0.87652504,0.87272465,0.47998312,0.28501573,0.16392118,0.8223444,0.761234,0.7927178,0.7222227]}"]
[2024/05/29 18:27:11.321 +08:00] [INFO] [base/milvus_client.go:37] [Request] [method=DescribeCollection] [reqs="{\"collection_name\":\"pre_rWHYWt\"}"]
[2024/05/29 18:27:11.329 +08:00] [INFO] [base/milvus_client.go:56] [Response] [method=DescribeCollection] [resp="{\"status\":{},\"schema\":{\"name\":\"pre_rWHYWt\",\"fields\":[{\"fieldID\":100,\"name\":\"int64\",\"is_primary_key\":true,\"data_type\":5},{\"fieldID\":101,\"name\":\"floatVec\",\"data_type\":101,\"type_params\":[{\"key\":\"dim\",\"value\":\"128\"}]}]},\"collectionID\":450070883630667076,\"virtual_channel_names\":[\"zong-go-test-rootcoord-d..."]
[2024/05/29 18:27:11.329 +08:00] [DEBUG] [base/milvus_client.go:57] [Cost] [method=DescribeCollection] [cost=7.665557ms]
[2024/05/29 18:27:11.401 +08:00] [INFO] [base/milvus_client.go:37] [Request] [method=Insert] [reqs="{\"collection_name\":\"pre_rWHYWt\",\"fields_data\":[{\"type\":5,\"field_name\":\"int64\",\"Field\":{\"Scalars\":{\"Data\":{\"LongData\":{\"data\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,..."]
[2024/05/29 18:27:11.573 +08:00] [INFO] [base/milvus_client.go:56] [Response] [method=Insert] [resp="{\"status\":{},\"IDs\":{\"IdField\":{\"IntId\":{\"data\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,..."]
[2024/05/29 18:27:11.573 +08:00] [DEBUG] [base/milvus_client.go:57] [Cost] [method=Insert] [cost=172.467182ms]
[2024/05/29 18:27:11.573 +08:00] [INFO] [base/milvus_client.go:201] [Insert] [result="{\"InsertCount\":3000,\"IDs\":{}}"]
  • rows data generation:
func GenInt64VecRows(nb int, enableDynamicField bool, autoID bool, option GenDataOption) []interface{} {
	dim := option.dim
	start := option.start

	rows := make([]interface{}, 0, nb)

	// BaseRow generate insert rows
	for i := start; i < start+nb; i++ {
		baseRow := BaseRow{
			FloatVec: common.GenFloatVector(dim),
		}
		if !autoID{
			baseRow.Int64 = int64(i)
		}
		if enableDynamicField{
			baseRow.Dynamic = GenDynamicRow(i)
		}
		rows = append(rows, &baseRow)
	}
	return rows
}

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

@ThreadDao ThreadDao added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 29, 2024
@yanliang567
Copy link
Contributor

/assign @congqixia
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 29, 2024
@yanliang567 yanliang567 added this to the 2.4.3 milestone May 29, 2024
@congqixia
Copy link
Contributor

congqixia commented May 29, 2024

Test with following code

package main

import (
	"context"
	"log"
	"math/rand"

	milvusclient "github.com/milvus-io/milvus/client/v2"
	"github.com/milvus-io/milvus/client/v2/row"
)

type Data struct {
	ID     int64     `milvus:"name:id;primary_key;auto_id"`
	Vector []float32 `milvus:"name:vector;dim:128"`
}

const (
	milvusAddr     = `localhost:19530`
	nEntities, dim = 10, 128
	collectionName = "hello_milvus"

	msgFmt                         = "==== %s ====\n"
	idCol, randomCol, embeddingCol = "id", "random", "vector"
	topK                           = 3
)

func main() {
	schema, err := row.ParseSchema(&Data{})
	if err != nil {
		log.Fatal("failed to parse schema from struct", err.Error())
	}

	for _, field := range schema.Fields {
		log.Printf("Field name: %s, FieldType %s, IsPrimaryKey: %t", field.Name, field.DataType, field.PrimaryKey)
	}
	schema.WithName(collectionName)

	ctx := context.Background()

	log.Printf(msgFmt, "start connecting to Milvus")
	c, err := milvusclient.New(ctx, &milvusclient.ClientConfig{
		Address: milvusAddr,
	})
	if err != nil {
		log.Fatal("failed to connect to milvus, err: ", err.Error())
	}
	defer c.Close(ctx)

	if has, err := c.HasCollection(ctx, milvusclient.NewHasCollectionOption(collectionName)); err != nil {
		log.Fatal("failed to check collection exists or not", err.Error())
	} else if has {
		c.DropCollection(ctx, milvusclient.NewDropCollectionOption(collectionName))
	}

	err = c.CreateCollection(ctx, milvusclient.NewCreateCollectionOption(collectionName, schema))
	if err != nil {
		log.Fatal("failed to create collection", err.Error())
	}

	var rows []any
	for i := 0; i < nEntities; i++ {
		vec := make([]float32, 0, dim)
		for j := 0; j < dim; j++ {
			vec = append(vec, rand.Float32())
		}
		rows = append(rows, &Data{
			Vector: vec,
		})
	}

	insertResult, err := c.Insert(ctx, milvusclient.NewRowBasedInsertOption(collectionName, rows...))
	if err != nil {
		log.Fatal("failed to insert data")
	}
	log.Println(insertResult.IDs)
}

the output was like follow

2024/05/29 20:58:28 Field name: id, FieldType int64, IsPrimaryKey: true
2024/05/29 20:58:28 Field name: vector, FieldType []float32, IsPrimaryKey: false
2024/05/29 20:58:28 ==== start connecting to Milvus ====
2024/05/29 20:58:28 &{{} id [450094383872654028 450094383872654029 450094383872654030 450094383872654031 450094383872654032 450094383872654033 450094383872654034 450094383872654035 450094383872654036 450094383872654037]}

@congqixia
Copy link
Contributor

It looks like the schema in your test case does not set auto id of pk field to true
/assign @ThreadDao
/unassign

@sre-ci-robot sre-ci-robot assigned ThreadDao and unassigned congqixia May 29, 2024
@ThreadDao
Copy link
Contributor Author

It looks like the schema in your test case does not set auto id of pk field to true /assign @ThreadDao /unassign

@congqixia Oh~ I only set schema AutoId to true.

In other words, the above test is equivalent to setting AutoID to false. In this case, inserting a row without pk data should also result in an error instead of returning zero IDs.

@yanliang567 yanliang567 modified the milestones: 2.4.3, 2.4.4 May 30, 2024
@congqixia
Copy link
Contributor

@ThreadDao BaseRow has field named Int64 the test case is equivalent to pass zero value pk column

@ThreadDao
Copy link
Contributor Author

@congqixia
I should not have passed an int64 value. My struct is set to ignore zero values omitempty . You can also see that I printed a line of logs showing that there is no int64 value

[2024/05/29 18:27:11.321 +08:00] [INFO] [testcases/insert_test.go:50] ["rows data"] ["rows[8]"="{\"floatVec\":[0.6178241,0.21087281,0.44867408,0.7325401,0.16834813,0.93801135,0.7424177,0.45748225,0.31714168,0.52351075,0.23468941,0.20924239,0.39932206,0.27672815,0.94168675,0.15122142,0.67313886,0.33818346,0.490186,0.37304404,0.8647382,0.984301,0.86432034,0.6576149,0.034932584,0.11069491,0.9180387,0.27059072,0.7769772,0.4492993,0.9091361,0.49744037,0.79990107,0.947381,0.79302114,0.9386312,0.08769191,0.08857079,0.6660659,0.63789123,0.8110448,0.6232179,0.06447095,0.7867628,0.011526427,0.23668459,0.31532583,0.930082,0.5526945,0.35251418,0.7554627,0.6498429,0.8466115,0.15021151,0.6578219,0.6485041,0.36646286,0.022119252,0.6632878,0.90742284,0.024551086,0.69591314,0.041073326,0.6479019,0.0280014,0.5167739,0.76998764,0.96678704,0.3818616,0.7679683,0.58915144,0.75805616,0.8849203,0.3163661,0.6954391,0.29520997,0.012191617,0.817875,0.84755325,0.89885163,0.9787547,0.7072435,0.14403586,0.124327034,0.67088,0.24942382,0.40041453,0.6156071,0.57584834,0.0073799007,0.52532995,0.416126,0.18008909,0.14110237,0.99673915,0.13406962,0.33921134,0.41489682,0.442895,0.5816908,0.80970913,0.6648036,0.26837,0.5643006,0.53988045,0.02402428,0.16577701,0.91346854,0.61975455,0.2552258,0.6679491,0.01729772,0.6181179,0.4244343,0.31080252,0.36212626,0.89749485,0.5521082,0.40784967,0.87652504,0.87272465,0.47998312,0.28501573,0.16392118,0.8223444,0.761234,0.7927178,0.7222227]}"]
type BaseRow struct {
	Bool       bool      `json:"bool,omitempty" milvus:"name:bool"`
	Int8       int8      `json:"int8,omitempty" milvus:"name:int8"`
	Int16      int16     `json:"int16,omitempty" milvus:"name:int16"`
	Int32      int32     `json:"int32,omitempty" milvus:"name:int32"`
	Int64      int64     `json:"int64,omitempty" milvus:"name:int64"`
	Float      float32   `json:"float,omitempty" milvus:"name:float"`
	Double     float64   `json:"double,omitempty" milvus:"name:double"`
	Varchar    string    `json:"varchar,omitempty" milvus:"name:varchar"`
	JSON       *JSONStruct `json:"json,omitempty" milvus:"name:json"`
	FloatVec   []float32 `json:"floatVec,omitempty" milvus:"name:floatVec"`
	Fp16Vec    []byte    `json:"fp16Vec,omitempty" milvus:"name:fp16Vec"`
	Bf16Vec     []byte    `json:"bf16Vec,omitempty" milvus:"name:bf16Vec"`
	BinaryVec    []byte    `json:"binaryVec,omitempty" milvus:"name:binaryVec"`
	SparseVec    entity.SparseEmbedding    `json:"sparseVec,omitempty" milvus:"name:sparseVec"`
	Array
	Dynamic
}

@congqixia
Copy link
Contributor

@ThreadDao json omitempty not working for client for now. maybe we shall add support later?

@yanliang567 yanliang567 modified the milestones: 2.4.4, 2.4.5 Jun 5, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.5, 2.4.6 Jun 26, 2024
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Jul 8, 2024
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Jul 8, 2024
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Jul 8, 2024
@ThreadDao ThreadDao removed their assignment Jul 11, 2024
@yanliang567 yanliang567 added this to the 2.4.11 milestone Sep 5, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.11, 2.4.12 Sep 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.12, 2.4.13 Sep 27, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.13, 2.4.14 Oct 15, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Oct 17, 2024
sre-ci-robot pushed a commit that referenced this issue Oct 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.14, 2.4.16 Nov 14, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.16, 2.4.17, 2.4.18 Nov 21, 2024
@ThreadDao ThreadDao modified the milestones: 2.4.18, 2.5.0 Dec 11, 2024
sre-ci-robot pushed a commit that referenced this issue Dec 12, 2024
- add log level for go sdk cases
- update cases for issue #33460 & #37853

---------

Signed-off-by: ThreadDao <[email protected]>
@yanliang567 yanliang567 modified the milestones: 2.4.18, 2.4.19, 2.4.20 Dec 24, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.20, 2.4.21 Jan 6, 2025
@yanliang567 yanliang567 modified the milestones: 2.4.21, 2.4.22 Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

3 participants