Skip to content

Commit

Permalink
READEME update and throw error when previously ingested data format m…
Browse files Browse the repository at this point in the history
…ismatch (#128)
  • Loading branch information
Big-Vi authored Oct 4, 2023
1 parent ec02b62 commit e86f3ce
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 3 deletions.
29 changes: 26 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,9 +362,13 @@ The consumer will try to fetch messages every ```pullInterval``` (that was given
```go
func handler(msgs []*memphis.Msg, err error, ctx context.Context) {
if err != nil {
m := msgs[0]
fmt.Println(string(m.Data()))
m.Ack()
fmt.Printf("Fetch failed: %v", err)
return
}

for _, msg := range msgs {
fmt.Println(string(msg.Data()))
msg.Ack()
}
}

Expand All @@ -373,6 +377,25 @@ consumer.Consume(handler,
)
```

#### Consumer schema deserialization
To get messages deserialized, use `msg.DataDeserialized()`.

```go
func handler(msgs []*memphis.Msg, err error, ctx context.Context) {
if err != nil {
fmt.Printf("Fetch failed: %v", err)
return
}

for _, msg := range msgs {
fmt.Println(string(msg.DataDeserialized()))
msg.Ack()
}
}
```

if you have ingested data into station in one format, afterwards you apply a schema on the station, the consumer won't deserialize the previously ingested data. For example, you have ingested string into the station and attached a protobuf schema on the station. In this case, consumer won't deserialize the string.

### Fetch a single batch of messages
```go
msgs, err := conn.FetchMessages("<station-name>", "<consumer-name>",
Expand Down
5 changes: 5 additions & 0 deletions consumer.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,11 @@ func (m *Msg) DataDeserialized() (any, error) {

msgBytes := m.msg.Data

_, err = sd.validateMsg(msgBytes)
if err != nil {
return nil, memphisError(errors.New("Deserialization has been failed since the message format does not align with the currently attached schema: " + err.Error()))
}

switch sd.schemaType {
case "protobuf":
pMsg := dynamicpb.NewMessage(sd.msgDescriptor)
Expand Down

0 comments on commit e86f3ce

Please sign in to comment.