Add functionality so Starcoder can send back arbitrary PMTs, not just blobs #78

reiinakano · 2018-09-10T07:54:13Z

Adds functionality so Starcoder can send back arbitrary PMTs, not just data bytes. Instead of sending a serialized PMT, we convert it to the appropriate serialized Protobuf format and send it up the string queue.

We also introduce the Blob PMT. On the GNURadio side, this is equivalent to a uint8_vector, but since our protobuf maps the uint8_vector to a repeated uint32 field, it introduces a lot of overhead and necessitates weird conversion code when working with it from the gRPC side (e.g. converting []uint32 to []byte in Go).

vaelen · 2018-09-10T08:13:28Z

server/starcoder.go

 					BlockId: blockName,
-					Payload: bytes,
-				}); err != nil {
+					Pmt:     message,


It looks like this code is duplicated above. Could this be made more DRY?

vaelen

Someone who knows C++ better than I do should also review this.

cqueue/string_queue.h

kenichi-fukushima · 2018-09-10T08:19:20Z

gr-starcoder/lib/pmt_to_proto.h

+
+void convert_pmt_to_proto(const pmt::pmt_t &pmt_msg,
+                          starcoder::BlockMessage *proto_msg);
+void convert_proto_uniform_vector(const pmt::pmt_t &pmt_msg,


It seems like this function is only used by convert_pmt_to_proto and does not need to be exposes in a header file.

kenichi-fukushima · 2018-09-10T08:35:39Z

gr-starcoder/lib/pmt_to_proto.h

+#include <pmt/pmt.h>
+#include "starcoder.pb.h"
+
+void convert_pmt_to_proto(const pmt::pmt_t &pmt_msg,


Feel free to pass BlockMessage back as a return value. I believe the compiler is smart enough to avoid copying an object when returning it (and actually this behavior is required by the language specification.)

This is what I did first. The problem is the C++ generated protobuf has no way to set embedded message fields other than through getting a pointer to the embedded message field with the mutable_xxx() method and directly modifying that pointer (there is no set_xxx() method). See https://stackoverflow.com/questions/43268845/do-i-need-to-delete-objects-passed-to-google-protocol-buffer-protobuf. For the recursion to work, not sure if there's another way than having the function take a pointer.

It seems you can call Swap() on a mutable message. I haven't verified this API on my side and leave the decision up to you.
https://developers.google.com/protocol-buffers/docs/reference/cpp-generated#message

AMessageType parentMessage; ... AMessageType submessage = convert_pmt_to_proto(...); parentMessage.get_mutable_field()->Swap(&submessage); .... return parentMessage;

I guess this would work. Is there a specific reason to prefer returning the BlockMessage?

This is more or less a matter of personal preference so I left it up to you. But do you think it is more straightforward to return a value via a pointer argument than to use a return value? You wouldn't do it if you are programming in another language.

Just for reference, found these about the protobuf move semantics. Not sure if copy when reallocating affects us

protocolbuffers/protobuf#2791
protocolbuffers/protobuf#3630

FYI. No explicit move is needed in this case. Its optimization for returning a local variable from a function.
https://gist.github.com/kenichi-fukushima/197238e0cd23b82666a27abfa5fbd540

@kenichi-fukushima Converted this to your suggested function signature. It does seem more intuitive to use.

kenichi-fukushima · 2018-09-10T08:38:01Z

gr-starcoder/lib/pmt_to_proto.cc

+    const std::vector<std::complex<float>> vector_elements =
+        pmt::c32vector_elements(pmt_msg);
+    std::transform(vector_elements.begin(), vector_elements.end(),
+                   c32_vector->mutable_value()->begin(),


Don't you need to use a back inserter? Is it guaranteed that c32_vector has enough buffer to store all transformed elements?
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.repeated_field

Okay, I modified it. Could you check if I did it correctly? Sorry, I don't fully understand STL containers yet...

STL algorithms write to output iterators. And a vector's iterator is just the address of underlying memory buffer. So outputting to an iterator is no more than writing to a memory buffer. It won't change the container size, it won't reallocate the memory buffer if there is enough room.

kenichi-fukushima · 2018-09-10T08:38:10Z

gr-starcoder/lib/pmt_to_proto.cc

+    const std::vector<std::complex<double>> vector_elements =
+        pmt::c64vector_elements(pmt_msg);
+    std::transform(vector_elements.begin(), vector_elements.end(),
+                   c64_vector->mutable_value()->begin(),


same as above

kenichi-fukushima · 2018-09-11T02:31:48Z

gr-starcoder/lib/pmt_to_proto.cc

+    u_vector->set_size(starcoder::IntSize::Size8);
+    const std::vector<uint8_t> vector_elements =
+        pmt::u8vector_elements(pmt_msg);
+    *u_vector->mutable_value() = { vector_elements.begin(),


My C++ knowledge hasn't been update for C++11 and I really don't know how initializer expressions work... Can you explain how this assignment works? My reasoning is the right hand side calls RepeatedPtrField::RepeatedPtrField(Iter begin, const Iter & end) and then the whole content is copied for the another time, into the left hand side, which is redundant. You used transform and an inserter below. Why this piece of code follows a different pattern?

I changed it to use std::copy. Wdyt?

Looks good 👍

server/starcoder.go

Reiichiro Nakano added 22 commits August 29, 2018 14:07

rename convert_pmt_proto to convert_proto_to_pmt

93c33b5

init commit of pmt_to_proto

68e50c0

finish pmt_to_proto?

3b353b5

clang

c7b6653

update proto

044f5ad

convert in enqueue_message_sink

c873e2d

update starcoder to parse the protobufs

e62c48d

modify waterfall plotter to pass up protobuf

dc1308a

update noaa

2efc6e4

introduce blob_value

359b1c1

add blob to proto_to_pmt

f8c4b65

Merge remote-tracking branch 'upstream/master' into convert-pmt-to-proto

2c33d2e

test for proto_to_pmt

a610968

use pmt in meteor decoder

dc5fc29

experimental drop payload

044768e

fix closing

e4fbdbc

Merge remote-tracking branch 'upstream/master' into convert-pmt-to-proto

bf3e2ee

Merge remote-tracking branch 'upstream/master' into convert-pmt-to-proto

c6e7ab7

Merge remote-tracking branch 'upstream/master' into convert-pmt-to-proto

ad27a4d

Merge remote-tracking branch 'upstream/master' into convert-pmt-to-proto

db77770

check if queue is closed() to exit goroutine

df2b028

revert

5686c67

reiinakano requested review from vaelen, anuraaga, kenichi-fukushima and koncha99 September 10, 2018 07:54

reiinakano changed the title ~~Convert PMT to Protocol buffer~~ Add functionality so Starcoder can send back arbitrary PMTs, not just blobs Sep 10, 2018

vaelen reviewed Sep 10, 2018

View reviewed changes

factor out function

1215f26

vaelen approved these changes Sep 10, 2018

View reviewed changes

kenichi-fukushima reviewed Sep 10, 2018

View reviewed changes

Reiichiro Nakano added 4 commits September 10, 2018 17:58

dont expose in header file

8c1522d

use back inserter

26a5a50

use repeatedptrfieldbackinsertiterator

55f1eea

clang

d9d22b9

kenichi-fukushima reviewed Sep 11, 2018

View reviewed changes

koncha99 reviewed Sep 11, 2018

View reviewed changes

server/starcoder.go Outdated Show resolved Hide resolved

Reiichiro Nakano added 2 commits September 11, 2018 12:57

use std::copy

3e95525

rename request to respose

c2b222e

kenichi-fukushima approved these changes Sep 11, 2018

View reviewed changes

koncha99 approved these changes Sep 11, 2018

View reviewed changes

Reiichiro Nakano added 2 commits September 11, 2018 14:57

change signature of pmt_to_proto

d0df957

changes to test.grc

a9b4413

reiinakano merged commit 7c119d8 into master Sep 11, 2018

reiinakano deleted the convert-pmt-to-proto branch September 11, 2018 06:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add functionality so Starcoder can send back arbitrary PMTs, not just blobs #78

Add functionality so Starcoder can send back arbitrary PMTs, not just blobs #78

reiinakano commented Sep 10, 2018 •

edited

Loading

vaelen Sep 10, 2018

vaelen left a comment

kenichi-fukushima Sep 10, 2018

kenichi-fukushima Sep 10, 2018

reiinakano Sep 10, 2018 •

edited

Loading

kenichi-fukushima Sep 10, 2018

reiinakano Sep 10, 2018 •

edited

Loading

kenichi-fukushima Sep 11, 2018

anuraaga Sep 11, 2018

kenichi-fukushima Sep 11, 2018

reiinakano Sep 11, 2018

kenichi-fukushima Sep 10, 2018

reiinakano Sep 10, 2018

kenichi-fukushima Sep 11, 2018

kenichi-fukushima Sep 10, 2018

kenichi-fukushima Sep 11, 2018

reiinakano Sep 11, 2018

kenichi-fukushima Sep 11, 2018

Add functionality so Starcoder can send back arbitrary PMTs, not just blobs #78

Add functionality so Starcoder can send back arbitrary PMTs, not just blobs #78

Conversation

reiinakano commented Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

vaelen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reiinakano Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reiinakano Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reiinakano commented Sep 10, 2018 •

edited

Loading

reiinakano Sep 10, 2018 •

edited

Loading

reiinakano Sep 10, 2018 •

edited

Loading