End of Over Data #35

drowe67 · 2024-11-22T01:10:01Z

An approach to get some sort of text support for RADE V1, to support FreeDV and PSK reporter messaging. Up to 180 bits are inserted in to the End of Over frame.

Limitations:

Only one frame of 180 bits (90 QPSK symbols) at the end of over (not continuously streamed)
Will only work down to moderate (>3dB AWGN, >6dB MPP) SNRs as EQ isn't as good as for RADE symbols - and data is harder to get through a channel than speech with RADE!
This will likely be discarded with future developments of RADE (6 month time frame) so we need to throttle the amount of work we put into it.

TODO PoC:

TODO completion:

PLT consensus on continuing
try it on a multipath channel at medium SNR - works OK at 6dB on MPP - ctest -V -R radae_eoo_data_mpp
fix failing ctests
is output soft decn symbols (e.g. for LDPC dec) or bits. Soft decn.
break out to C API, need funcs to set EOO bits, and read EOO bits, way of detecting EOO data has been updated

Demo Results:

ctest -V -R radae_eoo_data_mpp

Test on a MPP channel at 6dB SNR, 5 "overs" are sent, and we attempt to get at least one EOO data packet with a raw BER < 5%.

…ew bi errors at a reasonable SNR

…his is a weakness in the test designs

drowe67 · 2024-11-26T20:07:12Z

@tmiw re rade_api.h and programming model for returning received EOO bits/symbols:

I want to a avoid a callback between Python and C as it feels like pain and this needs to be quick and easy. So suggest we use polling, like a function that tells us if the EOO rx data has been recently received/updated.
I can't recall - do you need soft decision QPSK symbols as the input for the reliable text LDPC decoder? Alternatively I can supply you with bits.
I'll draft an API example for you for review.

drowe67 · 2024-11-26T20:50:59Z

@tmiw - take a look at rade_rx.c & rade_api.h and see if that work for you. Note n_eoo_features_out is the number of floats returned which is 2x the number of complex QPSK symbols. Note I haven't implemented the API just yet, so it won't actually work. Once we're agreed on the API I'll set to work on hooking everything up and add a ctest.

tmiw · 2024-11-27T09:59:50Z

I can't recall - do you need soft decision QPSK symbols as the input for the reliable text LDPC decoder? Alternatively I can supply you with bits.

I looked at the code again and it looks like it can do hard decision if it absolutely needed to. The relevant part from codec2:

  if (obj->bit_index == obj->sym_index * 2) {
    // Use soft decision for the LDPC decoder.

    int Npayloadsymsperpacket = LDPC_TOTAL_SIZE_BITS / 2;

    // Deinterleave symbols
    gp_deinterleave_comp(
        (COMP*)deinterleavedSyms,
        (COMP*)&obj->inbound_pending_syms[RELIABLE_TEXT_UW_LENGTH_BITS / 2],
        Npayloadsymsperpacket);
    gp_deinterleave_float(
        deinterleavedAmps,
        &obj->inbound_pending_amps[RELIABLE_TEXT_UW_LENGTH_BITS / 2],
        Npayloadsymsperpacket);

    float EsNo = 3.0;  // note: constant from freedv_700.c

    symbols_to_llrs(llr, (COMP*)deinterleavedSyms, deinterleavedAmps, EsNo,
                    obj->fdv->ofdm->mean_amp, Npayloadsymsperpacket);
  } else {
    // Deinterlace the received bits.
    gp_deinterleave_bits(deinterleavedBits, src, LDPC_TOTAL_SIZE_BITS / 2);

    // We don't have symbol data (likely due to incorrect mode), so we fall back
    // to hard decision.
    for (int bitIndex = 0; bitIndex < LDPC_TOTAL_SIZE_BITS; bitIndex++) {
      // fprintf(stderr, "rx bit %d: %d\n", bitIndex,
      // deinterleavedBits[bitIndex]);

      // Map to value expected by sd_to_llr()
      incomingData[bitIndex] = 1.0 - 2.0 * deinterleavedBits[bitIndex];
    }

    sd_to_llr(llr, incomingData, LDPC_TOTAL_SIZE_BITS);
  }
  run_ldpc_decoder(&obj->ldpc, output, llr, &parityCheckCount);

Not sure if the interleaving is needed, either. Regardless, if it's easy to provide the stuff needed for soft decision, that's fine.

@tmiw - take a look at rade_rx.c & rade_api.h and see if that work for you.

Looking now.

tmiw · 2024-11-27T10:05:08Z

src/rade_api.h

@@ -109,7 +110,9 @@ RADE_EXPORT int rade_tx_eoo(struct rade *r, RADE_COMP tx_eoo_out[]);
 RADE_EXPORT int rade_nin(struct rade *r);

 // returns non-zero if features_out[] contains valid output. The number
-// returned is the number of samples written to features_out[]
+// returned is the number of samples written to features_out[].  If the


I think it might be a bit clearer to have a separate eoo_bits_out argument and an indicator that this has been populated, i.e.

RADE_EXPORT int rade_rx(struct rade *r, float features_out[], bool* has_eoo_out, float eoo_out[], RADE_COMP rx_in[]);

i.e.

bool has_eoo_out = false; int ret = rade_rx(r, features_out, &has_eoo_out, eoo_out, rx_in); if (has_eoo_out) { // do something with the EOO bits }

Also, I don't see a way to populate the EOO bits for TX yet?

drowe67 · 2024-11-27T11:29:30Z

We should use soft decision if supported.

I think it might be a bit clearer to have a separate eoo_bits_out argument and an indicator that this has been populated, i.e.

What you are suggesting is significantly more work. In the spirit of this feature - is it really necessary or will my proposal do the job? This will all be tossed away in 6 months and replaced with a new system (and new API for txt that returns a constant low bit rate like existing FreeDV modes).

Oops I forgot Tx, will attend to that.

drowe67 · 2024-11-28T00:33:38Z

Thinking further 🤔 - we should be able to implement the API you have suggested, e.g.:

RADE_EXPORT int rade_rx(struct rade *r, float features_out[], bool* has_eoo_out, float eoo_out[], RADE_COMP rx_in[]);

With some re-arrangement on the C side only, i.e. avoiding moving all of that stuff over the C-Python interface (my main concern). I'll get something working with my original proposal then see if we can refactor.

drowe67 · 2024-11-29T01:23:26Z

Hi @tmiw - I think this is ready for you to try hooking up to the reliable text system. Pls see the PR description ☝️ for demo instructions and limitations.

tmiw · 2024-12-01T17:18:45Z

src/rade_api.h


 // note vocoder is not encapsulated in API in this version
 // returns number of RADE_COMP samples written to tx_out[]
 RADE_EXPORT int rade_tx(struct rade *r, RADE_COMP tx_out[], float features_in[]);

+// Set the rade_n_eoo_bits() bits to be sent in the EOO frame, which are
+// in +/- 1 float form (note NOT 1 or 0)
+RADE_EXPORT void rade_tx_set_eoo_bits(struct rade *r, float eoo_bits[]);


It seems that the code from codec2 just passes the bits directly for TX instead of using floats. Is that viable to do here too? If not, is there already some sort of way to convert to the form this function needs?

Can you pls describe the format of the bits generated by reliable text?

Basically, the existing codec2 reliable_text system ties into the varicode functionality. When you set a callsign (https://github.com/drowe67/codec2/blob/main/src/reliable_text.c#L384), it does the following:

Converts the ASCII callsign into a special six-bit character set. This effectively generates a 48-bit string of zeros and ones.

Calculates the CRC8 of the above 48-bit string and appends it to the end. This results in 56 bits of data.

Generates parity bits with the HRA_56_56 LDPC code and appends to the end of (2). (I have mentions of LDPC(112,56) in the comments but that might be wrong.) We now have 112 bits of data.

Interleaves the parity and data bits together.

Saves this result somewhere in memory.

When codec2 calls the TX callback that reliable_text injects (https://github.com/drowe67/codec2/blob/main/src/reliable_text.c#L340), the latter returns the next byte of data from (5), i.e.

Call 1: return txString[0]

Call 2: return txString[1]

...

Call 14: return txString[13]

Call 15: return txString[0]

...

This is then injected into the data transmitted over the air.

Anyway, I hope this helps. Let me know if you need further clarification.

OK so I think this is the array of bits to be sent to the modem:

char tx_text[LDPC_TOTAL_SIZE_BITS + RELIABLE_TEXT_UW_LENGTH_BITS];

So the format is one bit per char. We'll need some glue code to connect reliable text to the EOO modem. So on the tx side we want a binary 1 -> +1, and binary 0 -> -1, something like:

eoo_bit[i] = 2.0*tx_text[i] - 1.0;

The LDPC decoder expects the received symbol to be mapped to the QPSK constellation defined by (codec2/src/mp_decode.c:

static COMP S_matrix[] = { {1.0f, 0.0f}, {0.0f, 1.0f}, {0.0f, -1.0f}, {-1.0f, 0.0f}};

which is 45 degrees rotated from the EOO text constellation {1,1}, {-1,1}, {-1,-1}, {1,-1} . So on the rx side, something like (ROT45 from codec2/src/ofdm.c):

complex float symbol = (COMP)eoo_out[2*i]; reliable_text_demod_symbol[i] = symbol * cmplx(ROT45);

Good idea to have a unit tests that runs with no noise, make sure the LDPC decoding happens with 1 iteration, just to make sure we haven't messed up anything. Then add some noise, e.g. 2-3dB AWGN and make sure it hangs on. Not sure where to do the tests .. it's a mash up of several repos 🤔

Not sure where to do the tests .. it's a mash up of several repos

Considering the current API, I wonder if having this in freedv-gui would be best. That way, it'd have access to some of the codec2 functions (e.g. LDPC and interleaving) and we wouldn't need to reimplement versions in this repo. OTOH testing does become a bit harder but maybe doable with the current test framework. If that sounds good, I'll go ahead and do that.

Yes I agree, it's the only project with everything linked in. And you now have a nice command line test framework for freedv-gui to support automated tests :+1

OK, I created drowe67/freedv-gui#783 but I suspect there's something wrong with either the TX or RX symbol generation; BER is much too high when playing back files generated locally (i.e. no RF). I also disabled interleaving just to be sure that wasn't interfering. I'll have to investigate more when I have time.

tmiw · 2024-12-01T17:23:45Z

src/rade_api.c

@@ -426,7 +457,7 @@ int rade_tx_eoo(struct rade *r, RADE_COMP tx_eoo_out[]) {
  return r->Neoo;
 }

-int rade_rx(struct rade *r, float features_out[], RADE_COMP rx_in[]) {
+int rade_rx(struct rade *r, float features_out[], int *has_eoo_out, float eoo_out[], RADE_COMP rx_in[]) {


Just to make sure, eoo_out is basically structured so that odd indexed items are symbols and even indexed symbols are amplitudes, right? i.e.

eoo_out[0] = real value of symbol
eoo_out[1] = complex value of symbol
eoo_out[2] = amplitude
...

Do the notes above rade_rx() in rade_api.h help?

For the LDPC decoder you will need to calculate the symbol amplitudes. As a starting point, I'd suggest making one estimate for the entire vector (e.g. the RMS amplitude across all symbols) and filling all entries of the amplitude[] vector with that value.

I actually didn't see that comment originally, sorry about that. I'll think about it some more.

drowe67 added 10 commits November 22, 2024 11:38

wip insert some QPSK symbols in EOO frame

6fb88ba

bottleneck 3 for data symbols

91a77f0

wip EOO data basic rx

14fd1c4

inferernce.py wasn't adding phase and freq offsets to EOO!

46569da

based EOO data proof of concept (ML side) - getting data out with a f…

e364a63

…ew bi errors at a reasonable SNR

wip fix failing ctests

d851f8f

Merge branch 'main' into dr-eoo-data

0ab0061

used a custom RNG for EOO data bits, which stopped some test fails. T…

52b412b

…his is a weakness in the test designs

comment out #weights prints, to reduce noise when running tests

0bb2a12

allow for EOO not triggering at low SNR awgn test

a006b07

drowe67 added 3 commits November 27, 2024 07:05

prototype EOO rade_api.h

2dad457

Merge branch 'dr-eoo-data' of github.com:drowe67/radae into dr-eoo-data

8ccd3cb

n_eoo_features out is number of floats rather than symbols

238facf

tmiw reviewed Nov 27, 2024

View reviewed changes

drowe67 added 7 commits November 28, 2024 12:47

wip EOO ouput data through C API

a106dfc

wip EOO data - about to refactor tests

39bf054

1st pass EOO data unit test framework

048ecc7

initial attempt at passing in tx EOO bits via function call (Python)

3f8a23b

numpy interafce for setting eoo bits

bcb032f

first pass of EOO data through C API

09e9547

multipath channel test

1ca79a6

drowe67 changed the title ~~End of Over Data prototype~~ End of Over Data Nov 29, 2024

rade_rx() API change as per Mooneer's suggestion

5c02edf

tmiw reviewed Dec 1, 2024

View reviewed changes

rm resource_est.py (moved to papers repo)

d294bcb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

End of Over Data #35

End of Over Data #35

drowe67 commented Nov 22, 2024 •

edited

Loading

drowe67 commented Nov 26, 2024

drowe67 commented Nov 26, 2024

tmiw commented Nov 27, 2024

tmiw Nov 27, 2024

drowe67 commented Nov 27, 2024 •

edited

Loading

drowe67 commented Nov 28, 2024 •

edited

Loading

drowe67 commented Nov 29, 2024

tmiw Dec 1, 2024

drowe67 Dec 1, 2024

tmiw Dec 2, 2024

drowe67 Dec 2, 2024 •

edited

Loading

tmiw Dec 6, 2024

drowe67 Dec 6, 2024

tmiw Dec 7, 2024

tmiw Dec 1, 2024

drowe67 Dec 1, 2024

tmiw Dec 2, 2024

End of Over Data #35

Are you sure you want to change the base?

End of Over Data #35

Conversation

drowe67 commented Nov 22, 2024 • edited Loading

Demo Results:

drowe67 commented Nov 26, 2024

drowe67 commented Nov 26, 2024

tmiw commented Nov 27, 2024

Choose a reason for hiding this comment

drowe67 commented Nov 27, 2024 • edited Loading

drowe67 commented Nov 28, 2024 • edited Loading

drowe67 commented Nov 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drowe67 Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drowe67 commented Nov 22, 2024 •

edited

Loading

drowe67 commented Nov 27, 2024 •

edited

Loading

drowe67 commented Nov 28, 2024 •

edited

Loading

drowe67 Dec 2, 2024 •

edited

Loading