Hardware (CPLD-based) synchronisation of multiple HackRFs

[barto-]

Hardware (CPLD-based) synchronisation

This PR allows to synchronise multiple HackRFs with a synchronisation error below 1 sampling period #341

WARNING: Use this at your own risk. If you don't know what you are doing you may damage your HackRF.
The author takes no responsability for potential damages

Usage example: synchronise two HackRFs

1. Chose the master HackRF which will send the synchronisation pulse (HackRF0). HackRF1 will represent the slave hackrf.

2. Retreive the serial number of both HackRFs using hackrf_info

3. Use a wire to connect SYNC_CMD of HackRF0 to SYNC_IN of HackRF0 and HackRF1

4. Run hackrf_transfer with the argument -H 1 to enable hardware synchronisation:

   $ hackrf_tranfer ... -r rec1.bin -d HackRF1_serial -H 1 | hackrf_transfer ... -r rec0.bin -d HackRF0_serial -H 1
   

rec0.bin and rec1.bin will have a time offset below 1 sampling period.
The 1PPS output of GNSS receivers can be used to synchronise HackRFs even if they are far from each other.

DON'T APPLY INCOMPATIBLE VOLTAGE LEVELS TO THE CPLD PINS

Signal	Header	Pin	Description
SYNC_IN	P28	16	Synchronisation pulse input
SYNC_CMD	P28	15	Synchronisation pulse output

Note:

I had to remove CPLD-based decimation to use a GPIO for enabling hardware synchronisation.

More info:

M. Bartolucci, J. A. Del Peral-Rosado, R. Estatuet-Castillo, J. A. Garcia-Molina, M. Crisci and G. E. Corazza, "Synchronisation of low-cost open source SDRs for navigation applications," 2016 8th ESA Workshop on Satellite Navigation Technologies and European Workshop on GNSS Signals and Signal Processing (NAVITEC), Noordwijk, 2016, pp. 1-7.

Alternative link

[dodgymike]

Nice! Do you think it would be possible to add a second channel so sync can be chained? My use-case is four hackrfs.

[barto-]

If you want to synchronize 4 hackrfs just connect SYNC_IN of the 4 hackrfs in parallel.

[nononame]

I don't clearly understand these commits @barto- . My understanding is:

• you add a trigger-like to capture data.
• the trigger-like signal is an input of the CPLD.
• to do this you delete decimation in CPLD and timeout in host software

My miss understanding is why do you re-use the "--hw-sync" option? This option was not relied to the CLKIN/CLKOUT synchronisation? With do you not implement a new option?

[barto-]

@nononame
The previous implementation was using MCU GPIO pins. The MCU was reading the GPIOs in polling mode.
With this commit (using CPLD GPIOs) we can achieve synchronisation errors that orders of magnitude smaller with respect to the previous implementation. Moreover the latest implementation guarantees that the synchronisation error is limited and falls in the interval [-1 sample; +1sample]. We also know the probability distribution of the synchronisation error.
I'm reusing the --hw-sync option because, in my opinion, this is an improved version of the previous implementation.
The CLKIN/CLKOUT synchronisation is a completely different matter. These two ports are used to feed the same clock to multiple HackRFs. This, however, does not guarantee that multiple HackRFs start transmitting/receiving at the same time.
The CLKIN/CLKOUT synchronisation is still there and it's not modified in any way by this commit.

[nononame]

@barto-
Thanks for that clarification.

[mrroboto1420]

@barto- I was able to update the CPLD to the latest version. Two questions:
(1) When I try to run the hackrf_transfer test, it just "waits for sync" indefinitely. I have double-checked my connections and they seem to be correct. I have pin 15 of the master going to pin 16 of itself and the slave. The CLKOUT of the master SDR is connected to the CLKIN of the slave SDR. Is my setup correct?
(2) I have noticed that pin 15 and pin 16 on the master SDR must be connected for it to operate normally. It is my understanding that this is a requirement because of the new CPLD code, is that correct?

[barto-]

Have you updated the MCU firmware?

[mrroboto1420]

Yes, I did that a few months ago. Both boards have: Firmware Version: 2018.01.1 (API:1.02)

I updated the CPLD by forking the most recent version of the hackrf master, downloaded it, navigated to the firmware folder, and successfully ran: hackrf_cpldjtag -x firmware/cpld/sgpio_if/default.xsvf

This ran correctly based on the LEDs.

Is it a physical connection issue? Also, is the needing to connect P28 pin 15 and 16 of a single HackRF necessary with this new CPLD version? That is what I am seeing at the moment.

[barto-]

It could be a physical connection issue. If one or more hackrf get stuck waiting for sync it means they don’t receive the sync pulse. If you have access to a scope(or even a multimeter if you don’t find anything better) you can check if pin 16 goes high.
The connection of pin 15 and 16 on the same hackrf is required only when you run hackrf_transfer with the -H 1 option

[mrroboto1420]

Hmm ok I will try that probing the output later.

That is NOT the behavior I am seeing. For example, if I connect one HackRF to my computer it will not work with GQRX (i.e. it connects to the SDR, but there is no data being streamed). However, if I connect pin 15 and 16 on that HackRF, GQRX will show that the HackRF is now working. This behavior happens system wide with any app (GRC, GNU Radio, etc.)

Is there another version of the CPLD default.xsvf I should try?

[barto-]

Try to run hackrf_transfer with the option -H 0 and then see if gqrx works without making the connection between pins 15 and 16

[dominicgs]

The changes that @barto- made aren't in the 2018.01.1 release. You need to build firmware from git to get them.

[mrroboto1420]

@barto- I was able to try that yesterday, that method did not work.
@dominicgs ah my mistake, I will try that tonight. So to clarify, I just have to follow the procedure here, I just have to download the most recent firmware file from the master?

I am attempting to use these for coherent measurements "out-of-the-box." It really seems quite silly that the Hack RF only has CLKIN / CLKOUT ports. If you can't sync the sampling just as easily, what is the point? There are plenty of ways of generating a 10MHz clock in the lab. I guess this is something you would catch if you looked a little deeper into the Hack RF docs / if you take the "open-source nature" into account, but the "out-of-the-box" synchronization was one of the primary draws of the Hack RF (that and the ability to go up to 6 GHz of course). I do really hope I can get this method working!

By the way, are there any updates on implementing the hardware synchronization in the osmocom library / GRC? To me that was the real kicker, even after you make the hardware hacks like the CPLD tweak, it only works at the command-line.

It is good to see that development is very active though!

[dominicgs]

The hardware synchroinisation is a relatively new feature, so right now it requires you to build the firmware from the master branch until we get another release out.

If you're running Debian / Ubuntu Linux you can:

apt install gcc-arm-none-eabi cmake python-yaml dfu-util
cd firmware/libopencm3
git submodule init
git submodule update
make
cd ../firmware/hackrf_usb
mkdir build
cd build
cmake ..
make
'''

This should generate the latest firmware binary.  The latest CPLD image is in the repository.

[barto-]

@dominicgs thank you for clarifying that. I was not aware that my pull request wasn’t included in the latest release.

[mrroboto1420]

@dominicgs I am having difficulties compiling libopencm3 library. Am I messing up the initial step? I am able to run apt install gcc-arm-none-eabi cmake python-yaml dfu-util fine. Then I go to my downloaded hack-rf master directory and to the appropriate sub-directory to run the other code. But I cannot run git submodule init. I then tried downloading libopencm3 and ran make as per the readme for that library. That worked, but generating the .bin still fails.

[bjswift]

Hi, thanks SO much @barto- your prior work on this is exactly what I needed! One question, though, (to any/everyone, as well), @barto- you mention with a stern warning up top in the main comment to this "DON'T APPLY INCOMPATIBLE VOLTAGE LEVELS TO THE CPLD PINS", but then you don't offer any guidance. What would be an acceptable level here? I am a little new, sorry.

If I'm not mistaken, the CPLD runs on 1.8V logic in&out, so it's going to want only that? Or, is there something going on that I don't understand which will make it 3.3V- or 5V-safe.

Connecting to another board's HOST_SYNC_CMD pin I guess will naturally be safe. But I'd also like to like you did, and run off of a 1PPS line from GPSDO unit. In my case, this is 4V pulse. @barto- , did you level-shift or already have a ~1.8V 1PPS signal in your original experiment? Alternatively, I'd like to run off a GPIO pin from MCU or a Raspberry Pi or something like that, which are both definitely not 1.8V-safe.

Anyway, advice or experience from anyone in the crowd is surely welcome! :)

[mossmann]

The CPLD core runs at 1.8 V, but the I/O is at 3.3 V. I suggest using a resistor voltage divider to bring your 4 V pulse down to 3 V.

[bjswift]

Excellent, thank you for the insights, @mossmann! That's quite convenient, indeed.

I was thinking of a voltage divider, too, since the current is so low it'll not have a problem working properly.

So, I'm doing ok with all this so far, but no good sync, yet...

Took me a bit before realizing it shipped with release 2018.01.1 and the hackrf_transfer -H flag was now working as it "should". Thanks to the comments above, I figured that out, and then cloned the master branch, built the firmware, and uploaded it to my two boards (thanks especially @dominicgs for the explicit instructions!).

With all this, I did get them to take data at the same time using the piped commands as @barto- suggests (step 4, way up top, like "hackrf_transfer [other flags] -d $SERIAL1 -r rx1.dat -H1 | hackrf_transfer [other flags] -d $SERIAL2 -r rx2.dat -H1" ) with the board in the 2nd half of the command serving as the one that provides the HOST_SYNC_CMD signal to both of them. This made sense to me, since you want the command to the master to be issued last, so that the/any slave/s are ready to go.

... but it seems like they AREN'T actually synced. In the couple tests I've done (3 -- will do more when I get a chance soon), there are definitely offsets of hundreds to thousands of samples. In one of the 3 tests, it actually did seem close enough that it wasn't obvious they were sync'ed or not.

In the cases with the delay, the one that seems to lag behind is the one serving as master, which is extra-confusing to me.

Thoughts from the crowd? What am I doing wrong? Again, I'll get back to trying out some more ideas as soon as I can in a few hours.

(Running under Ubuntu 18.x on a brand-name laptop. To see rough timing, using a 3rd SDR as a CW source that is turned on/off in the middle of the HackRf rx recording. Plotting the magnitude of the sample data, I figure that should be good enough that to quickly visually see whether each rx antenna sees the tx signal come and go w/in a sample or few, or whether there is a huge offset.)

[mossmann]

I was just looking through your description and the instructions above to see if anything essential might be missing. One thing I noticed is that the instructions don't say that you should connect GND between the two HackRFs, but you certainly should. One way to do that would be to connect a CLKOUT-> CLKIN SMA cable which is probably desirable anyway. Have you connected GND?

[bjswift]

Yup, did that explicitly via headers, and had the same thought process regarding CLKIN & CLKOUT.

So I am connecting CLKOUT of the master to CLKIN of the slave (even though it doesn't really matter for the frequency-side of things, AFAIK), so ground is connected there, and I'm also connecting P28:pin2 between the boards, like in the photo of the proto board in the @barto- et al paper.

ETA - here's a photo

[bjswift]

Testing out the -H flag with and without the HOST_SYNC_CMD pin connected back to the HOST_SYNC pins does confirm that everything there seems to work correctly. With -H1 and nothing to HOST_SYNC, it stalls out, and with -H0 it does just fine. Certain the CPLD and the microcontroller are doing the job right.

Still getting major offsets of several thousand samples, but now sometimes the slave device tends to be later, and not consistently. I'm starting to suspect that their sampling clocks aren't actually synced up like I'm expecting from the CLKOUT -> CLKIN connection between the boards.

If I'm seeing an offset of a few thousand samples, a few tens of millions of samples into the recording, then that's a few parts in 10^4 difference in clock rates. Seems pretty big. I don't have a great way to test this theory that I can think of.

Am I supposed to do something more than I am, or maybe can I do something more explicitly to make sure that the one board is really using the clock signal of the other? I haven't noticed anything yet other than the basic "hook up CLKIN to CLKOUT" idea, so I was assuming it was supposed to be auto-detecting.

ETA (minutes after your response, I was still composing this) --
Success? I can get it to work, but only under very particular conditions: (1) hook an external 10 MHz to both CLKINs, (2) first measurement only after hitting RESET (tried only the button, not other methods for resetting).

I was planning on implementing (1) my ultimate setup, anyhow via a GPSDO to keep ~dozen of these radios in line. But it's not a very satisfying answer, and not what I expected from my understanding of these devices, so far.

Even more dissatisfying is (2) that it only works for a single measurement, after a hard reset. :/

Using master CLKOUT-->slave CLKIN does kinda work after a reset, but it's still off by a few (literally small, like 6-10) samples, whereas I see an offset of zero or 1 samples with the 10 MHz in to both separately. So that's puzzling.

Adding to the puzzle is that if I do an external 10 MHz into master CLKIN, and then link master CLKOUT to slave CLKIN, I'll get an offset of 1-2 thousand samples, even immediately after a reset. Additionally, it is very consistently a fractional offset of 2.00e-5.

(I'm wondering if this is diverging from being the appropriate forum for these kinds of questions and discussion..)

[mossmann]

You need to use the hackrf_clock utility to turn on CLKOUT (only when running firmware from git (until our next release)).

[bjswift]

Thanks. I just added a bunch of interesting observations to my previous comment, but this hackrf_clock thing you mention seems like it would clear up at least some of those.

Found it and tried building it using a similar recipe to @dominicgs 's helpful comment above (from Apr 13, 2018), but it's complaining when I make. Will have to come back to it tomorrow. Thanks again for all your helpful insights! :)

[barto-]

@bjswift
There was an annoying bug in the past, I had to reset the HackRFs before each capture. I have no idea if this problem has been fixed. I’m sorry, I don’t have time to look into it.

[bjswift]

No worries @barto- ...I'm sure that we'll figure it out soon here. Yes, the moment I realized it only worked after reset, I recalled the notes about that in your paper! I guess I'd assumed that issue was sorted out with your/this commit/merge, but as I'm discovering, not everything is all tightly implemented for this moderately-esoteric use-case. :) (That's not at all meant to be a criticism!)

Not sure yet that I'll have to make changes like you did to the firmware/software, since there are a few other options available to me right now...

I was just experimenting this morning with resetting using "hackrf_spiflash -d serialno -R", but sometimes the device doesn't come back up so I'm gonna look into trying to reconnect via some other command-line thing in those cases.

Squinting at the traces and schematics I found that the reset button connects back to P22:pin3, and ultimately I'll be running things with a Raspberry Pi, so I can always tickle that pin programmatically with the Pi's GPIO as if I was directly pushing the button.

If anyone knows of a better idea, I'm open to it. (Just joined the IRC channel right now, so maybe we can keep this comment thread a little less cluttered now.)

[barto-]

if the device doesn’t come back using the reset command it got stuck. See #394 . It is another issue unrelated with CPLD synchronization. As a quick and dirty workaround I was using the watchdog time to reset the device automatically if it got stuck.

[bjswift]

Ok thanks for the guidance, I'll have to digest and re-read that a few times. I don't know if I'd call that "quick and dirty" though, since you used a hardware feature on the microcontroller! My workaround idea to use the external host to keep kicking the reset pin is WAY quicker-and-dirtier. ;)

May be nice if your thing eventually gets implemented. Did you ever share the code for that with anyone?

ETA -- went to your repos, and poked around, which linked me back to @mossmann 's libopencm3. My reading of the commits there seems to indicate that a watchdog reset of some kind is implemented.

[barto-]

Actually it’s easier than it looks. Open the file firmware/hackrf_usb/hackrf_usb.c And locate the while(true) statement. Edit as follows:

// ...
	WWDT_TC = 0xffffff;
	WWDT_MOD = 0x0003;
	WWDT_FEED = 0xAA;
	WWDT_FEED = 0x55;

	unsigned int phase = 0;
	while(true) {
		WWDT_FEED = 0xAA;
		WWDT_FEED = 0x55;
//...

Be aware that, if you decide to use this code, you will need to use DFU for future firmware updates because the micro controller will reset due to the watchdog, before completing the firmware update.

[bjswift]

Ok, I read up on DFU stuff like you mentioned and feeling pretty confident. Was able to figure things out mostly from the fine wiki page https://github.com/mossmann/hackrf/wiki/Updating-Firmware

I tried your suggested changes, and seemed to have to add "#include <libopencm3/lpc43xx/wwdt.h>" up at the top of that file to get it to compile. ... and the boards still fail to come up from a reset press every so often. It definitely happens less frequently than before the change, so at least there's that. [Must have been doing something incorrectly, because it seems to work rather well now.]

In addition to that symptom, I can also hackrf_spiflash them still. Maybe the watchdog functionality isn't working as you expect anymore with respect to the rest of the codebase at this point? The sizes of hackrf_usb.bin and hackrf_usb.dfu are exactly the same both before and after, but their md5sums are different, so there are some bits in there that are different before-and-after the change. [I was fooling myself here, too. hackrf_spiflash did seem to go without complaining, but I don't think it was actually doing anything. Additionally, hackrf_spiflash DOES do something, and you should definitely still do it after a DFU update, if you want the Firmware Version to be correct, as seen under hackrf_info]

[barto-]

Right, I forgot to mention you need to include the wwdt.h header. The value WWDT_TC = 0xffffff translates to ~5.6 seconds (enough time to upload the firmware using hackrf_spiflash). Anyway your devices should reset automatically if they get stuck for 5.6 seconds.
You can change the value of WWDT_TC from 1024 to 2^24 and the resulting time period in seconds will be WWDT_TC * 4 / 12000000

[bjswift]

Maybe I hadn't done something properly the first (few?) times, but now I don't see it ever fail to come up from reset. Thanks, it seems to work great! (crosses fingers)

Also, for future people trying to set up synchronized boards, related the earlier problem I figured out what I was doing wrong to build hackrf_clock. Needed to build from source in a different place...

sudo apt-get install libusb-1.0-0-dev #didn't have this already, eyeroll
cd hackrf/host/
mkdir build ; cd build
cmake ..
make
sudo make install
Then for some reason had to explicitly sudo cp hackrf_clock /usr/local/bin/. as well for hackrf_info and hackrf_spiflash (and hackrf_operacake, but I don't care about that)

Now I can successfully enable the CLKOUT line with hackrf_clock -d serialno -o1, and finally sync the sampling/proc clocks without an external reference!

Still, have the same old problem of needing to reset every time to get full sampling sync. But now with the watchdog mods in place, I can't command-line reset the board like I did this morning via hackrf_spiflash -d serialno -R like before.

Struggling to find a command-line solution for this, if it doesn't have a timely firmware solution. Maybe will just use a hardware reset line, since I'll have a few of those in my ultimate implementation anyhow! :|

Thanks @mossmann and @barto- for all your insights the last few days! :)

[PhillipDo]

Hello, I have the CPLD Sync method working (two HackRFs) using command line:
hackrf_transfer -r - -d <serial # of slave> -H 1 | hackrf_transfer -r - -d <serial # of master> -H 1

Then, I wrote a C program using LibHackRF. Is it possible to get CPLD sync with your own C program, instead of using command line? I tried the following to enable HW Sync on two HackRFs:

hackrf_set_hw_sync_mode(device_master, HW_SYNC_MODE_ON);
hackrf_set_hw_sync_mode(device_slave, HW_SYNC_MODE_ON);

I suspect that the master HackRF runs and gets the sync pulse first, while the slave HackRF is waiting for a sync pulse.

[bunphotk]

Hi,
I implemented both default_sync.xsvf and hackrf_usb on two hackrf boards. Both have 1pps attached on P28 pin 16. It does not work for me when I called the 'hackrf_transfer' from python code. Both hackrf do not start receive at the same time and randomly shift lead/lag to each other. The 1pps signals are verified that they are sync by using oscilloscope. I think python may have some delay when it call the 'hackrf_transfer' cml but should not as slow as 1 sec.

[jnunezortuno]

Is there any posibility to synchronous record data from two HackRF units using GNURadio blocks (even Osmocomm or SoapySDR sources) instead of using the well know pipe trick?

hackrf_transfer ... -r rec1.bin -d HackRF1_serial -H 1 | hackrf_transfer ... -r rec0.bin -d HackRF0_serial -H 1

[srcdude]

Hi, can this be extended to > 2 HackRfOnes?
you mention applying +5V from primary pin3 to primary and secondary pin8 to simulate a 3rd HackRF.... which makes me think this could somehow be extended to synchronize additional SDRs

[qnaved099]

dear bjswift,
does Hardware (CPLD-based) synchronisation of multiple HackRFs makes it to be a full duplex

[martinling]

This is a pull request that was closed over five years ago. Please open a new issue to ask questions about hardware synchronisation.

[swapnachallagundla]

Hi Marco Bartolucci,
I am working on synchronization of multiple HackRf radios with 1pps generated using GPSDO. I have gone through your research article
"M. Bartolucci, J. A. Del Peral-Rosado, R. Estatuet-Castillo, J. A. Garcia-Molina, M. Crisci and G. E. Corazza, "Synchronisation of low-cost open source SDRs for navigation applications," 2016 8th ESA Workshop on Satellite Navigation Technologies and European Workshop on GNSS Signals and Signal Processing (NAVITEC), Noordwijk, 2016, pp. 1-7."

I got confused in experimental setup.

Do I need to change the VHDL code of CPLD in HackRf to connect 1PPS from external GPSDO?
If we have to change the VHDL code, how to do it?

[martinling]

@swapnachallagundla you are commenting on the pull request in which Marco's work was added to HackRF. As you can see this pull request was merged, over five years ago. It is no longer necessary to modify the code, the feature is already included.

For instructions on how to set up hardware triggering, see this section of the HackRF documentation.

[swapnachallagundla]

Thanks a lot for clarifying it.
So, I can directly connect 1 pps to sync_in, right?

[martinling]

Yes, so long as your 1PPS signal is at 3.3V and you want to trigger the HackRF on the rising edge.

If you have any difficulties please open a new issue in this repository, rather than commenting further on this old PR.