GUI v5.0.3 fails to send/update/maintain hardware settings

[marles77]

Problem

I'm trying to record EEG and EOG simultaneously following instructions from this official tutorial. I use Cyton+Daisy but in this case I just switch to 8 channels recording. The signal is then sent through LSL and intercepted by LabRecorder, since I need to synchronize it with a stream of markers.
My configuration:
A1 - bottom SRB
A2 - bottom BIAS
EOG - N1P (bipolar: top pin + bottom pin)
EEG - N8P

Because I dont't need the rest of the channels and I don't want my EOG channel to refer either to SRB or Bias, I go to Hardware settings and first disable unused channels 2-7, and then for channel 1 I switch off both SRB2 and Bias. I hit "Send" and get a message that it was sent to my board. I go back to Time series and it seems to work fine. However, after ~2-3 minutes the disabled channels (usually 3 and 4) start to send some high frequency noise. After another ~60 seconds all channels (including EEG and EOG) produce only HF noise. Updating settings at this stage is impossible. I tested this with GUI v4.2 and everything worked just fine. GUI 5.0.3 works without problems when I don't touch Hardware settings (but I really need to...).

Expected

Hardware settings are sent to Cyton and maintain until the end of current session or until they are updated (as it is in v4.2).

Operating System and Version

Windows 10

GUI Version

5.0.3

Running standalone app

I'm running the downloaded app

Type of OpenBCI Board

Cyton/Cyton+Daisy

Are you using a WiFi Shield?

No

[retiutut]

Please provide a console log and screenshot. Descriptions are not enough information in this case.

[marles77]

OK, here are some screenshots from playback:

1. Begining of recording - channels 1 (VEOG) and 8 (EEG) activated, the rest are disabled
   

2. Both signals as expected - no leaking from EOG channel, channels 2-7 muted
   

3. Around 03:13 disabled channel 4 activates
   

4. Around 03:18 disabled channel 3 activates
   

5. Around 04:08 channel 1 starts to send ~75 Hz noise (possibly generated by channels 3 and 4 ??)
   

6. Around 04:15 channel 8 joins with the same frequency of noise, disabled channels 3 and 4 remain active
   

7. Around 04:38 disabled channel 5 activates; there is clear leaking between channels
   

Log files:
Console_2021-03-10_16-17-31.txt
Brainflow_2021-03-10_16-17-52.txt

[retiutut]

When you make a recording, electrodes should be connected to a user, and other channels should be turned off.

Is this the case?

If not, I would try again with this in mind.

[retiutut]

FYI This is not an issue with the GUI, so I will close this issue. I can see from the console log that the commands were sent out from the Dongle with success while the board is not streaming. There isn't always an acknowledgment from the board.

I can try to help do some basic troublshooting though on the hardware.

[marles77]

I'm not sure what you mean. As I mentioned earlier the recordings were done with 5 electrodes connected (in this case to my own head) and channels 2-7 were turned off. Somehow, however, the turned-off channels activated themselves with no actions from my side. I tested it several times. The outcome was always the same. So, it's not just an occasional problem. If the channels are disabled, why their signal appears in the GUI at all?

[retiutut]

There is enough info here for me to try and replicate. Looking into this now....

[retiutut]

I was not able to replicate this issue using Cyton + Mac and the latest development copy of the GUI. There was no leaking of signals and I set the hardware settings almost the same as yours.

What we can do is send commands for channels with un-sent commands only, instead of sending commands for all channels.

Console_2021-03-12_15-09-48.txt

[retiutut]

@marles77

1. It's important to use GUI v5 now since it formats CSV data using the new BrainFlow backend.
2. We can send commands for channels with un-sent commands only, instead of sending commands for all channels. I will add this to the feature list for 5.0.4.
3. Try saving your hardware settings, and then load and send this. This will help save you time clicking buttons in the future.
4. Try making sure the battery is fully charged. The Cyton can behave weirdly when the battery is low. This may fix your issue.
5. If all of the above fails, please contact official OpenBCI support and be ready to share when you purchased this hardware.

[marles77]

I understand. In my case the problem only started after 3 minutes of recording. I run GUI on Win 10.

[retiutut]

I understand. In my case the problem only started after 3 minutes of recording. I run GUI on Win 10.

I am trying again and leaving it running for 10-15 minutes with a charged (and larger) LiPo battery. In this case, the operating system is not a factor.

[retiutut]

ACTION ITEMS FROM THIS THREAD:

• Improve hardware settings save/load to show that channels are changed or not after loading
• Send only changed channel settings (should have a blue highlight)

[retiutut]

@marles77 Two most important steps you can do now are:

1. Try making sure the battery is fully charged. The Cyton can behave weirdly when the battery is low. This may fix your issue.
2. If all of the above fails, please contact official OpenBCI support and be ready to share when you purchased this hardware.

[marles77]

I'll give it a try with totally fresh batteries after the weekend. I appreciate all the improvements in GUI since v4. If we manage to fix this issue I will be more than happy. Loading the hardware settings form a file seems convenient indeed.

[marles77]

So, I spent a few hours to try to document my issue. Apologies in advance for such a long comment.
First of all, it wasn't caused by batteries as they were quite fine (1.47v each). Anyway in the following tests (on GUI 4.2 and GUI 5.0.3; the same electrodes were connected to my arms and head all the time) I used fresh ones, with no apparent difference. I reduced the number of channels so that the situation is clearer. I started with two channels:

1. A2: BIAS
2. N1P: bipolar EMG/ECG following this tutorial

GUI 4.2
Hardware settings:

Recording (4th minute); no problems with the signal; disabled channels are turned off:

Log file:
1_gui4_Console_2021-03-16_12-17-08.txt

GUI 5.0.3
Hardware settings:

Recording (4th minute); activated channels which are supposed to be turned off; EMG/ECG signal disappears; ~75Hz noise; notch filter failure; channels railed:

Log file:
1_gui5_Console_2021-03-16_12-25-20.txt

Another test with additional EEG channel (N8P)
GUI 4.2
Hardware settings:

Recording (4th minute); no problems with signals; disabled channels are turned off:

Log file:
2_gui4_eeg_Console_2021-03-16_12-41-49.txt

GUI 5.0.3
Hardware settings:

Recording:
1st minute - OK but channels railed:

2nd minute - some disabled channels are activated:

3rd minute - activated channels which are supposed to be turned off; EMG/ECG signal disappears; ~75Hz noise; notch filter failure:

Log file:
2_gui5_eeg_Console_2021-03-16_12-56-28.txt

Is there any explanation of such a different behavior of GUI 4 and GUI 5 ? Is this some sort of a problem with the connection between GUI and the Cyton board (firmware, drivers, whatever else)? I tried to run these tests multiple times. The outcome was always the same.

[retiutut]

Please Save and Load Hardware Settings in GUI v5. Then, Send commands. This will reduce the number of commands sent to Cyton by half. I have already pushed code to shorten this even more. This is the difference.

[2] onOff released
Sending config string to board: x2161000X
[3] onOff released
Sending config string to board: x3161000X
[4] onOff released
Sending config string to board: x4161000X
[5] onOff released
Sending config string to board: x5161000X
[6] onOff released
Sending config string to board: x6161000X
[7] onOff released
Sending config string to board: x7161000X
HardwareSettings Toggle: true
Sending config string to board: x1040000X
Sending config string to board: x2161000X
Sending config string to board: x3161000X
Sending config string to board: x4161000X
Sending config string to board: x5161000X
Sending config string to board: x6161000X
Sending config string to board: x7161000X
Sending config string to board: x8060110X

If you save/load/send, this would only be 8 commands in GUI v5.0.3. Please do this with Data Streaming Off.

[retiutut]

Closing this issue as I have pushed code to resolve what can be fixed, and was unable to replicate the further issues.

[retiutut]

@Andrey1994 This could be an issue with BrainFlow and Cyton commands, or maybe an issue with how the firmware interprets multiple commands.

[marles77]

I used Save and Load option on other occasions. That did not change the outcome.

[wjcroft]

The link I previously gave,

https://docs.openbci.com/docs/02Cyton/CytonProgram

[marles77]

@wjcroft I can see the link but I can't see anything like "reflashing". Do you mean I should install arduino firmware manually?

[retiutut]

@marles77 It looks like @Andrey1994 has reviewed this thread.

The last thing you can do is try BrainFlow Python Binding. To configure the board the way you want, look at the Console Logs from GUI v5 and use these same commands in your Python Code.

https://brainflow.readthedocs.io/en/stable/BuildBrainFlow.html#python
https://brainflow.readthedocs.io/en/stable/Examples.html#python

Sending config string to board: x1040000X
Sending config string to board: x2161000X
Sending config string to board: x3161000X
Sending config string to board: x4161000X
Sending config string to board: x5161000X
Sending config string to board: x6161000X
Sending config string to board: x7161000X
Sending config string to board: x8060110X

Look at the data there and see if this issue is still happening. The channels that are turned off should show true 0.

If it still happens, something weird is happening with your board.

@marles77 Please try this first. After sending these configuration commands. You can confirm the ADS1299 registers using the ? command with Cyton.

If the registers are correct, and the channels that are Off still show signals, this will confirm that the error is in firmware/hardware.

[marles77]

@retiutut OK, I'll try to do that tomorrow. However, what about the recordings performed under v4.2? They are consistent. I posted just one sample but I have made tens of such recordings. Doesn't that prove that the hardware is OK?

[wjcroft]

@marles77, yes, the previous link shows how to upload and refresh the firmware on the Cyton. Cyton firmware is stored in flash memory on the CPU chip. So the informal term for this is 'flashing' the firmware.

As far as sending Cyton commands over Brainflow, (the test Richard suggests) you can see an example of the Brainflow function call used, here:

https://openbci.com/forum/index.php?p=/discussion/2703/disable-srb-with-brainflow-on-cyton-resolved

[retiutut]

@marles77 Here is the full Python code you need to bypass the GUI and stream every single sample over LSL with no delays:

https://github.com/OpenBCI/OpenBCI_GUI/blob/development/Networking-Test-Kit/LSL/brainflow_lsl.py

CODE AS OF MARCH 16, 2021, 5PM CST:

#############################################################################       
##                           BrainFlow + LSL                               ##
##      Use BrainFlow to read data from board send it as an LSL stream     ##
#############################################################################

# Install dependencies with:
# pip install --upgrade numpy brainflow pylsl

# Here are example commands using Cyton and get_exg_channels()from BrainFlow. This has only been tested with Cyton + Dongle, for now.

# Mac:
# python3 Networking-Test-Kit/LSL/brainflow_lsl.py --board-id 2 --serial-port /dev/cu.usbserial-DM00D7TW --name test --data-type EXG --channel-names 1,2,3,4,5,6,7,8 --uid brainflow

# Windows:
# python3 Networking-Test-Kit/LSL/brainflow_lsl.py --board-id 2 --serial-port COM3 --name test --data-type EXG --channel-names 1,2,3,4,5,6,7,8 --uid brainflow

import argparse
import time
import numpy as np

from queue import Queue

import brainflow
from brainflow.board_shim import BoardShim, BrainFlowInputParams
from brainflow.data_filter import DataFilter, FilterTypes, AggOperations

from pylsl import StreamInfo, StreamOutlet, local_clock

def channel_select(board, board_id, data_type): 
    switcher = { 
        'EXG': board.get_exg_channels(board_id),
        # can add more
    } 
 
    return switcher.get(data_type, "error") 

def main():
    BoardShim.enable_dev_board_logger()

    parser = argparse.ArgumentParser()

    # brainflow params - use docs to check which parameters are required for specific board, e.g. for Cyton set serial port
    parser.add_argument('--timeout', type=int, help='timeout for device discovery or connection', required=False, default=0)
    parser.add_argument('--ip-address', type=str, help='ip address', required=False, default='')
    parser.add_argument('--board-id', type=int, help='board id, check docs to get a list of supported boards', required=True)
    parser.add_argument('--serial-port', type=str, help='serial port', required=False, default='')
    parser.add_argument('--streamer-params', type=str, help='streamer params', required=False, default='')

    # LSL params 
    parser.add_argument('--name', type=str, help='name', required=True)
    parser.add_argument('--data-type', type=str, help='data type', required=True)
    parser.add_argument('--channel-names', type=str, help='channel names', required=True)
    parser.add_argument('--uid', type=str, help='uid', required=True)

    args = parser.parse_args()

    # brainflow initialization
    params = BrainFlowInputParams()
    params.serial_port = args.serial_port
    params.ip_address = args.ip_address
    board = BoardShim(args.board_id, params)

    # LSL initialization  
    channel_names = args.channel_names.split(',')
    n_channels = len(channel_names)
    srate = board.get_sampling_rate(args.board_id)
    info = StreamInfo(args.name, args.data_type, n_channels, srate, 'double64', args.uid)
    outlet = StreamOutlet(info)
    fw_delay = 0

    # prepare session
    board.prepare_session()

    # send commands to the board for every channel. Cyton has 8 Channels. Here, we turn off every channel except for 1 and 8.
    board.config_board("x1040000X")
    board.config_board("x2161000X")
    board.config_board("x3161000X")
    board.config_board("x4161000X")
    board.config_board("x5161000X")
    board.config_board("x6161000X")
    board.config_board("x7161000X")
    board.config_board("x8060110X")

    # start stream
    board.start_stream(45000, args.streamer_params)
    time.sleep(1)
    start_time = local_clock()
    sent_samples = 0
    queue = Queue(maxsize = 5*srate)
    chans = channel_select(board, args.board_id, args.data_type)

    # read data with brainflow and send it via LSL
    print("Now sending data...")
    while True:
        data = board.get_board_data()[chans]
        for i in range(len(data[0])):
            queue.put(data[:,i].tolist())
        elapsed_time = local_clock() - start_time
        required_samples = int(srate * elapsed_time) - sent_samples
        if required_samples > 0 and queue.qsize() >= required_samples:    
            mychunk = []

            for i in range(required_samples):
                mychunk.append(queue.get())
            stamp = local_clock() - fw_delay 
            outlet.push_chunk(mychunk, stamp)
            sent_samples += required_samples
        #time.sleep(1)


if __name__ == "__main__":
    main()

[marles77]

@retiutut Brilliant! It's still warm... I'll let know tomorrow how it worked. I need to crash, it's neerly midnight here.

[marles77]

@retiutut @wjcroft @Andrey1994
Sorry for no update from yesterday. I have just completed the tests (and almost completely removed the hair from my forearm ...) and I think we're getting somewhere. It seems, however, that after all the problem lies in your software and not my hardware (which is also your hardware obviously).

1. First I tried your code from Brainflow, streamed over LSL, received by LabRecorder and saved as XDF for further viewing in EEGLab. This was the effect:
   
   test1_log.txt

2. Then I tried the same code but this time I commented out the board config commands (I assume this should leave default configuration). And the effect:
   
   test4_log.txt

3. Finally, I decided to do the same thing with openbci_lsl which, as I understand, you do not recommend. And it did the job perfectly (I didn't change the default localizations, so it doesn't matter; channel one is just bipolar EMG/ECG):
   

One thing that can be important here is the log from my first attempt, when I tried to send multiple commands to the board:
test5_openbcilsl_log.txt
But when I did it one at a time waiting for success message from Cyton, it went smoothly. Actually, Richard @retiutut already mentioned that commands should be sent in smaller packages. I think that brainflow should check the messages from Cyton and at least put them in log files (I just see a message that config string was sent to the board, but was it successful?), that would clarify things. Anyway, I don't think it is necessary to reflash my board.

[retiutut]

For Cyton, we only know that the Dongle aka Serial Port was able to send the message.

Also, the data from the example I shared does not have filters applied. You can modify the code to add BrainFlow filters (same used in GUI) to the data before sending out via LSL.

[retiutut]

It seems, however, that after all the problem lies in your software and not my hardware (which is also your hardware obviously).

Not entirely true. The values that were non-zero were indeed coming from the board.

[retiutut]

I don't understand how openbci_lsl.py can output filtered data.

[marles77]

For Cyton, we only know that the Dongle aka Serial Port was able to send the message.

Are you absolutely positive? Take a look at lines 1060-1190 of OpenBCI_32bit_Library.cpp where incomming channel settings are processed. I can see many different messages that are sent depending on success or failure. That is exactly what I get when I send commands to the board by openbci_lsl.

Also, the data from the example I shared does not have filters applied. You can modify the code to add BrainFlow filters (same used in GUI) to the data before sending out via LSL.

OK, I can try.

[marles77]

I don't understand how openbci_lsl.py can output filtered data.

What do you mean by filtered data? It's just normalized in EEGLab. >60Hz frequencies are cut off because sample rate is only 125Hz. No other filters were applied.

[Andrey1994]

Some commands force device to write smth to serial port some of them not.
For example you can use command '?' to get values for registers or command 'v'(I dont remember what exactly this one returns)

You can write smth like this if you want:

board.config_board("x1040000X")
time.sleep(1)
resp = board.config_board("?")
print(resp)
# check resp here how you want
....

Also, I dont think that its a good idea to stream filered data like this. Because you read very small chunks and filter data in such chunks. It makes much more sense to filter it once in the end where you need it

[marles77]

@Andrey1994 Thanks for your reply. From what I can see 'v' calls soft reset and it just returns board ID.

Also, I dont think that its a good idea to stream filered data like this. Because you read very small chunks and filter data in such >chunks. It makes much more sense to filter it once in the end where you need it

Sorry, but I don't understand what filtered data you refer to. This is what I get after streaming by openbci_lsl and recorded by LabRecorder. I don't think they use filters. I searched for any filtering methods and didn't find anything.

Anyway, I will try your code together with brainflow filters as suggested.

[retiutut]

What do you mean by filtered data? It's just normalized in EEGLab. >60Hz frequencies are cut off because sample rate is only 125Hz.

Cyton 8Ch Sampling rate is ~250Hz.

[marles77]

Yes, I'm aware of that. Openbci_lsl automatically detects Daisy and I didn't disable it manually, so originally 16 channels were available. I turned off all but one. However, sample rate was reduced by half. I think that at this stage this discussion is pointless. Thank you for your time. I'll be monitoring the development of the new GUI and the Brainflow library because I believe that they have great potential. At the moment, however, I don't think they are reliable enough to use them in my work. Maybe it has something to do with my configuration. I have to figure it out.

[wjcroft]

Sorry to beat a dead horse here. ;-) I wanted to add one more thing.

Previous comment by Richard:

@marles77 Please understand that this is not really an issue with the GUI, but with firmware and low level communication with YOUR board.

Your board could also have an issue at the hardware level.

I would contact OpenBCI support, though I have requested @Andrey1994, the creator of BrainFlow, review this also.

Is it possible that some ADS1299's, are overly sensitive to the timing and sequencing of commands sent via the Cyton firmware. Such that Richard's tests with GUI 5 work fine, but Marcin's GUI 5 tests, produce flaky 75Hz noise after some minutes of running. BECAUSE, the rapid default GUI 5 loading of his Cyton channel settings, (sent one after the other with little / no delays between channel set commands) somehow confused / jangled / fried the marginal ADS1299 that Marcin owns.

So although technically his ADS1299 failure with GUI 5, is really a hardware bug in his ADS1299. It only 'works' with GUI 4, because the old Hub is so slow that it naturally inserts some small delays between each channel set command. Whereas GUI 5 using Brainflow is so efficient and fast, that the commands come faster into Marcin's ADS1299. And unfortunately his ADS1299 hardware is marginal, and fails after some minutes when 'pushed' in this fashion.

So... I realize Richard already made improvements in the GUI 5 channel setting logic, to only send the required channel set commands, not all at once. But, this may not be enough to handle ADS1299 chips that have this type of marginal behavior. And it is unknown how many such chips are in the field.

So, could a workaround be, that any time GUI 5 sends multiple channel set commands, that an arbitrary timing delay be inserted between such commands? Say some fraction of a second if that is possible with Processing language. Like 500 ms or 200 ms. If only 1 second delay is possible, then go with that.

Regards, William

[marles77]

Finally, something constructive. I believe that a 0.2 sec delay would not interfere with UX. I think that @retiutut Richard has already proposed another good workaround (redundancy seems not a bad thing in this case):

ACTION ITEMS FROM THIS THREAD:

• Improve hardware settings save/load to show that channels are changed or not after loading
• Send only changed channel settings (should have a blue highlight)

Sending only changed channels would let the user match the amount of data being sent, to the particular board.
Also, it's really bad that there are no traces of feedback from the board in log files. From what I can see in the firmware it should be possible.

[marles77]

@retiutut @Andrey1994 As I already mentioned on the forum adding delay does the trick and fixes the problem. Now I start to appreciate the simplicity of Brainflow. But can I have one last question? I use Cyton + Daisy and even if I send the command 'c' (with 0.5 sec delay) and receive the confirmation that Daisy is removed, the sampling rate is still only 125Hz. I send this particular config_board() before LSL initialization (and after board.prepare_session()), so srate should already be updated. Using 0 as board-id in params doesn't work either (noise at all channels). Is there any other way I can set a sample rate of 250Hz for the remaining 8 (i.e. Cyton) channels?
Marcin

[marles77]

I'm not sure if anyone still follows this issue, but just in case someone bumps into a similar problem with GUI 5 and Cyton+Daisy, the fix is just sending 'c' command through the Hardware settings in Expert mode (tested on v5.0.4). In my case it was necessary even if I chose 8 channels option in the Control panel. I intuitively assumed that selecting 8 channels should automatically deactivate Daisy. I was wrong. I think it should be either implemented or explained somewhere.

[retiutut]

@marles77 I haven't tried using 8 channels with the Daisy attached in some time. The main reason is that I would forget to change how the earclip electrodes are attached. Using Daisy requires Y-splitter cable for one of them.

I'm going to reopen this issue and maybe we should send "c" as you suggested for this use case.

• but just in case someone bumps into a similar problem with GUI 5 and Cyton+Daisy, the fix is just sending 'c' command through the Hardware settings in Expert mode (tested on v5.0.4). In my case it was necessary even if I chose 8 channels option in the Control panel. I intuitively assumed that selecting 8 channels should automatically deactivate Daisy. I was wrong. I think it should be either implemented or explained somewhere.