Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to resume CMUX (or other modem modes) when waking up from deep sleep (IDFGH-14047) #692

Open
jlittlewood opened this issue Nov 11, 2024 · 9 comments
Assignees
Labels
Status: Opened Issue is new Type: Feature Request Feature Request for esp-protocols

Comments

@jlittlewood
Copy link

Is your feature request related to a problem?

I am using the esp_modem library and my device needs to support deep sleep. I need access to commands and data through UART, so I need to use CMUX mode. I am not aware of any way to set the DCE to CMUX mode on wakeup, when the modem is already in CMUX mode. When the DCE is initialized after wakeup it is in an undefined mode, which allows switching to any mode, but with the modem already set up and communicating in CMUX mode, simply running the set_mode to CMUX will not work, since it is dependent on various responses from the modem which may be communicating on different channels due to the CMUX setting being active on the modem.

Describe the solution you'd like.

Ideally, there would be a way to resume a mode upon initialization. The resume will assume the modem is already in the perspective mode and set up the DCE, netif, and UART terminals based on the mode that is being resumed.

Describe alternatives you've considered.

I have attempted to make my own CMUX resume function in the DCE that would set up the primary and secondary terminals and start the netif, but would not fail based on responses from the modem. This method has not been successful. I have not been able to get data to work, and commands usually don't work either.

Another option is to reset the modem each time, but this is undesirable due to the longer time to boot up the modem and connect. I need the modem to be ready as fast as possible from deep sleep wakeup.

Similarly, I have tried changing the modes prior to entering sleep. This has been problematic because I need the modem to be in PPP mode (or CMUX) while the MCU is asleep, so I can get the GPIO wake-ups for incoming data. This method also slows down the sleep process.

Additional context.

I am using an ESP32-S3 and the esp_modem library for UART modem connectivity. The CMUX works fine after the modem is rebooted. The main issue is how to get the DCE mode to match the modem mode that was set prior to the deep sleep.

@jlittlewood jlittlewood added the Type: Feature Request Feature Request for esp-protocols label Nov 11, 2024
@github-actions github-actions bot changed the title Add the ability to resume CMUX (or other modem modes) when waking up from deep sleep Add the ability to resume CMUX (or other modem modes) when waking up from deep sleep (IDFGH-14047) Nov 11, 2024
@espressif-bot espressif-bot added the Status: Opened Issue is new label Nov 11, 2024
@david-cermak
Copy link
Collaborator

Have you tried the new mode-detection feature? -- released yesterday as v1.2.0

Add support for guessing mode (52598e5f)

The idea is that you issue dce->set_mode(AUTODETECT) after wakeup and the modem lib sends a few probes to match the actual mode. In case of CMUX, it needs to go over (all) virtual terminals, so is uses manual CMUX modes.

@jlittlewood
Copy link
Author

That release does seem to have a lot of features I have been needing. Unfortunately it does not seem to be working for me.

In my code, on startup the modem boots up and I set the mode to CMUX mode. This works great and I am able to get data and commands working without any problem. Then when the MCU enters deep sleep the netif and DCE are deinited. When the MCU wakes from deep sleep I set the mode to AUTODETECT, but I get the response that it failed. No log lines about the primary or secondary terminal are printed. I added a print in the set_unsafe to see where it was failing. It is getting inside this if statement and returning false (inside the DCE_Mode::set_unsafe function in esp_modem_dce.cpp):

case modem_mode::AUTODETECT: {
        auto guessed = guess_unsafe(dte, true);
        if (guessed == modem_mode::UNDEF) { // AUTODETECT failing because of this response
            printf("Guessed mode is undefined\n"); // Added print to see where it was failing
            return false;
        }

Also even if the AUTODETECT did work, it looks like the only CMUX resumes are the manual CMUX modes. I dont see a possible transition from manual CMUX to the regular CMUX. Is that something that is possible? I still need to maintain support for commands and data, without regularly swapping between the two modes.

Here are the results from trying to resume using the different resume modes that are provided:

  • AUTODETECT: returned fail, data and commands not working.
  • RESUME_MANUAL_CMUX_DATA: returned successful from the set_mode, but data was not working, neither were commands.
  • RESUME_CMUX_MANUAL_MODE: returned fail and data and commands were not working.
  • RESUME_DATA_MODE: returned success but data and commands were not working.
  • RESUME_COMMAND_MODE: returned success, but data and commands were not working.

I also tried resuming modes, and being in CMUX manual prior to deep sleep, and it still gave me the same results as above, with no modem commands or data working.

Is there something else I am missing or anything special needed prior to entering deep sleep or after wakeup?

I am using the C APIs. A brief overview of my modem setup and teardown is shown below:

C API function to allow resume modes from C

extern "C" esp_err_t esp_modem_resume_mode(esp_modem_dce_t *dce_wrap, esp_modem_dce_resume_mode_t mode)
{
    if (dce_wrap == nullptr || dce_wrap->dce == nullptr) {
        return ESP_ERR_INVALID_ARG;
    }
    switch (mode) {
    case ESP_MODEM_RESUME_DATA_MODE:
        return dce_wrap->dce->set_mode(modem_mode::RESUME_DATA_MODE) ? ESP_OK : ESP_FAIL;
    case ESP_MODEM_RESUME_COMMAND_MODE:
        return dce_wrap->dce->set_mode(modem_mode::RESUME_COMMAND_MODE) ? ESP_OK : ESP_FAIL;
    case ESP_MODEM_RESUME_CMUX_MANUAL_MODE:
        return dce_wrap->dce->set_mode(modem_mode::RESUME_CMUX_MANUAL_MODE) ? ESP_OK : ESP_FAIL;
    case ESP_MODEM_RESUME_CMUX_MANUAL_DATA:
        return dce_wrap->dce->set_mode(modem_mode::RESUME_CMUX_MANUAL_DATA) ? ESP_OK : ESP_FAIL;
    case ESP_MODEM_AUTODETECT:
        return dce_wrap->dce->set_mode(modem_mode::AUTODETECT) ? ESP_OK : ESP_FAIL;
    return ESP_ERR_NOT_SUPPORTED;
    }   
}

Inside wakeup/startup

esp_netif_config_t netif_ppp_config = ESP_NETIF_DEFAULT_PPP();
esp_netif_t * net_if = esp_netif_new(&netif_ppp_config);
esp_modem_dce_config_t dce_config = ESP_MODEM_DCE_DEFAULT_CONFIG(apn);
esp_modem_dte_config_t dte_config = ESP_MODEM_DTE_DEFAULT_CONFIG();
esp_modem_dce_t* dce = esp_modem_new_dev(ESP_MODEM_DCE_CUSTOM, &dte_config, &dce_config, net_if);

esp_err_t ret;
if (deep_sleep_wakeup) { 
    // Wake from deep sleep, modem is not rebooted and should resume previous mode
    ret = esp_modem_resume_mode(dce, ESP_MODEM_AUTODETECT); 
    // I have tried all of the available resume modes, but none make the commands and data work again, 
    // despite 3 of the 5 modes returning ESP_OK (see list above for results for each mode).
} else { 
    // Regular startup. Modem was rebooted
   ret = esp_modem_set_mode(dce,ESP_MODEM_MODE_CMUX);
}
printf("Modem mode setup: %s\n",esp_err_to_name(ret));

Inside prepare for deep sleep

esp_modem_destroy(dce);
esp_netif_destroy(net_if);

As a note, I am using the Qualcomm QCX216 modem.

@david-cermak
Copy link
Collaborator

I haven't tested it with many devices, but I'm surprised it won't work even with basic command and data modes?

Could you please share more logs after applying these changes:

--- a/components/esp_modem/src/esp_modem_uart.cpp
+++ b/components/esp_modem/src/esp_modem_uart.cpp
@@ -175,7 +175,9 @@ int UartTerminal::read(uint8_t *data, size_t len)
     uart_get_buffered_data_len(uart.port, &length);
     length = std::min(len, length);
     if (length > 0) {
-        return uart_read_bytes(uart.port, data, length, portMAX_DELAY);
+        int read_len = uart_read_bytes(uart.port, data, length, portMAX_DELAY);
+        ESP_LOG_BUFFER_HEXDUMP("read", data, read_len, ESP_LOG_INFO);
+        return read_len;
     }
     return 0;
@@ -182,6 +182,7 @@ int UartTerminal::read(uint8_t *data, size_t len)
 
 int UartTerminal::write(uint8_t *data, size_t len)
 {
+    ESP_LOG_BUFFER_HEXDUMP("write", data, len, ESP_LOG_INFO);
     return uart_write_bytes_compat(uart.port, data, len);
 }

Let's start with these simple scenarios (no CMUX for now):

  1. Resume command mode

    • leave the device in the command mode
    • go to sleep
    • wake-up + init
    • set mode to ESP_MODEM_AUTODETECT
  2. Resume data mode

    • switch the device into the PPP mode
    • go to sleep
    • wake-up + init
    • set mode to ESP_MODEM_AUTODETECT

And if you can share the logs from these two procedures above, it would be really helpful!

@jlittlewood
Copy link
Author

Thank you so much for your help with this issue, it has been blocking me for quite some time now.

I performed the tests you requested. In the resume CMD mode, there was no issues and I didnt expect there to one any problems with that mode.
When testing the resume PPP mode, it failed to resume, but I noticed that the sync that was called on startup was successful. So I repeated the test and tried entering PPP (not the resume data, just the regular data) and that was successful. I have provided logs for those 3 tests.

For the sake of efficiency, I also provided the logs for the same test using the CMUX and AUTO/CMUX resume modes. This is not working and is the use case I need.

In my project we need to be able to use the PPP data and send commands to the modem. Switching out of CMUX mode before sleeping is not preferred since the modem is reset when exiting CMUX, and I would need to wait for startup and connect to PPP. This causes a 10-30 second delay in the deep sleep setup, which will not work.

The other option would be to restart the modem every time the MCU wakes from deep sleep, but that also causes the delay in the startup process and I need the device to be able to make a quick connection.

I have tried the USB DCE to use the 2 interfaces for data and commands, which works, but this is not an option due to the higher power usage of USB compared to UART.

From an analysis of the modem log, it appears that in the case of the resume CMUX/AUTO/PPP modes, the modem did not receive any data from the MCU after wakeup from deep sleep. The init and startup process is not different based on the modes, as I am manually entering and resuming modes for these tests.

test-Resume CMUX mode.txt
test-Resume data mode.txt
test-Normal data mode.txt
test-Resume command mode.txt

@david-cermak
Copy link
Collaborator

So you're also setting your device to a low power mode, correct?
And is seems like it wakes up in the default command mode (your modem_sync command succeeds) -- then this command

dce->set_mode(ESP_MODEM_AUTODETECT);

should succeed and resume the command mode (i.e. do nothing).

@jlittlewood
Copy link
Author

I am not intentionally setting the modem to low power mode or changing anything on the modem. I am sending no commands to the modem as part of the enter sleep. I am not sure if the modem is timing out or reverting to a different state when the MCU is asleep/not responding. I am working with Qualcomm to better understand that, but from the logs I have reviewed and my understanding, that is not happening in the CMUX case even though something like that does appear to be happening in the PPP mode, but perhaps not when CMUX is enabled.

I did the repeat the test by setting data mode to enter PPP
sleep the MCU
wakeup the MCU
resume mode AUTO detect

This test was successfully and resumed to CMD mode, as we would expect since in earlier test it entered CMD mode on wakeup. So in the case of PPP and CMD, the resume options are working.

The real issue I am facing is with the CMUX. CMUX resume is not working and neither is AUTO, when the mode was in CMUX prior to sleeping.

The closest method I have to a successful case with CMUX is as follow:

  • MCU startup/MCU wakeup
  • set mode to CMUX manual
  • set mode to CMUX data
  • set mode to CMUX command
  • set mode to CMUX exit
  • deep sleep modem

In this test I did not use any of the resume functionality, just the standard set modes. As part of startup I enter CMUX manual then transition to CMUX manual data. This is the same behavior whether the device is waking up from deep sleep or initial startup. Then to prepare for deep sleep the mode is set to CMUX manual command then to CMUX exit. If the CMUX is properly exited prior to sleep, I am able to use the modem and enter CMUX manual again after wakeup.

The reason I need to go to CMUX manual command prior to the CMUX exit is due to the fact that the modem reboots if the CMUX is exited from an active CMUX data state, but not from a CMUX command state. This speeds up the prepare for sleep process.

There does seem to be an issue when transitioning from CMUX data to CMUX command. It sends the +++ and immediately gets the NO CARRIER response, but instead of returning success, it tries a few more times, then reports that the command failed. It does successfully transition out of data mode, but it slows it down since it keeps retying, which will fail because the data is already suspended. Please see the attached log that shows this issue.
test-cmux_manual_data_to_cmux_manaual_cmd_fails.txt

The successful CMUX test case mentioned above could be a viable solution if we can figure out why the +++ thinks it failed, when it didn't, and if I can work with Qualcomm to ensure we can get a GPIO wakeup while the data mode is suspended.

@david-cermak
Copy link
Collaborator

@jlittlewood Could you please test the CMUX manual data -> cmd transition with this update?

--- a/components/esp_modem/src/esp_modem_dce.cpp
+++ b/components/esp_modem/src/esp_modem_dce.cpp
@@ -18,7 +18,6 @@ namespace transitions {
 
 static bool exit_data(DTE &dte, ModuleIf &device, Netif &netif)
 {
-    netif.stop();
     auto signal = std::make_shared<SignalGroup>();
     std::weak_ptr<SignalGroup> weak_signal = signal;
     dte.set_read_cb([&netif, weak_signal](uint8_t *data, size_t len) -> bool {
@@ -44,6 +43,7 @@ static bool exit_data(DTE &dte, ModuleIf &device, Netif &netif)
         }
         return false;
     });
+    netif.stop();
     netif.wait_until_ppp_exits();
     if (!signal->wait(1, 2000)) {

(moving the netif.stop() after that lambda)

@jlittlewood
Copy link
Author

@david-cermak I applied the suggested changes, but it did not make a difference. I looked closer in the logs from today and the logs I provided on my previous post and noticed that the NO CARRIER is not in the command_lib response.

V (101571) command_lib: set_command_mode
D (101571) command_lib: generic_command command +++

I (101572) write: 0x3fce06d4   f9 09 ef 07                                       |....|
I (101584) write: 0x3fce081c   2b 2b 2b                                          |+++|
I (101595) write: 0x3fce06d8   35 f9                                             |5.|
I (104531) read: 0x3fcd0370   f9 07 ef 1d 0d 0a 4e 4f  20 43 41 52 52 49 45 52  |......NO CARRIER| <- NO CARRIER response is recieved, but 
I (104532) read: 0x3fcd0380   0d 0a 22 f9                                       |..".|
D (104544) CMUX: Payload frame: dlci:01 type:ef payload:14 available:16
I (104555) read: 0x3fcd0370   f9 07 ef 1d 0d 0a 2b 53  52 56 5f 49 4e 44 3a 31  |......+SRV_IND:1|
I (104557) read: 0x3fcd0380   0d 0a 22 f9 f9 07 ef 1d  0d 0a 2b 53 52 56 5f 49  |..".......+SRV_I|
I (104568) read: 0x3fcd0390   4e 44 3a 31 0d 0a 22 f9                           |ND:1..".|
D (104580) CMUX: Payload frame: dlci:01 type:ef payload:14 available:36
D (104581) CMUX: Payload frame: dlci:01 type:ef payload:14 available:16
V (107596) command_lib: sync
V (107596) command_lib: generic_command_common
V (107597) command_lib: generic_command
D (107597) command_lib: generic_command command AT

I (107608) write: 0x3fce0634   f9 09 ef 07                                       |....|
I (107620) write: 0x3fce086c   41 54 0d                                          |AT.|
I (107621) write: 0x3fce0638   35 f9                                             |5.|
I (107636) read: 0x3fcd0370   f9 0b ef 0b 2b 2b 2b 41  54 5d f9                 |....+++AT].|
D (107637) CMUX: Payload frame: dlci:02 type:ef payload:5 available:7
I (107649) read: 0x3fcd0370   f9 0b ef 13 0d 0a 45 52  52 4f 52 0d 0a 4f f9     |......ERROR..O.|
D (107660) CMUX: Payload frame: dlci:02 type:ef payload:9 available:11
D (107662) command_lib: Response: +++AT
ERROR

Here it is obvious that the modem sent the NO CARRIER response, but I believe it was sent on the data terminal, rather than the command terminal, so the command_lib didn't see that text. This seems to be an issue when CMUX is used. Then since the data is already suspended, sending +++ again causes the error, which is communicated on the same terminal as the commands, so the command_lib correctly perceives it as a failure, but missed the previous success.

@david-cermak
Copy link
Collaborator

Yes, you're right, the answer is coming from the other terminal.
I'm sorry, haven't noticed this from your previous post:

Test:
	CMUX Manual exit before sleep
		set device to CMUX manual
		set device to CMUX manual data
		set device to CMUX manual command

Could you please try to swap the terminal before going to CMUX-manual-data ?
and log the transition from manual-data to manual-command? -- and the whole transition, the last post contained only the retries, i.e. after this wait for closing PPPD timeouted with 2 seconds?

if (!signal->wait(1, 2000)) {

so the procedure would be (saw that you're using modem console):

  1. set_mode CMUX1 <- enter manual cmux
  2. set_mode CMUX3 <- swap terminals
  3. set_mode CMUX4 <- manual data
  4. start logging
  5. set_mode CMUX5 <- manual command
  6. stop logging

PS: I'm sorry, the manual cmux modes are not documented well; switching the terminals is necessary since we switch the current terminal to data and need to reserve the former one to commands (this actually mimics the standard CMUX transition).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Opened Issue is new Type: Feature Request Feature Request for esp-protocols
Projects
None yet
Development

No branches or pull requests

3 participants