Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LwIP seems to not correctly handle packages larger than MTU #2751

Closed
pljakobs opened this issue Mar 31, 2024 · 29 comments
Closed

LwIP seems to not correctly handle packages larger than MTU #2751

pljakobs opened this issue Mar 31, 2024 · 29 comments

Comments

@pljakobs
Copy link
Contributor

Due to an ever growing json structure, my client sends an ip package larger than the MTU of 1500 bytes (1960 bytes to be precise). Due to path MTU being on by default, my client ip stack happily sends this package out to the wire, expecting to be told if that's not what is expected.
On Sming, when I check the body length, I get 1497 bytes, so the rest is basically just cut off.

Expected behaviour:
the lwip stack should quietly discard the received package and respond to the sender with an ICMP "fragmentation required" package, allowing the sender to configure the best package size for this receiver.

I have not yet dug into the lwip debug for this, but the observed behaviour clearly points toward lwip not helping with path discovery.

@pljakobs
Copy link
Contributor Author

pljakobs commented Mar 31, 2024

(testing lwip options)

compiling with

define IP_REASSEMBLY                   1
define IP_FRAG                         1
define IP_REASS_MAXAGE                 3

in $SMING_HOME/Components/lwip/lwipopts.h did not change the behaviour.

request tcp header:

Frame 155: 1954 bytes on wire (15632 bits), 1954 bytes captured (15632 bits) on interface wlp4s0, id 0
Ethernet II, Src: IntelCor_7d:32:00 (74:13:ea:7d:32:00), Dst: Espressi_23:62:06 (84:f3:eb:23:62:06)
Internet Protocol Version 4, Src: 192.168.29.45, Dst: 192.168.29.153
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 1940
    Identification: 0x1e6e (7790)
    010. .... = Flags: 0x2, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: TCP (6)
    Header Checksum: 0x58df [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.29.45
    Destination Address: 192.168.29.153
Transmission Control Protocol, Src Port: 51972, Dst Port: 80, Seq: 2171, Ack: 3421, Len: 1900
    Source Port: 51972
    Destination Port: 80
    [Stream index: 2]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 1900]
    Sequence Number: 2171    (relative sequence number)
    Sequence Number (raw): 3750011435
    [Next Sequence Number: 4071    (relative sequence number)]
    Acknowledgment Number: 3421    (relative ack number)
    Acknowledgment number (raw): 10074
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
    Window: 31624
    [Calculated window size: 31624]
    [Window size scaling factor: -2 (no window scaling used)]
    Checksum: 0xc39d [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (1900 bytes)
Hypertext Transfer Protocol
JavaScript Object Notation: application/json

response:

Frame 157: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface wlp4s0, id 0
Ethernet II, Src: Espressi_23:62:06 (84:f3:eb:23:62:06), Dst: IntelCor_7d:32:00 (74:13:ea:7d:32:00)
Internet Protocol Version 4, Src: 192.168.29.153, Dst: 192.168.29.45
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 40
    Identification: 0x001e (30)
    000. .... = Flags: 0x0
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 255
    Protocol: TCP (6)
    Header Checksum: 0xff9a [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.29.153
    Destination Address: 192.168.29.45
Transmission Control Protocol, Src Port: 80, Dst Port: 51972, Seq: 3421, Ack: 3561, Len: 0
    Source Port: 80
    Destination Port: 51972
    [Stream index: 2]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 0]
    Sequence Number: 3421    (relative sequence number)
    Sequence Number (raw): 10074
    [Next Sequence Number: 3421    (relative sequence number)]
    Acknowledgment Number: 3561    (relative ack number)
    Acknowledgment number (raw): 3750012825
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x010 (ACK)
    Window: 5560
    [Calculated window size: 5560]
    [Window size scaling factor: -2 (no window scaling used)]
    Checksum: 0x6438 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]

so the firmware correctly acks for 1390 bytes of payload, which is just right for the default TCP_MSS at 1390 bytes.

But there still is no ICMP fragmentation required package being sent, which I would expect would be used to help determine the allowable MTU for this connection.

Is there another lwip setting for that somewhere? The lwip documentation indicates it can do it.

@pljakobs
Copy link
Contributor Author

pljakobs commented Mar 31, 2024

further checking with ENABLE_LWIPDEBUG=! I see

CONNECTION DROPPED
	(free heap: 7560)


15829943 tcp_abandon: sending RST

15830431 tcp_rst: seqno 6515 ackno -1212338775.

15830887 

CONNECTION DROPPED
	(free heap: 7848)


15831068 tcp_abandon: sending RST

15831469 tcp_rst: seqno 6516 ackno 1612911029.

but the sequence numbers don't match with what I see in wireshark (which might be because lwip debug outputs them as signed integers :o)

also, I can't see TCP RST packages on the packet trace, when filtered for the sming node's address.

The front end, however, seems to respond to the sming app only acknowledging part of the data by re-sending the missing part - which is never ack'd by sming.

@mikee47
Copy link
Contributor

mikee47 commented Mar 31, 2024

Maybe unrelated to your issue, but there's clearly memory pressure. Assuming this is your esp_rgbww_firmware, take a look in your .rodata segment: objdump -s -j .rodata out/Esp8266/debug/build/app_0.out. I'd guess there's 3K+ of RAM you can reclaim by moving strings into flash.

@pljakobs
Copy link
Contributor Author

Good point, also, the original design keeps the whole configuration in a large struct which alone is at least around 1.5kB and is not really used that much, so I should probably refractor this as well. 4.5kB more RAM is 25% more than now, so that should certainly be helpful.

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

I have moved almost all string literals into F() and now have mostly >20kB of heap free, so that was well worth while the effort. (as a side note: the F macro doesn't work for debug_ - why would that be?)
lwip still doesn't deal well with my oversized packets, though.
I am actually confused about this, in my network analysis days, senders would stick to their local MTU but it seems that modern stacks try path MTU whenever possible and negotiate the largest possible frame size.

@mikee47
Copy link
Contributor

mikee47 commented Apr 1, 2024

as a side note: the F macro doesn't work for debug_ - why would that be?

debug_ macros require a string literal which gets stored in flash memory. See also https://sming.readthedocs.io/en/latest/framework/core/pgmspace.html

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

I'm not 100% sure that my lwipopts.h is used - where should that be placed to enable specific options? in the build directory? in the ./include directory?

@mikee47
Copy link
Contributor

mikee47 commented Apr 1, 2024

lwipopts customisation isn't supported by Sming. What specific changes do you need?

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

from going through the lwipopts.h, I would think that

define IP_REASSEMBLY                   1
define IP_FRAG                         1
define IP_REASS_MAXAGE                 3

are what should allow fragmentation / reassembly (and with that, hopefully, path MTU)
IP_REASSEMBLY (receive) and IP_FRAG (send) are 0 by default

@mikee47
Copy link
Contributor

mikee47 commented Apr 1, 2024

You can add these to your project's component.mk file as global compiler flags, for example:

GLOBAL_CFLAGS += \
  -DIP_REASSEMBLY=1 \
  -DIP_FRAG=1 \
  -DIP_REASS_MAXAGE=3

This will require a full make clean components clean as it probably affects some header files too.

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

I'll give it a try

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

this is an unexpected result

Building /opt/Sming/Sming/out/Esp8266/debug/lib/clib-Storage-1325f1cb593ea10b1ba97cb1de3d8181.a
xtensa-lx106-elf-g++: error: 0: No such file or directory
xtensa-lx106-elf-g++: error: 0: No such file or directory
make[2]: *** [/opt/Sming/Sming/component-wrapper.mk:171: src/Debug.cpp.d] Fehler 1
make[2]: *** Es wird auf noch nicht beendete Prozesse gewartet …
make[2]: *** [/opt/Sming/Sming/component-wrapper.mk:171: src/Device.cpp.d] Fehler 1
xtensa-lx106-elf-g++: error: 0: No such file or directory
make[2]: *** [/opt/Sming/Sming/component-wrapper.mk:171: src/Iterator.cpp.d] Fehler 1
xtensa-lx106-elf-g++: error: 0: No such file or directory
xtensa-lx106-elf-g++: error: 0: No such file or directory
make[2]: *** [/opt/Sming/Sming/component-wrapper.mk:171: src/Partition.cpp.d] Fehler 1
make[2]: *** [/opt/Sming/Sming/component-wrapper.mk:171: src/PartitionStream.cpp.d] Fehler 1
make[1]: *** [/opt/Sming/Sming/project.mk:468: Storage-build] Fehler 2

with

GLOBAL_CFLAGS += \
  -DIP_REASSEMBLY=1 \
  -DIP_FRAG=1 \
  -DIP_REASS_MAXAGE=3 \
  -DLWIP_NETIF_TX_SINGLE_PBUF 0

in component.mk

with just

GLOBAL_CFLAGS += \
  -DIP_REASSEMBLY=1 \
  -DIP_REASS_MAXAGE=3

it compiles. That should be enough....

@mikee47
Copy link
Contributor

mikee47 commented Apr 1, 2024

-DLWIP_NETIF_TX_SINGLE_PBUF 0 missing =

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

oh dang! thanks

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 1, 2024

hmm.. no, that doesn't fix it.
It's really odd,
Client sends oversized packet, Sming ack's as much as it can, given the TCP_MSS config, client sends a new frame with the non-ack'd data but that's never ack'd by Sming.
Well.... maybe it's a good idea to not send almost 2k of json anyway.

@mikee47
Copy link
Contributor

mikee47 commented Apr 1, 2024

Due to an ever growing json structure, my client sends an ip package larger than the MTU of 1500 bytes (1960 bytes to be precise). Due to path MTU being on by default, my client ip stack happily sends this package out to the wire, expecting to be told if that's not what is expected.
On Sming, when I check the body length, I get 1497 bytes, so the rest is basically just cut off.

Note that serialising via JsonObjectStream is not memory-efficient. As you'll see from the source code, it's basically just a MemoryDataStream object. The first 'read' call triggers a full serialisation of the content, which will more than double the actual RAM in use.

You'll need a different approach to send large JSON serialisations. The following sample application does this using a custom stream and serialisation in parts. Source is in the DeviceListStream class. https://sming.readthedocs.io/en/latest/_inc/Sming/Libraries/HueEmulator/samples/Basic_Alexa/index.html

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 2, 2024

trying to figure out the odd ip/tcp behaviour - which lwip is built by default? https://sming.readthedocs.io/en/latest/_inc/Sming/Components/lwip/index.html#envvar-ENABLE_CUSTOM_LWIP docments ENABLE_CUSTOM_LWIP defaulting to 2 for lwip2?
however setting

ENABLE_CUSTOM_LWIP=2

in my component.mk yields strange build issues

Building /opt/Sming/Sming/out/Esp8266/debug/lib/liblwip2.a
In file included from include/lwip/debug.h:40,
                 from ../../glue-lwip/lwipopts.h:2987,
                 from include/lwip/opt.h:51,
                 from core/inet_chksum.c:48:
core/inet_chksum.c: In function 'inet_cksum_pseudo_base':
include/lwip/arch.h:158:15: error: expected ')' before '__INT32'
  158 | #define X32_F PRIx32
      |               ^~~~~~
include/lwip/arch.h:79:42: note: in definition of macro 'LWIP_PLATFORM_DIAG'
   79 | #define LWIP_PLATFORM_DIAG(x) do {printf x;} while(0)

(and I really hope it's not a forgotten = or something as stupid again)
seems that this might be a result of not including esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/inttypes.h?

[pjakobs@devbox Sming]$ grep PRIx32 * -R
Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/include/lwip/arch.h:#define X32_F PRIx32
esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/inttypes.h:#define PRIx32		__PRI32(x)
[pjakobs@devbox Sming]$ grep __PRI32\(x\) * -R
esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/inttypes.h:#define __PRI32(x) __INT32 __STRINGIFY(x)
esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/inttypes.h:#define PRIx32		__PRI32(x)

@mikee47
Copy link
Contributor

mikee47 commented Apr 2, 2024

trying to figure out the odd ip/tcp behaviour - which lwip is built by default? https://sming.readthedocs.io/en/latest/_inc/Sming/Components/lwip/index.html#envvar-ENABLE_CUSTOM_LWIP docments ENABLE_CUSTOM_LWIP defaulting to 2 for lwip2?

That page is for Host, see https://sming.readthedocs.io/en/latest/_inc/Sming/Arch/Esp8266/Components/sming-arch/index.html

however setting ENABLE_CUSTOM_LWIP=2 in my component.mk yields strange build issues

Yep, me too - requires ENABLE_LWIP_DEBUG=1 but that's a bug.

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 2, 2024

that alone doesn't do it for me, but I had a similar issue with enabling TCP_DEBUG which also required IP_DEBUG which is where the #defines were.

currently, I compile with

GLOBAL_CFLAGS += \
  -DIP_REASSEMBLY=1 \
  -DIP_FRAG=1 \
  -DIP_REASS_MAXAGE=3 \
  -DLWIP_NETIF_TX_SINGLE_PBUF=0 \
  -DIP_REASS_DEBUG=1 \
  -DIP_DEBUG=1 \
  -DTCP_DEBUG=1 \
  -DTCP_INPUT_DEBUG=1

and

ENABLE_CUSTOM_LWIP = 2
ENABLE_LWIP_DEBUG = 1

and still get those build errors.

btw: I am always amazed how complex the whole environment is. It never stops boggling the mind how much code and functionality fits on those tiny controllers nowadays (well, already did fit on it a decade ago, for the ESP8266).

@mikee47
Copy link
Contributor

mikee47 commented Apr 2, 2024

Fix for debug build errors, Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-lwip/arch/cc.h:

diff --git a/glue-lwip/arch/cc.h b/glue-lwip/arch/cc.h
index b6d402a..e4b618d 100644
--- a/glue-lwip/arch/cc.h
+++ b/glue-lwip/arch/cc.h
@@ -36,6 +36,16 @@ author: d. gauchard
 
 typedef signed short        sint16_t;
 
+#define X8_F  "02x"
+
+#define S16_F "d"
+#define U16_F "d"
+#define X16_F "x"
+
+#define S32_F "d"
+#define U32_F "d"
+#define X32_F "x"
+
 #ifdef LWIP_BUILD
 
 // define LWIP_BUILD only when building LWIP

PR #2754

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 3, 2024

that alone doesn't do it for me, but I had a similar issue with enabling TCP_DEBUG which also required IP_DEBUG which is where the #defines were.

currently, I compile with

GLOBAL_CFLAGS += \
  -DIP_REASSEMBLY=1 \
  -DIP_FRAG=1 \
  -DIP_REASS_MAXAGE=3 \
  -DLWIP_NETIF_TX_SINGLE_PBUF=0 \
  -DIP_REASS_DEBUG=1 \
  -DIP_DEBUG=1 \
  -DTCP_DEBUG=1 \
  -DTCP_INPUT_DEBUG=1

and

ENABLE_CUSTOM_LWIP = 2
ENABLE_LWIP_DEBUG = 1

and still get those build errors.

btw: I am always amazed how complex the whole environment is. It never stops boggling the mind how much code and functionality fits on those tiny controllers nowadays (well, already did fit on it a decade ago, for the ESP8266).

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 3, 2024

puh, not sure lwip2, feels like lwip will lead into the next rabbit hole:

[OS] lwESP: pbuf_alloc BAD CASE
[OS] lwESP: pbuf_alloc BAD CASE
[OS] lwESP: pbuf_alloc BAD CASE


***** Fatal exception 9 (UNALIGNED)
pc=0x40228c63 sp=0x3ffffec0 excvaddr=0x005e000d
ps=0x00000030 sar=0x00000010 vpri=0x402586e5
r00: 0x402181c9=1075937737 r01: 0x3ffffec0=1073741504 r02: 0x005e0001=   6160385 
r03: 0x00000000=         0 r04: 0x00000001=         1 r05: 0xfffffffd=-         3 
r06: 0x3ffeaac0=1073654464 r07: 0x00000028=        40 r08: 0x3fff4be0=1073695712 
r09: 0x00000470=      1136 r10: 0x00000045=        69 r11: 0x00000000=         0 
r12: 0x3fff4bf0=1073695728 r13: 0x3fff1928=1073682728 r14: 0x3fff4c22=1073695778 
r15: 0x3fff4bf0=1073695728 

stack decode:

0x402181c9: system_pp_recycle_rx_pkt at ??:?
0x401068fc: pbuf_free at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-esp/lwip-esp.c:885
0x4026e1b5: UdpConnection::staticOnReceive(void*, udp_pcb*, pbuf*, ip_addr*, unsigned short) at /opt/Sming/Sming/Components/Network/src/Network/UdpConnection.cpp:118
0x40234059: udp_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/udp.c:393
0x401013ae: malloc at ??:?
0x40236e43: ip4_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/ipv4/ip4.c:679
0x40228cb0: ppRecycleRxPkt at /home/xcg/workspace/debug/esp8266_nonos_sdk_core_20180510/app/pp/pp.c:768
0x40236ae9: ethernet_input_LWIP2 at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/netif/ethernet.c:182
0x40106adc: esp2glue_ethernet_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-lwip/lwip-git.c:377
0x4010697e: ethernet_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-esp/lwip-esp.c:424
0x401069ad: ethernet_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-esp/lwip-esp.c:432 (discriminator 3)
0x402288df: ppTask at /home/xcg/workspace/debug/esp8266_nonos_sdk_core_20180510/app/pp/pp.c:430
0x4021c8f3: ets_snprintf at ??:?
0x40000f49: ?? ??:0
0x40000f49: ?? ??:0

I found that pbuf_free is #defined as pbuf_free_LWIP2, but could not find that function anywhere in Sming/Sming.

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 3, 2024

same thing again (albeit without the OS errors )

0x402181c9: system_pp_recycle_rx_pkt at ??:?
0x401068fc: pbuf_free at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-esp/lwip-esp.c:886
0x4026f21d: TcpConnection::internalOnReceive(pbuf*, signed char) at /opt/Sming/Sming/Components/Network/src/Network/TcpConnection.cpp:489
0x4021090b: ieee80211_recv_action at ??:?
0x40235dfa: memp_free at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/memp.c:496
0x402349d1: tcp_seg_free at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/tcp.c:1396
0x402389fc: tcp_receive at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/tcp_in.c:1170
0x40235e24: mem_free at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/mem.c:152
0x4026f284: operator() at /opt/Sming/Sming/Components/Network/src/Network/TcpConnection.cpp:329
 (inlined by) _FUN at /opt/Sming/Sming/Components/Network/src/Network/TcpConnection.cpp:331
0x4023988c: tcp_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/tcp_in.c:443 (discriminator 1)
0x4022d540: set_pwm_debug_en at ??:?
0x401013ae: malloc at ??:?
0x40236e4d: ip4_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/core/ipv4/ip4.c:685
0x40228cb0: ppRecycleRxPkt at /home/xcg/workspace/debug/esp8266_nonos_sdk_core_20180510/app/pp/pp.c:768
0x40236ae9: ethernet_input_LWIP2 at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/lwip2-src/src/netif/ethernet.c:182
0x40106adc: esp2glue_ethernet_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-lwip/lwip-git.c:377
0x4010697e: ethernet_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-esp/lwip-esp.c:424
0x401069ad: ethernet_input at /opt/Sming/Sming/Arch/Esp8266/Components/lwip2/lwip2/glue-esp/lwip-esp.c:432 (discriminator 3)
0x4021c8f3: ets_snprintf at ??:?
0x40000f49: ?? ??:0
0x40000f49: ?? ??:0

This is 100% reproducible - but I can't see at a glance if it's a double free or really a free on an unaligned address (I can't imagine how that would happen as pbuf_free takes a pbuf pointer that has been used by other functions before)

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 3, 2024

(I'll open a new issue for that if you want)

@mikee47
Copy link
Contributor

mikee47 commented Apr 3, 2024

I get the same errors with lwip2, running in debugger:

rm -rf out $SMING_HOME/out
make clean components-clean
make -j ENABLE_GDB=1 COM_SPEED_SERIAL=115200
make flashpart PART=rom0 gdb

May I suggest that you refactor code so as much of it as possible runs under the Host emulator. Then run it through valgrind - it can pick up a lot of issues.

@mikee47
Copy link
Contributor

mikee47 commented Apr 4, 2024

Regarding the original subject of this post, if you can reproduce the issue with one of the sample applications that would be helpful.

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 6, 2024

sigh, I've replaced

request.getBody()

with

request.getBodyStream()

and that fixed the oversized json package issue.

@mikee47
Copy link
Contributor

mikee47 commented Apr 6, 2024

The LWIP2 stack is pretty dated for esp8266, fork is from 2017. Upstream at https://github.com/d-a-v/esp82xx-nonos-linklayer is more actively developed but porting it not a job not for the feint-hearted! It's always hacky due to the closed-source nature of Espressif's low-level stuff.

The esp-open-lwip implementation has always worked for me so more of a 'would like' than a necessity.

@pljakobs
Copy link
Contributor Author

pljakobs commented Apr 8, 2024

I was a bit worried that it maybe had fallen behind modern standard use cases, but as said, using the getBodyStream() method seems to have done the trick (I think I have mentioned before that I have inherited this code which in itself is probably 10 years old by now)
I'm happy with whatever works, but over time, would you agree that using what the arduino core uses would be the least maintainance effort?
I am not particularly happy with sending that large json anyway as, sooner or later, I will hit a brick wall when serializing / deserializing it. But changing the api would break compatibility with the home automation systems that use it.
Also, internally, the firmware reads the whole json structure into a global array (and there are two sets of serializing / deserializing code, one pair for the api, the other to store it in flash), I'm thinking I should probably create a configure object that encapsulates all that.

@pljakobs pljakobs closed this as completed Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants