f-stack client fails to establish TCP connections

[MincYu]

Hi,

To understand the performance benefits of f-stack, I started with a simple application where a client interacts with a server via TCP. Both client and server run on aws ec2 instances, each having two NICs where one for kernel driver and another for DPDK driver.

I followed the helloworld example to write a f-stack server, and it works as expected. However, the f-stack client cannot connect to any server (neither my f-stack server nor others). In particular, ff_connect failed and the socket gets blocked in SYN_SENT state.

To figure out the reason, I tested normal client and server (based on kernel stack) as well. When running normal client/server with f-stack server/client, I can track the tcp packages on normal ones to help understand the behavior of f-stack client (while currently I can only see the byte-level traffic of the DPDK NIC on ec2 instance using f-stack tools without high-level semantics, that's why I need normal client/server). Anyway, it turns out that the server can receive the SYN package from f-stack client and send out SYN_ACK, while the latter fails to receive SYN_ACK and always re-sends SYN to the server.

I feel that the problem can either come from routing issues or the f-stack client somehow dropping the package. To exclude the routing issue, I inspected the client-side traffic of both NICs, and found the DPDK NIC DID receive packages during the tcp handshake (based on rx packets and rx bytes from traffic command) and there is no package coming into the kernel NIC (so the routing might be correct). I thus doubt the second reason might be more possible.

Still, I'm not sure if my understanding is correct and how can I fix my problem. Here are a couple of questions. If I can inspect more package-level information (like tcpdump) with f-stack tools, instead of only the numbers of bytes and packages received and sent? If the f-stack client does drop the package, why it might happen and how can I solve this?

Any help will be greatly appreciated. Let me know in case more details are required.

[tinboxw]

我基于main_epoll.c例子，编写了一个基于epoll的异步tcp客户端，是可以连接成功-发送接收数据-正常关闭。

[jiangxiaosheng]

我基于main_epoll.c例子，编写了一个基于epoll的异步tcp客户端，是可以连接成功-发送接收数据-正常关闭。

any chance you could share your code and the way you establish the connection?

[BaiXious]

我基于main_epoll.c例子，编写了一个基于epoll的异步tcp客户端，是可以连接成功-发送接收数据-正常关闭。

any chance you could share your code and the way you establish the connection?Please

[ChiaoTeo]

可以分享一下代码吗

我基于main_epoll.c例子，编写了一个基于epoll的异步tcp客户端，是可以连接成功-发送接收数据-正常关闭。

[zhubingbing]

@jiangxiaosheng 能分享下例子吗

[Chithesus]

Hi,

To understand the performance benefits of f-stack, I started with a simple application where a client interacts with a server via TCP. Both client and server run on aws ec2 instances, each having two NICs where one for kernel driver and another for DPDK driver.

I followed the helloworld example to write a f-stack server, and it works as expected. However, the f-stack client cannot connect to any server (neither my f-stack server nor others). In particular, ff_connect failed and the socket gets blocked in SYN_SENT state.

To figure out the reason, I tested normal client and server (based on kernel stack) as well. When running normal client/server with f-stack server/client, I can track the tcp packages on normal ones to help understand the behavior of f-stack client (while currently I can only see the byte-level traffic of the DPDK NIC on ec2 instance using f-stack tools without high-level semantics, that's why I need normal client/server). Anyway, it turns out that the server can receive the SYN package from f-stack client and send out SYN_ACK, while the latter fails to receive SYN_ACK and always re-sends SYN to the server.

I feel that the problem can either come from routing issues or the f-stack client somehow dropping the package. To exclude the routing issue, I inspected the client-side traffic of both NICs, and found the DPDK NIC DID receive packages during the tcp handshake (based on rx packets and rx bytes from traffic command) and there is no package coming into the kernel NIC (so the routing might be correct). I thus doubt the second reason might be more possible.

Still, I'm not sure if my understanding is correct and how can I fix my problem. Here are a couple of questions. If I can inspect more package-level information (like tcpdump) with f-stack tools, instead of only the numbers of bytes and packages received and sent? If the f-stack client does drop the package, why it might happen and how can I solve this?

Any help will be greatly appreciated. Let me know in case more details are required.

I capture all the package from client and server, client send SYN to server and server reply a message, but client send SYN again with tcpdump; but bind certain port can establish tcp connection;

[vewe-richard]

@Chithesus Can you show me the code? I tried to bind certain port but still didn't working.

` memset(&client_addr, 0, sizeof(client_addr));
client_addr.sin_family = AF_INET;
client_addr.sin_port = htons(CLIENT_PORT);
client_addr.sin_addr.s_addr = htonl(INADDR_ANY);

if (ff_bind(sockfd, (struct linux_sockaddr *)&client_addr, sizeof(client_addr)) < 0) {
    printf("ff_bind failed\n");
    ff_close(sockfd);
    return -1;
}

`

[Chithesus]

Thanks for your mail.邮件已收，谢谢

[vewe-richard]

@tinboxw Can you share the code?

[vewe-richard]

I fixed it, we should disable kni in config.ini

Below is the code implement a client

`#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <fcntl.h>

#include "ff_config.h"
#include "ff_api.h"
#include "ff_epoll.h"

#define SERVER_IP "101.32.223.170"

#define SERVER_PORT 12345
#define BUFFER_SIZE 256
#define MAX_EVENTS 10

static struct epoll_event ev, events[MAX_EVENTS];
int epfd;
int sockfd;
//#define STANDARD
int loop(void *arg)
{
int nfds;
char send_buf[] = "Hello, Server!";
char recv_buf[BUFFER_SIZE];
#ifdef STANDARD
nfds = epoll_wait(epfd, events, MAX_EVENTS, -1);
#else
nfds = ff_epoll_wait(epfd, events, MAX_EVENTS, -1);
#endif
if (nfds < 0) {
printf("ff_epoll_wait failed\n");
} else if(nfds != 0){
printf("got events, nfds %d\n", nfds);
}

    for (int i = 0; i < nfds; ++i) {
        if (events[i].events & EPOLLOUT) {
            printf("before write\n");

#ifdef STANDARD
if (write(sockfd, send_buf, strlen(send_buf)) < 0) {
#else
if (ff_write(sockfd, send_buf, strlen(send_buf)) < 0) {
#endif
printf("ff_write failed: %s\n", strerror(errno));
} else {
ev.events = EPOLLIN;
printf("write a buffer\n");
#ifdef STANDARD
epoll_ctl(epfd, EPOLL_CTL_MOD, sockfd, &ev);
#else
ff_epoll_ctl(epfd, EPOLL_CTL_MOD, sockfd, &ev);
#endif
}
} else if (events[i].events & EPOLLIN) {
#ifdef STANDARD
int len = read(sockfd, recv_buf, BUFFER_SIZE - 1);
#else
int len = ff_read(sockfd, recv_buf, BUFFER_SIZE - 1);
#endif
if (len > 0) {
recv_buf[len] = '\0'; // 添加字符串结束符
printf("Received from server: %s\n", recv_buf);
} else {
if (len < 0) {
printf("ff_read failed: %s\n", strerror(errno));
} else {
printf("Connection closed by server\n");
}
#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
break;
}
} else if (events[i].events & (EPOLLHUP | EPOLLERR)) {
printf("epoll error\n");
#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
break;
}
}
}

int main(int argc, char *argv[]) {
int nfds;
struct sockaddr_in server_addr;
int on = 1;

#ifdef STANDARD
#else
ff_init(argc, argv);
#endif

#ifdef STANDARD
sockfd = socket(AF_INET, SOCK_STREAM, 0);
#else
sockfd = ff_socket(AF_INET, SOCK_STREAM, 0);
#endif
if (sockfd < 0) {
printf("ff_socket failed\n");
return -1;
}

#ifdef STANDARD
ioctl(sockfd, FIONBIO, &on);
#else
ff_ioctl(sockfd, FIONBIO, &on);
#endif

memset(&server_addr, 0, sizeof(server_addr));
server_addr.sin_family = AF_INET;
server_addr.sin_port = htons(SERVER_PORT);
server_addr.sin_addr.s_addr = inet_addr(SERVER_IP);
printf("just before ff_connect\n");

#ifndef STANDARD
//sleep(10);
#endif
#ifdef STANDARD
if (connect(sockfd, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
#else
if (ff_connect(sockfd, (struct linux_sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
#endif
if (errno != EINPROGRESS) {
printf("ff_connect failed: %s\n", strerror(errno));
#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
return -1;
} else {
printf("EINPROGRESS\n");
}
}

printf("end ff_connect\n");

#ifdef STANDARD
epfd = epoll_create(1);
#else
epfd = ff_epoll_create(1);
#endif
if (epfd < 0) {
printf("ff_epoll_create failed\n");

#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
return -1;
}

ev.events = EPOLLOUT | EPOLLIN;
ev.data.fd = sockfd;

#ifdef STANDARD
if (epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &ev) < 0) {
#else
if (ff_epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &ev) < 0) {
#endif
printf("ff_epoll_ctl failed\n");
#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
return -1;
}

#ifdef STANDARD
while (1) {
loop(NULL);
}
#else
ff_run(loop, NULL);
#endif
#if 0
while (1) {
#ifdef STANDARD
nfds = epoll_wait(epfd, events, MAX_EVENTS, -1);
#else
nfds = ff_epoll_wait(epfd, events, MAX_EVENTS, -1);
#endif
if (nfds < 0) {
printf("ff_epoll_wait failed\n");
if(nfds != 0) printf("got events, nfds %d\n", nfds);

    for (int i = 0; i < nfds; ++i) {
        if (events[i].events & EPOLLOUT) {
            printf("before write\n");

#ifdef STANDARD
if (write(sockfd, send_buf, strlen(send_buf)) < 0) {
#else
if (ff_write(sockfd, send_buf, strlen(send_buf)) < 0) {
#endif
printf("ff_write failed: %s\n", strerror(errno));
} else {
ev.events = EPOLLIN;
printf("write a buffer\n");
#ifdef STANDARD
epoll_ctl(epfd, EPOLL_CTL_MOD, sockfd, &ev);
#else
ff_epoll_ctl(epfd, EPOLL_CTL_MOD, sockfd, &ev);
#endif
}
} else if (events[i].events & EPOLLIN) {
#ifdef STANDARD
int len = read(sockfd, recv_buf, BUFFER_SIZE - 1);
#else
int len = ff_read(sockfd, recv_buf, BUFFER_SIZE - 1);
#endif
if (len > 0) {
recv_buf[len] = '\0'; // 添加字符串结束符
printf("Received from server: %s\n", recv_buf);
} else {
if (len < 0) {
printf("ff_read failed: %s\n", strerror(errno));
} else {
printf("Connection closed by server\n");
}
#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
goto cleanup;
}
} else if (events[i].events & (EPOLLHUP | EPOLLERR)) {
printf("epoll error\n");
#ifdef STANDARD
close(sockfd);
#else
ff_close(sockfd);
#endif
goto cleanup;
}
}
}
}

cleanup:
#ifdef STANDARD
close(epfd);
#else
ff_close(epfd);
#endif
#endif
return 0;
}
`