Supporting sendMsg and recvMsg. #433

kazu-yamamoto · 2019-12-04T05:03:11Z

Note that the quality of this code is good enough, I believe. But I don't expect that this PR will be merged quickly. I would like to merge this PR after discussion.

Cc: @snoyberg, @farnoy, @ocheron, @lukehoersten, @lpeterse, @vdukhovni

Motivation

We would like to use MSG_PEEK for TCP sockets to peek the received data without removing the input queue.
We would like to receive or specify, if possible, the control message such as IP_TTL, IP_TOS for ECN and IP_PKTINFO to specify the outgoing network interface for UDP packets.

So, I would like to discuss how to support sendmsg() and recvmsg(). Note that they can be used for TCP in addition to UDP.

Design

The current wrapper APIs for sendmsg() and recvmsg() are as follows:

sendMsg :: Socket       -- ^ Socket
        -> SockAddr     -- ^ Destination address
        -> [ByteString] -- ^ Data to be sent
        -> [Cmsg]       -- ^ Control messages
        -> [MsgFlag]    -- ^ Message flags
        -> IO Int       -- ^ The length actually sent

recvMsg :: Socket    -- ^ Socket
        -> [Int]     -- ^ A list of length of data to be received
                     --   If the total length is not large enough,
                     --   'MSG_TRUNC' is returned
        -> Int       -- ^ The buffer size for control messages.
                     --   If the length is not large enough,
                     --   'MSG_CTRUNC' is returned
        -> [MsgFlag] -- ^ Message flags
        -> IO (SockAddr, [ByteString], [Cmsg], [MsgFlag]) -- ^ Source address, received data, control messages and message flags

MsgFlag includes MSG_PEEK
Cmsg is a low level format for sendMsg and recvMsg. Their values can be converted into user-friendly type values such as IPv4TTL and IPv4PktInfo via Auxiliary class.

I believe we don't have to extend, for instance, recv to support MsgFlag since recvMsg is a superset.

Please read test cases to know how to use these APIs more concretely.

Discussion

@eborden Should sendMsg/recvMsg be supported in network. Or another package?
@Mistuke How can we support Windows? Does Windows provide similar APIs?

Note

IPv4TTL/IPv6HopLimit and IPv4TOS/IPv6TClass are reading-purpose only.
IPv4PktInfo and IPv6PktInfo for sendMsg are not tested yet.
This is relating to Support passing flags to c_recv #243.

maoe · 2019-12-04T05:11:22Z

Coincidentally, I wrote similar code for recvmmsg recently to use hardware timestamping. I'll review this PR when I have time.

Network/Socket/ByteString/Flag.hsc

Network/Socket/ByteString/IO.hsc

Network/Socket/Options.hsc

eborden · 2019-12-05T16:08:46Z

Should sendMsg/recvMsg be supported in network. Or another package?

If we can support it cross platform then I'm in favor of inclusion. @Mistuke's answer about Windows will be key in that decision.

kazu-yamamoto · 2019-12-05T23:59:20Z

@eborden I agree.

kazu-yamamoto · 2019-12-08T23:38:18Z

@Mistuke I would like to forward this issue. Would you give us your advice? Are you in mountains again? :-)

maoe

I suppose the API could have been lower-level to reduce some overhead and to make C-interop easier.

For example in the proposed API recvMsg returns lists of ByteStrings and Cmsgs. This can be a costly process to run in low-latency networking code because every payload and ancillary data needs to be copied over to the Haskell heap. Instead it would be nice if we had something analoguous to recvBuf.

Network/Socket/ByteString.hs

Network/Socket/ByteString/Cmsg.hsc

Network/Socket/ByteString/MsgHdr.hsc

Network/Socket/ByteString/Auxiliary.hsc

Network/Socket/ByteString/IO.hsc

Mistuke · 2019-12-22T08:55:44Z

@Mistuke I would like to forward this issue. Would you give us your advice? Are you in mountains again? :-)

Sorry, was very very busy with work. I will go over this today.

kazu-yamamoto · 2019-12-23T23:54:22Z

OK. I will try to implement this feature in network and resolve all issues pointed by the reviewers.

kazu-yamamoto · 2020-01-10T01:41:59Z

Additional plan:

Supporting Windows
Testing ancillary data especially for IPv4PktInfo and IPv6PktInfo
Generalizing sendBufMsg and recvBufMsg with SocketAddress
Replacing C implementation of sendFd and recvFd with this API
Implementing extendable socket option functions
Checking CChar vs CInt

kazu-yamamoto · 2020-01-10T08:00:22Z

@maoe @vdukhovni Would you review this again?

@Mistuke Now it's your turn. If time allows, please implement Windows part.

kazu-yamamoto · 2020-01-17T01:01:30Z

It appeared that IP_PKTINFO of macOS is buggy. If it used with sendMsg with either non-null ipi_ifindex or non-null ipi_spec_dst, the local address of the socket is overridden with 0.0.0.0.

Mistuke · 2020-01-19T17:44:08Z

@kazu-yamamoto Sorry missed your message, I will get started on this next Sunday (suspect to be able to get it done same week). Currently untangling some GHC release issues :(

kazu-yamamoto · 2020-01-19T23:34:03Z

@Mistuke No rush. But please keep in mind this issue. :-)

I'm currently testing this PR with my QUIC implementation.

kazu-yamamoto · 2020-01-22T01:46:29Z

Rebased onto master.

Network/Socket/Flag.hsc

Mistuke · 2020-02-18T12:52:41Z

I haven't forgotten about this! I'm mapping the changes I need to do out and will refactor it and hopefully have something soon!

Mistuke · 2020-02-23T18:34:47Z

WIP PR for Windows support at at kazu-yamamoto#2, should be able to finish it next weekend.

kazu-yamamoto · 2020-03-16T00:17:56Z

@Mistuke Gentle ping

Mistuke · 2020-03-16T08:49:58Z

looks like RecvMsg needs some work, I'll try to do that today after work so this is done :)

Mistuke · 2020-03-22T20:38:31Z

Haven't forgotten, still debugging WSARecvMsg the failure is a bit generic..

vdukhovni

Some mostly documentation-related comments. The code looks fine to me at first glance.

Network/Socket/Buffer.hsc

vdukhovni · 2020-03-22T23:05:47Z

Network/Socket/Posix/Cmsg.hsc

+
+----------------------------------------------------------------
+
+-- | Looking up control message. The following shows an example usage:


Instead of "Looking up control message", better might be:

Locate a control message of the given type in a list of control messages

But is this function really necessary? It is the same as Data.List.find . (. cmsgId) . (==).
I am not sure it needs to be provided here.

Changed the doc and the definition.

But is this function really necessary? It is the same as Data.List.find . (. cmsgId) . (==).

Not sure. But I think it's useful as suggested in the usage.

vdukhovni · 2020-03-22T23:18:32Z

Network/Socket/Posix/Cmsg.hsc

+  | cmsgId cmsg == cid = Just cmsg
+  | otherwise          = lookupCmsg cid cmsgs
+
+-- | Filtering control message.


This too might be better rephrased, but again, why not just drop it?

It would be nice if you provided the better doc.

vdukhovni · 2020-03-22T23:31:39Z

Network/Socket/Unix.hsc

-foreign import ccall SAFE_ON_WIN "recvFd" c_recvFd :: CInt -> IO CInt
+recvFd s = allocaBytes dummyBufSize $ \buf -> do
+    (NullSockAddr, _, cmsgs, _) <- recvBufMsg s [(buf,dummyBufSize)] 32 mempty
+    case (lookupCmsg CmsgIdFd cmsgs >>= decodeCmsg) :: Maybe Fd of


Here find ((== CmsgIdFd) . cmsgId) cmsgs could replace lookupCmsg.

vdukhovni · 2020-03-22T23:32:36Z

appveyor.yml

    - GHCVER: 8.0.2
    - GHCVER: 8.2.2
    - GHCVER: 8.4.4
    - GHCVER: 8.6.3
+    - GHCVER: 8.8.1


By now perhaps 8.8.2 or 8.8.3, whichever is the latest available on Appveyor?

Mistuke · 2020-04-02T21:13:09Z

Almost done, it's a bit slow going debugging the API. I have a working C example now fixing the Haskell side. hopefully done this weekend.

Mistuke · 2020-04-05T18:57:48Z

@kazu-yamamoto That should do it. Sorry for the delay. I've been a bit over stretched lately.

The only thing I was wondering about is if I can simplify the logic in recvBufMsg, particularly this part

#if !defined(mingw32_HOST_OS)
              , msgCtrl    = castPtr ctrlPtr
#else
              , msgCtrl    = if clen == 0 then nullPtr else castPtr ctrlPtr
#endif
              , msgCtrlLen = fromIntegral clen
#if !defined(mingw32_HOST_OS)
              , msgFlags   = 0
#else
              , msgFlags   = fromIntegral $ fromMsgFlag flags
#endif

Would it present a problem for Linux if when clen == 0 we always pass nullPtr and if we always encode the flag in msgFlags ?

vdukhovni · 2020-04-05T21:27:40Z

Would it present a problem for Linux if when clen == 0 we always pass nullPtr and if we always encode the flag in msgFlags ?

I've not checked Linux, but it would be OK to always encode the flags on FreeBSD where I checked the kernel source, and the recvmsg() flags argument is copied to msghdr.msg_flags when collecting the system call arguments, and later replaced with the output flags.

However, setting msgCtrl to NULL when clen == 0 changes semantics. With a non-null msgCtrl, but clen == 0 you get MSG_CTRUNC in the returned flags when there were control messages that were discarded. With msgCtrl == NULL the flag is not set. (Again FreeBSD).

I don't know whether anyone depends on this behaviour, but it is perhaps best to not make that change on Unix-like systems.

Breacking change: CustomSockOpt was deleted.

kazu-yamamoto · 2020-04-08T05:44:11Z

This PR has been merged. Thank you for your contribution! Let's discuss left issues in another PR.

kazu-yamamoto changed the title ~~Supportint sendMsg and recvMsg.~~ Supporting sendMsg and recvMsg. Dec 4, 2019

vdukhovni reviewed Dec 4, 2019

View reviewed changes

Network/Socket/ByteString/Flag.hsc Outdated Show resolved Hide resolved

vdukhovni reviewed Dec 4, 2019

View reviewed changes

Network/Socket/ByteString/IO.hsc Outdated Show resolved Hide resolved

vdukhovni reviewed Dec 4, 2019

View reviewed changes

Network/Socket/ByteString/IO.hsc Outdated Show resolved Hide resolved

vdukhovni reviewed Dec 4, 2019

View reviewed changes

Network/Socket/Options.hsc Outdated Show resolved Hide resolved

maoe reviewed Dec 9, 2019

View reviewed changes

kazu-yamamoto force-pushed the msg branch from d5bec3a to cfb6e81 Compare January 22, 2020 01:45

Mistuke reviewed Feb 1, 2020

View reviewed changes

Network/Socket/Flag.hsc Outdated Show resolved Hide resolved

codedmart mentioned this pull request Feb 14, 2020

Auth failure on FreeBSD rblaze/haskell-dbus#42

Closed

vdukhovni reviewed Mar 22, 2020

View reviewed changes

kazu-yamamoto and others added 22 commits April 8, 2020 14:41

SocketOption now contains CInt directly.

e255097

Breacking change: CustomSockOpt was deleted.

renaming: cmsg instead of ancillary

f8305c4

making fields strict.

7ff9f9f

fix for Linux.

b1df8f2

TTL is CInt on Linux.

5676bc9

Using Int in IPv4PktInfo.

5f3706f

fixing the Storable instance.

d040a33

fixing alignment.

031df10

IPv4PktInfo now contains ipi_spec_dst.

aef4e42

adding filterCmsg.

54f3afc

fixing pattern synonym.

3da86ef

network: Initial implementation

e1a4057

network: Add explicit cast

4ae61a3

First implementation Windows msg

834e6ff

Fix Win32 linking issues. now need to fixtest failures

9683add

Fix bindist missin header

fb8529f

Finish windows implementation

507a2c0

fixing a gap between Unix and Windows.

7cc239a

improving docs and definitions as suggested by vdukhovni.

f536d35

using GHC 8.8.3.

3aa7b18

improving docs as suggested by vdukhovni.

15193be

using GHC 8.6.5.

f6b1f7c

kazu-yamamoto force-pushed the msg branch from acfaf50 to f6b1f7c Compare April 8, 2020 05:42

kazu-yamamoto merged commit 54b872f into haskell:master Apr 8, 2020

kazu-yamamoto deleted the msg branch April 8, 2020 05:43

kazu-yamamoto mentioned this pull request May 14, 2020

recv, recvfrom and recvmsg #284

Closed

2 tasks

eborden mentioned this pull request May 17, 2020

Releasing version 3.1.2.0 #414

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supporting sendMsg and recvMsg. #433

Supporting sendMsg and recvMsg. #433

kazu-yamamoto commented Dec 4, 2019

maoe commented Dec 4, 2019

eborden commented Dec 5, 2019

kazu-yamamoto commented Dec 5, 2019

kazu-yamamoto commented Dec 8, 2019

maoe left a comment •

edited

Loading

Mistuke commented Dec 22, 2019

kazu-yamamoto commented Dec 23, 2019

kazu-yamamoto commented Jan 10, 2020 •

edited

Loading

kazu-yamamoto commented Jan 10, 2020

kazu-yamamoto commented Jan 17, 2020

Mistuke commented Jan 19, 2020

kazu-yamamoto commented Jan 19, 2020

kazu-yamamoto commented Jan 22, 2020

Mistuke commented Feb 18, 2020

Mistuke commented Feb 23, 2020

kazu-yamamoto commented Mar 16, 2020

Mistuke commented Mar 16, 2020

Mistuke commented Mar 22, 2020

vdukhovni left a comment

vdukhovni Mar 22, 2020

vdukhovni Mar 22, 2020

kazu-yamamoto Apr 8, 2020

vdukhovni Mar 22, 2020

kazu-yamamoto Apr 8, 2020

vdukhovni Mar 22, 2020

vdukhovni Mar 22, 2020

Mistuke commented Apr 2, 2020

Mistuke commented Apr 5, 2020

vdukhovni commented Apr 5, 2020

kazu-yamamoto commented Apr 8, 2020


		----------------------------------------------------------------

		-- \| Looking up control message. The following shows an example usage:

Supporting sendMsg and recvMsg. #433

Supporting sendMsg and recvMsg. #433

Conversation

kazu-yamamoto commented Dec 4, 2019

Motivation

Design

Discussion

Note

maoe commented Dec 4, 2019

eborden commented Dec 5, 2019

kazu-yamamoto commented Dec 5, 2019

kazu-yamamoto commented Dec 8, 2019

maoe left a comment • edited Loading

Choose a reason for hiding this comment

Mistuke commented Dec 22, 2019

kazu-yamamoto commented Dec 23, 2019

kazu-yamamoto commented Jan 10, 2020 • edited Loading

kazu-yamamoto commented Jan 10, 2020

kazu-yamamoto commented Jan 17, 2020

Mistuke commented Jan 19, 2020

kazu-yamamoto commented Jan 19, 2020

kazu-yamamoto commented Jan 22, 2020

Mistuke commented Feb 18, 2020

Mistuke commented Feb 23, 2020

kazu-yamamoto commented Mar 16, 2020

Mistuke commented Mar 16, 2020

Mistuke commented Mar 22, 2020

vdukhovni left a comment

Choose a reason for hiding this comment

vdukhovni Mar 22, 2020

Choose a reason for hiding this comment

vdukhovni Mar 22, 2020

Choose a reason for hiding this comment

kazu-yamamoto Apr 8, 2020

Choose a reason for hiding this comment

vdukhovni Mar 22, 2020

Choose a reason for hiding this comment

kazu-yamamoto Apr 8, 2020

Choose a reason for hiding this comment

vdukhovni Mar 22, 2020

Choose a reason for hiding this comment

vdukhovni Mar 22, 2020

Choose a reason for hiding this comment

Mistuke commented Apr 2, 2020

Mistuke commented Apr 5, 2020

vdukhovni commented Apr 5, 2020

kazu-yamamoto commented Apr 8, 2020

maoe left a comment •

edited

Loading

kazu-yamamoto commented Jan 10, 2020 •

edited

Loading