Allow changing blocking mode temporarily on socket/stream

[jd]

I'm trying to integrate cl-postgres and cl-async together. I've opened a bug and I am working on an implementatino, see marijnh/Postmodern#28

My current problem, is that all of cl-postgres is synchronous. That means I need to be able to put the socket into synchronous mode so it can write some requests, read its expected replies, and then give back the hand to the loop.

Currently, I provide a callback function to cl-postgres for it to call when it wants a stream to communicate with PostgreSQL. The problem is that it calls it and gets a stream from tcp-connect. Then it tries to authenticate right away, and it gets "end of file" really soon. I think this is due to the fact there's nothing to read at this precise moment, but I'd prefer the read function to block, in this case.

[orthecreedence]

You are entering down the long and difficult road of converting a synchronous driver to be asynchronous. Welcome to my world the past few weeks =].

Unfortunately, libevent provides an abstraction function for making a socket _non_blocking, but not one to make it block (I think because they block by default). In other words, there's not an easy, portable way for me to add it into the library.

Don't think I don't feel your pain, though. I worked on a solution via coroutines would be the best of both worlds, but the implementation maintainers I talked to basically said that coroutines are going to be a lot of trouble to program, and only a handful of lispers want them, so "no, unless you want to do it yourself."

You have a few options:

• Study the evutil_make_socket_nonblocking function (see below) and use cffi to mimic it (you can directly call C functions via lisp). This will let you do an "easier" conversion of cl-postgres, but a solution that only allows one call to the db per thread from your app at once (kind of defeats the purpose, and is another reason I'm hesitant to allow making sockets blocking).
• Wait for me to convert it. Postgres is one of the most used databases by lispers, so I do have the drivers scheduled for conversion (sorry this intention isn't noted in the drivers list). Once converted, it will have a very native interface, provided your app makes use of cl-async's futures.
• Work on the proper conversion yourself. This is a bitch and I wouldn't wish it on anyone BUT that said, you will be helping the viability of using Common Lisp for non-blocking communication with the rest of the world. You'd be doing your part!

// fd can be accessed via le:bufferevent-getfd

int evutil_make_socket_nonblocking(evutil_socket_t fd)
{
#ifdef WIN32
    {
        u_long nonblocking = 1;
        if (ioctlsocket(fd, FIONBIO, &nonblocking) == SOCKET_ERROR) {
            event_sock_warn(fd, "fcntl(%d, F_GETFL)", (int)fd);
            return -1;
        }
    }
#else
    {
        int flags;
        if ((flags = fcntl(fd, F_GETFL, NULL)) < 0) {
            event_warn("fcntl(%d, F_GETFL)", fd);
            return -1;
        }
        if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) {
            event_warn("fcntl(%d, F_SETFL)", fd);
            return -1;
        }
    }
#endif
    return 0;
}

Sorry if this is bad news, but being the early adopter of a library that forces switching to continuation-passing style has its bad parts =[. My recommendation is to not make your socket blocking since in the end, I think you'll hate this option more than you realize, but to either wait for a cl-postgres conversion or have a try at doing it yourself.

[jd]

Thanks @orthecreedence! Indeed, it's a tough situation, but I'm really getting used to teething troubles nowadays. :-)

I'm using cl-postgres into a second thread now to avoid blocking, but everything's not thread safe, so I'll have to add some simple locking mechanism. Something I would have love to avoid by using cl-async in this precise case, but I'll wait for the conversion to happen!

I don't think I could help that much about converting, I'm still discovering CL every day, but count on me to try and review any implementation you'll have! :-)

[orthecreedence]

Much appreciated, Julien. I'll do my best to let the world know when the conversion is completed, both via the cl-async homepage blog and reddit's /r/lisp.

In the meantime, good luck with the threaded approach, that seems like a fine pursuit. One thing I forgot to mention which may make your life a bit easier is pretend-event-loop, a library I wrote a while back which mimics an event loop using a thread pool. It has one active thread (the main thread that code runs in) and a configurable number of passive threads, used to run blocking operations. The idea is that whenever you have to do I/O, you schedule it in a passive thread and bind the result(s) to a variable available in the active thread such that the passive threads can be many more than your CPU cores since they are mostly doing blocking operations.

Right now it's kind of weak on error handling, as most of it has to happen in the passive thread the blocking operation runs on which is annoying since the passive operations should be as lean as absolutely possible to avoid context switches. It also adds a bit of cruft to the interface it uses. You might still find some use for it though, or even some patches if you can think of a better interface, which I'd gladly accept. If you'd like to give it a try, feel free to ask me any questions you run into while using it!