Authenticate not working w/ Openssl 3.0.2

[wawahuy]

Describe the bug
Can't login with novnc or remmina.
password: 12345678

To Reproduce

pthread_t tid = pthread_self();
int argc;
char *agv;
server = rfbGetScreen(&argc, &agv, maxx, maxy, 8, 3, bpp);
server->frameBuffer = (char*)malloc(maxx * maxy * bpp);
server->alwaysShared = TRUE;
server->newClientHook = newClient;
server->ptrAddEvent = mouseCallback;
server->kbdAddEvent = keyCallback;

char **passwordList = (char**) malloc(sizeof(char **) * 2);
const char cPassword[] = "12345678";
passwordList[0] = strdup(cPassword);
passwordList[1] = NULL;
server->authPasswdData = (void *) passwordList;
server->passwordCheck = rfbCheckPasswordByList;

or test command:

copyrecttest -passwd 12345678

Your environment (please complete the following information):

• OS and version: Ubuntu 22.04
• Compiler and version: 0.9.14

[bk138]

Can you provide a complete, minimal example that shows the issue for me to repro please?

[bk138]

Maybe you're doing something differently. If I do ./build/examples/server/example -passwd 12345678, this works.

[wawahuy]

Screencast from 13-07-2023 20:05:52.webm
not working!

[bk138]

What's the client you are using ? Can you please try with a publicly available one so I can repro?

Edit: sorry, didn't see it's Remmina. Works for me here w/ "org.remmina.Remmina - 1.4.29 (git n/a)".

Please share the complete libvncserver cmake .. log.

[wawahuy]

remmina 1.4.25 (git n/a)
cmake version 3.27.0-rc4
when run cmake ..

CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The C compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")  
-- Could NOT find LZO
-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR) 
-- Could NOT find PNG (missing: PNG_LIBRARY PNG_PNG_INCLUDE_DIR) 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Could NOT find SDL2 (missing: SDL2_LIBRARY SDL2_INCLUDE_DIR) 
-- Some or all of the gtk libraries were not found. (missing: GTK2_GTK_LIBRARY GTK2_GTK_INCLUDE_DIR GTK2_GDK_INCLUDE_DIR GTK2_GDKCONFIG_INCLUDE_DIR GTK2_GDK_LIBRARY) 
-- Could NOT find LibSSH2 (missing: LIBSSH2_LIBRARY LIBSSH2_INCLUDE_DIR) 
-- Could NOT find GnuTLS (missing: GNUTLS_LIBRARY GNUTLS_INCLUDE_DIR) (Required is at least version "3.4.0")
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.0.2")  
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'libsystemd'
--   No package 'libsystemd' found
-- Could NOT find FFMPEG (missing: FFMPEG_INCLUDE_DIRS FFMPEG_LIBRARIES avformat avcodec avutil swscale) (Required is at least version "3.1.0")
-- Looking for dirent.h
-- Looking for dirent.h - found
-- Looking for endian.h
-- Looking for endian.h - found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Looking for netinet/in.h
-- Looking for netinet/in.h - found
-- Looking for sys/endian.h
-- Looking for sys/endian.h - not found
-- Looking for sys/socket.h
-- Looking for sys/socket.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for sys/wait.h
-- Looking for sys/wait.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for sys/resource.h
-- Looking for sys/resource.h - found
-- Looking for vfork.h
-- Looking for vfork.h - not found
-- Looking for ws2tcpip.h
-- Looking for ws2tcpip.h - not found
-- Looking for arpa/inet.h
-- Looking for arpa/inet.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for vfork
-- Looking for vfork - found
-- Looking for vprintf
-- Looking for vprintf - found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for fork
-- Looking for fork - found
-- Looking for ftime
-- Looking for ftime - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for gethostname
-- Looking for gethostname - found
-- Looking for inet_ntoa
-- Looking for inet_ntoa - found
-- Looking for memmove
-- Looking for memmove - found
-- Looking for memset
-- Looking for memset - found
-- Looking for mkfifo
-- Looking for mkfifo - found
-- Looking for select
-- Looking for select - found
-- Looking for socket
-- Looking for socket - found
-- Looking for strchr
-- Looking for strchr - found
-- Looking for strcspn
-- Looking for strcspn - found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for strstr
-- Looking for strstr - found
-- Looking for htobe64
-- Looking for htobe64 - found
-- Looking for OSSwapHostToBigInt64
-- Looking for OSSwapHostToBigInt64 - not found
-- Building crypto with OpenSSL
-- Threads support is using pthreads
-- Check size of pid_t
-- Check size of pid_t - done
-- Check size of size_t
-- Check size of size_t - done
-- Check size of socklen_t
-- Check size of socklen_t - done
-- Check size of in_addr_t
-- Check size of in_addr_t - done
-- Building TLS with OpenSSL
-- Configuring done (3.2s)
-- Generating done (0.0s)
-- Build files have been written to: /home/yuh/Project/j2run/libvncserver/build

[bk138]

Thanks! If you change the crypto backend to GnuTLS or built-in, does the issue happen still?

[wawahuy]

Great, perhaps openssl 3.0.2 didn't work with libvnc. I used the built-in option with the command

 -DWITH_OPENSSL=OFF -DWITH_GCRYPT=ON

and it worked perfectly. Thank you very much.
libvnc.log