Making the window transparent (X Window System, X11)

[MadLittleMods]

These are just notes I can share around on this problem. I haven't gotten this working yet.

Background

To make sure this isn't an X/Y problem, I'm trying to make the window background transparent but it's always just opaque even when giving it 32-bit colors(ARGB, 0xaabbccdd). With a successful basic the window setup, we can set .parent_window_id = screen.root, .depth = 0 (which means inherit from parent), and .visual_id = screen.root_visual to get a opaque window showing on the screen. The reason it's 24-bit color-depth is because the root window is 24-bit depth and we inherit from the parent window. You can verify the color depth of the root window by running the xdpyinfo command on your system (see the command output in the dev notes).

When trying to provide .depth = 32 but still using the same .visual_id, we get a match error from the connection setup which is as expected according to the docs.

So we need to find the correct matching .visual_id that has 32-bit color depth and allows us to do transparency.

Problem

I'm trying to modify zigx to parse the following buffer of data that you get back when the connection is accepted during setup because it looks like it has the necessary info.

There is already code that successfully parses up to and including the first screen but without the sub-lists of ScreenDepths or VisualType. Each Screen has many ScreenDepths and each ScreenDepth has many VisualType.

Buffer structure (X11 protocol docs source):

     1     1                               Success
     1                                     unused
     2     CARD16                          protocol-major-version
     2     CARD16                          protocol-minor-version
     2     8+2n+(v+p+m)/4                  length in 4-byte units of
                                           "additional data"
     4     CARD32                          release-number
     4     CARD32                          resource-id-base
     4     CARD32                          resource-id-mask
     4     CARD32                          motion-buffer-size
     2     v                               length of vendor
     2     CARD16                          maximum-request-length
     1     CARD8                           number of SCREENs in roots
     1     n                               number for FORMATs in
                                           pixmap-formats
     1                                     image-byte-order
          0     LSBFirst
          1     MSBFirst
     1                                     bitmap-format-bit-order
          0     LeastSignificant
          1     MostSignificant
     1     CARD8                           bitmap-format-scanline-unit
     1     CARD8                           bitmap-format-scanline-pad
     1     KEYCODE                         min-keycode
     1     KEYCODE                         max-keycode
     4                                     unused
     v     STRING8                         vendor
     p                                     unused, p=pad(v)
     8n     LISTofFORMAT                   pixmap-formats
     m     LISTofSCREEN                    roots (m is always a multiple of 4)
FORMAT
     1     CARD8                           depth
     1     CARD8                           bits-per-pixel
     1     CARD8                           scanline-pad
     5                                     unused
SCREEN
     4     WINDOW                          root
     4     COLORMAP                        default-colormap
     4     CARD32                          white-pixel
     4     CARD32                          black-pixel
     4     SETofEVENT                      current-input-masks
     2     CARD16                          width-in-pixels
     2     CARD16                          height-in-pixels
     2     CARD16                          width-in-millimeters
     2     CARD16                          height-in-millimeters
     2     CARD16                          min-installed-maps
     2     CARD16                          max-installed-maps
     4     VISUALID                        root-visual
     1                                     backing-stores
          0     Never
          1     WhenMapped
          2     Always
     1     BOOL                            save-unders
     1     CARD8                           root-depth
     1     CARD8                           number of DEPTHs in allowed-depths
     n     LISTofDEPTH                     allowed-depths (n is always a
                                           multiple of 4)
DEPTH
     1     CARD8                           depth
     1                                     unused
     2     n                               number of VISUALTYPES in visuals
     4                                     unused
     24n     LISTofVISUALTYPE              visuals
VISUALTYPE
     4     VISUALID                        visual-id
     1                                     class
          0     StaticGray
          1     GrayScale
          2     StaticColor
          3     PseudoColor
          4     TrueColor
          5     DirectColor
     1     CARD8                           bits-per-rgb-value
     2     CARD16                          colormap-entries
     4     CARD32                          red-mask
     4     CARD32                          green-mask
     4     CARD32                          blue-mask
     4                                     unused

How the problem is solved in other X11 client libraries

• C: libx11 -> src/OpenDis.c#L59-570
• Java: x11-client -> src/main/java/com/github/moaxcp/x11client/X11Connection.java#L34-L67 and src/main/xcbXmls/xproto.xml#L94-L267
• Python: Xlib/protocol/display.py#L1024-L1059
• C: libxcb
• Rust: breadx
• Rust: x11rb

---

Keywords:

• reinterpret memory

[MadLittleMods]

It seems like the keywords I am after are "flexible array member" and "variable length array".

Related reading:

• https://www.reddit.com/r/Zig/comments/sr7p3f/does_zig_support_flexible_array_members/
• Imported C Structs with VLAs ziglang/zig#8759
• Variable length arrays in ntdll definitions ziglang/zig#6008
• Consider allowing variable length arrays ziglang/zig#3952
• use case: struct with embedded resizable last field ziglang/zig#173
• https://nathancraddock.com/blog/2022/deserialization-with-zig-metaprogramming/
• https://hero.handmade.network/forums/code-discussion/t/7826-has_anyone_else_given_zig_a_serious_try%252C_only_to_get_caught_up_in_language_debugging#24420

Related examples in other Zig code:

• https://github.com/ziglang/zig/blob/6f0a613b6f2d070196d47cb2932f7c728c63542a/lib/std/pdb.zig#L53-L69
• https://github.com/marler8997/netpunch/blob/f4128a23e4c910d9870911a253e78aa93d998e1f/eventing/selectwindows.zig#L6-L23
• https://github.com/marler8997/rpghelper/blob/e12dad597f91f6ac6bcac5a2c91ed8f01dddbebd/fdeventer.zig#L116-L128
• https://github.com/piec/ncdu/blob/0726f673cca55f50372ef84fd404bac5f366e4be/src/model.zig#L289-L292
• std.zig.c_translation.FlexibleArrayType added in translate-c: Implement flexible arrays ziglang/zig#8891

[MadLittleMods]

I can't quite figure out where I am going wrong in all of the pointer arithmetic. Without the sanity checks, I constantly run into panic: integer overflow, panic: invalid enum value, or segmentation faults but the sanity checks stop the program whenever it sees the unused blocks being anything other than 0.

I pulled the offending code out of zigx and made a barebones reproduction: https://gist.github.com/MadLittleMods/f6988a67b98108630cf34746a82f7d65/8a7cffea30693de234a070a056c5a2d0f869d34e

From the output, you can see the the first ScreenDepth looks ok but after that it's a garbled mess

$ zig run x11-flexible-array-member-parsing.zig
release_number: 12101008
resource_id_base: 396361728
resource_id_mask: 2097151
motion_buffer_size: 256
vendor_len: 20
max_request_len: 65535
root_screen_count: 1
format_count: 7
image_byte_order: x11-flexible-array-member-parsing.ConnectSetup.Fixed.ImageByteOrder.lsb_first
bitmap_format_bit_order: 0
bitmap_format_scanline_unit: 32
bitmap_format_scanline_pad: 32
min_keycode: 8
max_keycode: 255
unused: 0
debug: asdf allowed_depth_count 7
debug: asdf SCREENDEPTH | depth: 24
debug: asdf SCREENDEPTH | unused0: 0
debug: asdf SCREENDEPTH | visual_type_count: 480
debug: asdf SCREENDEPTH | unused1: 0
debug: asdf SCREENDEPTH | _visual_types_array_start: { x11-flexible-array-member-parsing.VisualType{ .id = 33, .class = x11-flexible-array-member-parsing.VisualType.Class.true_color, .bits_per_rgb_value = 8, .colormap_entries = 256, .red_mask = 16711680, .green_mask = 65280, .blue_mask = 255, .unused = 0 } }
debug: visual count 480, pointer_offset += 11528
debug: asdf SCREENDEPTH | depth: 0
debug: asdf SCREENDEPTH | unused0: 0
debug: asdf SCREENDEPTH | visual_type_count: 255
debug: asdf SCREENDEPTH | unused1: 65280
debug: asdf SCREENDEPTH | _visual_types_array_start: { x11-flexible-array-member-parsing.VisualType{ .id = 255, .class = x11-flexible-array-member-parsing.VisualType.Class.static_gray, .bits_per_rgb_value = 0, .colormap_entries = 0, .red_mask = 1910, .green_mask = 16779269, .blue_mask = 16711680, .unused = 65280 } }
debug: visual count 255, pointer_offset += 6128
debug: asdf SCREENDEPTH | depth: 58
debug: asdf SCREENDEPTH | unused0: 47
debug: asdf SCREENDEPTH | visual_type_count: 26978
debug: asdf SCREENDEPTH | unused1: 1966029422
debug: asdf SCREENDEPTH | _visual_types_array_start: { x11-flexible-array-member-parsing.VisualType{ .id = 1815048819, .class = x11-flexible-array-member-parsing.VisualType.Class.thread 169494 panic: invalid enum value
/home/eric/.zvm/0.11.0/lib/std/fmt.zig:527:37: 0x298d2a in formatType__anon_10664 (x11-flexible-array-member-parsing)
                try writer.writeAll(@tagName(value));
                                    ^
/home/eric/.zvm/0.11.0/lib/std/fmt.zig:596:31: 0x289ad9 in formatType__anon_10483 (x11-flexible-array-member-parsing)
                try formatType(@field(value, f.name), ANY, options, writer, max_depth - 1);
                              ^
/home/eric/.zvm/0.11.0/lib/std/fmt.zig:688:31: 0x27d807 in formatType__anon_10071 (x11-flexible-array-member-parsing)
                try formatType(elem, actual_fmt, options, writer, max_depth - 1);
                              ^
/home/eric/.zvm/0.11.0/lib/std/fmt.zig:184:23: 0x27d5ed in format__anon_10051 (x11-flexible-array-member-parsing)
        try formatType(
                      ^
/home/eric/.zvm/0.11.0/lib/std/io/writer.zig:28:34: 0x271560 in print__anon_9605 (x11-flexible-array-member-parsing)
            return std.fmt.format(self, format, args);
                                 ^
/home/eric/.zvm/0.11.0/lib/std/log.zig:154:27: 0x25fe1d in defaultLog__anon_8294 (x11-flexible-array-member-parsing)
    nosuspend stderr.print(level_txt ++ prefix2 ++ format ++ "\n", args) catch return;
                          ^
/home/eric/.zvm/0.11.0/lib/std/log.zig:125:22: 0x230008 in log__anon_4826 (x11-flexible-array-member-parsing)
    std.options.logFn(message_level, scope, format, args);
                     ^
/home/eric/.zvm/0.11.0/lib/std/log.zig:197:16: 0x229dd8 in debug__anon_4076 (x11-flexible-array-member-parsing)
            log(.debug, scope, format, args);
               ^
/home/eric/Documents/code/zig/random/x11-flexible-array-member-parsing/x11-flexible-array-member-parsing.zig:59:30: 0x2299de in getAllowedDepths (x11-flexible-array-member-parsing)
                std.log.debug("asdf SCREENDEPTH | {s}: {any}", .{ field.name, @field(depth_ptr, field.name) });
                             ^
/home/eric/Documents/code/zig/random/x11-flexible-array-member-parsing/x11-flexible-array-member-parsing.zig:299:55: 0x22a09a in getScreens (x11-flexible-array-member-parsing)
            const depths = try screen.getAllowedDepths(allocator);
                                                      ^
/home/eric/Documents/code/zig/random/x11-flexible-array-member-parsing/x11-flexible-array-member-parsing.zig:349:47: 0x22ae34 in main (x11-flexible-array-member-parsing)
    var screens = try connect_setup.getScreens(allocator);
                                              ^
/home/eric/.zvm/0.11.0/lib/std/start.zig:574:37: 0x22705e in posixCallMainAndExit (x11-flexible-array-member-parsing)
            const result = root.main() catch |err| {
                                    ^
/home/eric/.zvm/0.11.0/lib/std/start.zig:243:5: 0x226b41 in _start (x11-flexible-array-member-parsing)
    asm volatile (switch (native_arch) {
    ^
???:?:?: 0x0 in ??? (???)
Aborted (core dumped)

Another aspect I can't quite figure out is why the pointers don't line up. If I log something like the following, I'd expect the following but the math isn't adding up (perhaps this is an alignment thing?).

• &self and &self.root to be equal
• @intFromPtr(&self.allowed_depth_count) and @intFromPtr(&self) + @offsetOf(@TypeOf(self), "allowed_depth_count") to be equal

// asdf iterating through depth 0 - 140736350248287 vs 140736350248360 vs &self=140736350248392 &self.root=140736350248464 + 0 + 39 or 40 = 140736350248575 vs 140736350248648
std.log.debug("asdf iterating through depth {} - {} vs {} vs &self={} &self.root={} + {} + {} or {} = {} vs {}", .{
    i,
    @intFromPtr(&self.allowed_depth_count),
    @intFromPtr(&self._allowed_depths_array_start),
    @intFromPtr(&self),
    @intFromPtr(&self.root),
    @offsetOf(@TypeOf(self), "root"),
    @offsetOf(@TypeOf(self), "allowed_depth_count"),
    @offsetOf(@TypeOf(self), "_allowed_depths_array_start"),
    @intFromPtr(&self) + @offsetOf(@TypeOf(self), "allowed_depth_count"),
    @intFromPtr(&self) + @offsetOf(@TypeOf(self), "_allowed_depths_array_start"),
});

[MadLittleMods]

I figured it out! The problem boils down to self: @This() vs self: *@This() which I needed to use in order to get a reference instead of a copy. Working barebones setup: https://gist.github.com/MadLittleMods/f6988a67b98108630cf34746a82f7d65

Related post: https://zig.news/gowind/beware-the-copy-32l4

From the Zig docs:

Pass-by-value Parameters

Primitive types such as Integers and Floats passed as parameters are copied, and then the copy is available in the function body. This is called "passing by value". Copying a primitive type is essentially free and typically involves nothing more than setting a register.

Structs, unions, and arrays can sometimes be more efficiently passed as a reference, since a copy could be arbitrarily expensive depending on the size. When these types are passed as parameters, Zig may choose to copy and pass by value, or pass by reference, whichever way Zig decides will be faster. This is made possible, in part, by the fact that parameters are immutable.

-- https://ziglang.org/documentation/master/#Pass-by-value-Parameters

[MadLittleMods]

Submitted a PR to parse the rest of the connection setup buffer to get the screens, depths, and visual types -> marler8997/zigx#8

[MadLittleMods]

Another PR to create and specify custom color maps marler8997/zigx#9

I was able to get transparency/alpha (32-bit color depth) working with both of these changes! #2