add a lib-shdc?

[kevinw]

Being able to recompile shaders at run-time is a thing I would like in my sokol projects.

sokol-shdc has done a wonderful job of wrapping the glslang and SPIRV-cross libs. Would it be feasible to write a C API for a sokol-shdc.dll that, given some shader source, fills the appropriate sokol shader description structs?

[floooh]

Yeah I think that should work and wouldn't be too complicated (both as DLL or static link library, or with a bit of tinkering even as a WASM module). It won't be small though (the DLL will probably be between 5..6 MBytes, or in case of a static lib, add that much to the executable).

What's needed is an "API header/source pair" which defines a C library API and maybe move some code from main.cc into a common source used both by the sokol-shdc exe and the library.

Ideally the sokol-shdc executable would use the same library API internally too (I'd prefer this because then the library API is automatically tested by the sokol-shdc executable).

I had something similar in mind but for a different use case: eventually I want to get rid of the precompiled binaries in https://github.com/floooh/sokol-tools-bin and want to have a solution where the bulk of the shader compiler is compiled to WASM and then executed through some WASI-launcher, I think this would also benefit from having the core functionality wrapped in a separate library.

I'll see if I can start tinkering on this 'soon-ish' (want to do some smaller things first).

[rcases]

Also define callback functions for basic file operations (open, read, write, seek, tell ,close). If not defined, use the standard ones.
More or less as SDL_RWops

[kkukshtel]

Picking this back up — would also really like this. I'm binding Sokol headers to an external language (C#) for an engine, and being able to have runtime compilation of shaders (and also abstract away the process to end-users) would definitely be ideal.

I think something as simple as "pass in a glsl file and get back the shader in the target lang + uniforms in some info struct"would be great, don't need any of the header generation stuff.

Right now my plan was to ship shdc and invoke it as an external process and capture the output. A DLL would definitely be more ideal.

[RandyGaul]

Would it be possible to get some guidance on implementing a library for shdc? It seems a bit low priority for @floooh so perhaps I could take a stab at it. I’ve also wanted to compile shaders at runtime so they can be easily hotloaded, and to simplify build steps by not requiring build time shader compilation.

[floooh]

There's a couple of fairly simple steps, and one I don't know how to solve :)

• In the first step I would isolate all IO calls into a virtual interface which needs to be passed into the code which does IO, this would include fopen, fclose, fread, fwrite, fseek, fgets, ...? (but maybe on a higher level, for instance fseek is only used in the combination of fseek(f, 0, SEEK_END) + fseek(f, 0, SEEK_SET) to get the file size. The reason for this virtual interface would be to allow 'file IO' from memory buffers instead of directly touching the filesystem... this would be the only place where some experimentation might be required to 'make it feel right'...
• Error reporting/logging should probably also be tackled... not sure yet how
• In the next step, move most of the main function into a separate source and function which becomes the libraries 'main entry point':

  sokol-tools/src/shdc/main.cc

  Lines 28 to 126 in 9300010

  	// load the source and parse tagged blocks
  	const Input inp = Input::load_and_parse(args.input, args.module);
  	if (args.debug_dump) {
  	inp.dump_debug(args.error_format);
  	}
  	if (inp.out_error.valid()) {
  	inp.out_error.print(args.error_format);
  	return 10;
  	}
  	// compile source snippets to SPIRV blobs (multiple compilations is necessary
  	// because of conditional compilation by target language)
  	std::array<Spirv,Slang::Num> spirv;
  	for (int i = 0; i < Slang::Num; i++) {
  	Slang::Enum slang = Slang::from_index(i);
  	if (args.slang & Slang::bit(slang)) {
  	spirv[i] = Spirv::compile_glsl(inp, slang, args.defines);
  	if (args.debug_dump) {
  	spirv[i].dump_debug(inp, args.error_format);
  	}
  	if (!spirv[i].errors.empty()) {
  	bool has_errors = false;
  	for (const ErrMsg& err: spirv[i].errors) {
  	if (err.type == ErrMsg::ERROR) {
  	has_errors = true;
  	}
  	err.print(args.error_format);
  	}
  	if (has_errors) {
  	return 10;
  	}
  	}
  	if (args.save_intermediate_spirv) {
  	if (!spirv[i].write_to_file(args, inp, slang)) {
  	return 10;
  	}
  	}
  	}
  	}
  	// cross-translate SPIRV to shader dialects
  	std::array<Spirvcross,Slang::Num> spirvcross;
  	for (int i = 0; i < Slang::Num; i++) {
  	Slang::Enum slang = Slang::from_index(i);
  	if (args.slang & Slang::bit(slang)) {
  	spirvcross[i] = Spirvcross::translate(inp, spirv[i], slang);
  	if (args.debug_dump) {
  	spirvcross[i].dump_debug(args.error_format, slang);
  	}
  	if (spirvcross[i].error.valid()) {
  	spirvcross[i].error.print(args.error_format);
  	return 10;
  	}
  	}
  	}
  	// compile shader-byte code if requested (HLSL / Metal)
  	std::array<Bytecode, Slang::Num> bytecode;
  	if (args.byte_code) {
  	for (int i = 0; i < Slang::Num; i++) {
  	Slang::Enum slang = Slang::from_index(i);
  	if (args.slang & Slang::bit(slang)) {
  	bytecode[i] = Bytecode::compile(args, inp, spirvcross[i], slang);
  	if (args.debug_dump) {
  	bytecode[i].dump_debug();
  	}
  	if (!bytecode[i].errors.empty()) {
  	bool has_errors = false;
  	for (const ErrMsg& err: bytecode[i].errors) {
  	if (err.type == ErrMsg::ERROR) {
  	has_errors = true;
  	}
  	err.print(args.error_format);
  	}
  	if (has_errors) {
  	return 10;
  	}
  	}
  	}
  	}
  	}
  	// build merged Reflection info
  	const Reflection refl = Reflection::build(args, inp, spirvcross);
  	if (refl.error.valid()) {
  	refl.error.print(args.error_format);
  	return 10;
  	}
  	if (args.debug_dump) {
  	refl.dump_debug(args.error_format);
  	}
  	// generate output files
  	const GenInput gen_input(args, inp, spirvcross, bytecode, refl);
  	ErrMsg gen_error = generate(args.output_format, gen_input);
  	if (gen_error.valid()) {
  	gen_error.print(args.error_format);
  	return 10;
  	}

  • this new entry point should take an Args object as input, and it also needs a 'virtual IO object' which abstract away the differences between direct file IO and reading/writing memory buffers
• finally the required build system changes, the build process should generally build a static link library, and the executable would just be the main.cc source file and link with the library

...and now the tricky part: D3D and Metal byte code generation (which might be out of scope):

• for D3D this is done by attempting to load a d3dcompiler.dll and calling into that - sokol_gfx.h is doing the same btw when HLSL shader code is provided
• for Metal this is done by running the Metal shader compiler as command line tool

Both might not be an option (but maybe also not required?) for sokol-shdc as a library, so maybe this can simply be ignored (since at one point, shader source code needs to be compiled anyway, and this can just as well happen in sokol_gfx.h).

If anybody wants to tackle this, I would prefer it as a number of smaller PRs instead of a single big one, for instance:

1. implement the virtual IO interface and replace any direct IO usage with this
2. maybe another PR for the error reporting
3. move most of the main function into a new common entry point function, and change main() to call that function
4. implement the actual split into a library and executable

[kkukshtel]

Correct me if I'm wrong but bytecode generation would be the thing that would make the library most useful. Especially in terms of hotloading shaders, there would need to be some compile step as part of library functionality to actually consume the shaders, otherwise the library would only be about producing .c versions of the shaders, correct?

For me, the bytecode generation bit would definitely be the thing worth doing. Though I'm also not good enough (for now!) to actually build this func and would mostly be a consumer of whatever anyone else does until I could contribute to it.

[rcases]

I've been creating my own library for a while now.

The way I've done it revolves around making minimal changes to the shdc codebase to make it easier to maintain.
These changes are splitting the load_and_preprocess method into load and preprocess and adding parse_source.

The library can use either a filename or a pointer to the source code.
Unfortunately I haven't yet implemented a way to load includes from a virtual interface.

If you're interested in seeing the strategy I've used, you can find it here:

https://gist.github.com/rcases/8f65e7d54b2556fea60bc138af6cdb0d

[floooh]

otherwise the library would only be about producing .c versions of the shaders, correct?

No, there would be no code generation step, you would get GLSL, HLSL, MSL and WGSL source code as strings which you can feed into sokol-gfx in the sg_make_shader() call. The compilation of HLSL and MSL to byte code happens inside sokol_gfx.h then (and this compilation step needs to happen either way, because SPIRVCross cannot generate HLSL or Metal bytecode directly - so letting sokol_gfx.h do it isn't any less efficient than doing it yourself for a hot reload situation).

How to deal with reflection is indeed an interesting question though. One 'naive' way would be to have a new function which directly returns a populated sg_shader_desc struct, but that would make sokol-shdc dependent on sokol_gfx.h, which really isn't a good idea.

A cleaner way would be to have a new 'user-friendly' reflection struct which makes it easy to build a reflection_to_shader_desc() helper function on the user side.

[kkukshtel]

Ah thanks for the clarification!

but that would make sokol-shdc dependent on sokol_gfx.h, which really isn't a good idea.

One possible option here could be to provide a header similar to sokol_glue.h that acts as a go-between when both gfx and the (not yet existent) sokol_shdc.h are present that can return a populated sg_shader_desc struct?