Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FFIgenPad #1390

Draft
wants to merge 123 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
67b39c8
[ffigenpad] init project
TheComputerM Jul 25, 2024
af7d2e4
[ffigenpad] add libclang.wasm bindings
TheComputerM Jul 25, 2024
4ce5c06
[ffigenpad] fix ffi.Struct generation
TheComputerM Jul 25, 2024
e862969
[ffigenpad] fix missing publish_to in pubspec
TheComputerM Jul 25, 2024
9bf1b10
[ffigenpad] add remaining functions
TheComputerM Jul 25, 2024
a40ca94
[ffigenpad] add website skeleton
TheComputerM Jul 25, 2024
307e760
[ffigenpad] functional libclang app
TheComputerM Jul 25, 2024
46e3e23
[ffigenpad] port some utils
TheComputerM Jul 26, 2024
79cb95e
[ffigenpad] add wrapper functions
TheComputerM Jul 26, 2024
5e6c243
[ffigenpad] add llvm build folder
TheComputerM Jul 26, 2024
bd0122b
[ffigenpad] format wrapper.c
TheComputerM Jul 26, 2024
46ceb47
[ffigenpad] more wrapper functions
TheComputerM Jul 26, 2024
ed74146
[ffigenpad] add build script for libclang.wasm
TheComputerM Jul 26, 2024
4069841
[ffigenpad] more wrapper functions
TheComputerM Jul 26, 2024
fd84f55
[ffigenpad] fix build scripts
TheComputerM Jul 26, 2024
b5adb4f
[ffigenpad] generate enums as integers
TheComputerM Jul 27, 2024
3f38614
[ffigenpad] add clang_types stub
TheComputerM Jul 27, 2024
aa15b0d
[ffigenpad] add logging package
TheComputerM Jul 27, 2024
0de06c0
[ffigenpad] moar functions
TheComputerM Jul 27, 2024
3940d70
[ffigenpad] create a Clang class
TheComputerM Jul 27, 2024
22b5f83
[ffigenpad] remove wasm function
TheComputerM Jul 29, 2024
b8111ec
[ffigenpad] is life just copy pasting
TheComputerM Jul 29, 2024
463985e
[ffigenpad] modify visitChildren wrapper
TheComputerM Jul 29, 2024
1155d8e
[ffigenpad] rename to calloc and add emscriptens function interop
TheComputerM Aug 1, 2024
35e5f0e
[ffigenpad] test out visitChildren
TheComputerM Aug 1, 2024
bf85ca5
Merge branch 'main' into ffigenpad
TheComputerM Aug 1, 2024
3bed03c
[ffigenpad] moar utils
TheComputerM Aug 2, 2024
6180832
[ffigenpad] add data.dart
TheComputerM Aug 2, 2024
d0bbe2b
[ffigenpad] add code generator
TheComputerM Aug 2, 2024
75e6994
[ffigenpad] add config
TheComputerM Aug 2, 2024
a93e9fa
[ffigenpad] add more functions
TheComputerM Aug 2, 2024
e27bbbd
[ffigenpad] moar functions
TheComputerM Aug 3, 2024
0745838
[ffigenpad] more interop
TheComputerM Aug 3, 2024
6dd84d5
[ffigen] add header_parser
TheComputerM Aug 3, 2024
7c1622f
[ffigenpad] add code gen library.dart
TheComputerM Aug 3, 2024
7cfcc6e
[ffigenpad] add parsers
TheComputerM Aug 3, 2024
482bcd0
[ffigenpad] port FfiGen class
TheComputerM Aug 5, 2024
b1f3c8e
[ffigenpad] IOOverrides for emscriptens MemFS
TheComputerM Aug 5, 2024
17d3160
[ffigenpad] demo run
TheComputerM Aug 6, 2024
810a15d
[ffigenpad] do not export ccall
TheComputerM Aug 6, 2024
6299659
[ffigenpad] enable wasm bigint
TheComputerM Aug 6, 2024
1234beb
[ffigenpad] some workarounds
TheComputerM Aug 6, 2024
00bf3d5
[ffigenpad] convert to a getter
TheComputerM Aug 6, 2024
b3683d8
[ffigenpad] name the libclang asset
TheComputerM Aug 6, 2024
08093ef
Merge branch 'main' into ffigenpad
TheComputerM Aug 6, 2024
c8f4c7e
[ffigenpad] enable output formatting
TheComputerM Aug 6, 2024
e8aabd4
[ffigenpad] add yaml config provider
TheComputerM Aug 6, 2024
767a31b
[ffigenpad] add std libs to memfs
TheComputerM Aug 6, 2024
6e08436
[ffigenpad] yaml config through args
TheComputerM Aug 7, 2024
40aa56e
[ffigenpad] embed instead of preload
TheComputerM Aug 7, 2024
16ea78d
[ffigenpad] format generated file
TheComputerM Aug 7, 2024
707ddd7
[ffigenpad] add park-ui
TheComputerM Aug 7, 2024
1b9dd4e
[ffigenpad] add logging display
TheComputerM Aug 8, 2024
5fb9226
[ffigenpad] add typescript
TheComputerM Aug 8, 2024
29f2420
[ffigenpad] add basic ui
TheComputerM Aug 8, 2024
cbd724b
[ffigenpad] add worker as environment
TheComputerM Aug 8, 2024
30ccf19
[ffigenpad] detect nullptr when converting to String
TheComputerM Aug 9, 2024
599c7f3
[ffigenpad] add CXFile typedef
TheComputerM Aug 9, 2024
0fe1b5d
[ffigenpad] rebuild libclang
TheComputerM Aug 9, 2024
4a2770d
[ffigenpad] optimise libclang.wasm
TheComputerM Aug 9, 2024
f76f384
[ffigenpad] libclang.wasm only works till -O2
TheComputerM Aug 9, 2024
e939fe9
[ffigenpad] -O3 build
TheComputerM Aug 9, 2024
6e825dc
[ffigenpad] mock website
TheComputerM Aug 9, 2024
ed4c0b2
[ffigenpad] add logs all at once
TheComputerM Aug 9, 2024
28eb19e
[ffigenpad] change font
TheComputerM Aug 9, 2024
f9bb818
[ffigenpad] filter logs
TheComputerM Aug 9, 2024
c422826
[ffigenpad] remove console.log
TheComputerM Aug 9, 2024
dfa6ad4
[ffigenpad] rename to navbar
TheComputerM Aug 9, 2024
32da58a
[ffigenpad] clean up website
TheComputerM Aug 9, 2024
6e3ffbf
[ffigenpad] update packages
TheComputerM Aug 9, 2024
1173d97
[ffigenpad] add basic file tree
TheComputerM Aug 9, 2024
7629f14
[ffigenpad] fix ts errors
TheComputerM Aug 9, 2024
c62228e
[ffigenpad] add operations to fs
TheComputerM Aug 10, 2024
ddee6ae
[ffigenpad] remove nanostores
TheComputerM Aug 10, 2024
c1e8ee9
[ffigenpad] export FS events
TheComputerM Aug 10, 2024
d7e5a84
[ffigenpad] complete fs interop
TheComputerM Aug 10, 2024
ef7dd9e
[ffigenpad] lazyload big components
TheComputerM Aug 10, 2024
65c4cf5
[ffigenpad] make list not grow
TheComputerM Aug 10, 2024
1c914f5
[ffigenpad] add dark mode
TheComputerM Aug 11, 2024
0a1d16c
[ffigenpad] fix portal issue
TheComputerM Aug 11, 2024
64e3d54
[ffigenpad] make file explorer more performant
TheComputerM Aug 11, 2024
9df674a
[ffigenpad] minor qol
TheComputerM Aug 11, 2024
bb901de
[ffigenpad] add file upload
TheComputerM Aug 11, 2024
d93dfe1
[ffigenpad] remove unused imports
TheComputerM Aug 11, 2024
c832daf
[ffigenpad] add more info to navbar
TheComputerM Aug 11, 2024
4a50419
[ffigenpad] remove unused imports
TheComputerM Aug 11, 2024
9100d5f
[ffigenpad] change wiki icon
TheComputerM Aug 11, 2024
fcf5d22
[ffigenpad] refactor FS functions
TheComputerM Aug 11, 2024
4877bd9
[ffigenpad] refactor logs table
TheComputerM Aug 12, 2024
e378701
[ffigenpad] generate bindings on load
TheComputerM Aug 12, 2024
fad58d0
[ffigenpad] move ffigen docs link
TheComputerM Aug 12, 2024
7c8c7ae
[ffigenpad] change web app theme
TheComputerM Aug 12, 2024
b875ff2
Merge branch 'dart-lang:main' into ffigenpad
TheComputerM Aug 12, 2024
8d2d296
Merge branch 'ffigenpad' of https://github.com/TheComputerM/dart-nati…
TheComputerM Aug 12, 2024
c42f950
[ffigenpad] refactor navbar
TheComputerM Aug 12, 2024
1f34b13
[ffigenpad] organize imports
TheComputerM Aug 14, 2024
c9dab50
Merge branch 'main' into ffigenpad
TheComputerM Aug 14, 2024
a5bb1ab
[ffigenpad] add copyright headers to dart files
TheComputerM Aug 14, 2024
1596ae3
Merge branch 'main' into ffigenpad
TheComputerM Aug 15, 2024
e43f341
[ffigenpad] update packages and comment
TheComputerM Aug 18, 2024
e5af39a
[ffigenpad] add setup.dart
TheComputerM Aug 19, 2024
9fec5c5
[ffigenpad] change libclang build path
TheComputerM Aug 19, 2024
62ffe45
[ffigenpad] merge into main function
TheComputerM Aug 19, 2024
8e73e43
[ffigenpad] add copyright headers
TheComputerM Aug 19, 2024
f5b4f5e
[ffigenpad] update README
TheComputerM Aug 19, 2024
8e266bc
[ffigenpad] update packages
TheComputerM Aug 19, 2024
7c5fc5c
Merge branch 'dart-lang:main' into ffigenpad
TheComputerM Aug 19, 2024
945e1e0
[ffigenpad] add deploy to github actios
TheComputerM Aug 19, 2024
60d9799
[ffigenpad] change paths
TheComputerM Aug 20, 2024
8485fdc
[ffigenpad] add optimize build
TheComputerM Aug 20, 2024
9ef321d
[ffigenpad] remove build files
TheComputerM Aug 20, 2024
4df8cbf
[ffigenpad] remove gitignore
TheComputerM Aug 20, 2024
a5dac1f
[ffigenpad] optimize builds
TheComputerM Aug 20, 2024
6daf9a6
[ffigenpad] add llvm-project license
TheComputerM Aug 20, 2024
c3d84d5
[ffigenpad] move park-ui components to third_party
TheComputerM Aug 20, 2024
b39a9bf
[ffigenpad] tidy up
TheComputerM Aug 23, 2024
06925e6
[ffigenpad] correct command
TheComputerM Aug 23, 2024
01eacb6
[ffigenpad] add details on project structure
TheComputerM Aug 24, 2024
4806041
add missing entry
TheComputerM Aug 24, 2024
19d19ff
refactor
TheComputerM Sep 1, 2024
f4b654f
ffigenpad: refactor based on lint rules
TheComputerM Sep 3, 2024
1755e12
Merge branch 'dart-lang:main' into ffigenpad
TheComputerM Sep 3, 2024
2312d63
[ffigenpad] cleanup
TheComputerM Sep 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/ffigenpad.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: ffigenpad

on:
pull_request:
branches: [main]
paths:
- ".github/workflows/ffigenpad.yaml"
- "pkgs/ffigenpad/**"
push:
branches: [main]
paths:
- ".github/workflows/ffigenpad.yaml"
- "pkgs/ffigenpad/**"

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9a9194f87191a7e9055e3e9b95b8cfb13023bb08

- uses: dart-lang/setup-dart@e9a814f39d5452701455a47847735e0f482026c8
with:
sdk: dev

- run: dart pub get
working-directory: pkgs/ffigenpad

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run dart analyze and dart format on the code on the CI here.

- uses: mymindstorm/setup-emsdk@6ab9eb1bda2574c4ddb79809fc9247783eaf9021
with:
version: 3.1.61

- name: setup ffigenpad
working-directory: pkgs/ffigenpad
run: dart run tool/setup.dart

- name: build libclang.wasm
working-directory: pkgs/ffigenpad
run: dart run tool/build_libclang.dart --optimize

- name: build ffigenpad
working-directory: pkgs/ffigenpad
run: |
dart compile wasm ./lib/ffigenpad.dart -o ./bin/ffigenpad.wasm -O3 \
--extra-compiler-option=--enable-experimental-ffi \
--extra-compiler-option=--enable-experimental-wasm-interop

- uses: pnpm/action-setup@ac5bf11548bf5e19b8aadb8182072616590fa4a6
with:
version: 9.7.0
run_install: |
- cwd: pkgs/ffigenpad/web

- uses: actions/configure-pages@aabcbc432d6b06d1fd5e8bf3cf756880c35e014d

- name: build website
working-directory: pkgs/ffigenpad/web
run: pnpm build

- uses: actions/upload-pages-artifact@1780dfc2cece65a782cc86fa133f96aef8ad0345
with:
path: pkgs/ffigenpad/web/dist

deploy:
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gives an error here according to the editor, but seems to be consistent with the documentation.

url: ${{ steps.deployment.outputs.page_url }}
steps:
- uses: actions/deploy-pages@b74272834adc04f971da4b0b055c49fa8d7f90c9
3 changes: 3 additions & 0 deletions pkgs/ffigenpad/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## 0.1.0

- Initial version.
63 changes: 63 additions & 0 deletions pkgs/ffigenpad/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# FFIgenPad

A web interface in which you paste a C header in a textbox on the left, and it outputs `dart:ffi` bindings generated by `package:ffigen` in a textbox on the right. (Inspiration is https://godbolt.org/ which does this for a C file on the left and assembly on the right.)

It doesn't need a server cause it compiles the Dart code in `package:ffigen` and libclang to WASM. That way everything can run in the browser.

------

## Project Structure

Most of the files in `lib/src/` are copied from the source of *ffigen* with modifications to make it compatible with dart2wasm (many of the workarounds are listed in [this blog article](https://thecomputerm.hashnode.dev/dirty-deeds-done-dart-cheap-experiments-with-dart2wasm)).

However the files listed do not have an ffigen counterpart:
- [./lib/ffigenpad.dart](./lib/ffigenpad.dart)
- [./lib/memfs.dart](./lib/memfs.dart)
- [./lib/src/header_parser/calloc.dart](./lib/src/header_parser/calloc.dart)
- [./lib/src/header_parser/clang_bindings/clang_types.dart](./lib/src/header_parser/clang_bindings/clang_types.dart)
- [./lib/src/header_parser/clang_bindings/clang_wrapper.dart](./lib/src/header_parser/clang_bindings/clang_wrapper.dart)

## Building

### Prerequisites

- dart
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal version? (Fine to mention just today's dev release if that's good enough.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works with dart stable v3.5.1, should I also update it in github actions?

- emscripten (>= 3.1.61)
- for website
- nodejs (>= ^20.17.0)
- pnpm (optional but recommended)

### Steps

- Run `tool/setup.dart` to download LLVM archive files to build *libclang.wasm*

```sh
# in project root
dart run tool/setup.dart
```

- Run `tool/build_libclang.dart` to build *libclang.wasm* using functions exported from *third_party/libclang/libclang.exports*

```sh
# in project root
dart run tool/build_libclang.dart
```

- Build FFIgenPad using the experimental dart2wasm compiler

```sh
# in project root
dart compile wasm ./lib/ffigenpad.dart -o ./bin/ffigenpad.wasm \
--extra-compiler-option=--enable-experimental-ffi \
--extra-compiler-option=--enable-experimental-wasm-interop
```

- To build the website

```sh
# in project root
cd web
npm i # install dependencies
TheComputerM marked this conversation as resolved.
Show resolved Hide resolved
npm run build
TheComputerM marked this conversation as resolved.
Show resolved Hide resolved
# preview with: npm run preview
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

```
33 changes: 33 additions & 0 deletions pkgs/ffigenpad/analysis_options.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# This file configures the static analysis results for your project (errors,
# warnings, and lints).
#
# This enables the 'recommended' set of lints from `package:lints`.
# This set helps identify many issues that may lead to problems when running
# or consuming Dart code, and enforces writing Dart using a single, idiomatic
# style and format.
#
# If you want a smaller set of lints you can change this to specify
# 'package:lints/core.yaml'. These are just the most critical lints
# (the recommended set includes the core lints).
# The core lints are also what is used by pub.dev for scoring packages.

include: package:dart_flutter_team_lints/analysis_options.yaml

# Uncomment the following section to specify additional rules.

linter:
rules:
implementation_imports: false

analyzer:
errors:
todo: ignore
language:
strict-casts: true
strict-inference: true
strict-raw-types: true
# For more information about the core and recommended set of lints, see
# https://dart.dev/go/core-lints

# For additional information about configuring this file, see
# https://dart.dev/guides/language/analysis-options
3 changes: 3 additions & 0 deletions pkgs/ffigenpad/bin/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*.wasm
*.mjs
*.wasm.map
3 changes: 3 additions & 0 deletions pkgs/ffigenpad/bin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Overview

Contains build outputs for *libclang.wasm* and *ffigenpad.wasm*.
38 changes: 38 additions & 0 deletions pkgs/ffigenpad/lib/ffigenpad.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'dart:io';
import 'dart:js_interop';
import 'dart:js_interop_unsafe';

import 'package:logging/logging.dart';
import 'package:yaml/yaml.dart';

import 'memfs.dart';
import 'src/config_provider.dart';
import 'src/ffigen.dart';

@JS()
external void setLogs(JSObject logs);

void generate(String yaml) {
final ffigen = FfiGen(logLevel: Level.ALL);
final config = YamlConfig.fromYaml(loadYaml(yaml) as YamlMap);
ffigen.run(config);
}

void main(List<String> args) {
final logs = <JSObject>[];
Logger.root.onRecord.listen((record) {
final log = JSObject();
log.setProperty('level'.toJS, (record.level.value / 100).toJS);
log.setProperty('message'.toJS, record.message.toJS);

logs.add(log);
});
IOOverrides.runWithIOOverrides(() {
generate(args.first);
}, MemFSIOOverrides());
setLogs(logs.toJS);
}
122 changes: 122 additions & 0 deletions pkgs/ffigenpad/lib/memfs.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'dart:convert' as convert;
import 'dart:io';
import 'dart:js_interop';
import 'dart:js_interop_unsafe';
import 'dart:typed_data';

// adapted functions from https://emscripten.org/docs/api_reference/Filesystem-API.html#id2
extension type MemFS(JSObject _) implements JSObject {
external JSArray<JSString> readdir(String path);
external JSUint8Array readFile(String path, [JSObject? opts]);
external void writeFile(String path, String data);
external void unlink(String path);
external void mkdir(String path);
external void rmdir(String path);
external void rename(String oldpath, String newpath);
external String cwd();
external void chdir(String path);
external JSObject analyzePath(String path, bool dontResolveLastLink);
}

@JS('FS')
external MemFS get memfs;

class MemFSDirectory implements Directory {
@override
String path;

MemFSDirectory(this.path);

@override
void createSync({bool recursive = false}) {
memfs.mkdir(path);
}

@override
dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation);
}

class MemFSFile implements File {
@override
String path;

MemFSFile(this.path);

@override
MemFSFile get absolute => MemFSFile(path);

@override
void createSync({bool recursive = false, bool exclusive = false}) {
memfs.writeFile(path, '');
}

@override
void deleteSync({bool recursive = false}) {
memfs.unlink(path);
}

@override
bool existsSync() {
return memfs
.analyzePath(path, false)
.getProperty<JSBoolean>('exists'.toJS)
.toDart;
}

@override
void writeAsStringSync(String contents,
{FileMode mode = FileMode.write,
convert.Encoding encoding = convert.utf8,
bool flush = false}) {
memfs.writeFile(path, contents);
}

@override
Uint8List readAsBytesSync() {
return memfs.readFile(path).toDart;
}

@override
String readAsStringSync({convert.Encoding encoding = convert.utf8}) {
return encoding.decode(readAsBytesSync());
}

@override
dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation);
}

class MemFSIOOverrides extends IOOverrides {
@override
MemFSDirectory createDirectory(String path) {
return MemFSDirectory(path);
}

@override
MemFSFile createFile(String path) {
return MemFSFile(path);
}

@override
bool fsWatchIsSupported() {
return false;
}

@override
void setCurrentDirectory(String path) {
memfs.chdir(path);
}

@override
MemFSDirectory getCurrentDirectory() {
return MemFSDirectory(memfs.cwd());
}

@override
MemFSDirectory getSystemTempDirectory() {
return MemFSDirectory('/tmp');
}
}
29 changes: 29 additions & 0 deletions pkgs/ffigenpad/lib/src/code_generator.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

/// Generates FFI bindings for a given Library.
library;

export 'code_generator/binding.dart';
export 'code_generator/compound.dart';
export 'code_generator/constant.dart';
export 'code_generator/enum_class.dart';
export 'code_generator/func.dart';
export 'code_generator/func_type.dart';
export 'code_generator/global.dart';
export 'code_generator/handle.dart';
export 'code_generator/imports.dart';
export 'code_generator/library.dart';
export 'code_generator/native_type.dart';
export 'code_generator/objc_block.dart';
export 'code_generator/objc_built_in_functions.dart';
export 'code_generator/objc_interface.dart';
export 'code_generator/objc_methods.dart';
export 'code_generator/objc_nullable.dart';
export 'code_generator/objc_protocol.dart';
export 'code_generator/pointer.dart';
export 'code_generator/struct.dart';
export 'code_generator/type.dart';
export 'code_generator/typealias.dart';
export 'code_generator/union.dart';
Loading