reboot wit-bindgen-go (#1447)

* reboot `wit-bindgen-go`

This re-introduces Go support, with a new implementation written from scratch.
While the previous incarnation targeted TinyGo and reused the C generator for
lifting and lowering, this one targets "big" Go and handles lifting and lowering
itself.  In addition, it supports idiomatic, goroutine-based concurrency on top
of the component model async ABI, including streams and futures.

This implementation is also distinct from the
[go-modules](https://github.com/bytecodealliance/go-modules) project, which has
a number of limitations (e.g. no async support, not safe for use with "big" Go)
and is no longer being actively maintained.

Note that the async support currently requires [a small
patch](dicej/go@a1c8322)
to the Go `runtime` library.  I plan to work with the upstream project to make
that patch unecessary in the future.  Components that don't use async features
should work with stock, unpatched Go.

One of the tricky parts about lowering values and passing pointers to
e.g. `{stream,future}.{read,write}` is that we must tell the Go garbage
collector to pin the pointers (i.e. mark the pointee as both immovable and
uncollectable) for as long as the host has access to them, but then unpin them
promptly afterward to avoid leaks.  We do this using `runtime.Pinner` instances,
most of which are locally scoped and easy to reason about.  However, we must use
a couple of globally-scoped pinners as well: one for `cabi_realloc` allocations
by the host when it lowers parameters (which we unpin after lifting those
parameters) and another for returning values from sync-lifted exports (which we
unpin in post-return functions).

There are a couple of known missing features which I'll open GitHub issues for:

1. Resource handles are not being restored for unwritten items in the case of an incomplete stream or future write.
2. Importing and/or exporting multiple versions of the same package will cause name clashes.

In addition, I plan to expand the test coverage beyond the basics covered in
this commit.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

add ping-pong Go tests

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

address review feedback

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

add wit-bindgen-go to ci/publish.rs

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

enable Go in test CI matrix

I'm assuming the default version of Go on the GitHub worker image will be
sufficient; if not, we can install a different one explicitly.

I'm not enabling the async tests for Go yet, since that will require publishing
and installing a build of Go with [this
patch](dicej/go@a1c8322);
I'll need to do some homework to find out the best way to do that.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

fix codegen tests

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

update Go runtime tests to use `run` export

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

run cargo fmt

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

remove `publish = false` from crates/go/Cargo.toml

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

temporarily skip async codegen tests for Go

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

add `async = true` config directive to `async-trait-function.wit`

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

indicate that `async-trait-function.wit-no-std` works for Rust

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

add error contexts to `Go::compile`

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

add `actions/setup-go` to workflow

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

add finalizers where appropriate; tweak `StreamReader` API

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

update workspace to 2024 edition

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* remove indirection in `wit_types.Option`

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* add crates/go/README.md

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

---------

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

Author: dicej

.github/workflows/main.yml

@@ -59,7 +59,7 @@ jobs:
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
         # moonbit removed from language matrix for now - causing CI failures
-        lang: [c, rust, csharp, cpp]
+        lang: [c, rust, csharp, cpp, go]
         exclude:
           # For now csharp doesn't work on macos, so exclude it from testing.
           - os: macos-latest
@@ -85,6 +85,12 @@ jobs:
         dotnet-version: '9.x'
       if: matrix.lang == 'csharp'
 
+    - name: Setup Go
+      uses: actions/setup-go@v5
+      with:
+        go-version: 1.25.4
+      if: matrix.lang == 'go'
+
       # Hacky work-around for https://github.com/dotnet/runtime/issues/80619
     - run: dotnet new console -o /tmp/foo
       if: matrix.os != 'windows-latest' && matrix.lang == 'csharp'

Cargo.lock

@@ -15,11 +15,11 @@ CLI tool to generate bindings for WIT documents and the component model.
 resolver = "2"
 
 [workspace.package]
-edition = "2021"
+edition = "2024"
 version = "0.49.0"
 license = "Apache-2.0 WITH LLVM-exception OR Apache-2.0 OR MIT"
 repository = "https://github.com/bytecodealliance/wit-bindgen"
-rust-version = "1.82.0"
+rust-version = "1.85.0"
 
 [workspace.dependencies]
 anyhow = "1.0.72"
@@ -48,6 +48,7 @@ wit-bindgen-rust = { path = "crates/rust", version = "0.49.0" }
 wit-bindgen-csharp = { path = 'crates/csharp', version = '0.49.0' }
 wit-bindgen-markdown = { path = 'crates/markdown', version = '0.49.0' }
 wit-bindgen-moonbit = { path = 'crates/moonbit', version = '0.49.0' }
+wit-bindgen-go = { path = 'crates/go', version = '0.49.0' }
 wit-bindgen = { path = 'crates/guest-rust', version = '0.49.0', default-features = false }
 wit-bindgen-test = { path = 'crates/test', version = '0.49.0' }
 
@@ -64,6 +65,7 @@ wit-bindgen-cpp = { workspace = true, features = ['clap'], optional = true }
 wit-bindgen-markdown = { workspace = true, features = ['clap'], optional = true }
 wit-bindgen-moonbit = { workspace = true, features = ['clap'], optional = true }
 wit-bindgen-csharp = { workspace = true, features = ['clap'], optional = true }
+wit-bindgen-go = { workspace = true, features = ['clap'], optional = true }
 wit-bindgen-test = { workspace = true }
 wit-component = { workspace = true }
 wasm-encoder = { workspace = true }
@@ -84,7 +86,7 @@ c = ['dep:wit-bindgen-c']
 cpp = ['dep:wit-bindgen-cpp']
 rust = ['dep:wit-bindgen-rust']
 markdown = ['dep:wit-bindgen-markdown']
-go = []
+go = ['dep:wit-bindgen-go']
 csharp = ['dep:wit-bindgen-csharp']
 csharp-mono = ['csharp']
 moonbit = ['dep:wit-bindgen-moonbit']

Cargo.toml

@@ -362,7 +362,11 @@ Now you can build with:
 dotnet publish
 ```
 
-Checkout out [componentize-dotnet](https://github.com/bytecodealliance/componentize-dotnet) for a simplified experience.
+Check out [componentize-dotnet](https://github.com/bytecodealliance/componentize-dotnet) for a simplified experience.
+
+### Guest: Go
+
+See the [wit-bindgen-go README.md](crates/go/README.md) for details and generating and using Go bindings.
 
 ### Guest: Java
 

README.md

@@ -24,6 +24,7 @@ const CRATES_TO_PUBLISH: &[&str] = &[
     "wit-bindgen-csharp",
     "wit-bindgen-markdown",
     "wit-bindgen-moonbit",
+    "wit-bindgen-go",
     "wit-bindgen-rust-macro",
     "wit-bindgen-rt",
     "wit-bindgen",
@@ -65,11 +66,13 @@ fn main() {
                 bump_version(&krate, &crates, name == "bump-patch");
             }
             // update the lock file
-            assert!(Command::new("cargo")
-                .arg("fetch")
-                .status()
-                .unwrap()
-                .success());
+            assert!(
+                Command::new("cargo")
+                    .arg("fetch")
+                    .status()
+                    .unwrap()
+                    .success()
+            );
         }
 
         "publish" => {

ci/publish.rs

@@ -56,7 +56,7 @@ pub unsafe extern "C" fn $realloc(
     align: usize,
     new_len: usize,
 ) -> *mut u8 {
-    crate::rt::cabi_realloc(old_ptr, old_len, align, new_len)
+    unsafe { crate::rt::cabi_realloc(old_ptr, old_len, align, new_len) }
 }
 EOF
 

ci/rebuild-libwit-bindgen-cabi.sh

@@ -10,8 +10,8 @@ use wit_bindgen_core::abi::{
     self, AbiVariant, Bindgen, Bitcast, Instruction, LiftLower, WasmSignature, WasmType,
 };
 use wit_bindgen_core::{
-    dealias, uwrite, uwriteln, wit_parser::*, AnonymousTypeGenerator, AsyncFilterSet, Direction,
-    Files, InterfaceGenerator as _, Ns, WorldGenerator,
+    AnonymousTypeGenerator, AsyncFilterSet, Direction, Files, InterfaceGenerator as _, Ns,
+    WorldGenerator, dealias, uwrite, uwriteln, wit_parser::*,
 };
 use wit_component::StringEncoding;
 

crates/c/src/lib.rs

@@ -3,8 +3,8 @@ use std::iter;
 
 pub use wit_parser::abi::{AbiVariant, FlatTypes, WasmSignature, WasmType};
 use wit_parser::{
-    align_to_arch, Alignment, ArchitectureSize, ElementInfo, Enum, Flags, FlagsRepr, Function,
-    Handle, Int, Record, Resolve, Result_, SizeAlign, Tuple, Type, TypeDefKind, TypeId, Variant,
+    Alignment, ArchitectureSize, ElementInfo, Enum, Flags, FlagsRepr, Function, Handle, Int,
+    Record, Resolve, Result_, SizeAlign, Tuple, Type, TypeDefKind, TypeId, Variant, align_to_arch,
 };
 
 // Helper macro for defining instructions without having to have tons of
@@ -806,6 +806,12 @@ pub fn guest_export_needs_post_return(resolve: &Resolve, func: &Function) -> boo
         .unwrap_or(false)
 }
 
+pub fn guest_export_params_have_allocations(resolve: &Resolve, func: &Function) -> bool {
+    func.params
+        .iter()
+        .any(|(_, t)| needs_deallocate(resolve, &t, Deallocate::Lists))
+}
+
 fn needs_deallocate(resolve: &Resolve, ty: &Type, what: Deallocate) -> bool {
     match ty {
         Type::String => true,
@@ -1134,7 +1140,10 @@ impl<'a, B: Bindgen> Generator<'a, B> {
                     for (param_name, ty) in func.params.iter() {
                         let Some(types) = flat_types(self.resolve, ty, Some(max_flat_params))
                         else {
-                            panic!("failed to flatten types during direct parameter lifting ('{param_name}' in func '{}')", func.name);
+                            panic!(
+                                "failed to flatten types during direct parameter lifting ('{param_name}' in func '{}')",
+                                func.name
+                            );
                         };
                         for _ in 0..types.len() {
                             self.emit(&Instruction::GetArg { nth: offset });

crates/core/src/abi.rs

@@ -1,4 +1,4 @@
-use anyhow::{bail, Result};
+use anyhow::{Result, bail};
 use std::collections::HashSet;
 use std::fmt;
 use wit_parser::{Function, FunctionKind, Resolve, WorldKey};

crates/core/src/async_.rs

@@ -148,7 +148,7 @@ pub trait InterfaceGenerator<'a> {
     fn type_record(&mut self, id: TypeId, name: &str, record: &Record, docs: &Docs);
     fn type_resource(&mut self, id: TypeId, name: &str, docs: &Docs);
     fn type_flags(&mut self, id: TypeId, name: &str, flags: &Flags, docs: &Docs);
-    fn type_tuple(&mut self, id: TypeId, name: &str, flags: &Tuple, docs: &Docs);
+    fn type_tuple(&mut self, id: TypeId, name: &str, tuple: &Tuple, docs: &Docs);
     fn type_variant(&mut self, id: TypeId, name: &str, variant: &Variant, docs: &Docs);
     fn type_option(&mut self, id: TypeId, name: &str, payload: &Type, docs: &Docs);
     fn type_result(&mut self, id: TypeId, name: &str, result: &Result_, docs: &Docs);