Add support for all CLP FFI functionality and refactor existing code: (…

…#3)

- Add CLP encoding/decoding functionalities
- Add CLP IR wildcard query support
- Add basic GitHub templates and workflows
- Rewrite CLP IR serialization/deserialization functionalities to match the latest CLP glossary, and simplify resource management
- Add reader/writer interfaces for CLP IR
- Update formatting and lining tooling
- Switch to use static library

.github/ISSUE_TEMPLATE/bug-report.yml

@@ -0,0 +1,40 @@
+name: "Bug Report"
+description: Report software deficiencies
+labels: ["bug"]
+body:
+- type: markdown
+  attributes:
+    value: |
+      Use this form to report any functional or performance bugs you've found in the software.
+
+      Be sure to check if your [issue](https://github.com/y-scope/clp-ffi-go/issues) has already been reported.
+
+- type: textarea
+  attributes:
+    label: Bug
+    description: "Describe what's wrong and if applicable, what you expected instead."
+  validations:
+    required: true
+
+- type: input
+  attributes:
+    label: clp-ffi-go version
+    description: "The release version number or development commit hash that has the bug."
+    placeholder: "Version number or commit hash"
+  validations:
+    required: true
+
+- type: textarea
+  attributes:
+    label: Environment
+    description: "The environment in which you're running/using clp-ffi-go."
+    placeholder: "OS version, docker version, etc."
+  validations:
+    required: true
+
+- type: textarea
+  attributes:
+    label: Reproduction steps
+    description: "List each step required to reproduce the bug."
+  validations:
+    required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/feature-request.yml

@@ -0,0 +1,23 @@
+name: "Feature/Change Request"
+description: Request a feature or change
+labels: ["enhancement"]
+body:
+- type: markdown
+  attributes:
+    value: |
+      Use this form to request a feature/change in the software, or the project as a whole. 
+
+- type: textarea
+  attributes:
+    label: Request
+    description: "Describe your request and why it's important."
+  validations:
+    required: true
+
+- type: textarea
+  attributes:
+    label: Possible implementation
+    description: "Describe any implementations you have in mind."
+  validations:
+    required: true
+

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,9 @@
+# References
+<!-- Any issues or pull requests relevant to this pull request -->
+
+# Description
+<!-- Describe what this request will change/fix and provide any details necessary for reviewers -->
+
+# Validation performed
+<!-- What tests and validation you performed on the change -->
+

.github/workflows/build.yml

@@ -0,0 +1,66 @@
+name: Build
+
+on:
+  pull_request:
+  push:
+  workflow_call:
+
+# TODO: add separate jobs for building, linting, and testing c++
+jobs:
+  prebuilt-test:
+    strategy:
+      matrix:
+        os: [macos-latest, ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: recursive
+
+      - uses: actions/setup-go@v4
+        with:
+          go-version: '1.20.x'
+          check-latest: true
+
+      - run: go clean -cache && go build ./...
+
+      - run: go test -count=1 ./...
+
+  build-lint-test:
+    strategy:
+      matrix:
+        os: [macos-latest, ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: recursive
+
+      - uses: actions/setup-go@v4
+        with:
+          go-version: '1.20.x'
+          check-latest: true
+
+      - if: ${{ 'macos-latest' == matrix.os }}
+        run: |
+          brew update
+          brew install llvm
+
+      - name: Remove repo's generated c++ libraries and go code
+        run: |
+          rm ./lib/* ./**/*_string.go
+
+      - run: |
+          go install mvdan.cc/gofumpt@latest
+          go install github.com/segmentio/golines@latest
+          go install golang.org/x/tools/cmd/stringer@latest
+
+      - run: go clean -cache && go generate ./...
+
+      - run: |
+          diff="$(golines -m 100 -t 4 --base-formatter='gofumpt' --dry-run .)"
+          if [[ -n "$diff"  ]]; then echo "$diff"; exit 1; fi
+
+      # - run: cmake -S ./cpp -B ./cpp/build -DCMAKE_EXPORT_COMPILE_COMMANDS=1
+
+      - run: go test -count=1 ./...

.gitignore

@@ -1 +1,4 @@
 build/
+cpp/.cache/
+cpp/clp/
+**/compile_commands.json

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "clp"]
-	path = cpp/clp
-	url = [email protected]:y-scope/clp.git

BUILD.bazel

@@ -1,53 +1,18 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
 load("@rules_cc//cc:defs.bzl", "cc_library")
 
 cc_library(
     name = "libclp_ffi",
-    srcs = select({
-        "@io_bazel_rules_go//go/platform:android_amd64": [
-            "lib/libclp_ffi_linux_amd64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:android_arm64": [
-            "lib/libclp_ffi_linux_arm64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:darwin_amd64": [
-            "lib/libclp_ffi_darwin_amd64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:darwin_arm64": [
-            "lib/libclp_ffi_darwin_arm64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:ios_amd64": [
-            "lib/libclp_ffi_darwin_amd64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:ios_arm64": [
-            "lib/libclp_ffi_darwin_arm64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:linux_amd64": [
-            "lib/libclp_ffi_linux_amd64.so",
-        ],
-        "@io_bazel_rules_go//go/platform:linux_arm64": [
-            "lib/libclp_ffi_linux_arm64.so",
-        ],
-        "//conditions:default": [],
-    }),
-    hdrs = glob([
-        "cpp/src/**/*.h",
-    ]),
+    srcs = glob(["cpp/src/ffi_go/**"]) + [
+    ],
+    hdrs = glob(["cpp/src/ffi_go/**/*.h"]),
     includes = [
         "cpp/src",
     ],
-    visibility = ["//visibility:public"],
-)
-
-go_library(
-    name = "clp-ffi-go",
-    srcs = ["generate.go"],
-    importpath = "github.com/y-scope/clp-ffi-go",
-    visibility = ["//visibility:public"],
-)
-
-alias(
-    name = "go_default_library",
-    actual = ":clp-ffi-go",
+    deps = [
+        "@com_github_y_scope_clp//:libclp_core",
+    ],
+    copts = [
+        "-std=c++20",
+    ],
     visibility = ["//visibility:public"],
 )

LICENSE

@@ -1,3 +1,4 @@
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
@@ -186,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2022 YScope Inc.
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.rst

@@ -15,7 +15,7 @@ Getting started
 To add the module to your project run: ``go get github.com/y-scope/clp-ffi-go``
 
 Here's an example showing how to decode each log event containing "ERROR" from
-a CLP IR stream.
+a CLP IR byte stream.
 
 .. code:: golang
 
@@ -24,21 +24,29 @@ a CLP IR stream.
     "time"
 
     "github.com/klauspost/compress/zstd"
+    "github.com/y-scope/clp-ffi-go/ffi"
     "github.com/y-scope/clp-ffi-go/ir"
   )
 
   file, _ := os.Open("log-file.clp.zst")
+  defer file.Close()
   zstdReader, _ := zstd.NewReader(file)
+  defer zstdReader.Close()
+  irReader, _ := ir.NewReader(zstdReader)
+  defer irReader.Close()
 
-  irReader, _ := ir.ReadPreamble(zstdReader, 4096)
+  var err error
   for {
-    // To read every log event replace ReadToContains with
-    // ReadNextLogEvent(zstdReader)
-    log, err := irReader.ReadToContains(zstdReader, []byte("ERROR"))
-    if ir.Eof == err || io.EOF == err {
+    var log *ffi.LogEventView
+    // To read every log event replace ReadToContains with Read()
+    log, err = irReader.ReadToContains("ERROR")
+    if nil != err {
       break
     }
-    fmt.Printf("%v %v", time.UnixMilli(int64(log.Timestamp)), string(log.Msg))
+    fmt.Printf("%v %v", time.UnixMilli(int64(log.Timestamp)), log.LogMessageView)
+  }
+  if ir.EndOfIr != err {
+    fmt.Printf("Reader.Read failed: %v", err)
   }
 
 Building
@@ -49,15 +57,11 @@ as well as stringify ``Enum`` style types.
 1. Install requirements:
 
    a. A C++ compiler that supports C++17
-   #. CMake 3.5.1 or higher
+   #. CMake 3.11 or higher
    #. The Stringer tool: https://pkg.go.dev/golang.org/x/tools/cmd/stringer
 
       - ``go install golang.org/x/tools/cmd/stringer@latest``
 
-#. ``git submodule update --init --recursive``
-
-   - Pull all submodules in preparation for building
-
 #. ``go generate ./...``
 
    - Run all generate directives (note the 3 dots after '/')
@@ -70,22 +74,22 @@ arguments or modifications.
 
 __ https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/rules.md#go_library-deps
 
-Testing
--------
-To run all unit tests run: ``go test ./... -args $(readlink -f clp-ir-stream.clp.zst)``
-
-- The ``ir`` package's tests currently requries an existing CLP IR file
-  compressed with zstd. This file's path is taken as the only argument to the
-  test and is supplied after ``-args``. It can be an absolute path or a path
-  relative to the ``ir`` directory.
-
 Why not build with cgo?
 '''''''''''''''''''''''
 The primary reason we choose to build with CMake rather than directly with cgo,
 is to ease code maintenance by maximizing the reuse of CLP's code with no
 modifications. If a platform you use is not supported by the pre-built
 libraries, please open an issue and we can integrate it into our build process.
 
+Testing
+-------
+To run all unit tests run: ``go_test_ir="/path/to/my-ir.clp.zst" go test ./...``
+
+- Some of the ``ir`` package's tests currently require an existing CLP IR file
+  compressed with zstd. This file's path is taken as an environment variable
+  named ``go_test_ir``. It can be an absolute path or a path relative to the
+  ``ir`` directory.
+
 Using an external C++ library
 -----------------------------
 Use the ``external`` build tag to link with different CLP FFI library instead
@@ -100,5 +104,6 @@ For example, to run the tests using the ``external`` you can run:
 
 .. code:: bash
 
-  CGO_LDFLAGS="-L./lib -lclp_ffi_linux_amd64 -lstdc++" \
-  go test -tags external,test ./... -args $(readlink -f clp-ir-stream.clp.zst)
+  CGO_LDFLAGS="-L/path/to/external_libs -lclp_ffi_linux_amd64 -Wl,-rpath=/path/to/external_libs" \
+  go_test_ir="/path/to/my-ir.clp.zst" \
+  go test -tags external ./...