Skip to content

Commit

Permalink
feat: C daemon (#1084)
Browse files Browse the repository at this point in the history
* feat: add argparsing

* feat: checkin progress

* feat: add pkam auth

* feat: checking progress with sshnpd

* chore: restructure c packages

- moved 3rdparty/argparse up a level so it could be shared between sshnpd & srv

* chore: remove 'h' abbr from HELP flag

* feat: checkin socket-connector progress

* feat: checkin sshnpd progress

* feat: demoable socket_connector

hey... it's working... cool!

* chore: disable --bind-local-port for now

- much easier to not deal with it, and it's not needed for an mvp with sshnpd

* fix: stop srv from dropping idle connections

* feat: checkin aes_ctr tranformer for c srv

* feat: checkin progress

* fix: weird counting behaviour of argparse lib

* feat: rv-auth working

* chore: remove commented free calls

* fix: make sure to set the encrypter/decrypter in the side_hints

* fix: transformer allocation

* chore: remove temp tool

* chore: cleanup logging

* fix: found the issue with stream transformation

doesnt do decrypt correctly...

* chore: merge some files for easier navigation

* fix: make sure srv cleans itself up

* feat: checkin code

* chore: don't allocate the net context on the heap

* docs: not some important design decisions

* chore: cleanup

* style: adopt .clang-format style file

* chore: simplify clang formatting

* style: use new formatting style

* fix: srv concurrency

* docs: document side.h

* docs: add documentation for all headers in srv

* feat: checkin tool script bootstrap

switching focuses to sshnpd in c

* chore: disable encrypt traffic in srv for now

* feat: parse for multiple manager atsigns

* chore: improve the robustness of threading in srv

* feat: heartbeat working

* chore: debugging progress

* feat: working encrypt on srv

* chore: retry sending socket data when a partial send occurs

* chore(wip): c_daemon new atsdk uptake

* feat: More robust downloading

* feat: Check for unzip and tar

* chore: ignore generated development binaries

* feat: sshpublickey permissions

fix: brackets

fix

* chore: cleanup sshpublickey formatting

* chore: ignore compile_commands.json

* chore: build maintenance

* feat: add --local-host flag to srv

Tested against dart sshnpd, also did a custom build of sshnpd which has
--local-host disabled to make sure that the default value is respected.

* chore: simplify arg defaults

* feat: checkpoint

* chore: checkpoint for atlogger bug

* feat: ping request working

* feat: add sshpublickey type

* feat: add heartbeat (noop) to worker atclient

* chore: checkin mid-ssh request progress

* feat: handle username key put/deletes on startup

* fix: username key now users username value from environment

* WIP

* chore: debugging for c-daemon

* chore: cleanup logging

* fix: solved a few more bugs

* feat: v4 mvp of c_daemon

* chore: remove environment logs which contain sensitive info

* feat: v5 functionality support

* chore: remove logs

* chore: srv-lib build in sshnpd

* chore: add srv-lib to sshnpd build

* feat: sshnpd as single binary

(srv rolled up inside as a child process, but built in the binary)

* chore: build c sshnpd against a git ref

* chore: remove accidentally added temporary keys

* feat: add hidden args by hiding entries with a null description

* chore: fixes for end2end tests

* chore(WIP): end 2 end tests for c

* docs: add a brief summary of c sshnpd status

* fix: macos thing

* chore: better sshpublickey logging

* chore: cleanup file handling

* chore: trailing new line in log message

* chore: add c to description for versions

* chore: separate test list with ";"

* test: fix: ensure the cmake build dir exists

* fix: add explicit stddef.h include

* chore: more imports

* chore: more explicit headers

* chore: use atclient long_strlen instead of separate one for sshnpd

* fix: remove utils.c from cmakelists

* fix: remove unused header

* ci(temp): run only c:current daemon

* chore: update atsdk

* test: temporarily disable removal of output files so that I can view them all after the test run

* test: uncommenting dart daemons for full test run to attempt to get failure

* test: e2e_all main.sh: if start_daemons fails, then call stop_daemons before exiting

* test: just run with C daemon

* test: uncommenting dart daemons for full test run to reproduce pkam failure

* test: fix start_daemons.sh so it resets the 'extraFlags' variable for each daemon

* chore: use timestamps in atlogger

* fix: snake case validation for device name

* fix: can start with underscore (oops)

* chore: cleanup

* fix: max 15 chars

* build(deps): bump github/codeql-action in the github-actions group

Bumps the github-actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 3.25.6 to 3.25.7
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@9fdb3e4...f079b84)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump the pub group across 1 directory with 2 updates

Bumps the pub group with 2 updates in the /packages/dart/sshnoports directory: [at_onboarding_cli](https://github.com/atsign-foundation/at_libraries) and [test](https://github.com/dart-lang/test/tree/master/pkgs).


Updates `at_onboarding_cli` from 1.5.0 to 1.6.0
- [Release notes](https://github.com/atsign-foundation/at_libraries/releases)
- [Commits](https://github.com/atsign-foundation/at_libraries/commits)

Updates `test` from 1.25.5 to 1.25.6
- [Release notes](https://github.com/dart-lang/test/releases)
- [Commits](https://github.com/dart-lang/test/commits/test-v1.25.6/pkgs)

---
updated-dependencies:
- dependency-name: at_onboarding_cli
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pub
- dependency-name: test
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pub
...

Signed-off-by: dependabot[bot] <[email protected]>

* chore: WIP

* fix: includes for linux

* docs: add details about cc vs clang compile flags

* chore: update atsdk version

* chore: fix memory allocations for handle_ssh_request

* fix: move closing of a pipe to correct location

* fix: disable -r test for c daemon

* fix: invert comparison

* fix: flag tests

* fix: missing quote

* chore: uptake new changes

* fix: missing quotes

* docs: Note about statically linking cjson

* fix: help text for po

* chore: remove valgrind output

* chore: ignore valgrind files

* chore: better atclient lock management

* feat(wip): reconnect on network drop

* test: fix v4_dart_inline and v4_openssh_print scripts to get daemonVersion and clientVersion from args

* test: fix: add client version check to v5 tests when testing against the C daemon

* chore: additional logs

* fix: use new enable_hooks function

* chore: more logs

* chore: even more logs

* chore: use clang to build e2e tests

* chore: test cc for \r

* chore: use new hook name

* use latest atsdk

* feat: network connectivity

* chore: use latest trunk

* chore: more accurate log notation

* fix: allocate space for null-terminator

* chore: sigint handler and memory cleanup

* chore: remove printf

* fix: wrong variable

* chore: remove sshkeygen

* chore: set srv logging level to info

* docs: note about CJSON's override build shared libs options

* ci: Workflow and Dockerfile for C sshnpd builds

* build(deps): Pin Debian and add Dockerfile to Dependabot

* chore: more memory cleanup

* chore: uptake monitor changes

* chore: use latest commit of atsdk

* docs: cc -> gcc correction

* docs: add changelog

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Chris Swan <[email protected]>
Co-authored-by: gkc <[email protected]>
Co-authored-by: Xavier Lin <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
5 people authored Jun 12, 2024
1 parent d3edbe6 commit aadeec9
Show file tree
Hide file tree
Showing 66 changed files with 4,405 additions and 217 deletions.
1 change: 1 addition & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ updates:
- "*"
- package-ecosystem: "docker"
directories:
- "/packages/c/sshnpd/tools/"
- "/packages/dart/sshnoports/tools/"
- "/tests/end2end_tests/image/"
schedule:
Expand Down
165 changes: 165 additions & 0 deletions .github/workflows/c_release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
name: Release_Automation

on:
push:
tags:
- 'c*.*.*'
workflow_dispatch:
inputs:
main_build_only:
description: "Run non-dockerx builds only"
required: true
default: false
type: boolean

permissions: # added using https://github.com/step-security/secure-repo
contents: read

jobs:
main_build:
runs-on: ${{ matrix.os }}
defaults:
run:
working-directory: ./packages/c/sshnpd
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macOS-latest]
# os: [ubuntu-latest, macOS-latest, windows-latest]
include:
- os: ubuntu-latest
output-name: sshnp-linux-x64
ext: ''
bundle: 'shell'
- os: macOS-latest
output-name: sshnp-macos-x64
ext: ''
bundle: 'shell'
# - os: macos-14
# output-name: sshnp-macos-arm64
# ext: ''
# bundle: 'shell'
# - os: windows-latest
# output-name: sshnp-windows-x64
# ext: '.exe'
# bundle: 'windows'
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
# compile binaries
- run: |
cmake -B build -S . -DBUILD_SHARED_LIBS=off -DCMAKE_C_COMPILER=clang
cmake --build build
mkdir tarball
mv build/sshnpd .
# zip the build
- if: ${{ matrix.os == 'macOS-latest' || matrix.os == 'macos-14'}}
run: ditto -c -k --keepParent sshnpd tarball/${{ matrix.output-name }}.zip
- if: ${{ matrix.os == 'ubuntu-latest' }}
run: tar -cvzf tarball/${{ matrix.output-name }}.tgz sshnpd
- if: ${{ matrix.os == 'windows-latest' }}
run: Compress-Archive -Path sshnpd -Destination tarball/${{ matrix.output-name }}.zip
# upload the build
- uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
with:
name: ${{ matrix.output-name }}_${{ matrix.compiler }}-${{github.ref_name}}-${{github.run_number}}-${{github.run_attempt}}
path: ./packages/c/sshnpd/tarball
if-no-files-found: error

other_build:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./packages/c
strategy:
matrix:
platform: [linux/arm/v7, linux/arm64]
# platform: [linux/arm/v7, linux/arm64, linux/riscv64]
include:
- platform: linux/arm/v7
output-name: sshnpd-linux-arm
- platform: linux/arm64
output-name: sshnpd-linux-arm64
# - platform: linux/riscv64
# output-name: sshnpd-linux-riscv64
steps:
- if: ${{ ! inputs.main_build_only }}
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- if: ${{ ! inputs.main_build_only }}
uses: docker/setup-qemu-action@68827325e0b33c7199eb31dd4e31fbe9023e06e3 # v3.0.0
- if: ${{ ! inputs.main_build_only }}
uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb # v3.3.0
- if: ${{ ! inputs.main_build_only }}
run: |
docker buildx build -t cpswan/sshnpdc -f sshnpd/tools/Dockerfile.package \
--platform ${{ matrix.platform }} -o type=tar,dest=bins.tar .
mkdir tarballs
tar -xvf bins.tar -C tarballs
- if: ${{ ! inputs.main_build_only }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
with:
name: ${{ matrix.output-name }}-${{github.ref_name}}-${{github.run_number}}-${{github.run_attempt}}
path: ./packages/c/tarballs/${{ matrix.output-name }}.tgz

github-release:
name: >-
Upload artifacts and generate checksums for provenance
needs: [main_build, other_build]
runs-on: ubuntu-latest
outputs:
hashes: ${{ steps.hash.outputs.hashes }}
permissions:
contents: write # IMPORTANT: mandatory for making GitHub Releases
id-token: write # IMPORTANT: mandatory for sigstore
attestations: write
steps:
- name: Download all the tarballs
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
path: tarballs/
# - name: Generate SBOMs
# run: |
# syft scan file:./packages/dart/sshnoports/pubspec.lock \
# -o 'spdx-json=tarballs/dart_sshnoports_sbom.spdx.json' \
# -o 'cyclonedx-json=tarballs/dart_sshnoports_sbom.cyclonedx.json'
- name: Move packages for signing
run: |
cd tarballs
echo "Before:"
ls -latrR *
mv */*.tgz .
mv */*.zip .
rm -Rf -- */
echo "After:"
ls -latrR *
- name: Generate SHA256 checksums
working-directory: tarballs
run: sha256sum * > checksums.txt
- name: Upload artifacts to GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
# Upload to GitHub Release using the `gh` CLI.
# `dist/` contains the built packages, and the
# sigstore-produced signatures and certificates.
run: >-
gh release upload
'${{ github.ref_name }}' tarballs/**
--repo '${{ github.repository }}'
- id: hash
name: Pass artifact hashes for SLSA provenance
working-directory: tarballs
run: |
echo "hashes=$(cat checksums.txt | base64 -w0)" >> "$GITHUB_OUTPUT"
- uses: actions/attest-build-provenance@49df96e17e918a15956db358890b08e61c704919 # v1.2.0
with:
subject-path: 'tarballs/**'

provenance:
needs: [github-release]
permissions:
actions: read # Needed for detection of GitHub Actions environment.
id-token: write # Needed for provenance signing and ID
contents: write # Needed for release uploads
uses: slsa-framework/slsa-github-generator/.github/workflows/[email protected] # 5a775b367a56d5bd118a224a811bba288150a563
with:
base64-subjects: "${{ needs.github-release.outputs.hashes }}"
upload-assets: true
41 changes: 32 additions & 9 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
// Put your args to sshnpd in a .env file:
// For example:
// TO="@my_device"
// FROM="@my_client"
// DEVICE="foobardevice"

"version": "0.2.0",
"configurations": [
{
Expand All @@ -22,14 +25,34 @@
"console": "integratedTerminal",
"justMyCode": true,
"envFile": "${workspaceFolder}/.env",
//put your args in a .env
"args": [
"--atsign", "${env:TO}",
"--manager", "${env:FROM}",
"--device", "${env:DEVICE}",
"--atsign",
"${env:TO}",
"--manager",
"${env:FROM}",
"--device",
"${env:DEVICE}",
"-u"
]
},
{
"name": "C: SSHNPD",
"type": "lldb",
"request": "launch",
"program": "packages/c/sshnpd/build/sshnpd",
"preLaunchTask": "BuildSshnpdC",
"envFile": "${workspaceFolder}/.env",
"env": {
"PATH": "${env:PATH}"
},
"args": [
"-s",
"-v",
"-a", "${env:TO}",
"-m", "${env:FROM}",
"-d", "${env:DEVICE}"
],

"cwd": ""
}
]
}
}
4 changes: 4 additions & 0 deletions packages/c/.clang-format
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
Language: Cpp # Covers C and C++
BasedOnStyle: LLVM # Added this to ensure a sane set of defaults
ColumnLimit: 120 # LLVM uses 80, but we mostly use 120 at Atsign
7 changes: 7 additions & 0 deletions packages/c/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.cache
build*

compile_commands.json

core
valgrind-out.txt*
38 changes: 38 additions & 0 deletions packages/c/3rdparty/argparse/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@

# targets
# -------
# argparse::argparse-static
# argparse::argparse-shared

include(GNUInstallDirs)

set(ARGPARSE_STATIC_LIB argparse-static)
set(ARGPARSE_SHARED_LIB argparse-shared)

set(ARGPARSE_SOURCES
${CMAKE_CURRENT_LIST_DIR}/src/argparse.c
)

set(ARGPARSE_INCLUDE_DIR ${CMAKE_CURRENT_LIST_DIR}/include)

add_library(${ARGPARSE_STATIC_LIB} STATIC
${ARGPARSE_SOURCES}
)

add_library(${ARGPARSE_SHARED_LIB} SHARED
${ARGPARSE_SOURCES}
)

target_include_directories(${ARGPARSE_STATIC_LIB} PUBLIC
$<BUILD_INTERFACE:${ARGPARSE_INCLUDE_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>
)

target_include_directories(${ARGPARSE_SHARED_LIB} PUBLIC
$<BUILD_INTERFACE:${ARGPARSE_INCLUDE_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>
)

add_library(argparse::${ARGPARSE_STATIC_LIB} ALIAS ${ARGPARSE_STATIC_LIB})
add_library(argparse::${ARGPARSE_SHARED_LIB} ALIAS ${ARGPARSE_SHARED_LIB})

6 changes: 6 additions & 0 deletions packages/c/3rdparty/argparse/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# 3rdparty/argparse

> Copyright (c) 2012-2013 Yecheng Fu <[email protected]>
The files within this directory are from the [argparse](https://github.com/cofyc/argparse) project.
Their [license](https://github.com/cofyc/argparse/blob/master/LICENSE) hereby grants any person to redistribute and/or modify this software under the terms of the MIT license.
File renamed without changes.
Loading

0 comments on commit aadeec9

Please sign in to comment.