Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build tags for smaller binaries that don't need net/http or encoding/json #63

Merged
merged 27 commits into from
Jul 1, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a89783b
no_http/no_net and no_json buildtags (wip)
ldemailly Jun 30, 2024
a761e94
better with git add
ldemailly Jun 30, 2024
7830993
passing tests but not happy about the nil error Invalid case
ldemailly Jun 30, 2024
3a022cf
adding info about no_json and github.com/Zxilly/go-size-analyzer/cmd/gsa
ldemailly Jun 30, 2024
64bc184
linter alerted indirectly about wrongly named test file
ldemailly Jun 30, 2024
4dd0c32
so much cleaner, thanks polyscone (gophers discord)
ldemailly Jun 30, 2024
a3ee518
update comments and add test for nil *string
ldemailly Jun 30, 2024
34330f1
adding make coverage now that https://github.com/fortio/workflows/pul…
ldemailly Jun 30, 2024
d8deb75
lets see if it work with just cat and codecov
ldemailly Jun 30, 2024
c4f7b5e
codecov already looks for *coverage*
ldemailly Jun 30, 2024
507d00e
actually not... trying to specify files
ldemailly Jun 30, 2024
fe0dc98
using gocovmerge
ldemailly Jun 30, 2024
d0d83d1
added tests for struct etc
ldemailly Jun 30, 2024
8c8c166
retrigger codecov
ldemailly Jun 30, 2024
647912d
minor improvement to readme
ldemailly Jun 30, 2024
5efce32
lint the no_json path too
ldemailly Jul 1, 2024
6e2d299
json->JSON
ldemailly Jul 1, 2024
410f436
more json->JSON
ldemailly Jul 1, 2024
3a4c6bc
verify through test that byte = uint8 works and added log.Rune(k,v) t…
ldemailly Jul 1, 2024
bab5090
make the rune test slightly more readable
ldemailly Jul 1, 2024
4b32ab6
more efficient conversion rune to string
ldemailly Jul 1, 2024
9c02bb3
Apply suggestions from code review
ldemailly Jul 1, 2024
8f090f7
don't break standard lib message checks
ldemailly Jul 1, 2024
8c8094b
put variable in makefile for go so I can run 'make GOBIN=go1.23rc1'
ldemailly Jul 1, 2024
a085108
better to not search and replace ALL without testing
ldemailly Jul 1, 2024
e462489
thanks @ccoVeille - was using a defined go env var (GOBIN) instead of…
ldemailly Jul 1, 2024
7ec8cc2
fix comment / another bad search and replace
ldemailly Jul 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
.DS_Store
.golangci.yml
fullsize
smallsize
coverage.out
coverage?.out
14 changes: 14 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"version": "0.2.0",
"configurations": [
{
"name": "Launch test function with tags",
"type": "go",
"request": "launch",
"mode": "test",
"program": "${workspaceFolder}",
"buildFlags": "-tags=no_json"
}

]
}
28 changes: 27 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,22 @@
all: test example

test:
go test . -race ./...
go test -race ./...
go test -tags no_json ./...
go test -tags no_http ./...

local-coverage: coverage
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

coverage:
go test -coverprofile=coverage1.out ./...
go test -tags no_net -coverprofile=coverage2.out ./...
go test -tags no_json -coverprofile=coverage3.out ./...
go test -tags no_http,no_json -coverprofile=coverage4.out ./...
# cat coverage*.out > coverage.out
go install github.com/wadey/gocovmerge@b5bfa59ec0adc420475f97f89b58045c721d761c
gocovmerge coverage?.out > coverage.out

example:
@echo "### Colorized (default) ###"
Expand All @@ -17,9 +32,20 @@ line:
screenshot: line example
@echo

size-check:
@echo "### Size of the binary:"
CGO_ENABLED=0 go build -ldflags="-w -s" -trimpath -o ./fullsize ./levelsDemo
ls -lh ./fullsize
CGO_ENABLED=0 go build -tags no_net -ldflags="-w -s" -trimpath -o ./smallsize ./levelsDemo
ls -lh ./smallsize
CGO_ENABLED=0 go build -tags no_http,no_json -ldflags="-w -s" -trimpath -o ./smallsize ./levelsDemo
ls -lh ./smallsize
gsa ./smallsize # go install github.com/Zxilly/go-size-analyzer/cmd/gsa@master


lint: .golangci.yml
golangci-lint run
golangci-lint run --build-tags no_json

.golangci.yml: Makefile
curl -fsS -o .golangci.yml https://raw.githubusercontent.com/fortio/workflows/main/golangci.yml
Expand Down
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ log.S(log.Info, "msg", log.Attr("key1", value1)...)

See the `Config` object for options like whether to include line number and file name of caller or not etc

New since 1.4 server logging (as used in [fortio.org/scli](https://pkg.go.dev/fortio.org/scli#ServerMain) for instance) is now structured (json), client logging (as setup by [fortio.org/cli](https://pkg.go.dev/fortio.org/scli#ServerMain) remains as before.
New since 1.4 server logging (as used in [fortio.org/scli](https://pkg.go.dev/fortio.org/scli#ServerMain) for instance) is now structured (JSON), client logging (as setup by [fortio.org/cli](https://pkg.go.dev/fortio.org/scli#ServerMain) remains as before.

One can also revert server to not be JSON through config.

Expand All @@ -46,7 +46,7 @@ Which can be converted to JSONEntry but is also a fixed, optimized format (ie ts

The timestamp `ts` is in seconds.microseconds since epoch (golang UnixMicro() split into seconds part before decimal and microseconds after)

Since 1.8 the Go Routine ID is present in json (`r` field) or colorized log output (for multi threaded server types).
Since 1.8 the Go Routine ID is present in JSON (`r` field) or colorized log output (for multi threaded server types).

Optional additional `KeyValue` pairs can be added to the base structure using the new `log.S` or passed to `log.LogRequest` using `log.Any` and `log.Str`. Note that numbers, as well as arrays of any type and maps of string keys to any type are supported (but more expensive to serialize recursively).

Expand All @@ -63,7 +63,7 @@ When output is redirected, JSON output:
{"ts":1689986143.4634,"level":"err","r":1,"file":"levels.go","line":23,"msg":"This is an error message"}
{"ts":1689986143.463403,"level":"crit","r":1,"file":"levels.go","line":24,"msg":"This is a critical message"}
{"ts":1689986143.463406,"level":"fatal","r":1,"file":"levels.go","line":25,"msg":"This is a fatal message"}
This is a non json output, will get prefixed with a exclamation point with logc
This is a non JSON output, will get prefixed with a exclamation point with logc
ldemailly marked this conversation as resolved.
Show resolved Hide resolved
```

When on console:
Expand Down Expand Up @@ -100,3 +100,11 @@ LOGGER_GOROUTINE_ID=false
LOGGER_COMBINE_REQUEST_AND_RESPONSE=true
LOGGER_LEVEL='Info'
```

# Small binaries

If you're never logging http requests/responses, use `-tags no_http` (or `-tags no_net`) to exclude the http/https logging utilities (which pulls in a lot of dependencies because of `net/http` init).

If you never need to JSON log complex structures/types that have a special `json.Marshaler` then you can use `-tags no_net,no_json` for the smallest executables

(see `make size-check`)
6 changes: 6 additions & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
coverage:
ignore:
- "levelsDemo"
# not used (is nothing at all used?)
# files:
# - "coverage*.out"
3 changes: 3 additions & 0 deletions http_logging.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.

//go:build !no_http && !no_net
// +build !no_http,!no_net

package log

import (
Expand Down
3 changes: 3 additions & 0 deletions http_logging_test.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
//go:build !no_http && !no_net
// +build !no_http,!no_net

package log // import "fortio.org/fortio/log"

import (
Expand Down
67 changes: 67 additions & 0 deletions json_logging.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
// Copyright 2017-2024 Fortio Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Moved json logging out so it can be skipped for smallest binaries based on build tags.
// only difference is with nested struct/array logging or logging of types with json Marchaller interface.

//go:build !no_json

package log // import "fortio.org/log"

import (
"encoding/json"
"fmt"
"strconv"
)

var fullJSON = true

func toJSON(v any) string {
bytes, err := json.Marshal(v)
if err != nil {
return strconv.Quote(fmt.Sprintf("ERR marshaling %v: %v", v, err))
}
str := string(bytes)
// We now handle errors before calling toJSON: if there is a marshaller we use it
// otherwise we use the string from .Error()
return str
}

func (v ValueType[T]) String() string {
// if the type is numeric, use Sprint(v.val) otherwise use Sprintf("%q", v.Val) to quote it.
switch s := any(v.Val).(type) {
case bool, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64,
float32, float64:
return fmt.Sprint(s)
case string:
return fmt.Sprintf("%q", s)
case error:
// Sadly structured errors like nettwork error don't have the reason in
// the exposed struct/JSON - ie on gets
// {"Op":"read","Net":"tcp","Source":{"IP":"127.0.0.1","Port":60067,"Zone":""},
// "Addr":{"IP":"127.0.0.1","Port":3000,"Zone":""},"Err":{}}
// instead of
// read tcp 127.0.0.1:60067->127.0.0.1:3000: i/o timeout
// Noticed in https://github.com/fortio/fortio/issues/913
_, hasMarshaller := s.(json.Marshaler)
if hasMarshaller {
return toJSON(v.Val)
} else {
ldemailly marked this conversation as resolved.
Show resolved Hide resolved
return fmt.Sprintf("%q", s.Error())
}
/* It's all handled by json fallback now even though slightly more expensive at runtime, it's a lot simpler */
default:
return toJSON(v.Val) // was fmt.Sprintf("%q", fmt.Sprint(v.Val))
}
}
20 changes: 20 additions & 0 deletions json_logging_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
//go:build !no_json
// +build !no_json

package log // import "fortio.org/fortio/log"

import (
"fmt"
"testing"
)

func TestToJSON_MarshalError(t *testing.T) {
badValue := make(chan int)

expected := fmt.Sprintf("\"ERR marshaling %v: %v\"", badValue, "json: unsupported type: chan int")
ldemailly marked this conversation as resolved.
Show resolved Hide resolved
actual := toJSON(badValue)

if actual != expected {
t.Errorf("Expected %q, got %q", expected, actual)
}
}
6 changes: 3 additions & 3 deletions levelsDemo/levels.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ func main() {
log.Config.GoroutineID = false
log.Debugf("This is a debug message without goroutine id, file:line nor prefix (cli style)")
log.Config = log.DefaultConfig()
// So log fatal doesn't panic nor exit (so we can print the non json last line).
// So log fatal doesn't panic nor exit (so we can print the non JSON last line).
ldemailly marked this conversation as resolved.
Show resolved Hide resolved
log.Config.FatalPanics = false
log.Config.FatalExit = func(int) {}
// Meat of the example: (some of these are reproducing fixed issues in `logc` json->console attributes detection)
// Meat of the example: (some of these are reproducing fixed issues in `logc` JSON->console attributes detection)
log.Debugf("Back to default (server) logging style with a debug message ending with backslash \\")
log.LogVf("This is a verbose message")
log.Printf("This an always printed, file:line omitted message (and no level in console)")
Expand All @@ -28,5 +28,5 @@ func main() {
log.Errf("This is an error message")
log.Critf("This is a critical message")
log.Fatalf("This is a fatal message") //nolint:revive // we disabled exit for this demo
fmt.Println("This is a non json output, will get prefixed with a exclamation point with logc")
fmt.Println("This is a non JSON output, will get prefixed with a exclamation point with logc")
ldemailly marked this conversation as resolved.
Show resolved Hide resolved
}
47 changes: 6 additions & 41 deletions logger.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ package log // import "fortio.org/log"

import (
"bytes"
"encoding/json"
"flag"
"fmt"
"io"
Expand Down Expand Up @@ -349,7 +348,7 @@ func Logf(lvl Level, format string, rest ...interface{}) {
logPrintf(lvl, format, rest...)
}

// Used when doing our own logging writing, in JSON/structured mode.
// Used when doing our own logging writing, in JSON/structured mode (and some color variants as well, misnomer).
var (
jWriter = jsonWriter{w: os.Stderr, tsBuf: make([]byte, 0, 32)}
)
Expand Down Expand Up @@ -617,6 +616,11 @@ func Bool(key string, value bool) KeyVal {
return Any(key, value)
}

func Rune(key string, value rune) KeyVal {
// Special case otherwise rune is printed as int32 number
return Any(key, string(value)) // similar to "%c".
}

func (v *KeyVal) StringValue() string {
if !v.Cached {
v.StrValue = v.Value.String()
Expand All @@ -631,45 +635,6 @@ type ValueType[T ValueTypes] struct {
Val T
}

func toJSON(v any) string {
bytes, err := json.Marshal(v)
if err != nil {
return strconv.Quote(fmt.Sprintf("ERR marshaling %v: %v", v, err))
}
str := string(bytes)
// We now handle errors before calling toJSON: if there is a marshaller we use it
// otherwise we use the string from .Error()
return str
}

func (v ValueType[T]) String() string {
// if the type is numeric, use Sprint(v.val) otherwise use Sprintf("%q", v.Val) to quote it.
switch s := any(v.Val).(type) {
case bool, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64,
float32, float64:
return fmt.Sprint(s)
case string:
return fmt.Sprintf("%q", s)
case error:
// Sadly structured errors like nettwork error don't have the reason in
// the exposed struct/JSON - ie on gets
// {"Op":"read","Net":"tcp","Source":{"IP":"127.0.0.1","Port":60067,"Zone":""},
// "Addr":{"IP":"127.0.0.1","Port":3000,"Zone":""},"Err":{}}
// instead of
// read tcp 127.0.0.1:60067->127.0.0.1:3000: i/o timeout
// Noticed in https://github.com/fortio/fortio/issues/913
_, hasMarshaller := s.(json.Marshaler)
if hasMarshaller {
return toJSON(v.Val)
} else {
return fmt.Sprintf("%q", s.Error())
}
/* It's all handled by json fallback now even though slightly more expensive at runtime, it's a lot simpler */
default:
return toJSON(v.Val) // was fmt.Sprintf("%q", fmt.Sprint(v.Val))
}
}

// Our original name, now switched to slog style Any.
func Attr[T ValueTypes](key string, value T) KeyVal {
return Any(key, value)
Expand Down
Loading
Loading