Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix crash on startup #312

Merged
merged 2 commits into from
Sep 22, 2022
Merged

Fix crash on startup #312

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0afae26
Fix crash on startup
jotak Sep 22, 2022
f1e8240
Mock KubeData
jotak Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions cmd/flowlogs-pipeline/main_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,14 @@
package main

import (
"encoding/json"
"os"
"os/exec"
"testing"

"github.com/netobserv/flowlogs-pipeline/pkg/config"
"github.com/netobserv/flowlogs-pipeline/pkg/pipeline"
"github.com/stretchr/testify/require"
)

func TestTheMain(t *testing.T) {
Expand All @@ -36,3 +41,25 @@ func TestTheMain(t *testing.T) {
}
t.Fatalf("process ran with err %v, want exit status 1", err)
}

func TestPipelineConfigSetup(t *testing.T) {
js := `{
"PipeLine": "[{\"name\":\"grpc\"},{\"follows\":\"grpc\",\"name\":\"enrich\"},{\"follows\":\"enrich\",\"name\":\"loki\"},{\"follows\":\"enrich\",\"name\":\"prometheus\"}]",
"Parameters": "[{\"ingest\":{\"grpc\":{\"port\":2055},\"type\":\"grpc\"},\"name\":\"grpc\"},{\"name\":\"enrich\",\"transform\":{\"network\":{\"rules\":[{\"input\":\"SrcAddr\",\"output\":\"SrcK8S\",\"type\":\"add_kubernetes\"},{\"input\":\"DstAddr\",\"output\":\"DstK8S\",\"type\":\"add_kubernetes\"},{\"input\":\"DstPort\",\"output\":\"Service\",\"parameters\":\"Proto\",\"type\":\"add_service\"},{\"input\":\"SrcAddr\",\"output\":\"SrcSubnet\",\"parameters\":\"/16\",\"type\":\"add_subnet\"}]},\"type\":\"network\"}},{\"name\":\"loki\",\"write\":{\"loki\":{\"batchSize\":102400,\"batchWait\":\"1s\",\"clientConfig\":{\"follow_redirects\":false,\"proxy_url\":null,\"tls_config\":{\"insecure_skip_verify\":false}},\"labels\":[\"SrcK8S_Namespace\",\"SrcK8S_OwnerName\",\"DstK8S_Namespace\",\"DstK8S_OwnerName\",\"FlowDirection\"],\"maxBackoff\":\"5m0s\",\"maxRetries\":10,\"minBackoff\":\"1s\",\"staticLabels\":{\"app\":\"netobserv-flowcollector\"},\"tenantID\":\"netobserv\",\"timeout\":\"10s\",\"timestampLabel\":\"TimeFlowEndMs\",\"timestampScale\":\"1ms\",\"url\":\"http://loki.netobserv.svc:3100/\"},\"type\":\"loki\"}},{\"encode\":{\"prom\":{\"metrics\":[{\"buckets\":null,\"filter\":{\"key\":\"\",\"value\":\"\"},\"labels\":[\"Service\",\"SrcK8S_Namespace\"],\"name\":\"bandwidth_per_network_service_per_namespace\",\"type\":\"counter\",\"valueKey\":\"Bytes\"},{\"buckets\":null,\"filter\":{\"key\":\"\",\"value\":\"\"},\"labels\":[\"SrcSubnet\"],\"name\":\"bandwidth_per_source_subnet\",\"type\":\"counter\",\"valueKey\":\"Bytes\"},{\"buckets\":null,\"filter\":{\"key\":\"\",\"value\":\"\"},\"labels\":[\"Service\"],\"name\":\"network_service_total\",\"type\":\"counter\",\"valueKey\":\"\"}],\"port\":9102,\"prefix\":\"netobserv_\"},\"type\":\"prom\"},\"name\":\"prometheus\"}]",
"Health": {
"Port": "8080"
},
"Profile": {
"Port": 0
}
}`
var opts config.Options
err := json.Unmarshal([]byte(js), &opts)
require.NoError(t, err)
cfg, err := config.ParseConfig(opts)
require.NoError(t, err)
require.NotNil(t, cfg)
mainPipeline, err := pipeline.NewPipeline(&cfg)
require.NoError(t, err)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is failing here with:

can't access kubenetes. Tried using config from: config parameter, KUBECONFIG env, homedir and InClusterConfig. Got: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

I wonder if using os.Setenv is enough...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option would be to remove the add_kubernetes rules from the test config.

Copy link

@mariomac mariomac Sep 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, Ronen's suggestion is much cleaner for a unit test

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm rather mocking the kube client loading

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(else, kube client initialization would fail without a real kube cluster available)

require.NotNil(t, mainPipeline)
}
12 changes: 12 additions & 0 deletions pkg/api/transform_network.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,18 @@ type TransformNetwork struct {
ProtocolsFile string `yaml:"protocolsFile,omitempty" json:"protocolsFile,omitempty" doc:"path to protocols file (optional, default: /etc/protocols)"`
}

func (tn *TransformNetwork) GetServiceFiles() (string, string) {
p := tn.ProtocolsFile
if p == "" {
p = "/etc/protocols"
Copy link
Member Author

@jotak jotak Sep 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by the way, I used the previous default, as you can see here: https://github.com/netobserv/flowlogs-pipeline/pull/304/files#diff-803cd3d555b315c0ffdf0ac4905fd947085c6edf178137e7734e7af8a1218214L55 it was previously using these paths.

But maybe this isn't super safe to write in /etc/protocols? maybe rather use /tmp/flp-protocols and /tmp/flp-services instead? @KalmanMeth @ronensc @mariomac ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

never mind, I thought it was a file that we would write into, I didn't know it was a standardized thingy

}
s := tn.ServicesFile
if s == "" {
s = "/etc/services"
}
return p, s
}

type TransformNetworkOperationEnum struct {
AddRegExIf string `yaml:"add_regex_if" json:"add_regex_if" doc:"add output field if input field satisfies regex pattern from parameters field"`
AddIf string `yaml:"add_if" json:"add_if" doc:"add output field if input field satisfies criteria from parameters field"`
Expand Down
9 changes: 5 additions & 4 deletions pkg/pipeline/transform/transform_network.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,15 +189,16 @@ func NewTransformNetwork(params config.StageParam) (Transformer, error) {

var servicesDB *netdb.ServiceNames
if needToInitNetworkServices {
pFilename, sFilename := jsonNetworkTransform.GetServiceFiles()
var err error
protos, err := os.Open(jsonNetworkTransform.ProtocolsFile)
protos, err := os.Open(pFilename)
if err != nil {
return nil, fmt.Errorf("opening %q: %w", jsonNetworkTransform.ProtocolsFile, err)
return nil, fmt.Errorf("opening protocols file %q: %w", pFilename, err)
}
defer protos.Close()
services, err := os.Open(jsonNetworkTransform.ServicesFile)
services, err := os.Open(sFilename)
if err != nil {
return nil, fmt.Errorf("opening %q: %w", jsonNetworkTransform.ServicesFile, err)
return nil, fmt.Errorf("opening services file %q: %w", sFilename, err)
Comment on lines +196 to +201
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for improving the error message

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to find the error message in the e2e test of the PR that introduced the bug. But, despite that, the test passes.
https://github.com/netobserv/flowlogs-pipeline/actions/runs/3082582328/jobs/4982452400
image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah seems like there's some improvement to bring to the e2e test too

}
defer services.Close()
servicesDB, err = netdb.LoadServicesDB(protos, services)
Expand Down