Automated smoke test that mimics user behavior UI and CLI

[alexmt]

Summary

We need an automated test that simulates user behavior and can be run to catch transient errors related to racing conditions in code.

Proposal

• Implement an automation script that periodically re-creates and synchronizes a specified number of Argo CD applications.
• It should be possible to configure application project, source and destination
• The script should support CLI and UI mode:
  • CLI mode creates sync and deletes applications using argocd app create/argocd app sync/argocd app wait commands
  • UI mode uses webdriver and headless chrome

[keithchong]

Hi @alexmt, @jessesuen , I could start on the initial automated UI tests using selenium-webdriver. This item could be divided into a couple of child items IMHO. Please assign this to me for now.

[dewan-ahmed]

I can take on the CLI testing part. @jessesuen please let me know if there will be a separate task for CLI testing.

[alexmt]

Ops, missed your message @keithchong . Please go ahead.

@dewan-ahmed is it ok to use this ticket to track CLI and UI testing, and coordinate in comments? I can create separate ticket otherwise .

[dewan-ahmed]

Hi @alexmt that's fine; I'll use this same issue to ask questions and track the work.

[dewan-ahmed]

These are the steps I'm planning for CLI testing part:

Pseudo script for argocd app create

1. On a container, install all pre-reqs including k3d and spin up an argocd instance.
2. execute argocd app create
3. From the docs, understand what could be the validation after the above command executes e.g. some specific pods would be created. The script will query for those resources for the test to pass/fail.

@alexmt please advise.

[dewan-ahmed]

@alexmt is the assumption that argocd is already installed and the user/session is connected to a K8s cluster? Or do we need to install the req'd dependency within the script?

[dewan-ahmed]

This is the verification I'm thinking for argocd app create:

1. argocd app list --insecure checking if an app with already exists. If yes, exist the program and request the user to delete existing app.
2. If an app with does not exist, execute:
   argocd app create <some-app-name> --repo <some-repo-name>.git --path <some-path> --dest-server https://kubernetes.default.svc --dest-namespace default --insecure
3. `argocd app list --insecure | grep "< some-app-name >" && echo "test passed" '

This is the verification I could think of. Please suggest other verifications I could do for this scenario.

[dewan-ahmed]

Another approach for CLI testing would be to have source code level testing added here

[keithchong]

Hi @alexmt , just wanted to give you an update.

Here's a 480p video that demonstrates the test in action.

• Create the app
• Sync it
• Wait for status flags (Healthy and Synced - see console on right side of the video)
• Delete the app
• Wait for the delete operation to complete

This can also be run in headless mode (no browser is launched).

initialUiTest-480p.zip

Any feedback is appreciated.

Points to consider and possible follow-on enhancements

• Need to customize it so you can specify any number of apps to be created. Right now it is just one app.
• Add better error handling (edit: I added more to this in the PR. Stack traces are now more meaningful.)
• Add better validation/verification after each step
• Use mocha; launch a test suite of tests
• the CLI will still need to be created to invoke the UI test or launch the test suites.
• Add better error recovery. eg. how to return to the base state, say, no applications in the list, after one test is run but failed. and an app hasn't been deleted.

For the first point, if multiple apps are shown in the application list, there needs to be a way to correctly identify the Sync or Delete button in the tile of the desired app.

[dewan-ahmed]

This is myworking file for the first test - app creation. Per the discussion from community call, these are my assumptions:

• argocd cli is installed
• argocd server variable is set
• argocd cli is authenticated with username 'admin'

However, I'm getting the following error when running my test script:
time="2020-10-25T19:35:20-04:00" level=fatal msg="rpc error: code = Unauthenticated desc = Invalid username or password"

From my understand on the community call discussion, I should not be worrying about the authentication part and should assume that the session is already authenticated.

What I've tried:

• Added another code block above the first RunCli to authenticate. Still same error.
• Tried adding EnsureCleanState(t) code block beforehand. Still same error.

Please let me know if I can try something else to ensure I can properly test this code on my local environment.

[jannfis]

@dewan-ahmed Looking at what's happening in test/e2e/fixture/fixture.go, I'd assume that authentication only works with hard-coded credentials also on the server side. That means your server must be set-up with an admin password of password - then RunCli() should work as expected.

[dewan-ahmed]

Hi @jannfis that was the issue, thank you! Now the authentication error is gone.

[keithchong]

Hi @alexmt, please have a look at my commit and my comment. I think the ui-tests should be 'independent' of the rest of the ArgoCD code.

[keithchong]

Hi @alexmt, I created the PR for step 3. #4694. See PR for instructions on how to run the tests. When reviewing, take into account the points I made in one of my earlier comments in this issue,

Points to consider and possible follow-on enhancements

• Need to customize it so you can specify any number of apps to be created. Right now it is just one app.
• Add better error handling (edit: I added more to this in the PR. Stack traces are now more meaningful.)
• Add better validation/verification after each step
• Use mocha; launch a test suite of tests
• the CLI will still need to be created to invoke the UI test or launch the test suites.
• Add better error recovery. eg. how to return to the base state, say, no applications in the list, after one test is run but failed. and an app hasn't been deleted.

For the first point, if multiple apps are shown in the application list, there needs to be a way to correctly identify the Sync or Delete button in the tile of the desired app.

[dewan-ahmed]

Further discussion on design of smoke test: https://argoproj.slack.com/archives/CEWCP8TV5/p1604506062335700

[dewan-ahmed]

#4819 and #4820 issues have been created to break down the task regarding CLI part.

[dewan-ahmed]

Note on argo-cd slack regarding smoke test scripts

cc: @alexmt

[dewan-ahmed]

Hi @alexmt , I'm struggling to verify argocd app wait... from the CLI. While the app create.. and app sync.. are working (code here)[https://github.com/dewan-ahmed/argo-cd/tree/dewan-4820/test/smoke], the following piece of code is panic-ing

Running tool: /usr/local/bin/go test -timeout 30s github.com/argoproj/argo-cd/test/smoke -run ^TestWaitApp$

panic: test timed out after 30s

goroutine 19 [running]:
testing.(*M).startAlarm.func1()
/usr/local/Cellar/go/1.15.3/libexec/src/testing/testing.go:1618 +0xe5

func TestWaitApp(t *testing.T) {
	expectedBefore := "OutOfSync"
	// Create the guestbook app if not already created
	_, err := RunCmd("argocd", "app", "create", "guestbook", "--repo", "https://github.com/argoproj/argocd-example-apps.git", "--path", "guestbook", "--dest-server", "https://kubernetes.default.svc", "--dest-namespace", "default")
	errors.CheckError(err)
	time.Sleep(1 * time.Second)
	// Check that OutOfSync shows in the argocd app wait output
	outputBefore, err := RunCmd("argocd", "app", "wait", "guestbook")
	errors.CheckError(err)
	outputBeforeAsString := BytesToString(outputBefore)
	passBefore := OutputContains(outputBeforeAsString, expectedBefore)
	fmt.Println(passBefore)
	assert.Equal(t, true, passBefore)

	/* second block of code will go here that will sync the app and then expect "Synced" in the 'argocd app wait...' output */ 
}

[dewan-ahmed]

Added an agenda item on contributor call today to discuss this @alexmt . Since we went over the time, this issue was not discussed. Hoping to discuss during one of the future calls.