Failing test: X-Pack Security API Integration Tests (Session Idle Timeout).x-pack/test/security_api_integration/tests/session_idle/cleanup·ts - security APIs - Session Idle Session Idle cleanup should properly clean up session expired because of idle timeout #121482

kibanamachine · 2021-12-17T04:59:40Z

A test failed on a tracked branch

Error: expected 200 "OK", got 401 "Unauthorized"
    at Test._assertStatus (/opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/lib/test.js:131:12)
    at /opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
    at /opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
    at IncomingMessage.<anonymous> (/opt/local-ssd/buildkite/builds/kb-n2-4-c3b96ff0bbb5ca35/elastic/kibana-hourly/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at IncomingMessage.emit (node:events:402:35)
    at endReadableNT (node:internal/streams/readable:1343:12)

First failure: CI Build - main

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-12-17T10:28:20Z

Pinging @elastic/kibana-security (Team:Security)

azasypkin · 2021-12-17T13:28:52Z

Error: expected 200 "OK", got 401 "Unauthorized"

I cannot think of any reason why we'd get error like that in this test. Here is the assertion that fails:

kibana/x-pack/test/security_api_integration/tests/session_idle/cleanup.ts

Lines 86 to 95 in 788db0d

    
           const response = await supertest 
        
             .post('/internal/security/login') 
        
             .set('kbn-xsrf', 'xxx') 
        
             .send({ 
        
               providerType: 'basic', 
        
               providerName: 'basic1', 
        
               currentURL: '/', 
        
               params: { username: basicUsername, password: basicPassword }, 
        
             }) 
        
             .expect(200);

It's a simple basic login API request with the credentials of the pre-populated admin test user. I'll keep eye on this issue and either dig deeper if it happens again or close it as an intermittent if we don't see failures during next few weeks.

kibanamachine · 2021-12-18T22:03:27Z

New failure: CI Build - 8.0

azasypkin · 2021-12-22T15:53:05Z

Investigating the failure...

azasypkin · 2021-12-24T08:22:37Z

In this test we take the following steps:

Log in via '/internal/security/login'
Verify that session cookie is valid via '/internal/security/me'
Wait until session expires (idle timeout is 5s) and the cleanup job removes it
Verify that session is no longer valid via '/internal/security/me'

The test fails on the step 2 likely because sometimes Kibana is too slow on CI and it takes more than 5s between steps 1 and 2.

I'm doubling timeouts in configuration for these tests and will see if it helps: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/37

kibanamachine · 2021-12-28T01:03:28Z

New failure: CI Build - 8.0

kibanamachine · 2021-12-28T04:02:08Z

New failure: CI Build - 8.0

azasypkin · 2021-12-28T08:52:04Z

New failure: CI Build - 8.0

Closing again since 8.0/7.x backports haven't been merged yet due to merge conflicts (CI run pending).

kibanamachine · 2022-01-20T23:01:35Z

New failure: CI Build - main

azasypkin · 2022-01-24T07:35:15Z

New failure: CI Build - main

Before the fix, the symptom in all failed builds was Error: expected 200 "OK", got 401 "Unauthorized", now it's different: Error: expected 1 to equal 0. Let's wait and see if it's a transient issue or not. If not, we'll need to make sure that's not related to the recent changes in the cleanup logic in #122419

kibanamachine · 2022-02-11T03:08:13Z

New failure: CI Build - main

azasypkin · 2022-02-11T10:14:59Z

сс @thomheymann, just in case you can notice what can be the reason here.

kibanamachine · 2022-03-04T21:12:04Z

New failure: CI Build - 8.1

jportner · 2022-03-04T21:21:36Z

New failure: CI Build - 8.1

This is also the "new error":

└-: security APIs - Session Idle
--
  | └-> "before all" hook: beforeTestSuite.trigger in "security APIs - Session Idle"
  | └-: Session Idle cleanup
  | └-> "before all" hook: beforeTestSuite.trigger for "should properly clean up session expired because of idle timeout"
  | └-> should properly clean up session expired because of idle timeout
  | └-> "before each" hook: global before each for "should properly clean up session expired because of idle timeout"
  | └-> "before each" hook for "should properly clean up session expired because of idle timeout"
  | └- ✖ fail: security APIs - Session Idle Session Idle cleanup should properly clean up session expired because of idle timeout
  | │      Error: expected 1 to equal 0
  | │       at Assertion.assert (/opt/local-ssd/buildkite/builds/kb-n2-4-a6f049047d437185/elastic/kibana-hourly/kibana/node_modules/@kbn/expect/expect.js:100:11)
  | │       at Assertion.be.Assertion.equal (/opt/local-ssd/buildkite/builds/kb-n2-4-a6f049047d437185/elastic/kibana-hourly/kibana/node_modules/@kbn/expect/expect.js:227:8)
  | │       at Assertion.be (/opt/local-ssd/buildkite/builds/kb-n2-4-a6f049047d437185/elastic/kibana-hourly/kibana/node_modules/@kbn/expect/expect.js:69:22)
  | │       at Context.<anonymous> (test/security_api_integration/tests/session_idle/cleanup.ts:111:54)
  | │       at runMicrotasks (<anonymous>)
  | │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
  | │       at Object.apply (/opt/local-ssd/buildkite/builds/kb-n2-4-a6f049047d437185/elastic/kibana-hourly/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

The assertion that's failing is on line 111 below:

kibana/x-pack/test/security_api_integration/tests/session_idle/cleanup.ts

Lines 100 to 111 in 660023b

    
           const sessionCookie = parseCookie(response.headers['set-cookie'][0])!; 
        
           await checkSessionCookie(sessionCookie, basicUsername, { type: 'basic', name: 'basic1' }); 
        
           expect(await getNumberOfSessionDocuments()).to.be(1); 
        
           // Cleanup routine runs every 20s, and idle timeout threshold is three times larger than 10s 
        
           // idle timeout, let's wait for 60s to make sure cleanup routine runs when idle timeout 
        
           // threshold is exceeded. 
        
           log.debug('Waiting for cleanup job to run...'); 
        
           await setTimeoutAsync(60000); 
        
           // Session info is removed from the index and cookie isn't valid anymore 
        
           expect(await getNumberOfSessionDocuments()).to.be(0);

I think I know what's causing the test failure. We changed deleteByQuery to use a combination of find + bulk-delete so that we can log audit records for each deleted session document. In the delete step, we intentionally use refresh: false to ensure it's as fast as possible in case we have to delete multiple batches of sessions:

kibana/x-pack/plugins/security/server/session_management/session_index.ts

Lines 460 to 467 in 6627bd8

    
           const bulkResponse = await this.options.elasticsearchClient.bulk( 
        
             { 
        
               index: this.indexName, 
        
               operations, 
        
               refresh: false, 
        
             }, 
        
             { ignore: [409, 404] } 
        
           );

However, we don't appear to be triggering an index refresh after the whole cleanup process is finished. I believe we intended to add that but it got lost in the mix.

kibanamachine · 2022-03-08T23:09:19Z

New failure: CI Build - main

kibanamachine · 2022-03-10T04:07:12Z

New failure: CI Build - main

kibanamachine · 2023-03-12T23:33:22Z

New failure: CI Build - main

kibanamachine · 2023-03-13T14:42:11Z

New failure: CI Build - main

kibanamachine · 2023-03-15T05:54:17Z

New failure: CI Build - main

jbudz · 2023-03-15T17:36:37Z

/skip

kibanamachine · 2023-03-15T17:37:12Z

Skipped

main: 3dabc61

Closes #152260 Closes #121482 Closes #136688 ## Description - Adds security index refresh to `getSessionInfo` to ensure each query is running on updated data - Triggers the cleanup routine just before checking idle session timeouts to increase determinism (same methodology used in concurrent sessions tests) - Adds a short static delay when testing session extend to ensure the original session time has somewhat elapsed (when this was failing it was only by a few milliseconds) ### Tests x-pack/test/security_api_integration/session_idle.config.ts... - 'should properly clean up session expired because of idle timeout' - 'should properly clean up session expired because of idle timeout when providers override global session config' - 'should extend the session' ### Flaky Test Runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2031

Closes elastic#152260 Closes elastic#121482 Closes elastic#136688 ## Description - Adds security index refresh to `getSessionInfo` to ensure each query is running on updated data - Triggers the cleanup routine just before checking idle session timeouts to increase determinism (same methodology used in concurrent sessions tests) - Adds a short static delay when testing session extend to ensure the original session time has somewhat elapsed (when this was failing it was only by a few milliseconds) ### Tests x-pack/test/security_api_integration/session_idle.config.ts... - 'should properly clean up session expired because of idle timeout' - 'should properly clean up session expired because of idle timeout when providers override global session config' - 'should extend the session' ### Flaky Test Runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2031

kibanamachine · 2023-05-12T19:38:21Z

New failure: CI Build - main

kibanamachine · 2023-05-12T19:43:05Z

New failure: CI Build - main

kibanamachine · 2023-05-12T19:53:37Z

New failure: CI Build - 8.8

kibanamachine · 2023-05-12T20:00:00Z

New failure: CI Build - 8.8

jeramysoucy · 2023-05-16T20:49:00Z

Ran another flaky test runner just to be sure, but this looks tied to a series of CI failures on Friday.

kibanamachine · 2024-08-05T18:59:52Z

New failure: kibana-on-merge - 7.17

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Dec 17, 2021

botelastic bot added the needs-team Issues missing a team label label Dec 17, 2021

afharo added the Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! label Dec 17, 2021

botelastic bot removed the needs-team Issues missing a team label label Dec 17, 2021

azasypkin mentioned this issue Dec 23, 2021

Improve session cleanup tests stability. #121961

Merged

azasypkin closed this as completed in #121961 Dec 27, 2021

kibanamachine reopened this Dec 28, 2021

azasypkin closed this as completed Dec 28, 2021

kibanamachine reopened this Jan 20, 2022

jportner mentioned this issue Mar 4, 2022

Fix session cleanup test #126966

Merged

jportner linked a pull request Mar 4, 2022 that will close this issue

Fix session cleanup test #126966

Merged

jportner closed this as completed in #126966 Mar 7, 2022

kibanamachine reopened this Mar 8, 2022

mistic added a commit that referenced this issue Mar 10, 2022

skip flaky suite (#121482)

25d6e64

kibanamachine removed v8.2.0 skipped-test labels Apr 13, 2022

kibanamachine reopened this Mar 12, 2023

kibanamachine added blocker skipped-test v8.8.0 labels Mar 15, 2023

kibanamachine added a commit that referenced this issue Mar 15, 2023

skip failing test suite (#121482)

3dabc61

jeramysoucy self-assigned this Mar 20, 2023

jeramysoucy mentioned this issue Mar 20, 2023

Fixes session idle timeout API test #153303

Merged

jeramysoucy closed this as completed in #153303 Mar 23, 2023

kibanamachine removed skipped-test blocker v8.8.0 labels Mar 23, 2023

kibanamachine reopened this May 12, 2023

jeramysoucy closed this as completed May 16, 2023

kibanamachine reopened this Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing test: X-Pack Security API Integration Tests (Session Idle Timeout).x-pack/test/security_api_integration/tests/session_idle/cleanup·ts - security APIs - Session Idle Session Idle cleanup should properly clean up session expired because of idle timeout #121482

Failing test: X-Pack Security API Integration Tests (Session Idle Timeout).x-pack/test/security_api_integration/tests/session_idle/cleanup·ts - security APIs - Session Idle Session Idle cleanup should properly clean up session expired because of idle timeout #121482

kibanamachine commented Dec 17, 2021 •

edited

Loading

elasticmachine commented Dec 17, 2021

azasypkin commented Dec 17, 2021

kibanamachine commented Dec 18, 2021

azasypkin commented Dec 22, 2021

azasypkin commented Dec 24, 2021 •

edited

Loading

kibanamachine commented Dec 28, 2021

kibanamachine commented Dec 28, 2021

azasypkin commented Dec 28, 2021

kibanamachine commented Jan 20, 2022

azasypkin commented Jan 24, 2022

kibanamachine commented Feb 11, 2022

azasypkin commented Feb 11, 2022

kibanamachine commented Mar 4, 2022

jportner commented Mar 4, 2022

kibanamachine commented Mar 8, 2022

kibanamachine commented Mar 10, 2022

kibanamachine commented Mar 12, 2023

kibanamachine commented Mar 13, 2023

kibanamachine commented Mar 15, 2023

jbudz commented Mar 15, 2023

kibanamachine commented Mar 15, 2023

kibanamachine commented May 12, 2023

kibanamachine commented May 12, 2023

kibanamachine commented May 12, 2023

kibanamachine commented May 12, 2023

jeramysoucy commented May 16, 2023

kibanamachine commented Aug 5, 2024

Failing test: X-Pack Security API Integration Tests (Session Idle Timeout).x-pack/test/security_api_integration/tests/session_idle/cleanup·ts - security APIs - Session Idle Session Idle cleanup should properly clean up session expired because of idle timeout #121482

Failing test: X-Pack Security API Integration Tests (Session Idle Timeout).x-pack/test/security_api_integration/tests/session_idle/cleanup·ts - security APIs - Session Idle Session Idle cleanup should properly clean up session expired because of idle timeout #121482

Comments

kibanamachine commented Dec 17, 2021 • edited Loading

elasticmachine commented Dec 17, 2021

azasypkin commented Dec 17, 2021

kibanamachine commented Dec 18, 2021

azasypkin commented Dec 22, 2021

azasypkin commented Dec 24, 2021 • edited Loading

kibanamachine commented Dec 28, 2021

kibanamachine commented Dec 28, 2021

azasypkin commented Dec 28, 2021

kibanamachine commented Jan 20, 2022

azasypkin commented Jan 24, 2022

kibanamachine commented Feb 11, 2022

azasypkin commented Feb 11, 2022

kibanamachine commented Mar 4, 2022

jportner commented Mar 4, 2022

kibanamachine commented Mar 8, 2022

kibanamachine commented Mar 10, 2022

kibanamachine commented Mar 12, 2023

kibanamachine commented Mar 13, 2023

kibanamachine commented Mar 15, 2023

jbudz commented Mar 15, 2023

kibanamachine commented Mar 15, 2023

kibanamachine commented May 12, 2023

kibanamachine commented May 12, 2023

kibanamachine commented May 12, 2023

kibanamachine commented May 12, 2023

jeramysoucy commented May 16, 2023

kibanamachine commented Aug 5, 2024

kibanamachine commented Dec 17, 2021 •

edited

Loading

azasypkin commented Dec 24, 2021 •

edited

Loading