[Onboarding] Implement new design for the AWS service list #191989
Assignees
Labels
Comments
|
Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs) |
|
Pinging @elastic/obs-ux-onboarding-team (Feature: Observability Onboarding) |
mykolaharmash
added a commit
that referenced
this issue
Sep 18, 2024
Closes #191989 Closes #190799 Closes #191731 This change implements the design changes done to improve the AWS service discovery in the new Firehose flow. [Figma](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=454-24601&t=Y7saMXwJfMinghMq-1) https://github.com/user-attachments/assets/57e0bbb3-1ace-42df-ae6d-5e34d0fd9368 ### How To Test You going to need an AWS account. You can use a personal one or "Elastic Observability" account which you can access through Okta (type "AWS" in Okta's search and you should see "AWS - Elastic Observability"). In case you decide to use the shared "Elastic Observability" account, make sure it does not already have `Elastic-CloudwatchLogsAndMetricsToFirehose` CloudFormation stack left from the previous tester. Feel free to delete it if it's there. 1. In AWS account, create a few entities that generate logs and put them into a CloudWatch log group (see instructions below for a few services). 1. Generate some logs by accessing the entities that you've created and make sure they appear in CloudWatch (mind that there is a ~1 minute delay). **If you don't see anything in CloudWatch, there is no point in proceeding further, make sure to fix your AWS setup before starting the flow in Kibana.** 1. Go to the serverless Kibana instance deployed from this PR (see the latest `[Deploy Serverless Kibana] ...` comment by ` kibanamachine`) 1. Add Data → Collect and analyze logs → View AWS Collection → Firehose quickstart 1. Open the Firehose flow and create CloudFormation stack using one of the two options. 1. Wait for the stack to finish creating. 1. Generate some some logs by accessing the AWS services you've created. 1. Go back to the Kibana screen, after a minute or so incoming logs should be detected and corresponding AWS service will be appear. ### Example AWS Services Configs **Before creating any resources, make sure you're in the same region (top right corner in AWS Console) you've used while configuring AWS CLI.** #### API Gateway 1. [Create an IAM role](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html#set-up-access-logging-permissions) to grant API Gateway permissions to write into a CloudWatch log groups 1. Copy the ARN of the created role 1. Open "CloudWatch" in AWS and select "Log groups" in the sidebar 1. Create a new log group with default parameters 1. Copy the ARN of the new group 1. Open **API Gateway** in AWS 1. Navigate to "Settings" in the sidebar 1. In the "Logging" section click "Edit" and paste the ARN of the IAM role you created in step 1. Hit "Save changes" 1. Now go back to "APIs" in the sidebar and click "Create API" 1. In "REST API" click "Build" 1. Select "Example API" and click "Create API" 1. Click on "Deploy API" 1. For "Stage" dropdown select "New stage", give it any name and click "Deploy" 1. You will now see "Invoke URL", you can use it later to access this API and generate logs 1. Scroll to "Logs and tracing" section and click "Edit" 1. In the dropdown select "Full request and response logs" 1. Toggle "Custom access logging" 1. Paste the ARN of the CloudWatch log group you've created in step 4. But make sure to not include ":*" symbols at the end. 1. In the log format input paste [this format from our docs](https://www.elastic.co/docs/current/integrations/aws/apigateway#data-streams) and click "Save" ``` {"requestId":"$context.requestId","ip":"$context.identity.sourceIp","caller":"$context.identity.caller","user":"$context.identity.user","requestTime":"$context.requestTime","httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath","status":"$context.status","protocol":"$context.protocol","responseLength":"$context.responseLength","apiId":"$context.apiId","domainName":"$context.domainName","stage":"$context.stage"} ``` 1. Now when you access this API, you should see logs coming into the CloudWatch group. #### WAF **This sets up WAF for an API Gateway, see above if you don't have one already.** 1. Open WAF in AWS 3. Click "Web ACLs" in the sidebar 4. Click "Create web ACL" 5. Select the region where you've created your API Gateway and give ACL any name 6. In the "Associated AWS resources" section click "Add AWS resources" 7. Select you API Gateway and click "Add" 8. Click "Next" 9. Create some basic rule, for example to block requests that have a specific parameter in the URL 10. Click through the other configuration step leaving everything as is and then finally click "Create web ACL" 11. Select the created ACL and click on the "Logging and metrics" tab 12. Click "Edit" in "Logging" section 13. Click "Create new" in the "Amazon CloudWatch Logs log group" section 14. Create a new log group. **The log group name should start with `aws-waf-logs-`**. 15. Select the new group in the dropdown and click "Save" 16. Now you should have logs generated and saved into the log group when you access your API gateway #### VPC 1. [Create an IAM role](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html#flow-logs-iam) to write flow logs to CloudWatch log groups. 3. Create and EC2 instance and configure there some HTTP server like Nginx. Using Docker would probably be the fastest way. 4. Create a CloudWatch log group with default parameters 5. Open "VPC" in AWS and select the VPC where you've created the EC2 instance. 6. Click the "Flow logs" tab and click "Create flow logs" 7. In "Maximum aggregation interval" select 1 minute to see logs faster 8. In "Destination log group" select the log group you've created in step 3 9. In "IAM role" select the role you've created in step 1 10. Click "Create flow log" 11. Now when you access your EC2 instance, you should see logs in the CloudWatch log group
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Sep 18, 2024
…ic#192860) Closes elastic#191989 Closes elastic#190799 Closes elastic#191731 This change implements the design changes done to improve the AWS service discovery in the new Firehose flow. [Figma](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=454-24601&t=Y7saMXwJfMinghMq-1) https://github.com/user-attachments/assets/57e0bbb3-1ace-42df-ae6d-5e34d0fd9368 ### How To Test You going to need an AWS account. You can use a personal one or "Elastic Observability" account which you can access through Okta (type "AWS" in Okta's search and you should see "AWS - Elastic Observability"). In case you decide to use the shared "Elastic Observability" account, make sure it does not already have `Elastic-CloudwatchLogsAndMetricsToFirehose` CloudFormation stack left from the previous tester. Feel free to delete it if it's there. 1. In AWS account, create a few entities that generate logs and put them into a CloudWatch log group (see instructions below for a few services). 1. Generate some logs by accessing the entities that you've created and make sure they appear in CloudWatch (mind that there is a ~1 minute delay). **If you don't see anything in CloudWatch, there is no point in proceeding further, make sure to fix your AWS setup before starting the flow in Kibana.** 1. Go to the serverless Kibana instance deployed from this PR (see the latest `[Deploy Serverless Kibana] ...` comment by ` kibanamachine`) 1. Add Data → Collect and analyze logs → View AWS Collection → Firehose quickstart 1. Open the Firehose flow and create CloudFormation stack using one of the two options. 1. Wait for the stack to finish creating. 1. Generate some some logs by accessing the AWS services you've created. 1. Go back to the Kibana screen, after a minute or so incoming logs should be detected and corresponding AWS service will be appear. ### Example AWS Services Configs **Before creating any resources, make sure you're in the same region (top right corner in AWS Console) you've used while configuring AWS CLI.** #### API Gateway 1. [Create an IAM role](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html#set-up-access-logging-permissions) to grant API Gateway permissions to write into a CloudWatch log groups 1. Copy the ARN of the created role 1. Open "CloudWatch" in AWS and select "Log groups" in the sidebar 1. Create a new log group with default parameters 1. Copy the ARN of the new group 1. Open **API Gateway** in AWS 1. Navigate to "Settings" in the sidebar 1. In the "Logging" section click "Edit" and paste the ARN of the IAM role you created in step 1. Hit "Save changes" 1. Now go back to "APIs" in the sidebar and click "Create API" 1. In "REST API" click "Build" 1. Select "Example API" and click "Create API" 1. Click on "Deploy API" 1. For "Stage" dropdown select "New stage", give it any name and click "Deploy" 1. You will now see "Invoke URL", you can use it later to access this API and generate logs 1. Scroll to "Logs and tracing" section and click "Edit" 1. In the dropdown select "Full request and response logs" 1. Toggle "Custom access logging" 1. Paste the ARN of the CloudWatch log group you've created in step 4. But make sure to not include ":*" symbols at the end. 1. In the log format input paste [this format from our docs](https://www.elastic.co/docs/current/integrations/aws/apigateway#data-streams) and click "Save" ``` {"requestId":"$context.requestId","ip":"$context.identity.sourceIp","caller":"$context.identity.caller","user":"$context.identity.user","requestTime":"$context.requestTime","httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath","status":"$context.status","protocol":"$context.protocol","responseLength":"$context.responseLength","apiId":"$context.apiId","domainName":"$context.domainName","stage":"$context.stage"} ``` 1. Now when you access this API, you should see logs coming into the CloudWatch group. #### WAF **This sets up WAF for an API Gateway, see above if you don't have one already.** 1. Open WAF in AWS 3. Click "Web ACLs" in the sidebar 4. Click "Create web ACL" 5. Select the region where you've created your API Gateway and give ACL any name 6. In the "Associated AWS resources" section click "Add AWS resources" 7. Select you API Gateway and click "Add" 8. Click "Next" 9. Create some basic rule, for example to block requests that have a specific parameter in the URL 10. Click through the other configuration step leaving everything as is and then finally click "Create web ACL" 11. Select the created ACL and click on the "Logging and metrics" tab 12. Click "Edit" in "Logging" section 13. Click "Create new" in the "Amazon CloudWatch Logs log group" section 14. Create a new log group. **The log group name should start with `aws-waf-logs-`**. 15. Select the new group in the dropdown and click "Save" 16. Now you should have logs generated and saved into the log group when you access your API gateway #### VPC 1. [Create an IAM role](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html#flow-logs-iam) to write flow logs to CloudWatch log groups. 3. Create and EC2 instance and configure there some HTTP server like Nginx. Using Docker would probably be the fastest way. 4. Create a CloudWatch log group with default parameters 5. Open "VPC" in AWS and select the VPC where you've created the EC2 instance. 6. Click the "Flow logs" tab and click "Create flow logs" 7. In "Maximum aggregation interval" select 1 minute to see logs faster 8. In "Destination log group" select the log group you've created in step 3 9. In "IAM role" select the role you've created in step 1 10. Click "Create flow log" 11. Now when you access your EC2 instance, you should see logs in the CloudWatch log group (cherry picked from commit c56281c)
kibanamachine
referenced
this issue
Sep 18, 2024
…192860) (#193278) # Backport This will backport the following commits from `main` to `8.x`: - [[Onboarding] AWS Service detection re-design for Firehose flow (#192860)](#192860) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Mykola Harmash","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-18T11:53:23Z","message":"[Onboarding] AWS Service detection re-design for Firehose flow (#192860)\n\nCloses https://github.com/elastic/kibana/issues/191989\r\nCloses https://github.com/elastic/kibana/issues/190799\r\nCloses https://github.com/elastic/kibana/issues/191731\r\n\r\nThis change implements the design changes done to improve the AWS\r\nservice discovery in the new Firehose flow.\r\n\r\n\r\n[Figma](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=454-24601&t=Y7saMXwJfMinghMq-1)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/57e0bbb3-1ace-42df-ae6d-5e34d0fd9368\r\n\r\n### How To Test\r\n\r\nYou going to need an AWS account. You can use a personal one or \"Elastic\r\nObservability\" account which you can access through Okta (type \"AWS\" in\r\nOkta's search and you should see \"AWS - Elastic Observability\").\r\n\r\nIn case you decide to use the shared \"Elastic Observability\" account,\r\nmake sure it does not already have\r\n`Elastic-CloudwatchLogsAndMetricsToFirehose` CloudFormation stack left\r\nfrom the previous tester. Feel free to delete it if it's there.\r\n\r\n1. In AWS account, create a few entities that generate logs and put them\r\ninto a CloudWatch log group (see instructions below for a few services).\r\n1. Generate some logs by accessing the entities that you've created and\r\nmake sure they appear in CloudWatch (mind that there is a ~1 minute\r\ndelay). **If you don't see anything in CloudWatch, there is no point in\r\nproceeding further, make sure to fix your AWS setup before starting the\r\nflow in Kibana.**\r\n1. Go to the serverless Kibana instance deployed from this PR (see the\r\nlatest `[Deploy Serverless Kibana] ...` comment by ` kibanamachine`)\r\n1. Add Data → Collect and analyze logs → View AWS Collection → Firehose\r\nquickstart\r\n1. Open the Firehose flow and create CloudFormation stack using one of\r\nthe two options.\r\n1. Wait for the stack to finish creating.\r\n1. Generate some some logs by accessing the AWS services you've created.\r\n1. Go back to the Kibana screen, after a minute or so incoming logs\r\nshould be detected and corresponding AWS service will be appear.\r\n\r\n\r\n### Example AWS Services Configs\r\n\r\n**Before creating any resources, make sure you're in the same region\r\n(top right corner in AWS Console) you've used while configuring AWS\r\nCLI.**\r\n\r\n#### API Gateway\r\n\r\n1. [Create an IAM\r\nrole](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html#set-up-access-logging-permissions)\r\nto grant API Gateway permissions to write into a CloudWatch log groups\r\n1. Copy the ARN of the created role\r\n1. Open \"CloudWatch\" in AWS and select \"Log groups\" in the sidebar\r\n1. Create a new log group with default parameters\r\n1. Copy the ARN of the new group\r\n1. Open **API Gateway** in AWS\r\n1. Navigate to \"Settings\" in the sidebar\r\n1. In the \"Logging\" section click \"Edit\" and paste the ARN of the IAM\r\nrole you created in step 1. Hit \"Save changes\"\r\n1. Now go back to \"APIs\" in the sidebar and click \"Create API\"\r\n1. In \"REST API\" click \"Build\"\r\n1. Select \"Example API\" and click \"Create API\"\r\n1. Click on \"Deploy API\"\r\n1. For \"Stage\" dropdown select \"New stage\", give it any name and click\r\n\"Deploy\"\r\n1. You will now see \"Invoke URL\", you can use it later to access this\r\nAPI and generate logs\r\n1. Scroll to \"Logs and tracing\" section and click \"Edit\"\r\n1. In the dropdown select \"Full request and response logs\"\r\n1. Toggle \"Custom access logging\"\r\n1. Paste the ARN of the CloudWatch log group you've created in step 4.\r\nBut make sure to not include \":*\" symbols at the end.\r\n1. In the log format input paste [this format from our\r\ndocs](https://www.elastic.co/docs/current/integrations/aws/apigateway#data-streams)\r\nand click \"Save\"\r\n```\r\n{\"requestId\":\"$context.requestId\",\"ip\":\"$context.identity.sourceIp\",\"caller\":\"$context.identity.caller\",\"user\":\"$context.identity.user\",\"requestTime\":\"$context.requestTime\",\"httpMethod\":\"$context.httpMethod\",\"resourcePath\":\"$context.resourcePath\",\"status\":\"$context.status\",\"protocol\":\"$context.protocol\",\"responseLength\":\"$context.responseLength\",\"apiId\":\"$context.apiId\",\"domainName\":\"$context.domainName\",\"stage\":\"$context.stage\"}\r\n```\r\n1. Now when you access this API, you should see logs coming into the\r\nCloudWatch group.\r\n\r\n#### WAF\r\n\r\n**This sets up WAF for an API Gateway, see above if you don't have one\r\nalready.**\r\n\r\n1. Open WAF in AWS\r\n3. Click \"Web ACLs\" in the sidebar\r\n4. Click \"Create web ACL\"\r\n5. Select the region where you've created your API Gateway and give ACL\r\nany name\r\n6. In the \"Associated AWS resources\" section click \"Add AWS resources\"\r\n7. Select you API Gateway and click \"Add\"\r\n8. Click \"Next\"\r\n9. Create some basic rule, for example to block requests that have a\r\nspecific parameter in the URL\r\n10. Click through the other configuration step leaving everything as is\r\nand then finally click \"Create web ACL\"\r\n11. Select the created ACL and click on the \"Logging and metrics\" tab\r\n12. Click \"Edit\" in \"Logging\" section \r\n13. Click \"Create new\" in the \"Amazon CloudWatch Logs log group\" section\r\n14. Create a new log group. **The log group name should start with\r\n`aws-waf-logs-`**.\r\n15. Select the new group in the dropdown and click \"Save\"\r\n16. Now you should have logs generated and saved into the log group when\r\nyou access your API gateway\r\n\r\n#### VPC\r\n\r\n1. [Create an IAM\r\nrole](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html#flow-logs-iam)\r\nto write flow logs to CloudWatch log groups.\r\n3. Create and EC2 instance and configure there some HTTP server like\r\nNginx. Using Docker would probably be the fastest way.\r\n4. Create a CloudWatch log group with default parameters\r\n5. Open \"VPC\" in AWS and select the VPC where you've created the EC2\r\ninstance.\r\n6. Click the \"Flow logs\" tab and click \"Create flow logs\"\r\n7. In \"Maximum aggregation interval\" select 1 minute to see logs faster\r\n8. In \"Destination log group\" select the log group you've created in\r\nstep 3\r\n9. In \"IAM role\" select the role you've created in step 1\r\n10. Click \"Create flow log\"\r\n11. Now when you access your EC2 instance, you should see logs in the\r\nCloudWatch log group","sha":"c56281ce80d35e616c0e734a8a008eb8af0987fe","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-major","ci:project-deploy-observability","v8.16.0"],"title":"[Onboarding] AWS Service detection re-design for Firehose flow","number":192860,"url":"https://github.com/elastic/kibana/pull/192860","mergeCommit":{"message":"[Onboarding] AWS Service detection re-design for Firehose flow (#192860)\n\nCloses https://github.com/elastic/kibana/issues/191989\r\nCloses https://github.com/elastic/kibana/issues/190799\r\nCloses https://github.com/elastic/kibana/issues/191731\r\n\r\nThis change implements the design changes done to improve the AWS\r\nservice discovery in the new Firehose flow.\r\n\r\n\r\n[Figma](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=454-24601&t=Y7saMXwJfMinghMq-1)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/57e0bbb3-1ace-42df-ae6d-5e34d0fd9368\r\n\r\n### How To Test\r\n\r\nYou going to need an AWS account. You can use a personal one or \"Elastic\r\nObservability\" account which you can access through Okta (type \"AWS\" in\r\nOkta's search and you should see \"AWS - Elastic Observability\").\r\n\r\nIn case you decide to use the shared \"Elastic Observability\" account,\r\nmake sure it does not already have\r\n`Elastic-CloudwatchLogsAndMetricsToFirehose` CloudFormation stack left\r\nfrom the previous tester. Feel free to delete it if it's there.\r\n\r\n1. In AWS account, create a few entities that generate logs and put them\r\ninto a CloudWatch log group (see instructions below for a few services).\r\n1. Generate some logs by accessing the entities that you've created and\r\nmake sure they appear in CloudWatch (mind that there is a ~1 minute\r\ndelay). **If you don't see anything in CloudWatch, there is no point in\r\nproceeding further, make sure to fix your AWS setup before starting the\r\nflow in Kibana.**\r\n1. Go to the serverless Kibana instance deployed from this PR (see the\r\nlatest `[Deploy Serverless Kibana] ...` comment by ` kibanamachine`)\r\n1. Add Data → Collect and analyze logs → View AWS Collection → Firehose\r\nquickstart\r\n1. Open the Firehose flow and create CloudFormation stack using one of\r\nthe two options.\r\n1. Wait for the stack to finish creating.\r\n1. Generate some some logs by accessing the AWS services you've created.\r\n1. Go back to the Kibana screen, after a minute or so incoming logs\r\nshould be detected and corresponding AWS service will be appear.\r\n\r\n\r\n### Example AWS Services Configs\r\n\r\n**Before creating any resources, make sure you're in the same region\r\n(top right corner in AWS Console) you've used while configuring AWS\r\nCLI.**\r\n\r\n#### API Gateway\r\n\r\n1. [Create an IAM\r\nrole](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html#set-up-access-logging-permissions)\r\nto grant API Gateway permissions to write into a CloudWatch log groups\r\n1. Copy the ARN of the created role\r\n1. Open \"CloudWatch\" in AWS and select \"Log groups\" in the sidebar\r\n1. Create a new log group with default parameters\r\n1. Copy the ARN of the new group\r\n1. Open **API Gateway** in AWS\r\n1. Navigate to \"Settings\" in the sidebar\r\n1. In the \"Logging\" section click \"Edit\" and paste the ARN of the IAM\r\nrole you created in step 1. Hit \"Save changes\"\r\n1. Now go back to \"APIs\" in the sidebar and click \"Create API\"\r\n1. In \"REST API\" click \"Build\"\r\n1. Select \"Example API\" and click \"Create API\"\r\n1. Click on \"Deploy API\"\r\n1. For \"Stage\" dropdown select \"New stage\", give it any name and click\r\n\"Deploy\"\r\n1. You will now see \"Invoke URL\", you can use it later to access this\r\nAPI and generate logs\r\n1. Scroll to \"Logs and tracing\" section and click \"Edit\"\r\n1. In the dropdown select \"Full request and response logs\"\r\n1. Toggle \"Custom access logging\"\r\n1. Paste the ARN of the CloudWatch log group you've created in step 4.\r\nBut make sure to not include \":*\" symbols at the end.\r\n1. In the log format input paste [this format from our\r\ndocs](https://www.elastic.co/docs/current/integrations/aws/apigateway#data-streams)\r\nand click \"Save\"\r\n```\r\n{\"requestId\":\"$context.requestId\",\"ip\":\"$context.identity.sourceIp\",\"caller\":\"$context.identity.caller\",\"user\":\"$context.identity.user\",\"requestTime\":\"$context.requestTime\",\"httpMethod\":\"$context.httpMethod\",\"resourcePath\":\"$context.resourcePath\",\"status\":\"$context.status\",\"protocol\":\"$context.protocol\",\"responseLength\":\"$context.responseLength\",\"apiId\":\"$context.apiId\",\"domainName\":\"$context.domainName\",\"stage\":\"$context.stage\"}\r\n```\r\n1. Now when you access this API, you should see logs coming into the\r\nCloudWatch group.\r\n\r\n#### WAF\r\n\r\n**This sets up WAF for an API Gateway, see above if you don't have one\r\nalready.**\r\n\r\n1. Open WAF in AWS\r\n3. Click \"Web ACLs\" in the sidebar\r\n4. Click \"Create web ACL\"\r\n5. Select the region where you've created your API Gateway and give ACL\r\nany name\r\n6. In the \"Associated AWS resources\" section click \"Add AWS resources\"\r\n7. Select you API Gateway and click \"Add\"\r\n8. Click \"Next\"\r\n9. Create some basic rule, for example to block requests that have a\r\nspecific parameter in the URL\r\n10. Click through the other configuration step leaving everything as is\r\nand then finally click \"Create web ACL\"\r\n11. Select the created ACL and click on the \"Logging and metrics\" tab\r\n12. Click \"Edit\" in \"Logging\" section \r\n13. Click \"Create new\" in the \"Amazon CloudWatch Logs log group\" section\r\n14. Create a new log group. **The log group name should start with\r\n`aws-waf-logs-`**.\r\n15. Select the new group in the dropdown and click \"Save\"\r\n16. Now you should have logs generated and saved into the log group when\r\nyou access your API gateway\r\n\r\n#### VPC\r\n\r\n1. [Create an IAM\r\nrole](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html#flow-logs-iam)\r\nto write flow logs to CloudWatch log groups.\r\n3. Create and EC2 instance and configure there some HTTP server like\r\nNginx. Using Docker would probably be the fastest way.\r\n4. Create a CloudWatch log group with default parameters\r\n5. Open \"VPC\" in AWS and select the VPC where you've created the EC2\r\ninstance.\r\n6. Click the \"Flow logs\" tab and click \"Create flow logs\"\r\n7. In \"Maximum aggregation interval\" select 1 minute to see logs faster\r\n8. In \"Destination log group\" select the log group you've created in\r\nstep 3\r\n9. In \"IAM role\" select the role you've created in step 1\r\n10. Click \"Create flow log\"\r\n11. Now when you access your EC2 instance, you should see logs in the\r\nCloudWatch log group","sha":"c56281ce80d35e616c0e734a8a008eb8af0987fe"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192860","number":192860,"mergeCommit":{"message":"[Onboarding] AWS Service detection re-design for Firehose flow (#192860)\n\nCloses https://github.com/elastic/kibana/issues/191989\r\nCloses https://github.com/elastic/kibana/issues/190799\r\nCloses https://github.com/elastic/kibana/issues/191731\r\n\r\nThis change implements the design changes done to improve the AWS\r\nservice discovery in the new Firehose flow.\r\n\r\n\r\n[Figma](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=454-24601&t=Y7saMXwJfMinghMq-1)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/57e0bbb3-1ace-42df-ae6d-5e34d0fd9368\r\n\r\n### How To Test\r\n\r\nYou going to need an AWS account. You can use a personal one or \"Elastic\r\nObservability\" account which you can access through Okta (type \"AWS\" in\r\nOkta's search and you should see \"AWS - Elastic Observability\").\r\n\r\nIn case you decide to use the shared \"Elastic Observability\" account,\r\nmake sure it does not already have\r\n`Elastic-CloudwatchLogsAndMetricsToFirehose` CloudFormation stack left\r\nfrom the previous tester. Feel free to delete it if it's there.\r\n\r\n1. In AWS account, create a few entities that generate logs and put them\r\ninto a CloudWatch log group (see instructions below for a few services).\r\n1. Generate some logs by accessing the entities that you've created and\r\nmake sure they appear in CloudWatch (mind that there is a ~1 minute\r\ndelay). **If you don't see anything in CloudWatch, there is no point in\r\nproceeding further, make sure to fix your AWS setup before starting the\r\nflow in Kibana.**\r\n1. Go to the serverless Kibana instance deployed from this PR (see the\r\nlatest `[Deploy Serverless Kibana] ...` comment by ` kibanamachine`)\r\n1. Add Data → Collect and analyze logs → View AWS Collection → Firehose\r\nquickstart\r\n1. Open the Firehose flow and create CloudFormation stack using one of\r\nthe two options.\r\n1. Wait for the stack to finish creating.\r\n1. Generate some some logs by accessing the AWS services you've created.\r\n1. Go back to the Kibana screen, after a minute or so incoming logs\r\nshould be detected and corresponding AWS service will be appear.\r\n\r\n\r\n### Example AWS Services Configs\r\n\r\n**Before creating any resources, make sure you're in the same region\r\n(top right corner in AWS Console) you've used while configuring AWS\r\nCLI.**\r\n\r\n#### API Gateway\r\n\r\n1. [Create an IAM\r\nrole](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html#set-up-access-logging-permissions)\r\nto grant API Gateway permissions to write into a CloudWatch log groups\r\n1. Copy the ARN of the created role\r\n1. Open \"CloudWatch\" in AWS and select \"Log groups\" in the sidebar\r\n1. Create a new log group with default parameters\r\n1. Copy the ARN of the new group\r\n1. Open **API Gateway** in AWS\r\n1. Navigate to \"Settings\" in the sidebar\r\n1. In the \"Logging\" section click \"Edit\" and paste the ARN of the IAM\r\nrole you created in step 1. Hit \"Save changes\"\r\n1. Now go back to \"APIs\" in the sidebar and click \"Create API\"\r\n1. In \"REST API\" click \"Build\"\r\n1. Select \"Example API\" and click \"Create API\"\r\n1. Click on \"Deploy API\"\r\n1. For \"Stage\" dropdown select \"New stage\", give it any name and click\r\n\"Deploy\"\r\n1. You will now see \"Invoke URL\", you can use it later to access this\r\nAPI and generate logs\r\n1. Scroll to \"Logs and tracing\" section and click \"Edit\"\r\n1. In the dropdown select \"Full request and response logs\"\r\n1. Toggle \"Custom access logging\"\r\n1. Paste the ARN of the CloudWatch log group you've created in step 4.\r\nBut make sure to not include \":*\" symbols at the end.\r\n1. In the log format input paste [this format from our\r\ndocs](https://www.elastic.co/docs/current/integrations/aws/apigateway#data-streams)\r\nand click \"Save\"\r\n```\r\n{\"requestId\":\"$context.requestId\",\"ip\":\"$context.identity.sourceIp\",\"caller\":\"$context.identity.caller\",\"user\":\"$context.identity.user\",\"requestTime\":\"$context.requestTime\",\"httpMethod\":\"$context.httpMethod\",\"resourcePath\":\"$context.resourcePath\",\"status\":\"$context.status\",\"protocol\":\"$context.protocol\",\"responseLength\":\"$context.responseLength\",\"apiId\":\"$context.apiId\",\"domainName\":\"$context.domainName\",\"stage\":\"$context.stage\"}\r\n```\r\n1. Now when you access this API, you should see logs coming into the\r\nCloudWatch group.\r\n\r\n#### WAF\r\n\r\n**This sets up WAF for an API Gateway, see above if you don't have one\r\nalready.**\r\n\r\n1. Open WAF in AWS\r\n3. Click \"Web ACLs\" in the sidebar\r\n4. Click \"Create web ACL\"\r\n5. Select the region where you've created your API Gateway and give ACL\r\nany name\r\n6. In the \"Associated AWS resources\" section click \"Add AWS resources\"\r\n7. Select you API Gateway and click \"Add\"\r\n8. Click \"Next\"\r\n9. Create some basic rule, for example to block requests that have a\r\nspecific parameter in the URL\r\n10. Click through the other configuration step leaving everything as is\r\nand then finally click \"Create web ACL\"\r\n11. Select the created ACL and click on the \"Logging and metrics\" tab\r\n12. Click \"Edit\" in \"Logging\" section \r\n13. Click \"Create new\" in the \"Amazon CloudWatch Logs log group\" section\r\n14. Create a new log group. **The log group name should start with\r\n`aws-waf-logs-`**.\r\n15. Select the new group in the dropdown and click \"Save\"\r\n16. Now you should have logs generated and saved into the log group when\r\nyou access your API gateway\r\n\r\n#### VPC\r\n\r\n1. [Create an IAM\r\nrole](https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html#flow-logs-iam)\r\nto write flow logs to CloudWatch log groups.\r\n3. Create and EC2 instance and configure there some HTTP server like\r\nNginx. Using Docker would probably be the fastest way.\r\n4. Create a CloudWatch log group with default parameters\r\n5. Open \"VPC\" in AWS and select the VPC where you've created the EC2\r\ninstance.\r\n6. Click the \"Flow logs\" tab and click \"Create flow logs\"\r\n7. In \"Maximum aggregation interval\" select 1 minute to see logs faster\r\n8. In \"Destination log group\" select the log group you've created in\r\nstep 3\r\n9. In \"IAM role\" select the role you've created in step 1\r\n10. Click \"Create flow log\"\r\n11. Now when you access your EC2 instance, you should see logs in the\r\nCloudWatch log group","sha":"c56281ce80d35e616c0e734a8a008eb8af0987fe"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Mykola Harmash <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Design
The text was updated successfully, but these errors were encountered: