|
| 1 | +# CloudWatch Logs large query - Technical specification |
| 2 | + |
| 3 | +This document contains the technical specifications for _CloudWatch Logs large query_, |
| 4 | +a feature scenario that showcases AWS services and SDKs. It is primarily intended for the AWS code |
| 5 | +examples team to use while developing this example in additional languages. |
| 6 | + |
| 7 | +This document explains the following: |
| 8 | + |
| 9 | +- Deploying AWS resources. |
| 10 | +- Adding sample data. |
| 11 | +- Setting up a large query. |
| 12 | + |
| 13 | +For an introduction, see the [README.md](README.md). |
| 14 | + |
| 15 | +--- |
| 16 | + |
| 17 | +### Table of contents |
| 18 | + |
| 19 | +- [Architecture](#architecture) |
| 20 | +- [User input](#user-input) |
| 21 | +- [Common resources](#common-resources) |
| 22 | +- [Building the queries](#building-the-queries) |
| 23 | +- [Output](#output) |
| 24 | +- [Metadata](#metadata) |
| 25 | + |
| 26 | +## Architecture |
| 27 | + |
| 28 | +- Amazon CloudWatch Logs group |
| 29 | +- Amazon CloudWatch Logs stream |
| 30 | + |
| 31 | +--- |
| 32 | + |
| 33 | +## User input |
| 34 | + |
| 35 | +The example should allow the configuration of a query start date, query end date, and results limit. It's up to you to decide how to allow this configuration. |
| 36 | + |
| 37 | +### Suggested variable names |
| 38 | + |
| 39 | +- `QUERY_START_DATE` - The oldest date that will be queried. |
| 40 | +- `QUERY_END_DATE` - The newest date that will be queried. |
| 41 | +- `QUERY_LIMIT` - The maximum number of results to return. CloudWatch has a maximum of 10,000. |
| 42 | + |
| 43 | +--- |
| 44 | + |
| 45 | +## Common resources |
| 46 | + |
| 47 | +This example has a set of common resources that are stored in the [resources](resources) folder. |
| 48 | + |
| 49 | +- [stack.yaml](resources/stack.yaml) is an AWS CloudFormation template containing the resources needed to run this example. |
| 50 | +- [make-log-files.sh](resources/make-log-files.sh) is a bash script that creates log data. **Five minutes of logs, starting at the time of execution, will be created. Wait at least five minutes after running this script before attempting to query.** |
| 51 | +- [put-log-events](resources/put-log-events.sh) is a bash script that ingests log data and uploads it to CloudWatch. |
| 52 | + |
| 53 | +--- |
| 54 | + |
| 55 | +## Building the queries |
| 56 | + |
| 57 | +### Building and waiting for single query |
| 58 | + |
| 59 | +The query itself is a "CloudWatch Logs Insights query syntax" string. The query must return the `@timestamp` field so follow-up queries can use that information. Here's a sample query string: `fields @timestamp, @message | sort @timestamp asc`. Notice it sorts in ascending order. You can sort in either `asc` or `desc`, but the recursive strategy described later will need to match accordingly. |
| 60 | + |
| 61 | +Queries are jobs. You can start a query with `StartQuery`, but it immediately returns the `queryId`. You must poll a query using `GetQueryResults` until the query has finished. For the purpose of this example, a query has "finished" when `GetQueryResults` has returned a status of one of "Complete", "Failed", "Cancelled", "Timeout", or "Unknown". |
| 62 | + |
| 63 | +`StartQuery` responds with an error if the query's start or end date occurs out of bounds of the log group creation date. The error message starts with "Query's end date and time". |
| 64 | + |
| 65 | +Start the query and wait for it to "finish". Store the `results`. If the count of the results is less than the configured LIMIT, return the results. If the results are greater than or equal to the limit, go to [Recursive queries](#recursive-queries). |
| 66 | + |
| 67 | +--- |
| 68 | + |
| 69 | +### Recursive queries |
| 70 | + |
| 71 | +If the result count from the previous step is 10000 (or the configured LIMIT), it is very likely that there are more results. **The example must do a binary search of the remaining logs**. To do this, get the date of the last log (earliest or latest, depending on sort order). Use that date as the start date of a new date range. The end date can remain the same. |
| 72 | + |
| 73 | +Split that date range in half, resulting in two new date ranges. Call your query function twice; once for each new date range. |
| 74 | + |
| 75 | +Concatenate the results of the first query with the results of the two new queries. |
| 76 | + |
| 77 | +The following pseudocode illustrates this. |
| 78 | + |
| 79 | +```pseudocode |
| 80 | +func large_query(date_range): |
| 81 | + query_results = get_query_results(date_range) |
| 82 | +
|
| 83 | + if query_results.length < LIMIT |
| 84 | + return query_results |
| 85 | + else |
| 86 | + date_range = [query_results.end, date_range.end] |
| 87 | + d1, d2 = split(date_range) |
| 88 | + return concat(query_results, large_query(d1), large_query(d2)) |
| 89 | +``` |
| 90 | + |
| 91 | +## Output |
| 92 | + |
| 93 | +To illustrate the search, log the date ranges for each query made and the number of logs that were found. |
| 94 | + |
| 95 | +Example: |
| 96 | + |
| 97 | +``` |
| 98 | +Starting a recursive query... |
| 99 | +Query date range: 2023-12-22T19:08:42.000Z to 2023-12-22T19:13:41.994Z. Found 10000 logs. |
| 100 | +Query date range: 2023-12-22T19:09:41.995Z to 2023-12-22T19:11:41.994Z. Found 10000 logs. |
| 101 | +Query date range: 2023-12-22T19:11:41.995Z to 2023-12-22T19:13:41.994Z. Found 10000 logs. |
| 102 | +Query date range: 2023-12-22T19:10:41.995Z to 2023-12-22T19:11:11.994Z. Found 5000 logs. |
| 103 | +Query date range: 2023-12-22T19:11:11.995Z to 2023-12-22T19:11:41.994Z. Found 5000 logs. |
| 104 | +Query date range: 2023-12-22T19:12:41.995Z to 2023-12-22T19:13:11.994Z. Found 5000 logs. |
| 105 | +Query date range: 2023-12-22T19:13:11.995Z to 2023-12-22T19:13:41.994Z. Found 5000 logs. |
| 106 | +Queries finished in 11.253 seconds. |
| 107 | +Total logs found: 50000 |
| 108 | +``` |
| 109 | + |
| 110 | +--- |
| 111 | + |
| 112 | +## Metadata |
| 113 | + |
| 114 | +| action / scenario | metadata file | metadata key | |
| 115 | +| ----------------- | ----------------------------- | --------------------------------- | |
| 116 | +| `GetQueryResults` | cloudwatch-logs_metadata.yaml | cloudwatch-logs_GetQueryResults | |
| 117 | +| `StartQuery` | cloudwatch-logs_metadata.yaml | cloudwatch-logs_StartQuery | |
| 118 | +| `Large Query` | cloudwatch-logs_metadata.yaml | cloudwatch-logs_Scenario_LargeQuery | |
0 commit comments