Adding Partition Execution Logging for DimensionalTimeSliceCrawler #6362

chrisale000 · 2025-12-18T06:01:45Z

Description

Implemented enhanced partition logging to facilitate debugging and observability of partition processing workflows in DimensionalTimeSliceCrawler and added a 2 minute limitter for partition batch processing.

Testing

M365 Pipeline Validation

Successfully deployed and validated the M365 pipeline with the following observations:

2026-01-06T10:13:04,117 [pool-7-thread-4] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-06T18:08:03.989022Z to 2026-01-06T18:08:04.036494Z
2026-01-06T10:13:04,117 [pool-7-thread-3] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-06T18:08:03.989022Z to 2026-01-06T18:08:04.036494Z
2026-01-06T10:13:04,119 [pool-7-thread-2] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-06T18:08:03.989022Z to 2026-01-06T18:08:04.036494Z
2026-01-06T10:13:04,119 [pool-7-thread-5] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-06T18:08:03.989022Z to 2026-01-06T18:08:04.036494Z
2026-01-06T10:14:04,052 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:15:04,061 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:15:04,123 [pool-7-thread-3] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-06T18:08:03.989022Z to 2026-01-06T18:08:04.036494Z
2026-01-06T10:15:04,123 [pool-7-thread-4] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-06T18:08:04.036494Z to 2026-01-06T18:09:04.051336Z
2026-01-06T10:15:04,125 [pool-7-thread-5] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-06T18:08:04.036494Z to 2026-01-06T18:09:04.051336Z
2026-01-06T10:15:04,125 [pool-7-thread-2] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-06T18:08:04.036494Z to 2026-01-06T18:09:04.051336Z
2026-01-06T10:16:04,070 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:17:04,075 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:17:04,128 [pool-7-thread-5] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-06T18:08:04.036494Z to 2026-01-06T18:09:04.051336Z
2026-01-06T10:17:04,129 [pool-7-thread-4] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-06T18:08:04.036494Z to 2026-01-06T18:09:04.051336Z
2026-01-06T10:17:04,129 [pool-7-thread-3] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-06T18:09:04.051336Z to 2026-01-06T18:10:04.057166Z
2026-01-06T10:17:04,131 [pool-7-thread-2] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-06T18:09:04.051336Z to 2026-01-06T18:10:04.057166Z
2026-01-06T10:18:04,086 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:19:04,094 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:19:04,134 [pool-7-thread-5] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-06T18:09:04.051336Z to 2026-01-06T18:10:04.057166Z
2026-01-06T10:19:04,134 [pool-7-thread-3] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-06T18:09:04.051336Z to 2026-01-06T18:10:04.057166Z
2026-01-06T10:19:04,135 [pool-7-thread-4] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-06T18:09:04.051336Z to 2026-01-06T18:10:04.057166Z
2026-01-06T10:19:04,137 [pool-7-thread-2] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-06T18:10:04.057166Z to 2026-01-06T18:11:04.067885Z
2026-01-06T10:20:04,107 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:21:04,117 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:21:04,139 [pool-7-thread-3] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-06T18:10:04.057166Z to 2026-01-06T18:11:04.067885Z
2026-01-06T10:21:04,139 [pool-7-thread-5] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-06T18:10:04.057166Z to 2026-01-06T18:11:04.067885Z
2026-01-06T10:21:04,142 [pool-7-thread-4] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: DLP.All, TimeRange: 2026-01-06T18:10:04.057166Z to 2026-01-06T18:11:04.067885Z
2026-01-06T10:21:04,142 [pool-7-thread-2] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-06T18:10:04.057166Z to 2026-01-06T18:11:04.067885Z
2026-01-06T10:22:04,126 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:23:04,133 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0
2026-01-06T10:23:04,145 [pool-7-thread-5] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.AzureActiveDirectory, TimeRange: 2026-01-06T18:11:04.067885Z to 2026-01-06T18:12:04.073043Z
2026-01-06T10:23:04,146 [pool-7-thread-3] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.Exchange, TimeRange: 2026-01-06T18:11:04.067885Z to 2026-01-06T18:12:04.073043Z
2026-01-06T10:23:04,146 [pool-7-thread-2] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.SharePoint, TimeRange: 2026-01-06T18:11:04.067885Z to 2026-01-06T18:12:04.073043Z
2026-01-06T10:23:04,147 [pool-7-thread-4] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Processing partition - DimensionType: Audit.General, TimeRange: 2026-01-06T18:11:04.067885Z to 2026-01-06T18:12:04.073043Z
2026-01-06T10:24:04,142 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Total partitions created in this crawl: 5.0

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

wandna-amazon

Thank for this! Just want to raise that sometimes the Total partitions created in this crawl: 0.0 shows 0 when it actually makes partitions if you look at the metric. Not sure why.

wjyao0316 · 2025-12-19T19:13:16Z

...g/opensearch/dataprepper/plugins/source/source_crawler/base/DimensionalTimeSliceCrawler.java

    @Override
    public void executePartition(DimensionalTimeSliceWorkerProgressState state, Buffer<Record<Event>> buffer, AcknowledgementSet acknowledgementSet) {
+        log.info("Processing partition - DimensionType: {}, TimeRange: {} to {}, ProcessingTime: {}", 
+                 state.getDimensionType(), state.getStartTime(), state.getEndTime(), Instant.now());


The log.info has the now timestamp. I don't think the last one is needed.

This will be a super noisy log. Do we need to log this? if yes then may be change it to log.debug

Since the partitions are small, I do think this is too noisy, even if it's helpful. debug makes more sense.

Log debug makes sense, we also want to add this logging to another crawler. I will make sure both are debug.

I do think log.info is necessary to help us debug. Otherwise it would not be as helpful.
1 log line per minute might not be too bad.

Are you ok we make each partition duration longer like 2 minutes and change to log.info?

wjyao0316 · 2025-12-22T17:57:52Z

...va/org/opensearch/dataprepper/plugins/source/source_crawler/base/LeaderOnlyTokenCrawler.java

    @Override
    public void executePartition(PaginationCrawlerWorkerProgressState state, Buffer buffer, AcknowledgementSet acknowledgementSet) {
+        String firstToken = !state.getItemIds().isEmpty() ? state.getItemIds().get(0) : "";
+        String lastToken = !state.getItemIds().isEmpty() ? state.getItemIds().get(state.getItemIds().size()-1) : "";


Please add necessary log information calling out this is for RetryPartition.

wjyao0316 · 2025-12-22T18:02:43Z

...va/org/opensearch/dataprepper/plugins/source/source_crawler/base/LeaderOnlyTokenCrawler.java

    private void processBatch(List<ItemInfo> batch,
                              LeaderPartition leaderPartition,
                              EnhancedSourceCoordinator coordinator) {
+        String firstToken = !batch.isEmpty() ? batch.get(0).getItemId() : "(No First Token in this batch)";


Please take a look at the LeaderOnlyTokenCrawler design. The main processing is in Leader Thread crawl function.

Please log the lastToken in https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/saas-source-plugins/source-crawler/src/main/java/org/opensearch/dataprepper/plugins/source/source_crawler/base/LeaderOnlyTokenCrawler.java#L82 so the system record necessary information before an process.

We don't record it in processing so the system can catch error when encountering the failure when retrieving the data.

The last item Id is the last token, additionally the last token is also emitted here on startup and here on updates. However I noticed during initial runs, the first token used is not logged.

wjyao0316 · 2026-01-06T19:09:55Z

...g/opensearch/dataprepper/plugins/source/source_crawler/base/DimensionalTimeSliceCrawler.java

+        Instant executionStart = Instant.now();
        partitionProcessLatencyTimer.record(() -> client.executePartition(state, buffer, acknowledgementSet));
+
+        // Enforce 2 minute minimum execution duration


I don't recall we do this any where. In addition, we don't want to add 2 min delay inside partition execution because it blocks the thread and prevent it from doing other things.

There is already 5 min delay added before creating the partition.

Based on offline discussion, I misunderstood the previous comment #6362 (comment)

graytaylor0 · 2026-01-06T22:26:24Z

...g/opensearch/dataprepper/plugins/source/source_crawler/base/DimensionalTimeSliceCrawler.java


    @Override
    public void executePartition(DimensionalTimeSliceWorkerProgressState state, Buffer<Record<Event>> buffer, AcknowledgementSet acknowledgementSet) {
+        log.info("Processing partition - DimensionType: {}, TimeRange: {} to {}", 


While this log is useful, it is somewhat expensive since it will be 1 log per minute

san81 · 2026-01-06T23:38:55Z

@chrisale000 Seems like the commit is not signed. Please fix it

chrisale000 · 2026-01-07T17:11:59Z

Signed-off-by: Alexander Christensen <alchrisk@amazon.com>

chrisale000 requested review from KarstenSchnitter, chenqi0805, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners December 18, 2025 06:01

vecheka approved these changes Dec 18, 2025

View reviewed changes

wandna-amazon approved these changes Dec 18, 2025

View reviewed changes

chrisale000 changed the title ~~Adding Partition Timestamp Log~~ Adding Partition Execution Log for DimensionalTimeSliceCrawler Dec 19, 2025

chrisale000 force-pushed the timeSliceLog branch from dfc1690 to 8badf25 Compare December 19, 2025 18:01

wjyao0316 reviewed Dec 19, 2025

View reviewed changes

chrisale000 changed the title ~~Adding Partition Execution Log for DimensionalTimeSliceCrawler~~ Adding Partition Execution Logging for DimensionalTimeSliceCrawler and LeaderOnlyTokenCrawler Dec 22, 2025

chrisale000 force-pushed the timeSliceLog branch 2 times, most recently from dcc52e6 to 2ede0d9 Compare December 22, 2025 17:25

wjyao0316 reviewed Dec 22, 2025

View reviewed changes

chrisale000 force-pushed the timeSliceLog branch from 2ede0d9 to d68311b Compare January 6, 2026 19:01

chrisale000 changed the title ~~Adding Partition Execution Logging for DimensionalTimeSliceCrawler and LeaderOnlyTokenCrawler~~ Adding Partition Execution Logging for DimensionalTimeSliceCrawler and 2 minute Partition Execution Delay for Each Partition Batch Jan 6, 2026

chrisale000 force-pushed the timeSliceLog branch from d68311b to 3135327 Compare January 6, 2026 19:04

wjyao0316 reviewed Jan 6, 2026

View reviewed changes

chrisale000 force-pushed the timeSliceLog branch from 3135327 to 5ac6802 Compare January 6, 2026 19:31

chrisale000 changed the title ~~Adding Partition Execution Logging for DimensionalTimeSliceCrawler and 2 minute Partition Execution Delay for Each Partition Batch~~ Adding Partition Execution Logging for DimensionalTimeSliceCrawler Jan 6, 2026

chrisale000 force-pushed the timeSliceLog branch from 5ac6802 to 2e606c4 Compare January 6, 2026 19:35

graytaylor0 reviewed Jan 6, 2026

View reviewed changes

graytaylor0 approved these changes Jan 6, 2026

View reviewed changes

san81 approved these changes Jan 6, 2026

View reviewed changes

chrisale000 force-pushed the timeSliceLog branch 2 times, most recently from 910037e to b150055 Compare January 7, 2026 18:24

Adding Partition Execution Logging for DimensionalTimeSliceCrawler

eaa551d

Signed-off-by: Alexander Christensen <alchrisk@amazon.com>

chrisale000 force-pushed the timeSliceLog branch from b150055 to eaa551d Compare January 7, 2026 21:53

san81 merged commit 440fb43 into opensearch-project:main Jan 7, 2026
47 of 48 checks passed

Adding Partition Execution Logging for DimensionalTimeSliceCrawler #6362

Adding Partition Execution Logging for DimensionalTimeSliceCrawler #6362

Conversation

chrisale000 commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

M365 Pipeline Validation

Check List

Uh oh!

wandna-amazon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisale000 Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

san81 commented Jan 6, 2026

Uh oh!

chrisale000 commented Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chrisale000 commented Dec 18, 2025 •

edited

Loading

wandna-amazon left a comment •

edited

Loading

chrisale000 Dec 29, 2025 •

edited

Loading