Skip to content

Conversation

@rbs333
Copy link
Collaborator

@rbs333 rbs333 commented Dec 8, 2025

No description provided.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates how indexing time is measured across the retrieval optimizer codebase. Instead of relying on Redis's index.info()["total_indexing_time"], the code now manually measures wall-clock time from when indexing starts until completion (percent_indexed == 1). The measured time is persisted to Redis for reuse in subsequent runs that don't reload data.

Key changes:

  • Introduced new utility functions get_last_indexing_time() and set_last_indexing_time() to persist and retrieve indexing time measurements
  • Updated bayes_study.py, grid_study.py, and search_study.py to measure indexing time using time.time() instead of querying index.info()
  • Added documentation clarifying that total_indexing_time is measured in seconds using wall-clock time

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
redis_retrieval_optimizer/utils.py Added two new utility functions for getting/setting last indexing time in Redis
redis_retrieval_optimizer/search_study.py Updated to retrieve persisted indexing time instead of using index.info()
redis_retrieval_optimizer/grid_study.py Added wall-clock timing measurement when indexing data and persistence of timing
redis_retrieval_optimizer/bayes_study.py Added wall-clock timing measurement with conditional logic for recreate vs. reuse scenarios
README.md Added documentation explaining the wall-clock timing methodology for indexing time
docs/examples/search_study/search_study_walkthrough.ipynb New example notebook demonstrating search study workflow with existing index

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

"source": [
"# Search study\n",
"\n",
"Let's say you have an existing Redis database with a search index seeded. You may wish to quickly test different search method against the existing index without having to recreate data and/or recreate the index. This demo will walk you though how to set this up and get going.\n",
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spelling of "though" is incorrect. It should be "through" in this context.

Suggested fix: Change "though" to "through"

Suggested change
"Let's say you have an existing Redis database with a search index seeded. You may wish to quickly test different search method against the existing index without having to recreate data and/or recreate the index. This demo will walk you though how to set this up and get going.\n",
"Let's say you have an existing Redis database with a search index seeded. You may wish to quickly test different search method against the existing index without having to recreate data and/or recreate the index. This demo will walk you through how to set this up and get going.\n",

Copilot uses AI. Check for mistakes.
"\n",
"For the search_study we are assuming that the search index already exists. The cell below will create a Redis search index and populate it with our test data for example purposes but is assumed with a search study is populated and running within your data. \n",
"\n",
"Note: the demo assumes you have a instance of redis running on localhost:6379. If this is not the case, update the redis_url to direct to your running instance or start a local instance with the following command. \n",
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation states the "demo assumes you have a instance of redis" which should be "an instance" instead of "a instance".

Suggested fix: Change "a instance" to "an instance"

Suggested change
"Note: the demo assumes you have a instance of redis running on localhost:6379. If this is not the case, update the redis_url to direct to your running instance or start a local instance with the following command. \n",
"Note: the demo assumes you have an instance of redis running on localhost:6379. If this is not the case, update the redis_url to direct to your running instance or start a local instance with the following command. \n",

Copilot uses AI. Check for mistakes.
@rbs333 rbs333 force-pushed the feat/update-total-index-time branch from f1c7663 to 40a6fbd Compare December 12, 2025 14:41
rbs333 and others added 4 commits December 12, 2025 09:42
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@rbs333 rbs333 merged commit 0f495df into main Dec 12, 2025
2 checks passed
@rbs333 rbs333 deleted the feat/update-total-index-time branch December 12, 2025 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants