feat: new model_reuse_type to enable sync by peers and creation by leader #170
Conversation
samika98
changed the title
| &small_model_id, | ||
| &large_model_id, | ||
| config.params.number_of_documents, | ||
| global_leader, |
There was a problem hiding this comment.
This should be true so that each worker creates MIDs instead of just the global leader.
| global_leader, | |
| true, |
There was a problem hiding this comment.
Basically, we need the ReuseType::Shared code when creating models, but ReuseType::PerUser code when creating MIDs.
| } | ||
| }; | ||
|
|
||
| if lead_user { |
There was a problem hiding this comment.
Because of this if-block, one user per worker will call the model subscription API, which means each node will already be subscribed to the models. This means that we shouldn't need to pass the should_subscribe flag as you're doing below.
| redis_cli: &redis::Client, | ||
| should_create: bool, | ||
| redis_postfix: Option<String>, | ||
| should_subscribe: bool, |
There was a problem hiding this comment.
This might not be necessary - see my comment above. One user on each worker is already making sure that the corresponding node is subscribed to the models.
Added a Model Reuse Type to help us configure Goose Scenario. This Reuse type lets the global leader (worker with id 0) create models and all the users including the leader (ceramic nodes) subscribe to that model.