Krea VACE 14b Pipeline #297
Conversation
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
BuffMcBigHuge
force-pushed
the
marco/feat/krea-vace-14b
branch
from
4bc5216 to
6d40951
Compare
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
| .cursorindexingignore | ||
|
|
||
| .cursor/ | ||
| .specstory/ |
There was a problem hiding this comment.
@yondonfu I can remove this if you like. Spec story is a plugin for cursor chat history export that I use.
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
|
FWIW, I've played with this PR and I'm seeing this weird loop. @ryanontheinside mentioned it's probably that we're sending the reference image to each chunk. demo_krea.mp4 |
| # - Only start skipping once the cache is in steady-state: | ||
| # current_start_frame >= kv_cache_num_frames | ||
| # - Recompute every N "blocks" (a block produces num_frame_per_block frames). | ||
| every_n = int(os.getenv("SCOPE_KV_CACHE_RECOMPUTE_EVERY", "1") or "1") |
There was a problem hiding this comment.
I'm not sure if we use env variables in other blocks, but I think we may want to try to use input params instead. This would help someone wanting to reuse the block or the pipeline to interact with it.
There was a problem hiding this comment.
this is interesting - if you do move to input params would be good so see some documentation explaining what different values can be used
It could also be a problem with the vae after looking at another example. I had claude replicate the test_vace.py script from longlive and tested krea offline. The demarcation at chunk boundaries is familiar and makes me think vae issue. "She jumps": output_r2v.mp4I also tested longlive; there is no regression there. |
| WARMUP_PROMPT = [{"text": "a majestic sunset", "weight": 1.0}] | ||
|
|
||
|
|
||
| class KreaRealtimeVideoPipeline(Pipeline, LoRAEnabledPipeline): |
There was a problem hiding this comment.
Can we use VACEEnabledPipeline here instead of the lazy loading? We switched to the mixin recently
| # This fixes the chicken-and-egg problem where VACE isn't enabled until vace_input_frames arrives | ||
| vace_enabled = self.parameters.get("vace_enabled", False) | ||
|
|
||
| # Robustness fallback: If vace_enabled is missing (e.g. old frontend build), |
There was a problem hiding this comment.
we could expect the front end to match for the build here instead of adding fallbacks, wdyt?
| # - Only start skipping once the cache is in steady-state: | ||
| # current_start_frame >= kv_cache_num_frames | ||
| # - Recompute every N "blocks" (a block produces num_frame_per_block frames). | ||
| every_n = int(os.getenv("SCOPE_KV_CACHE_RECOMPUTE_EVERY", "1") or "1") |
There was a problem hiding this comment.
this is interesting - if you do move to input params would be good so see some documentation explaining what different values can be used
| from scope.core.pipelines.wan2_1.utils import initialize_kv_cache | ||
|
|
||
|
|
||
| def _get_block_mask_model(model: torch.nn.Module) -> torch.nn.Module: |
There was a problem hiding this comment.
could this be at the class level as opposed to the module level?
| description="Input frames for VACE conditioning (if present, indicates video input is enabled)", | ||
| ), | ||
| InputParam( | ||
| "soft_transition_active", |
There was a problem hiding this comment.
Assuming this is vestigial from experimentation - AFAIK this is never set to true
This PR implements full VACE (Video All-In-One Creation and Editing) support for the KREA Realtime Video pipeline. It addresses significant architectural challenges in adapting the VACE system—originally designed for the 1.3B "Longlive" model—to work with the much larger Wan 2.1 14B model used by KREA.