The transformation matrix between cameras in TAPVid-3D

[RunsenXu]

Dear Authors,

Thank you so much for your great work! It's really a great contribution to the filed.

I am writing to ask is it possible for you to provide the transformation matrix (delta pose) of the cameras of the DriveTrack and ADT splits? With these transformation matrix, we can decouple camera motion and object motion, which would benefit greatly for the 3D vision field.

Best,
Runsen

[skoppula]

Hey Runsen,

Thanks for reaching out and the kind words.

Yup -- the camera extrinsics matrices is something we were thinking of releasing as well, which I think we already have for both those splits, buried in the scripts. As you probably noted, for Panoptic, the camera is fixed.

It's very helpful to know that there is demand for this, so thanks for writing in.

I can't promise a timeline for releasing this updated version with extrinsics unfortunately -- as the team is super busy with the ICLR and other deadlines right now, but it's something that we ourselves would find useful, so added high to our to-do list.

[skoppula]

I'll leave this issue open, until we get around to uploading those as well.

[RunsenXu]

Hi Skanda,

Thanks for your reply! Really look forward to that!

Best,
Runsen

[Charles-Lu]

Dear Authors,

Thank you for your exceptional work and this wonderful dataset!

I have a similar question: based on my understanding, the released 3D trajectories of key points are in the camera coordinate space. I was wondering if it is possible to release the camera extrinsic parameters or 3D trajectories in world coordinates. Any updates on this matter would be greatly appreciated.

Thank you once again for your valuable time!

[RunsenXu]

Dear Authors,

Me too. Still waiting for the release of camera extrinsic. Hope I am able to use it for my CVPR 2025 project. :)

Thanks again for your wonderful work.

Best,
Runsen

[RunsenXu]

Dear authors,

Really look forward to the updates here! Your work is amazing, that's why I am still waiting for your release. :)
Thank you again.

Best,
Runsen

[yuantianyuan01]

Same issue here. Thank you for this great work! It would be very helpful if you could provide camera poses or the points 3D trajectory in the world coordinates.

[yuantianyuan01]

After checking the data structure, I find the extrinsics of ADT can be obtained by using the gt_provider.get_aria_3d_pose_by_timestamp_ns method in the script. However, it is still impossible to get the the extrinsics of the drivetrack split. I understand that releasing the ground truth extrinsics might involve significant effort, but would it be possible to at least provide a mapping between the *.npz filenames and the original scene IDs (and timestamps) in the Waymo Open Dataset? This would be sufficient for us to retrieve the ground truth poses on our end. Thanks.

[ignacio-rocco]

Hi all, thanks for your patience on this. Extrinsics for ADT and DriveTrack are planned to be released soon, probably next week. I believe the filenames of the DriveTrack npz should match those from the original DriveTrack data, but Skanda can say more about this. Le ven. 22 nov. 2024, 14:14, Tianyuan Yuan ***@***.***> a écrit :

…

After checking the data structure, I find the extrinsics of ADT can be obtained by using the gt_provider.get_aria_3d_pose_by_timestamp_ns method in the script. However, it is still impossible to get the the extrinsics of the drivetrack split. I understand that releasing the ground truth extrinsics might involve significant effort, but would it be possible to at least provide a mapping between the *.npz filenames and the original scene IDs (and timestamps) in the Waymo Open Dataset? This would be sufficient for us to retrieve the ground truth poses on our end. Thanks. — Reply to this email directly, view it on GitHub <#115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC3LX3FH7FYYDON44VA3XJD2B435VAVCNFSM6AAAAABMS3TZJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJTHA3DEMZZG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[yuantianyuan01]

Thanks for your reply! That would be great. :)

[skoppula]

Hi Runsen and Tianyuan,

Hope all is well! Huge thanks for your patience! We've just released TAPVid-3D with extrinsics (for the videos with moving camera -- Waymo Open videos and ADT videos). 🎉

You should be able to see a new extrinsics_w2c matrix in the *.npz files in the rc5 release version (e.g. https://storage.googleapis.com/dm-tapnet/tapvid3d/release_files/rc5/) and a visualization of the extrinsics is also available at the bottom of the demo Colab (https://colab.research.google.com/drive/1Ro2sE0lAvq-h0lixrUBB0oTYXEwXNr66?usp=sharing)

[skoppula]

I'll close this for now, but feel free to re-open if you have questions.

[yuantianyuan01]

Hi, thanks for releasing the extrinsics! Does it mean that I need to re-run the download command for each dataset to get the latest *.npz files with extrinsics? Btw the link https://storage.googleapis.com/dm-tapnet/tapvid3d/release_files/rc5/ reports NoSuchKeyThe specified key does not exist.No such object: dm-tapnet/tapvid3d/release_files/rc5/.