Output Format

Hi everyone, I’m preparing a submission for the VLM3D Challenge Task 3 and ran into a format inconsistency. Hoping someone here (or the organizers) can clarify.

Submission page format requirement: https://abnloc.vlm3dchallenge.com/evaluation/test/submissions/create/

Evaluation reference code (GitHub) expected input format: https://github.com/forithmus/VLM3D-Dockers/tree/main/abnloc_evaluation

These two seem to describe different formats for Task 3, which is a bit confusing while finalizing our submission. Which format should we follow to ensure the evaluation runs correctly?

Thanks in advance!

Hi @songtx, Sorry for the confusion. You don’t need to (and should not) wrap the predictions under outputs key (as in the evaluation docker expected input), this is handled automatically by the API. I’m aware that the example algorithm docker includes wrapping, but please ignore that as it’s incorrect. Some keys are not used in the evaluation docker, which is why they are omitted from the expected input for evaluation docker. Others (like outputs) are added by the API. You can simply follow the expected output format provided on the submission page. We will run everything in the cluster regardless, so it should not matter even though you wrap under outputs key. I hope this clarifies things.

Best,

Sezgin

Thank you for your detailed reply!@vlm3dchallenge