Error message: "Output file 'metrics.json' was not produced"

Hello everyone,

I recently submitted my algorithm, but received the following error:

Error message: "Output file 'metrics.json' was not produced"

Based on my understanding of the submission requirements, my code correctly creates a file named output.json, and everything works fine in my local testing. However, the error message indicates that the system expects to find a file named metrics.json.

Do the submission requirements explicitly state that we need to generate metrics.json rather than output.json? Are these two files supposed to be the same file, just with different naming requirements? Is metrics.json a file that needs to be generated by evaluation code that participants are expected to write themselves?

I had been under the impression that the evaluation process would be handled automatically by the platform, but now I'm unsure whether I need to write my own evaluation code to generate this metrics.json file.

I would greatly appreciate if the organizers or experienced participants could clarify this issue.

Thank you for your help!

Hi @songtx, metric.json is generated by the evaluation docker. The output file name for your algorithm should not matter (prediction.json is definitely correct). That said, the issue with your docker is that you used the "scores" key in your JSON instead of "probabilities", which causes the evaluation docker to fail. I usually fix these issues when I run the dockers externally. Since there are many submissions right now, most failed dockers are being handled this way. If you’d like, you can update your docker and submit a new image, or I can fix it for you and update the leaderboard in the next batch. Best, Sezgin

Thank you very much for your detailed reply!@vlm3dchallenge

I noticed an inconsistency that might confuse participants. In the abnclass_example_docker example provided on GitHub, the output file uses the key name probabilities, but on the submission page https://abnclass.vlm3dchallenge.com/evaluation/test/submissions/create/, the output requirements list the key name as scores.

To prevent other participants from encountering similar confusion, I suggest unifying the terminology across both resources, or explicitly stating on the page that both key names are acceptable. Would it be possible to modify the evaluation code to accept both probabilities and scores as valid keys?

If you do update the evaluation code to accept both keys and if you have retained my submission results, would it be possible to help me retest it with the updated code? I would greatly appreciate your assistance with this.

Thank you again for your support!

Hi Songtx,

Thank you very much for showing this. I am fixing it. It is possible to update the evaluation code for sure but you will still need to resubmit if you want to use the local execution. I'll run in the cluster anyways today so you don't need to do anything.

Best, Sezgin

Thank you for your reply and fix! This is very helpful for me and other participants. @vlm3dchallenge