How to handle images with no ground truth box

Hey @dbolya, 
I used my own dataset according to #9. However, I got slightly better results compared to pycocotools because gt images with no objects are ignored. How can I include these empty images in score calculation? 
Thank you!