HEIM stands for Holistic Evaluation of Text-To-Image Models. It is a comprehensive benchmark designed to assess the capabilities and risks of text-to-image generation models. Unlike previous evaluations that primarily focused on image-text alignment and image quality, HEIM considers 12 different aspects that are crucial for real-world model deployment:
By curating scenarios that encompass these aspects, HEIM evaluates state-of-the-art text-to-image models. Interestingly, no single model excels in all aspects; different models demonstrate strengths in different areas. For transparency, all prompts, generated images, and results are available on the HEIM website for exploration and study. Additionally, the GitHub repository provides a collection of models accessible via a unified API, along with metrics beyond accuracy, such as efficiency, bias, and toxicity.
Paper | Code | Results | Date | Stars |
---|