Towards universal evaluation of image annotation interfaces

Andrew M. Vernier, Jean Y. Song, Edward Sun, Allison Kench, Walter S. Lasecki

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

To guide the design of interactive image annotation systems that generalize to new domains and applications, we need ways to evaluate the capabilities of new annotation tools across a range of different types of image, content, and task domains. In this work, we introduce Corsica, a test harness for image annotation tools that uses calibration images to evaluate a tool's capabilities on general image properties and task requirements. Corsica is comprised of sets of three key components: 1) synthesized images with visual elements that are not domain-specific, 2) target microtasks that connects the visual elements and tools for evaluation, and 3) ground truth data for each microtask and visual element pair. By introducing a specification for calibration images and microtasks, we aim to create an evolving repository that allows the community to propose new evaluation challenges. Our work aims to help facilitate the robust verification of image annotation tools and techniques.

Original languageEnglish
Title of host publicationUIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology
PublisherAssociation for Computing Machinery, Inc
Pages81-83
Number of pages3
ISBN (Electronic)9781450368179
DOIs
StatePublished - 14 Oct 2019
Event32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019 - New Orleans, United States
Duration: 20 Oct 201923 Oct 2019

Publication series

NameUIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology

Conference

Conference32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019
Country/TerritoryUnited States
CityNew Orleans
Period20/10/1923/10/19

Bibliographical note

Publisher Copyright:
© 2019 Copyright is held by the owner/author(s).

Keywords

  • Crowdsourcing
  • Evaluation
  • Image annotation
  • Tools

Fingerprint

Dive into the research topics of 'Towards universal evaluation of image annotation interfaces'. Together they form a unique fingerprint.

Cite this