Hallmarks of Human-Machine Collaboration: A framework for assessment in\n the DARPA Communicating with Computers Program

Abstract

There is a growing desire to create computer systems that can communicate\neffectively to collaborate with humans on complex, open-ended activities.\nAssessing these systems presents significant challenges. We describe a\nframework for evaluating systems engaged in open-ended complex scenarios where\nevaluators do not have the luxury of comparing performance to a single right\nanswer. This framework has been used to evaluate human-machine creative\ncollaborations across story and music generation, interactive block building,\nand exploration of molecular mechanisms in cancer. These activities are\nfundamentally different from the more constrained tasks performed by most\ncontemporary personal assistants as they are generally open-ended, with no\nsingle correct solution, and often no obvious completion criteria.\n We identified the Key Properties that must be exhibited by successful\nsystems. From there we identified "Hallmarks" of success -- capabilities and\nfeatures that evaluators can observe that would be indicative of progress\ntoward achieving a Key Property. In addition to being a framework for\nassessment, the Key Properties and Hallmarks are intended to serve as goals in\nguiding research direction.\n

Ähnliche Arbeiten