[Paper Reading] Beyond Prompts: Exploring the Design Space of Mixed-Initiative Co-Creativity Systems
by: Zhiyu Lin, Upol Ehsan, Rohan Agarwal, Samihan Dani, Vidushi Vashishth, Mark Riedl
If you’ve ever tried out ChatGPT or Bard, you’ve engineered a prompt to generate an output from a Large Language Model (LLM). In this case, it’s a chat interface on top of an LLM, which is one of many ways of working with Deep Generative Models (or Neural Generative Systems, as described in this paper). If you want to know more about prompting Kory Mathewson wrote a great breakdown here. This paper looks at the design space around human-machine interactions. Specifically, they call this Mixed-Initiative Co-Creative (MI-CC) interfaces.
Before the 2020 Generative AI boom and before the 2010 Deep Learning boom, I remember learning about areas of Mixed-Initiative and Co-Creative interfaces. There were conferences in the Human-Computer Interactions space as well as groups dedicated to working towards Computational Creativity. My own PhD lab worked on Generative AI with rules, search, and planning based systems, which did not involve Machine Learning (ML). AI has a diverse history with various lanes, and I see this paper as a way of integrating the frameworks from the creative spaces with the most recent advancements in the ML space. (Here’s a position paper from me in 2018 on MI-CC).
Abstract
Generative Artificial Intelligence systems have been developed for image, code, story, and game generation with the goal of facilitating human creativity. Recent work on neural generative systems has emphasized one particular means of interacting with AI systems: the user provides a specification, usually in the form of prompts, and the AI system generates the content. However, there are other configurations of human and AI coordination, such as co-creativity (CC) in which both human and AI systems can contribute to content creation, and mixed-initiative (MI) in which both human and AI systems can initiate content changes. In this paper, we define a hypothetical human-AI configuration design space consisting of different means for humans and AI systems to communicate creative intent to each other. We conduct a human participant study with 185 participants to understand how users want to interact with differently configured MI-CC systems. We find out that MI-CC systems with more extensive coverage of the design space are rated higher or on par on a variety of creative and goal-completion metrics, demonstrating that wider coverage of the design space can improve user experience and achievement when using the system; Preference varies greatly between expertise groups, suggesting the development of adaptive, personalized MI-CC systems; Participants identified new design space dimensions including scrutability -- the ability to poke and prod at models -- and explainability.
Paper Link: https://arxiv.org/abs/2305.07465