August 8, 2018 feature
Evaluating creativity in computational co-creative systems
Computer programs assist humans in a variety of ways, including in their creative endeavors. Researchers at UNC Charlotte and the University of Sydney have recently developed a new framework for evaluating creativity in co-creative systems in which humans and computers collaborate on creative tasks.
The team of researchers has been investigating how to develop AI-based creativity for several years. One of their previous studies questioned the source of creativity in instances where computer programs produce creative works alone or in collaboration with human agents, while another study developed AI to evaluate the creativity of individual designs.
"Creative systems are intelligent systems that can perform creative tasks alone or in collaboration," the researchers explained in their paper. "There are three main strategies by which the role of computers in creative systems can be characterized: fully autonomous systems, creativity support tools, and co-creative systems."
Fully autonomous systems are designed to generate creative artifacts without human assistance, using technological tools that include machine learning, production rules, and evolutionary approaches. On the other hand, creativity support tools are applications or programs that support a human user's creativity, allowing them to express their talent using technology-based methods.
In their recent study, however, the researchers focused on the third type, co-creative systems, in which computers and humans collaborate with one another to produce shared creative works. These types of systems have been applied within a variety of fields, including art, humor, games, and robotics. One example is Drawing Apprentice, a co-creative application that entails collaboration between a human user and an AI agent on completing drawing tasks.
In their paper, the researchers specifically define a co-creative system as an "interaction between at least one AI agent and at least one human where they take action based on the response of their partner and their own conceptualization of creativity during the co-creative task."
Despite the growing popularity of these systems, there is currently no standard method for evaluating them. To address this lack, the researchers developed a theoretical framework based on four main questions: Who is evaluating the creativity? What is being evaluated? When does evaluation occur, and how is it performed?
Applying these questions to six examples of co-creative systems, they concluded that existing co-creative systems typically focus on evaluating the user experience, as well as the product of this experience. In other words, many existing co-creative systems broaden creativity support tools to include more pro-active contributions on behalf of the computer program.
"Unlike creativity support tools, co-creative systems have the potential for self-evaluation by embedding a self-awareness of the creativity of the AI agent," the researchers explained in their paper. "The capacity for self-evaluation can guide users toward or away from particular regions of the space of possibilities intentionally based on the AI agent's concept of creativity."
The researchers also concluded that compared to autonomous systems, co-creative systems benefit from human interaction, introducing human insight and perception of the creative product during the process. Integrating evaluation methods that are employed by autonomous creative systems could result in co-creative systems becoming self-aware and intentional.
"Unlike fully autonomous creative systems and creativity support tools, the creative process used by co-creative systems is not the result of a single agent, instead it is a collaboration," the researchers explained. "This means existing approaches to evaluating computational creativity or HCI (human-computer interaction) approaches to evaluate creativity support are insufficient. "
In future, the researchers plan to develop the framework further and use it to compare the evaluation of co-creative systems across different examples or implementations, as well as to compare them with other types of creative systems, such as autonomous and creativity support tools.
This paper provides a framework for evaluating creativity in co-creative systems: those that involve computer programs collaborating with human users on creative tasks. We situate co-creative systems within a broader context of computational creativity and explain the unique qualities of these systems. We present four main questions that can guide evaluation in co-creative systems: Who is evaluating the creativity, what is being evaluated, when does evaluation occur and how the evaluation is performed. These questions provide a framework for comparing how existing co-creative systems evaluate creativity, and we apply them to examples of co-creative systems in art, humor, games and robotics. We conclude that existing co-creative systems tend to focus on evaluating the user experience. Adopting evaluation methods from autonomous creative systems may lead to co-creative systems that are self-aware and intentional.
© 2018 Tech Xplore