Stepping Back to Progress Forwards: Setting Standards for Meta-Evaluation of Computational Creativity

Aus de_evolutionary_art_org
Wechseln zu: Navigation, Suche


Anna Jordanous: Stepping Back to Progress Forwards: Setting Standards for Meta-Evaluation of Computational Creativity. In: Computational Creativity 2014 ICCC 2014, 129-136.



There has been increasing attention paid to the ques- tion of how to evaluate the creativity of computational creativity systems. A number of different evaluation methods, strategies and approaches have been proposed recently, causing a shift in focus: which methodology should be used to evaluate creative systems? What are the pros and cons of using each method? In short: how can we evaluate the different creativity evaluation methodologies? To answer this question, five meta-evaluation criteria have been devised from cross-disciplinary research into good evaluative prac- tice. These five criteria are: correctness; usefulness; faithfulness as a model of creativity; usability of the methodology; generality. In this paper, the criteria are used to compare and contrast the performance of five various evaluation methods. Together, these meta- evaluation criteria help us explore the advantages and disadvantages of each creativity evaluation methodol- ogy, helping us develop the tools we have available to us as computational creativity researchers.

Extended Abstract


author = {Anna Jordanous},
title = {Stepping Back to Progress Forwards: Setting Standards for Meta-Evaluation of Computational Creativity},
booktitle = {Proceedings of the Fifth International Conference on Computational Creativity},
series = {ICCC2014},
year = {2014},
month = {Jun},
location = {Ljubljana, Slovenia},
pages = {129-136},
url = {, },
publisher = {International Association for Computational Creativity},
keywords = {computational, creativity},

Used References

Blanke, T. 2011. Using Situation Theory to evaluate XML retrieval. Dissertations in Database and Information Sys- tems. Heidelberg, Germany: IOS Press.

Colton, S.; Gow, J.; Torres, P.; and Cairns, P. 2010. Ex- periments in objet trouv ́e browsing. In Proceedings of the International Conference on Computational Creativity.

Colton, S.; Charnley, J.; and Pease, A. 2011. Computational Creativity Theory: The FACE and IDEA descriptive mod- els. In Proceedings of the 2nd International Conference on Computational Creativity, 90–95.

Colton, S. 2008. Creativity versus the perception of cre- ativity in computational systems. In Proceedings of AAAI Symposium on Creative Systems, 14–20.

Gillick, J.; Tang, K.; and Keller, R. M. 2010. Machine learn- ing of jazz grammars. Computer Music Journal 34(3):56– 66.

Jordanous, A., and Keller, B. 2014. What makes musical improvisation creative? Journal of Interdisciplinary Music Studies Forthcoming.

Jordanous, A. 2011. Evaluating evaluation: Assessing progress in computational creativity research. In Proceed- ings of the Second International Conference on Computa- tional Creativity (ICCC-11).

Jordanous, A. 2012a. Evaluating Computational Creativity: A Standardised Procedure for Evaluating Creative Systems and its Application. Ph.D. Dissertation, University of Sus- sex, Brighton, UK.

Jordanous, A. 2012b. A standardised procedure for evalu- ating creative systems: Computational creativity evaluation based on what it is to be creative. Cognitive Computation 4(3):246–279.

Kaufman, J. C. 2009. Creativity 101. The Psych 101 series. New York: Springer.

Kazai, G., and Lalmas, M. 2005. Notes on what to measure in INEX. In INEX 2005 Workshop on Element Retrieval Methodology.

Pease, A.; Winterstein, D.; and Colton, S. 2001. Evaluating machine creativity. In Proceedings of Workshop Program of ICCBR-Creative Systems: Approaches to Creativity in AI and Cognitive Science, 129–137.

Rhodes, M. 1961. An analysis of creativity. Phi Delta Kappan 42(7):305–310.

Ritchie, G. 2007. Some empirical criteria for attributing cre- ativity to a computer program. Minds and Machines 17:67– 99.

Sloman, A. 1978. The computer revolution in philosophy. Hassocks, Sussex: Harvester Press.

Thagard, P. 1988. Computational Philosophy of Science. Cambridge, MA: MIT Press.

Zhu, X.; Xu, Z.; and Khot, T. 2009. How creative is your writing? a linguistic creativity measure from computer sci- ence and cognitive psychology perspectives. In Proceedings of NAACL HLT Workshop on Computational Approaches to Linguistic Creativity (ACL), 87–93.


Full Text

intern file

Sonstige Links