Stepping Back to Progress Forwards: Setting Standards for Meta-Evaluation of Computational Creativity
Inhaltsverzeichnis
Reference
Anna Jordanous: Stepping Back to Progress Forwards: Setting Standards for Meta-Evaluation of Computational Creativity. In: Computational Creativity 2014 ICCC 2014.
DOI
Abstract
There has been increasing attention paid to the ques- tion of how to evaluate the creativity of computational creativity systems. A number of different evaluation methods, strategies and approaches have been proposed recently, causing a shift in focus: which methodology should be used to evaluate creative systems? What are the pros and cons of using each method? In short: how can we evaluate the different creativity evaluation methodologies? To answer this question, five meta-evaluation criteria have been devised from cross-disciplinary research into good evaluative prac- tice. These five criteria are: correctness; usefulness; faithfulness as a model of creativity; usability of the methodology; generality. In this paper, the criteria are used to compare and contrast the performance of five various evaluation methods. Together, these meta- evaluation criteria help us explore the advantages and disadvantages of each creativity evaluation methodol- ogy, helping us develop the tools we have available to us as computational creativity researchers.
Extended Abstract
Bibtex
Used References
Blanke, T. 2011. Using Situation Theory to evaluate XML retrieval. Dissertations in Database and Information Sys- tems. Heidelberg, Germany: IOS Press.
Colton, S.; Gow, J.; Torres, P.; and Cairns, P. 2010. Ex- periments in objet trouv ́e browsing. In Proceedings of the International Conference on Computational Creativity.
Colton, S.; Charnley, J.; and Pease, A. 2011. Computational Creativity Theory: The FACE and IDEA descriptive mod- els. In Proceedings of the 2nd International Conference on Computational Creativity, 90–95.
Colton, S. 2008. Creativity versus the perception of cre- ativity in computational systems. In Proceedings of AAAI Symposium on Creative Systems, 14–20.
Gillick, J.; Tang, K.; and Keller, R. M. 2010. Machine learn- ing of jazz grammars. Computer Music Journal 34(3):56– 66.
Jordanous, A., and Keller, B. 2014. What makes musical improvisation creative? Journal of Interdisciplinary Music Studies Forthcoming.
Jordanous, A. 2011. Evaluating evaluation: Assessing progress in computational creativity research. In Proceed- ings of the Second International Conference on Computa- tional Creativity (ICCC-11).
Jordanous, A. 2012a. Evaluating Computational Creativity: A Standardised Procedure for Evaluating Creative Systems and its Application. Ph.D. Dissertation, University of Sus- sex, Brighton, UK.
Jordanous, A. 2012b. A standardised procedure for evalu- ating creative systems: Computational creativity evaluation based on what it is to be creative. Cognitive Computation 4(3):246–279.
Kaufman, J. C. 2009. Creativity 101. The Psych 101 series. New York: Springer.
Kazai, G., and Lalmas, M. 2005. Notes on what to measure in INEX. In INEX 2005 Workshop on Element Retrieval Methodology.
Pease, A.; Winterstein, D.; and Colton, S. 2001. Evaluating machine creativity. In Proceedings of Workshop Program of ICCBR-Creative Systems: Approaches to Creativity in AI and Cognitive Science, 129–137.
Rhodes, M. 1961. An analysis of creativity. Phi Delta Kappan 42(7):305–310.
Ritchie, G. 2007. Some empirical criteria for attributing cre- ativity to a computer program. Minds and Machines 17:67– 99.
Sloman, A. 1978. The computer revolution in philosophy. Hassocks, Sussex: Harvester Press.
Thagard, P. 1988. Computational Philosophy of Science. Cambridge, MA: MIT Press.
Zhu, X.; Xu, Z.; and Khot, T. 2009. How creative is your writing? a linguistic creativity measure from computer sci- ence and cognitive psychology perspectives. In Proceedings of NAACL HLT Workshop on Computational Approaches to Linguistic Creativity (ACL), 87–93.
Links
Full Text
http://computationalcreativity.net/iccc2014/wp-content/uploads/2014/06//8.3_Jordanous.pdf
