Evaluating Evaluation: Assessing Progress in Computational Creativity Research
Inhaltsverzeichnis
Reference
Anna Jordanous: Evaluating Evaluation: Assessing Progress in Computational Creativity Research. In: Computational Creativity 2011 ICCC 2011, pp. 102-107.
DOI
Abstract
Computational creativity research has produced many computational systems that are described as creative. A comprehensive literature survey reveals that although such systems are labelled as creative, there is a distinct lack of evaluation of the creativity of creative systems. As a research community, we should adopt a more sci- entific approach to evaluation of the creativity of our systems if we are to progress in understanding creativ- ity and modelling it computationally. A methodology for creativity evaluation should accommodate differ- ent manifestations of creativity but also require a clear, definitive statement of the standards used for evaluation. This paper proposes Evaluation Guidelines, a standard but flexible approach to evaluation of the creativity of computational systems and argues that this approach should be taken up as standard practice in computa- tional creativity research. The approach is outlined and discussed, then illustrated through a comparative evalu- ation of the creativity of jazz improvisation systems.
Extended Abstract
Bibtex
@inproceedings{
author = {Anna Jordanous},
title = {Evaluating Evaluation: Assessing Progress in Computational Creativity Research},
editor = {Dan Ventura, Pablo Gervás, D. Fox Harrell, Mary Lou Maher, Alison Pease and Geraint Wiggins},
booktitle = {Proceedings of the Second International Conference on Computational Creativity},
series = {ICCC2011},
year = {2011},
month = {April},
location = {México City, México},
pages = {102-107},
url = {http://iccc11.cua.uam.mx/proceedings/the_foundational/jordanous_iccc11.pdf, http://de.evo-art.org/index.php?title=Evaluating_Evaluation:_Assessing_Progress_in_Computational_Creativity_Research },
publisher = {International Association for Computational Creativity},
keywords = {computational, creativity},
}
Used References
Berliner, P. F. 1994. Thinking in Jazz: The Infinite Art of Impro- visation. Chicago Studies in Ethnomusicology. Chicago, IL: The University of Chicago Press.
Biles, J. A. 2007. Improvising with genetic algorithms: GenJam. In Miranda, E. R., and Biles, J. A., eds., Evolutionary Computer Music. London, UK: Springer-Verlag. chapter 7, 137–169.
Colton, S. 2008. Creativity versus the perception of creativity in computational systems. In Proceedings of AAAI Symposium on Creative Systems, 14–20.
Dunning, T. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics 19(1):61–74.
Gibbs, L. 2010. Evaluating creative (jazz) improvisation: Distin- guishing invention and creativity. In Gibbs, L., ed., Proceedings of Leeds International Jazz Conference 2010: Improvisation - jazz in the creative moment.
Hodgson, P. 2006. Learning and the evolution of melodic com- plexisty in virtuoso jazz improvisation. In Proceedings of the 28th Annual Conference of the Cognitive Science Society (CogSci 2006), 1506–1510.
Jennings, K. E. 2010. Developing creativity: Artificial barriers in artificial intelligence. Minds and Machines 1–13. Article in Press.
Jordanous, A. 2010a. Defining creativity: Finding keywords for creativity using corpus linguistics techniques. In Ventura, D.; Pease, A.; P ́erez y P ́erez, R.; Ritchie, G.; and Veale, T., eds., Pro- ceedings of the International Conference on Computational Cre- ativity, 278–287.
Jordanous, A. 2010b. A fitness function for creativity in jazz im- provisation and beyond. In Ventura, D.; Pease, A.; P ́erez y P ́erez, R.; Ritchie, G.; and Veale, T., eds., Proceedings of the International Conference on Computational Creativity, 223–227.
Jordanous, A. forthcoming. Evaluating Computational Creativ- ity: A Standardised Evaluation Methodology and its Application to Case Studies. Ph.D. Dissertation, University of Sussex, Brighton, UK.
Le ́on, C., and Gerv ́as, P. 2010. The role of evaluation-driven rejec- tion in the successful exploration of a conceptual space of stories. Minds and Machines. Article in Press.
Lewis, G. E. 2000. Too many notes: Computers, complexity and culture in Voyager. Leonardo Music Journal 33–39.
Pearce, M. T.; Meredith, D.; and Wiggins, G. A. 2002. Motivations and methodologies for automation of the compositional process. Musicae Scientae 6(2):119–147.
Pease, A.; Winterstein, D.; and Colton, S. 2001. Evaluating machine creativity. In Proceedings of ICCBR Workshop on Ap- proaches to Creativity, 129–137.
Peinado, F., and Gervas, P. 2006. Evaluation of automatic genera- tion of basic stories. New Generation Computing 24(3):289–302.
P ́erez y P ́erez, R. 1999. MEXICA: A Computer Model of Creativity in Writing. Ph.D. Dissertation, University of Sussex, Brighton, UK.
Plucker, J. A.; Beghetto, R. A.; and Dow, G. T. 2004. Why isn’t creativity more important to educational psychologists? potentials, pitfalls, and future directions in creativity research. Educational Psychologist 39(2):83–96.
Ritchie, G. 2007. Some empirical criteria for attributing creativity to a computer program. Minds and Machines 17:67–99.
Widmer, G.; Flossmann, S.; and Grachten, M. 2009. YQX plays Chopin. AI Magazine 30(3):35–48.
Wiggins, G. A. 2006. A preliminary framework for description, analysis and comparison of creative systems. Knowledge-Based Systems 19(7):449–458.
Links
Full Text
http://iccc11.cua.uam.mx/proceedings/the_foundational/jordanous_iccc11.pdf
