Human Competence in Creativity Evaluation
Inhaltsverzeichnis
Reference
Carolyn Lamb, Daniel G. Brown and Charles Clarke: Human Competence in Creativity Evaluation. In: Computational Creativity 2015 ICCC 2015, 102-109.
DOI
Abstract
We investigate the performance of non-expert judges in using leading computational poetry evaluation metrics to evaluate poetry written by humans. We find that regardless of the model used, non-expert judges are very poor at using metrics to evaluate creativity, even displaying the reverse of the desired rating pattern, preferring novice poetry to professional poetry. We discuss likely reasons for this finding and the implications for the evaluation of computational creativity. Researchers using human judges should be aware that using a metric or structured evaluation does not negate the need for judge expertise.
Extended Abstract
Bibtex
@inproceedings{
author = {Lamb, Carolyn and Brown, Daniel G. and Clarke, Charles},
title = {Human Competence in Creativity Evaluation},
booktitle = {Proceedings of the Sixth International Conference on Computational Creativity},
series = {ICCC2015},
year = {2015},
month = {Jun},
location = {Park City, Utah, USA},
pages = {102-109},
url = {http://computationalcreativity.net/iccc2015/proceedings/5_2Lamb.pdf http://de.evo-art.org/index.php?title=Human_Competence_in_Creativity_Evaluation },
publisher = {International Association for Computational Creativity},
keywords = {computational, creativity},
}
Used References
Aguilar, W., and P´erez y P´erez, R. 2014. Criteria for evaluating early creative behavior in computational agents. In Proceedings of the Fifth International Conference on Computational Creativity, 284–287.
Amabile, T. 1983. In The social psychology of creativity, 37–64. Springer-Verlag New York.
Binsted, K.; Pain, H.; and Ritchie, G. 1997. Children’s evaluation of computer-generated punning riddles. Pragmatics & Cognition 5(2):305–354.
Boden, M. A. 1990. The creative mind: Myths and mechanisms. Psychology Press.
Bown, O. 2014. Empirically grounding the evaluation of creative systems: incorporating interaction design. In Proceedings of the Fifth International Conference on Computational Creativity.
Brown, D. 2009. Computational artistic creativity and its evaluation. Number Dagstuhl Seminar 09291, 1–8.
Burns, K. 2006. Atoms of eve: A bayesian basis for esthetic analysis of style in sketching. AIE EDAM: Artificial Intelligence for Engineering Design, Analysis, and Manufacturing 20(03):185–199.
Burns, K. 2012. Eve s energy in aesthetic experience: a bayesian basis for haiku humour. Journal of Mathematics and the Arts 6(2-3):77–87.
Burns, K. 2015. Computing the creativeness of amusing advertisements: A Bayesian model of Burma-Shave’s muse.
AIE EDAM: Artificial Intelligence for Engineering Design, Analysis, and Manufacturing 29(01):109–128.
Chan, H., and Ventura, D. 2008. Automatic composition of themed mood pieces. In Proceedings of the 5th International Joint Workshop on Computational Creativity, 109–115.
Colton, S.; Pease, A.; Corneli, J.; Cook, M.; and Llano, T. 2014. Assessing progress in building autonomously creative systems. In Proceedings of the Fifth International Conference on Computational Creativity, 137–145.
Colton, S.; Goodwin, J.; and Veale, T. 2012. Full face poetry generation. In Proceedings of the Third International Conference on Computational Creativity, 95–102.
Colton, S.; Pease, A.; and Charnley, J. 2011. Computational creativity theory: The FACE and IDEA descriptive models. In Proceedings of the Second International Conference on Computational Creativity, 90–95.
Colton, S. 2008a. Creativity versus the perception of creativity in computational systems. In AAAI Spring Symposium: Creative Intelligent Systems, 14–20.
Colton, S. 2008b. Experiments in constraint-based automated scene generation. Proceedings of the Fifth International Workshop on Computational Creativity 2008 127.
Das, A., and Gamb¨ack, B. 2014. Poetic machine: Computational creativity for automatic poetry generation in bengali. In Proceedings of the Fifth International Conference on Computational Creativity.
Galanter, P. 2012. Computational aesthetic evaluation: Past and future. In Computers and Creativity. Springer. 255–293.
Gerv´as, P. 2002. Exploring quantitative evaluations of the creativity of automatic poets. In Proc. of the 2nd Workshop on Creative Systems, Approaches to Creativity in Artificial Intelligence and Cognitive Science, the 15th European Conf. on Artificial Intelligence (ECAI 2002).
Gerv´as, P. 2007. On the fly collaborative story-telling: Revising contributions to match a shared partial story line. International JointWorkshop on Computational Creativity 13.
Jordanous, A. 2012. A standardised procedure for evaluating creative systems: Computational creativity evaluation based on what it is to be creative. Cognitive Computation 4(3):246–279.
Karampiperis, P.; Koukourikos, A.; and Koliopoulou, E. 2014. Towards machines for measuring creativity: The use of computational tools in storytelling activities. In Proceedings of the 14th International Conference on Advanced Learning Technologies (ICALT), 508–512.
Kaufman, J. C.; Baer, J.; and Cole, J. C. 2009. Expertise, domains, and the consensual assessment technique. The Journal of creative behavior 43(4):223–233.
Lehman, J.,and Stanley, K. O. 2012. Beyond openendedness: Quantifying impressiveness. In Artificial Life, volume 13, 75–82.
Llano, M. T.; Hepworth, R.; Colton, S.; Gow, J.; Charnley, J.; Granroth-Wilding, M.; and Clark, S. 2014. Baseline methods for automated fictional ideation. In Proceedings of the Fifth International Conference on Computational Creativity, 211–219.
Monteith, K.; Brown, B.; Ventura, D.; and Martinez, T. 2013. Automatic generation of music for inducing physiological response. In Annual Meeting of the Cognitive Science Society, 3098–3103.
Monteith, K.; Martinez, T.; and Ventura, D. 2010. Automatic generation of music for inducing emotive response. In Proceedings of the International Conference on Computational Creativity, 140–149.
Norton, D.; Heath, D.; and Ventura, D. 2010. Establishing appreciation in a creative system. In Proceedings of the International Conference on Computational Creativity, 26– 35.
Norton, D.; Heath, D.; and Ventura, D. 2013. Finding creativity in an artificial artist. The Journal of Creative Behavior 47(2):106–124.
Pearce, M. T., and Wiggins, G. A. 2007. Evaluating cognitive models of musical composition. In Proceedings of the 4th International JointWorkshop on Computational Creativity, 73–80.
Pease, A.; Winterstein, D.; and Colton, S. 2001. Evaluating machine creativity. In Workshop on Creative Systems, 4th International Conference on Case Based Reasoning, 129– 137.
Poetry Foundation. 2014. Poetry magazine discussion guide. http://www.poetryfoundation.org/poetrymagazine/guide/89. Accessed: 2015-02-03.
Rashel, F., and Manurung, R. 2014. Pemuisi: a constraint satisfaction-based generator of topical Indonesian poetry. In Proceedings of the Fifth International Conference on Computational Creativity, 82–90.
Riedl, M. O., and Young, R. M. 2006. Story planning as exploratory creativity: Techniques for expanding the narrative search space. New Generation Computing 24(3):303–323.
Ritchie, G.; Munro, R.; Pain, H.; and Binsted, K. 2008. Evaluating humorous properties of texts. In AISB 2008 Convention Communication, Interaction and Social Intelligence, volume 1, 17.
Ritchie, G. 2001. Assessing creativity. In Proc. of AISB01 Symposium.
Ritchie, G. 2007. Some empirical criteria for attributing creativity to a computer program. Minds and Machines 17(1):67–99.
Rom´an, I. G., and y P´erez, R. P. 2014. Social Mexica: A computer model for social norms in narratives. In Proceedings of the Fifth International Conference on Computational Creativity, 192–200.
Smith, M. R.; Hintze, R. S.; and Ventura, D. 2014. Nehovah: A neologism creator nomen ipsum. In Proceedings of the Fifth International Conference on Computational Creativity, 193–181.
Tearse, B.; Mawhorter, M. M. P.; and Wardrip-Fruin, N. 2011. Experimental results from a rational reconstruction of MINSTREL. In Proceedings of the Second International Conference on Computational Creativity.
Ventura, D. 2008. A reductio ad absurdum experiment in sufficiency for evaluating (computational) creative systems. In Proceedings of the 5th International Joint Workshop on Computational Creativity, 11–19.
Young, M. W.; Bown, O.; et al. 2010. Clap-along: A negotiation strategy for creative musical interaction with computational systems. In Proceedings of the International Conference on Computational Creativity 2010, 215–222.
Links
Full Text
http://computationalcreativity.net/iccc2015/proceedings/5_2Lamb.pdf
