Aesthetics and emotions in images

Aus de_evolutionary_art_org
Wechseln zu: Navigation, Suche


Joshi, D., Datta, R., Fedorovskaya, E., Luong, Q.T., Wang, J.Z., Li, J., Luo, J.: Aesthetics and emotions in images. Signal Processing Magazine, IEEE 28(5) (2011) 94-115



In this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics and emotion from images. We begin with a background discussion on philosophy, photography, paintings, visual arts, and psychology. This is followed by introduction of a set of key computational problems that the research community has been striving to solve and the computational framework required for solving them. We also describe data sets available for performing assessment and outline several real-world applications where research in this domain can be employed. A significant number of papers that have attempted to solve problems in aesthetics and emotion inference are surveyed in this tutorial. We also discuss future directions that researchers can pursue and make a strong case for seriously attempting to solve problems in this research domain.

Extended Abstract


author={D. Joshi and R. Datta and E. Fedorovskaya and Q. T. Luong and J. Z. Wang and J. Li and J. Luo},
journal={IEEE Signal Processing Magazine},
title={Aesthetics and Emotions in Images},
keywords={art;image processing;inference mechanisms;painting;philosophical aspects;photography;psychology;aesthetics;computational inference;emotion inference;paintings;philosophy;photography;psychology;visual arts;Data visualization;Emotion recognition;Human factors;Painting;Photography;Semantics},

Used References

L. von Ahn, "Games with a purpose", IEEE Comput. Mag., vol. 39, no. 6, pp. 96-98, 2006

L. von Ahn and L. Dabbish, "Designing games with a purpose", Commun. ACM, vol. 51, no. 8, pp. 58-67, 2008

I. Arapakis, I. Konstas and J. M. Jose, "Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance", Proc. ACM Multimedia, pp. 461-470

R. Arnheim, "Art and Visual Perception" in A Psychology of the Creative Eye, 1954, Univ. California Press

O. Axelsson, "Towards a psychology of photography: Dimensions underlying aesthetic appeal of photographs", Percept. Mot. Skills, vol. 105, no. 2, pp. 411-434, 2007

M. Balabanovic and Y. Shoham, "Fab: Content-based, collaborative recommendation", Commun. ACM, vol. 40, no. 3, pp. 66-72, 1997

B. Barry, "The mindful camera: Common sense for documentary videography", Proc. ACM Multimedia, pp. 648-649

B. Barry and G. Davenport, "Documenting life: Videography and common sense", Proc. IEEE Int. Conf. Multimedia (ICME), pp. 197-200

D. E. Berlyne, Aesthetics and Psychobiology, 1971, Appleton-Century-Crofts

N. Bianchi-Berthouze, "K-dime: An affective image filtering system", IEEE Multimedia, vol. 10, no. 3, pp. 103-106, 2003

S. Bhattacharya, R. Sukthankar and M. Shah, "A framework for photo-quality assessment and enhancement based on visual aesthetics", Proc. ACM Multimedia, pp. 271-280

M. Bressan, C. Cifarelli and F. Perronnin, "An analysis of the relationship between painters based on their work", Proc. IEEE ICIP, pp. 113-116

I. E. Berezhnoy, E. O. Postma and H. J. van den Herik, "Computerized visual analysis of paintings", Proc. 16th Int. Conf. Association History and Computing, pp. 28-32

I. E. Berezhnoy, E. O. Postma and H. J. van den Herik, "Computer analysis of Van Gogh's complementary colors", Pattern Recognit. Lett., vol. 28, no. 6, pp. 703-709, 2007

G. H. Bower, "Mood and memory", Amer. Psychol., vol. 36, no. 2, pp. 129-148, 1981

C. Cerosaletti and A. Loui, "Measuring the perceived aesthetic quality of photographic images", Proc. 1st Int. Workshop Quality Multimedia Experience, pp. 47-52

B. Cheng, B. Ni, S. Yan and Q. Tian, "Learning to photograph", Proc. ACM Multimedia, pp. 291-300

S. Daly, "The visible differences predictor: An algorithm for the assessment of image fidelity" in Digital Image Hum. Vis., pp. 179-206, 1993, MIT Press

R. Datta, D. Joshi, J. Li and J. Z. Wang, "Studying aesthetics in photographic images using a computational approach", Proc. ECCV, pp. 288-301

R. Data, D. Joshi, J. Li and J. Z. Wang, "Image retrieval: Ideas, influences, and trends of the new age", ACM Comput. Surv., vol. 40, no. 2, pp. 51-60, 2008

R. Datta, J. Li and J. Z. Wang, "Learning the consensus on visual quality for next generation image management", Proc. ACM Multimedia, pp. 533-536

R. Datta, J. Li and J. Z. Wang, "Algorithmic inferencing of aesthetics and emotion in natural images: An exposition", Proc. ICIP, pp. 105-108

B. C. Davis and S. Lazebnik, "Analysis of human attractiveness using manifold kernel regression", Proc. ICIP, pp. 109-112

J. O'Doherty, J. Winston, H. Critchley, D. Perrett, D. M. Burt and R. J. Dolan, "Beauty in a smile: The role of medial orbitofrontal cortex in facial attractiveness", Neuropsychologia, vol. 41, no. 2, pp. 147-155, 2003

D. Dutton, "The Art Instinct: Beauty, Pleasure, and Human Evolution", 2009, Bloomsbury Press

J. Elkins, The Domain of Images, 1999, Cornell Univ. Press

J. Elkins, J. Jansen, T. O'Connor and, "Aesthetics and the two cultures: Why art and science should be allowed to go their separate ways" in Rediscovering Aesthetics: Transdisciplinary Voices from Art History, Philosophy, and Art Practice (Cultural Memory in the Present), pp. 34-50, 2009, Columbia Univ. Press

Y. Eisenthal, G. Dror and E. Ruppin, "Facial attractiveness: Beauty and the machine", Neural Comput., vol. 18, no. 1, pp. 119-142, 2006

C. M. Falco, "Computer vision and art", IEEE Multimedia, vol. 14, no. 2, pp. 8-11, 2007

Y. Fang, D. Geman and N. Boujemaa, "An interactive system for mental face retrieval", Proc. ACM SIGMM Int. Workshop Multimedia Information Retrieval, pp. 193-200

G. T. Fechner, "Zur experimentalen Ästhetik (On experimental aesthetics)", Abhandlungen der Königlich Sächsischen Gesellschaft der Wissenschaften, vol. 9, pp. 555-635, 1871

G. T. Fechner, "Vorschule der aesthetik", Breitkopf und Härtel, Leipzig, Breitkopf und Härtel, Leipzig, vol. 1¿¿¿2, 1876

E. A. Fedorovskaya, C. Neustaedter and W. Hao, "Image harmony for consumer images", Proc. ICIP, pp. 121-124

M. Freeman, The Photographer's Eye: Composition and Design for Better Digital Photos, 2007, Focal Press, Elsevier Inc.

S. Freud, The Interpretation of Dreams, 1913, The Macmillan Company

A. Gallagher, D. Joshi, J. Yu and J. Luo, "Geo-location inference from image content and user tags", Proc. IEEE Int. Workshop Internet Vision (CVPR), pp. 55-62

A. Gallagher and T. Chen, "Using context to recognize people in consumer images", IPSJ Trans. Comput. Vis. Applicat., vol. 1, pp. 115-126, 2009

D. Goldberg, D. Nichols, B. M. Oki and D. Terry, "Using collaborative filtering to weave an information tapestry", Commun. ACM, vol. 35, no. 12, pp. 61-70, 1992

N. Goodman, Languages of Art: An Approach to a Theory of Symbols, 1976, Hackett Publishing Co.

C. Greenberg, Art and Culture Critical Essays, 1971, Beacon Press

J. Hays and A. Efros, "Scene completion using millions of photographs", ACM Trans. Graphics, vol. 26, no. 2, 2007

J. Hays and A. Efros, "IM2GPS: Estimating geographic information from a single image", Proc. CVPR, pp. 1-8

P. Hekkert, "Beauty in the eye of expert and non-expert beholders: A study in the appraisal of art", Amer. J. Psychol., vol. 109, no. 3, pp. 389-407, 1997

J. Howe, "The rise of crowdsourcing", Wired Mag., vol. 14, no. 6, 2006

C. R. Johnson, Jr. E. Hendriks, I. J. Berezhnoy, E. Brevdo, S. M. Hughes, I. Daubechies, J. Li, E. Postma and J. Z. Wang, "Image processing for artist identification: computerized analysis of Vincent van Gogh's painting brushstrokes", IEEE Signal Processing Mag. (Special Issue on Visual Cultural Heritage), vol. 25, no. 4, pp. 37-48, 2008

H. Kawabata and S. Zeki, "Neural correlates of beauty", J. Neurophysiol., vol. 91, no. 4, pp. 1699-1705, 2004

Y. Ke, X. Tang and F. Jing, "The design of high-level features for photo quality assessment", Proc. CVPR, pp. 419-426

L. Kennedy and M. Naaman, "How flicker helps us make sense of the world: Context and content in community-contributed media collections", Proc. ACM Multimedia, pp. 631-640

L. Kennedy and M. Naaman, "Generating diverse and representative image search results for landmarks", Proc. 17th Int. Conf. World Wide Web, pp. 297-306

U. Kirk, M. Skov, O. Hulme, M. S. Christensen and S. Zeki, "Modulation of aesthetic value by semantic context: An fMRI study", Neuroimage, vol. 44, no. 3, pp. 1125-1132, 2009

K. Koffka, Gestalt Psychology, 1935, Harcourt Brace Jovanovic

S. Kroner and A. Lattner, "Authentication of free hand drawings by pattern recognition methods", Proc. IEEE ICPR, pp. 462-464

A. Kushki, P. Androutsos, K. Plataniotis and A. Venetsanopoulos, "Retrieval of images from artistic repositories using a decision fusion framework", IEEE Trans. Image Process., vol. 13, no. 3, pp. 277-292, 2004

P. J. Lang, M. K. Greenwald, M. M. Bradley and A. O. Hamm, "Looking at pictures: Affective, facial, visceral, and behavioral reactions", Psychophysiology, vol. 30, no. 3, pp. 261-273, 1993

P. J. Lang, M. M. Bradley and B. N. Cuthbert, "International affective picture system (IAPS): Technical, manual, and affective ratings", NIMH Center for the Study of Emotion and Attention, Gainsville, FL, 1997

R. Latto, "The brain of the beholder" in The Artful Eye, pp. 66-94, 1995, Oxford Univ. Press

C. C. Li and T. Chen, "Aesthetic visual quality assessment of paintings", IEEE J. Select. Topics Signal Process., vol. 3, no. 2, pp. 236-252, 2009

J. Z. Wang, "Studying digital imagery of ancient paintings by mixtures of stochastic models", IEEE Trans. Image Process., vol. 13, no. 3, pp. 340-353, 2004

Y. Liu, K. L. Schmidt, J. F. Cohn and S. Mitra, "Facial asymmetry quantification for expression invariant human identification", Proc. CVPR, pp. 198-204

A. Louis, M. D. Wood, A. Scalise and J. Birkelund, "Multidimensional image value assessment and rating for automated albuming and retrieval", Proc. ICIP, pp. 97-100

P. J. Lu and P. J. Steinhardt, "Decagonal and quasi-crystalline tilings in medieval islamic architecture", Science, vol. 315, no. 5815, pp. 1106-1110, 2007

P. J. Lu, "Early precision compound machine from ancient China", Science, vol. 304, no. 5677, pp. 1638, 2004

J. Luo, A. Savakis, S. Etz and A. Singhal, "On the application of Bayes networks to semantic understanding of consumer photographs", Proc. ICIP, pp. 512-515

J. Luo, M. Boutell and C. Brown, "Exploiting context for semantic scene content understanding", IEEE Signal Processing Mag. (Special Issue on Semantic Retrieval of Multimedia), vol. 23, no. 2, pp. 101-114, 2006

J. Machajdik and A. Hanbury, "Affective image classification using features inspired by psychology and art theory", Proc. ACM Multimedia, pp. 83-92

W. J. T. Mitchell, Iconology: Image Text and Ideology, 1986, Univ. Chicago Press

W. J. T. Mitchell, Picture Theory, 1994, Univ. Chicago Press

C. F. Nodien, P. J. Locher and E. A. Krupinski, "The role of formal art training on perception and aesthetic judgment of art compositions", Leonardo, vol. 26, no. 3, pp. 219-227, 1993

S. E. Palmer, "Aesthetic science: Human preferences for spatial composition", Proc. IS&T/SPIE Electronic Imaging Conf.

D. I. Perrett, K. A. May and S. Yoshikawa, "Facial shape and judgments of female attractiveness", Nature, vol. 368, pp. 239-242, 1994

G. Peters, "Aesthetic primitives of images for visualization", Proc. IEEE Int. Conf. Information Visualization, pp. 316-325

V. S. Ramachandran and W. Hirstein, "Science of art: A neurological theory of aesthetic experience", J. Consciousness Stud., vol. 6, no. 6/7, pp. 15-51, 1999

S. Ramanathan, H. Katti, R. Huang, T.-S. Chua and M. Kankanhalli, "Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis", Proc. ACM Multimedia, pp. 729-732

R. N. Reber, N. Schwarts and P. Winkielman, "Processing fluency and aesthetic pleasure: Is beauty in the perceiver's processing experience?", Pers. Social Psychol. Rev., vol. 8, no. 4, pp. 364-382, 2004

D. Rockmore, S. Lyu and H. Farid, "A digital technique for authentication in the visual arts", Int. Found. Art Res., vol. 8, no. 2, pp. 12-23, 2006

A. Savakis, S. Etz and A. Loui, "Evaluation of image appeal in consumer photography", Proc. SPIE Human Vision and Electronic Imaging, pp. 111-120

J. E. Scheib, S. W. Gangestad and R. Thornhill, "Facial attractiveness, symmetry, and cues of good genes", Proc. Royal Soc. London, Biol Sci, vol. 266, no. 1431, pp. 1913-1917, 1999

H. R. Sheikh, A. C. Bovik and L. Cormack, "No-reference quality assessment using natural scene statistics: JPEG2000", IEEE Trans. Image Processing, vol. 14, no. 11, pp. 1918-1927, 2005

B. Shevade, H. Sundaram and L. Xie, "Modeling personal and social network context for event annotation in images", Proc. Joint Conf. Digital Libraries, pp. 127-134

P. Singh and B. Barry, "Teaching machines about everyday life", BT Technol J., vol. 22, no. 4, pp. 227-240, 2004

A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain, "Content-based image retrieval at the end of early years", IEEE Trans. Pattern Anal. Machine Intell., vol. 22, no. 12, pp. 1349-1380, 2000

R. L. Solso, The Psychology of Art and the Evolution of the Conscious Brain, 2003, MIT Press

I. Stamos and P. Allen, "3-D model construction using range and image data", Proc. CVPR, pp. 531-536

D. Stork, "Computer vision and computer graphics analysis of paintings and drawings: An introduction to the literature" in Proc Int. Conf. Computer Analysis of Images and Patterns, pp. 9-24, 2009, Springer-Verlag

X. Sun, H. Yao, R. Ji and S. Liu, "Photo assessment based on computational visual attention model", Proc. ACM Multimedia, pp. 541-544

J. P. Swaddle and I. C. Cuthill, "Asymmetry and human facial attractiveness: Symmetry may not always be beautiful", Proc. Royal Soc. London, Biol Sci, vol. 261, no. 1360, pp. 111-116

R. Taylor, A. P. Micolich and D. Jones, "Fractal analysis of pollock's drip paintings", Nature, vol. 399, no. 6735, pp. 422, 1999

R. Taylor, "Pollock, Mondrian and the nature: Recent scientific investigations", Chaos Complexity Lett., vol. 1, no. 3, pp. 265-277, 2004

A. Torralba, R. Fergus and W. T. Freeman, "80 million tiny images: A large dataset for nonparametric object and scene recognition", IEEE Trans. Pattern Anal. Machine Intell., vol. 30, no. 11, pp. 1958-1970, 2008

R. S. Ulrich and L. Gilpin, "Healing arts: Nutrition for the soul" in Putting Patients First: Designing and Practicing Patient-Centered Care, pp. 117-146, 2003, (Wiley)

R. Valenti, N. Sebe and T. Gevers, "Facial expression recognition: A fully integrated approach", Proc. Int. Workshop Visual and Multimedia Digital Libraries, pp. 125-130

R. Valenti, A. Jaimes and N. Sebe, "Sonify your face: Facial expressions for sound generation", Proc. ACM Multimedia, pp. 1363-1372

C. W. Valentine, "The Experimental Psychology of Beauty", 1962, Methuen and Co. Ltd Publishers

W. Wang and Q. He, "A survey on emotional semantic image retrieval", Proc. ICIP, pp. 117-120

X. J. Wang, L. Zhang, F. Jing and W. Y. Ma, "Annosearch: Image auto-annotation by search", Proc. CVPR, pp. 1483-1490

A. B. Watson, "Toward a perceptual video quality metric", Proc. SPIE, vol. 3299, pp. 139-147, 1998

A. R. Willis and D. B. Cooper, "Computational reconstruction of ancient artifacts—From ruins to relics", IEEE Signal Processing Mag., vol. 25, no. 4, pp. 65-83, 2008

A. S. Winston and G. C. Cupchik, "The evaluation of high art and popular art by naive and experienced viewers", Vis. Arts Res., vol. 18, no. 1, pp. 1-14, 1992

R. Wollheim, Painting as an Art, 1987, Princeton Univ. Press

J. Wypijewski, Painting by the Numbers: Komar and Melamid's Scientific Guide to Art, 1997, Farrar, Straus and Giroux

Y. Yang, M. Song, N. Li, J. Bu and C. Chen, "Visual attention analysis by pseudo gravitational field", Proc. ACM Multimedia, pp. 553-556

V. Yanulevskaya, J. C. van Gemert, K. Roth, A. K. Herbold, N. Sebe and J. M. Geusebroek, "Emotional valence categorization using holistic image features", Proc. ICIP, pp. 101-104

D. W. Zaidel and J. A. Cohen, "The face, beauty, and symmetry: Perceiving asymmetry in beautiful faces", Int. J. Neurosci., vol. 115, no. 8, pp. 1165-1173, 2005

S. Zeki, Inner Vision: An Exploration of Art and the Brain, 1999, Oxford Univ. Press

A. Zunjarwad, H. Sundaram and L. Xie, "Contextual wisdom: Social relations and correlations for multimedia event annotation", Proc. ACM Multimedia, pp. 615-624

"Special issue on image processing for cultural heritage", IEEE Trans. Image Processing, vol. 13, no. 3, 2004

"Special issue on semantic retrieval of multimedia", IEEE Signal Processing Mag., vol. 23, no. 2, 2006

"Special issue on visual cultural heritage", IEEE Signal Processing Mag., vol. 25, no. 4, 2008

"ACQUINE", [online] Available: online

"ALIPR", [online] Available: online

"DPChallenge", [online] Available: online

"Encyclopedia Britannica", [online] Available: online

"Flickr", [online] Available: online

"Nadia Camera", [online] Available: online

"", [online] Available: online

"Terragalleria", [online] Available: online

"USA Today", [online] Available: online


Full Text

internal file

Sonstige Links