Aesthetics and emotions in images
Inhaltsverzeichnis
Referenz
Joshi, D., Datta, R., Fedorovskaya, E., Luong, Q.T., Wang, J.Z., Li, J., Luo, J.: Aesthetics and emotions in images. Signal Processing Magazine, IEEE 28(5) (2011) 94-115
DOI
http://dx.doi.org/10.1109/MSP.2011.941851
Abstract
In this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics and emotion from images. We begin with a background discussion on philosophy, photography, paintings, visual arts, and psychology. This is followed by introduction of a set of key computational problems that the research community has been striving to solve and the computational framework required for solving them. We also describe data sets available for performing assessment and outline several real-world applications where research in this domain can be employed. A significant number of papers that have attempted to solve problems in aesthetics and emotion inference are surveyed in this tutorial. We also discuss future directions that researchers can pursue and make a strong case for seriously attempting to solve problems in this research domain.
Extended Abstract
Bibtex
@ARTICLE{5999579,
author={D. Joshi and R. Datta and E. Fedorovskaya and Q. T. Luong and J. Z. Wang and J. Li and J. Luo},
journal={IEEE Signal Processing Magazine},
title={Aesthetics and Emotions in Images},
year={2011},
volume={28},
number={5},
pages={94-115},
keywords={art;image processing;inference mechanisms;painting;philosophical aspects;photography;psychology;aesthetics;computational inference;emotion inference;paintings;philosophy;photography;psychology;visual arts;Data visualization;Emotion recognition;Human factors;Painting;Photography;Semantics},
doi={10.1109/MSP.2011.941851},
url={},
ISSN={1053-5888},
month={Sept},
}
Used References
L. von Ahn, "Games with a purpose", IEEE Comput. Mag., vol. 39, no. 6, pp. 96-98, 2006 http://dx.doi.org/10.1109/MC.2006.196
L. von Ahn and L. Dabbish, "Designing games with a purpose", Commun. ACM, vol. 51, no. 8, pp. 58-67, 2008 http://dx.doi.org/10.1145/1378704.1378719
I. Arapakis, I. Konstas and J. M. Jose, "Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance", Proc. ACM Multimedia, pp. 461-470 http://dx.doi.org/10.1145/1631272.1631336
R. Arnheim, "Art and Visual Perception" in A Psychology of the Creative Eye, 1954, Univ. California Press
O. Axelsson, "Towards a psychology of photography: Dimensions underlying aesthetic appeal of photographs", Percept. Mot. Skills, vol. 105, no. 2, pp. 411-434, 2007 http://dx.doi.org/10.2466/pms.105.2.411-434
M. Balabanovic and Y. Shoham, "Fab: Content-based, collaborative recommendation", Commun. ACM, vol. 40, no. 3, pp. 66-72, 1997 http://dx.doi.org/10.1145/245108.245124
B. Barry, "The mindful camera: Common sense for documentary videography", Proc. ACM Multimedia, pp. 648-649 http://dx.doi.org/10.1145/957013.957152
B. Barry and G. Davenport, "Documenting life: Videography and common sense", Proc. IEEE Int. Conf. Multimedia (ICME), pp. 197-200 http://dx.doi.org/10.1109/ICME.2003.1221587
D. E. Berlyne, Aesthetics and Psychobiology, 1971, Appleton-Century-Crofts
N. Bianchi-Berthouze, "K-dime: An affective image filtering system", IEEE Multimedia, vol. 10, no. 3, pp. 103-106, 2003 http://dx.doi.org/10.1109/MMUL.2003.1218262
S. Bhattacharya, R. Sukthankar and M. Shah, "A framework for photo-quality assessment and enhancement based on visual aesthetics", Proc. ACM Multimedia, pp. 271-280 http://dx.doi.org/10.1145/1873951.1873990
M. Bressan, C. Cifarelli and F. Perronnin, "An analysis of the relationship between painters based on their work", Proc. IEEE ICIP, pp. 113-116 http://dx.doi.org/10.1109/ICIP.2008.4711704
I. E. Berezhnoy, E. O. Postma and H. J. van den Herik, "Computerized visual analysis of paintings", Proc. 16th Int. Conf. Association History and Computing, pp. 28-32
I. E. Berezhnoy, E. O. Postma and H. J. van den Herik, "Computer analysis of Van Gogh's complementary colors", Pattern Recognit. Lett., vol. 28, no. 6, pp. 703-709, 2007 http://dx.doi.org/10.1016/j.patrec.2006.08.002
G. H. Bower, "Mood and memory", Amer. Psychol., vol. 36, no. 2, pp. 129-148, 1981 http://dx.doi.org/10.1037/0003-066X.36.2.129
C. Cerosaletti and A. Loui, "Measuring the perceived aesthetic quality of photographic images", Proc. 1st Int. Workshop Quality Multimedia Experience, pp. 47-52 http://dx.doi.org/10.1109/QOMEX.2009.5246977
B. Cheng, B. Ni, S. Yan and Q. Tian, "Learning to photograph", Proc. ACM Multimedia, pp. 291-300 http://dx.doi.org/10.1145/1873951.1873992
S. Daly, "The visible differences predictor: An algorithm for the assessment of image fidelity" in Digital Image Hum. Vis., pp. 179-206, 1993, MIT Press
R. Datta, D. Joshi, J. Li and J. Z. Wang, "Studying aesthetics in photographic images using a computational approach", Proc. ECCV, pp. 288-301 http://dx.doi.org/10.1007/11744078_23
R. Data, D. Joshi, J. Li and J. Z. Wang, "Image retrieval: Ideas, influences, and trends of the new age", ACM Comput. Surv., vol. 40, no. 2, pp. 51-60, 2008
R. Datta, J. Li and J. Z. Wang, "Learning the consensus on visual quality for next generation image management", Proc. ACM Multimedia, pp. 533-536 http://dx.doi.org/10.1145/1291233.1291364
R. Datta, J. Li and J. Z. Wang, "Algorithmic inferencing of aesthetics and emotion in natural images: An exposition", Proc. ICIP, pp. 105-108 http://dx.doi.org/10.1109/ICIP.2008.4711702
B. C. Davis and S. Lazebnik, "Analysis of human attractiveness using manifold kernel regression", Proc. ICIP, pp. 109-112 http://dx.doi.org/10.1109/ICIP.2008.4711703
J. O'Doherty, J. Winston, H. Critchley, D. Perrett, D. M. Burt and R. J. Dolan, "Beauty in a smile: The role of medial orbitofrontal cortex in facial attractiveness", Neuropsychologia, vol. 41, no. 2, pp. 147-155, 2003 http://dx.doi.org/10.1016/S0028-3932(02)00145-8
D. Dutton, "The Art Instinct: Beauty, Pleasure, and Human Evolution", 2009, Bloomsbury Press
J. Elkins, The Domain of Images, 1999, Cornell Univ. Press
J. Elkins, J. Jansen, T. O'Connor and, "Aesthetics and the two cultures: Why art and science should be allowed to go their separate ways" in Rediscovering Aesthetics: Transdisciplinary Voices from Art History, Philosophy, and Art Practice (Cultural Memory in the Present), pp. 34-50, 2009, Columbia Univ. Press
Y. Eisenthal, G. Dror and E. Ruppin, "Facial attractiveness: Beauty and the machine", Neural Comput., vol. 18, no. 1, pp. 119-142, 2006 http://dx.doi.org/10.1162/089976606774841602
C. M. Falco, "Computer vision and art", IEEE Multimedia, vol. 14, no. 2, pp. 8-11, 2007 http://dx.doi.org/10.1109/MMUL.2007.31
Y. Fang, D. Geman and N. Boujemaa, "An interactive system for mental face retrieval", Proc. ACM SIGMM Int. Workshop Multimedia Information Retrieval, pp. 193-200 http://dx.doi.org/10.1145/1101826.1101858
G. T. Fechner, "Zur experimentalen Ästhetik (On experimental aesthetics)", Abhandlungen der Königlich Sächsischen Gesellschaft der Wissenschaften, vol. 9, pp. 555-635, 1871
G. T. Fechner, "Vorschule der aesthetik", Breitkopf und Härtel, Leipzig, Breitkopf und Härtel, Leipzig, vol. 1¿¿¿2, 1876
E. A. Fedorovskaya, C. Neustaedter and W. Hao, "Image harmony for consumer images", Proc. ICIP, pp. 121-124 http://dx.doi.org/10.1109/ICIP.2008.4711706
M. Freeman, The Photographer's Eye: Composition and Design for Better Digital Photos, 2007, Focal Press, Elsevier Inc.
S. Freud, The Interpretation of Dreams, 1913, The Macmillan Company http://dx.doi.org/10.1037/10561-000
A. Gallagher, D. Joshi, J. Yu and J. Luo, "Geo-location inference from image content and user tags", Proc. IEEE Int. Workshop Internet Vision (CVPR), pp. 55-62 http://dx.doi.org/10.1109/CVPRW.2009.5204168
A. Gallagher and T. Chen, "Using context to recognize people in consumer images", IPSJ Trans. Comput. Vis. Applicat., vol. 1, pp. 115-126, 2009 http://dx.doi.org/10.2197/ipsjtcva.1.115
D. Goldberg, D. Nichols, B. M. Oki and D. Terry, "Using collaborative filtering to weave an information tapestry", Commun. ACM, vol. 35, no. 12, pp. 61-70, 1992 http://dx.doi.org/10.1145/138859.138867
N. Goodman, Languages of Art: An Approach to a Theory of Symbols, 1976, Hackett Publishing Co.
C. Greenberg, Art and Culture Critical Essays, 1971, Beacon Press
J. Hays and A. Efros, "Scene completion using millions of photographs", ACM Trans. Graphics, vol. 26, no. 2, 2007 http://dx.doi.org/10.1145/1275808.1276382
J. Hays and A. Efros, "IM2GPS: Estimating geographic information from a single image", Proc. CVPR, pp. 1-8 http://dx.doi.org/10.1109/CVPR.2008.4587784
P. Hekkert, "Beauty in the eye of expert and non-expert beholders: A study in the appraisal of art", Amer. J. Psychol., vol. 109, no. 3, pp. 389-407, 1997 http://dx.doi.org/10.2307/1423013
J. Howe, "The rise of crowdsourcing", Wired Mag., vol. 14, no. 6, 2006
C. R. Johnson, Jr. E. Hendriks, I. J. Berezhnoy, E. Brevdo, S. M. Hughes, I. Daubechies, J. Li, E. Postma and J. Z. Wang, "Image processing for artist identification: computerized analysis of Vincent van Gogh's painting brushstrokes", IEEE Signal Processing Mag. (Special Issue on Visual Cultural Heritage), vol. 25, no. 4, pp. 37-48, 2008 http://dx.doi.org/10.1109/MSP.2008.923513
H. Kawabata and S. Zeki, "Neural correlates of beauty", J. Neurophysiol., vol. 91, no. 4, pp. 1699-1705, 2004 http://dx.doi.org/10.1152/jn.00696.2003
Y. Ke, X. Tang and F. Jing, "The design of high-level features for photo quality assessment", Proc. CVPR, pp. 419-426
L. Kennedy and M. Naaman, "How flicker helps us make sense of the world: Context and content in community-contributed media collections", Proc. ACM Multimedia, pp. 631-640
L. Kennedy and M. Naaman, "Generating diverse and representative image search results for landmarks", Proc. 17th Int. Conf. World Wide Web, pp. 297-306 http://dx.doi.org/10.1145/1367497.1367539
U. Kirk, M. Skov, O. Hulme, M. S. Christensen and S. Zeki, "Modulation of aesthetic value by semantic context: An fMRI study", Neuroimage, vol. 44, no. 3, pp. 1125-1132, 2009 http://dx.doi.org/10.1016/j.neuroimage.2008.10.009
K. Koffka, Gestalt Psychology, 1935, Harcourt Brace Jovanovic
S. Kroner and A. Lattner, "Authentication of free hand drawings by pattern recognition methods", Proc. IEEE ICPR, pp. 462-464 http://dx.doi.org/10.1109/ICPR.1998.711180
A. Kushki, P. Androutsos, K. Plataniotis and A. Venetsanopoulos, "Retrieval of images from artistic repositories using a decision fusion framework", IEEE Trans. Image Process., vol. 13, no. 3, pp. 277-292, 2004 http://dx.doi.org/10.1109/TIP.2003.821350
P. J. Lang, M. K. Greenwald, M. M. Bradley and A. O. Hamm, "Looking at pictures: Affective, facial, visceral, and behavioral reactions", Psychophysiology, vol. 30, no. 3, pp. 261-273, 1993 http://dx.doi.org/10.1111/j.1469-8986.1993.tb03352.x
P. J. Lang, M. M. Bradley and B. N. Cuthbert, "International affective picture system (IAPS): Technical, manual, and affective ratings", NIMH Center for the Study of Emotion and Attention, Gainsville, FL, 1997
R. Latto, "The brain of the beholder" in The Artful Eye, pp. 66-94, 1995, Oxford Univ. Press
C. C. Li and T. Chen, "Aesthetic visual quality assessment of paintings", IEEE J. Select. Topics Signal Process., vol. 3, no. 2, pp. 236-252, 2009 http://dx.doi.org/10.1109/JSTSP.2009.2015077
J. Z. Wang, "Studying digital imagery of ancient paintings by mixtures of stochastic models", IEEE Trans. Image Process., vol. 13, no. 3, pp. 340-353, 2004
Y. Liu, K. L. Schmidt, J. F. Cohn and S. Mitra, "Facial asymmetry quantification for expression invariant human identification", Proc. CVPR, pp. 198-204 http://dx.doi.org/10.1126/science.1135491
A. Louis, M. D. Wood, A. Scalise and J. Birkelund, "Multidimensional image value assessment and rating for automated albuming and retrieval", Proc. ICIP, pp. 97-100
P. J. Lu and P. J. Steinhardt, "Decagonal and quasi-crystalline tilings in medieval islamic architecture", Science, vol. 315, no. 5815, pp. 1106-1110, 2007 http://dx.doi.org/10.1126/science.1096588
P. J. Lu, "Early precision compound machine from ancient China", Science, vol. 304, no. 5677, pp. 1638, 2004 http://dx.doi.org/10.1126/science.1096588
J. Luo, A. Savakis, S. Etz and A. Singhal, "On the application of Bayes networks to semantic understanding of consumer photographs", Proc. ICIP, pp. 512-515 http://dx.doi.org/10.1109/ICIP.2000.899479
J. Luo, M. Boutell and C. Brown, "Exploiting context for semantic scene content understanding", IEEE Signal Processing Mag. (Special Issue on Semantic Retrieval of Multimedia), vol. 23, no. 2, pp. 101-114, 2006
J. Machajdik and A. Hanbury, "Affective image classification using features inspired by psychology and art theory", Proc. ACM Multimedia, pp. 83-92 http://dx.doi.org/10.1145/1873951.1873965
W. J. T. Mitchell, Iconology: Image Text and Ideology, 1986, Univ. Chicago Press
W. J. T. Mitchell, Picture Theory, 1994, Univ. Chicago Press
C. F. Nodien, P. J. Locher and E. A. Krupinski, "The role of formal art training on perception and aesthetic judgment of art compositions", Leonardo, vol. 26, no. 3, pp. 219-227, 1993 http://dx.doi.org/10.2307/1575815
S. E. Palmer, "Aesthetic science: Human preferences for spatial composition", Proc. IS&T/SPIE Electronic Imaging Conf.
D. I. Perrett, K. A. May and S. Yoshikawa, "Facial shape and judgments of female attractiveness", Nature, vol. 368, pp. 239-242, 1994 http://dx.doi.org/10.1038/368239a0
G. Peters, "Aesthetic primitives of images for visualization", Proc. IEEE Int. Conf. Information Visualization, pp. 316-325 http://dx.doi.org/10.1109/IV.2007.20
V. S. Ramachandran and W. Hirstein, "Science of art: A neurological theory of aesthetic experience", J. Consciousness Stud., vol. 6, no. 6/7, pp. 15-51, 1999
S. Ramanathan, H. Katti, R. Huang, T.-S. Chua and M. Kankanhalli, "Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis", Proc. ACM Multimedia, pp. 729-732 http://dx.doi.org/10.1145/1631272.1631399
R. N. Reber, N. Schwarts and P. Winkielman, "Processing fluency and aesthetic pleasure: Is beauty in the perceiver's processing experience?", Pers. Social Psychol. Rev., vol. 8, no. 4, pp. 364-382, 2004 http://dx.doi.org/10.1207/s15327957pspr0804_3
D. Rockmore, S. Lyu and H. Farid, "A digital technique for authentication in the visual arts", Int. Found. Art Res., vol. 8, no. 2, pp. 12-23, 2006
A. Savakis, S. Etz and A. Loui, "Evaluation of image appeal in consumer photography", Proc. SPIE Human Vision and Electronic Imaging, pp. 111-120
J. E. Scheib, S. W. Gangestad and R. Thornhill, "Facial attractiveness, symmetry, and cues of good genes", Proc. Royal Soc. London, Biol Sci, vol. 266, no. 1431, pp. 1913-1917, 1999 http://dx.doi.org/10.1098/rspb.1999.0866
H. R. Sheikh, A. C. Bovik and L. Cormack, "No-reference quality assessment using natural scene statistics: JPEG2000", IEEE Trans. Image Processing, vol. 14, no. 11, pp. 1918-1927, 2005http://dx.doi.org/10.1109/TIP.2005.854492
B. Shevade, H. Sundaram and L. Xie, "Modeling personal and social network context for event annotation in images", Proc. Joint Conf. Digital Libraries, pp. 127-134 http://dx.doi.org/10.1145/1255175.1255200
P. Singh and B. Barry, "Teaching machines about everyday life", BT Technol J., vol. 22, no. 4, pp. 227-240, 2004 http://dx.doi.org/10.1023/B:BTTJ.0000047601.53388.74
A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain, "Content-based image retrieval at the end of early years", IEEE Trans. Pattern Anal. Machine Intell., vol. 22, no. 12, pp. 1349-1380, 2000 http://dx.doi.org/10.1109/34.895972
R. L. Solso, The Psychology of Art and the Evolution of the Conscious Brain, 2003, MIT Press
I. Stamos and P. Allen, "3-D model construction using range and image data", Proc. CVPR, pp. 531-536 http://dx.doi.org/10.1109/CVPR.2000.855865
D. Stork, "Computer vision and computer graphics analysis of paintings and drawings: An introduction to the literature" in Proc Int. Conf. Computer Analysis of Images and Patterns, pp. 9-24, 2009, Springer-Verlag
X. Sun, H. Yao, R. Ji and S. Liu, "Photo assessment based on computational visual attention model", Proc. ACM Multimedia, pp. 541-544 http://dx.doi.org/10.1145/1631272.1631351
J. P. Swaddle and I. C. Cuthill, "Asymmetry and human facial attractiveness: Symmetry may not always be beautiful", Proc. Royal Soc. London, Biol Sci, vol. 261, no. 1360, pp. 111-116 http://dx.doi.org/10.1098/rspb.1995.0124
R. Taylor, A. P. Micolich and D. Jones, "Fractal analysis of pollock's drip paintings", Nature, vol. 399, no. 6735, pp. 422, 1999 http://dx.doi.org/10.1038/20833
R. Taylor, "Pollock, Mondrian and the nature: Recent scientific investigations", Chaos Complexity Lett., vol. 1, no. 3, pp. 265-277, 2004
A. Torralba, R. Fergus and W. T. Freeman, "80 million tiny images: A large dataset for nonparametric object and scene recognition", IEEE Trans. Pattern Anal. Machine Intell., vol. 30, no. 11, pp. 1958-1970, 2008 http://dx.doi.org/10.1109/TPAMI.2008.128
R. S. Ulrich and L. Gilpin, "Healing arts: Nutrition for the soul" in Putting Patients First: Designing and Practicing Patient-Centered Care, pp. 117-146, 2003, (Wiley)
R. Valenti, N. Sebe and T. Gevers, "Facial expression recognition: A fully integrated approach", Proc. Int. Workshop Visual and Multimedia Digital Libraries, pp. 125-130 http://dx.doi.org/10.1109/ICIAPW.2007.25
R. Valenti, A. Jaimes and N. Sebe, "Sonify your face: Facial expressions for sound generation", Proc. ACM Multimedia, pp. 1363-1372 http://dx.doi.org/10.1145/1873951.1874219
C. W. Valentine, "The Experimental Psychology of Beauty", 1962, Methuen and Co. Ltd Publishers
W. Wang and Q. He, "A survey on emotional semantic image retrieval", Proc. ICIP, pp. 117-120 http://dx.doi.org/10.1109/ICIP.2008.4711705
X. J. Wang, L. Zhang, F. Jing and W. Y. Ma, "Annosearch: Image auto-annotation by search", Proc. CVPR, pp. 1483-1490
A. B. Watson, "Toward a perceptual video quality metric", Proc. SPIE, vol. 3299, pp. 139-147, 1998
A. R. Willis and D. B. Cooper, "Computational reconstruction of ancient artifacts—From ruins to relics", IEEE Signal Processing Mag., vol. 25, no. 4, pp. 65-83, 2008 http://dx.doi.org/10.1109/MSP.2008.923101
A. S. Winston and G. C. Cupchik, "The evaluation of high art and popular art by naive and experienced viewers", Vis. Arts Res., vol. 18, no. 1, pp. 1-14, 1992
R. Wollheim, Painting as an Art, 1987, Princeton Univ. Press
J. Wypijewski, Painting by the Numbers: Komar and Melamid's Scientific Guide to Art, 1997, Farrar, Straus and Giroux
Y. Yang, M. Song, N. Li, J. Bu and C. Chen, "Visual attention analysis by pseudo gravitational field", Proc. ACM Multimedia, pp. 553-556 http://dx.doi.org/10.1145/1631272.1631354
V. Yanulevskaya, J. C. van Gemert, K. Roth, A. K. Herbold, N. Sebe and J. M. Geusebroek, "Emotional valence categorization using holistic image features", Proc. ICIP, pp. 101-104 http://dx.doi.org/10.1109/ICIP.2008.4711701
D. W. Zaidel and J. A. Cohen, "The face, beauty, and symmetry: Perceiving asymmetry in beautiful faces", Int. J. Neurosci., vol. 115, no. 8, pp. 1165-1173, 2005 http://dx.doi.org/10.1080/00207450590914464
S. Zeki, Inner Vision: An Exploration of Art and the Brain, 1999, Oxford Univ. Press
A. Zunjarwad, H. Sundaram and L. Xie, "Contextual wisdom: Social relations and correlations for multimedia event annotation", Proc. ACM Multimedia, pp. 615-624 http://dx.doi.org/10.1145/1291233.1291382
"Special issue on image processing for cultural heritage", IEEE Trans. Image Processing, vol. 13, no. 3, 2004
"Special issue on semantic retrieval of multimedia", IEEE Signal Processing Mag., vol. 23, no. 2, 2006
"Special issue on visual cultural heritage", IEEE Signal Processing Mag., vol. 25, no. 4, 2008
"ACQUINE", [online] Available: online
"ALIPR", [online] Available: online
"DPChallenge", [online] Available: online
"Encyclopedia Britannica", [online] Available: online
"Flickr", [online] Available: online
"Nadia Camera", [online] Available: online
"Photo.net", [online] Available: online
"Terragalleria", [online] Available: online
"USA Today", [online] Available: online
Links
Full Text
