A Commonsense Reasoning Framework for Explanatory Emotion Attribution, Generation and Re-classification

Antonio Lieto

We present DEGARI (Dynamic Emotion Generator And ReclassIfier), an explainable system for emotion attribution and recommendation. This system relies on a recently introduced commonsense reasoning framework, the T CL logic, which is based on a human-like procedure for the automatic generation of novel concepts in a Description Logics knowledge base. Starting from an ontological formalization of emotions based on the Plutchik model, known as ArsEmotica, the system exploits the logic T CL to automatically generate novel commonsense semantic representations of compound emotions (e.g. Love as derived from the combination of Joy and Trust according to Plutchik). The generated emotions correspond to prototypes, i.e. commonsense representations of given concepts, and have been used to reclassify emotion-related contents in a variety of artistic domains, ranging from art datasets to the editorial contents available in RaiPlay, the online platform of RAI Radiotelevisione Italiana (the Italian public broadcasting company). We show how the reported results (evaluated in the light of the obtained reclassifications, the user ratings assigned to such reclassifications, and their explainability) are encouraging, and pave the way to many further research directions.

KNOSYS: 107166 Model 5G pp. 1–16 (col. fig: NIL) Knowledge-Based Systems xxx (xxxx) xxx Contents lists available at ScienceDirect Knowledge-Based Systems journal homepage: www.elsevier.com/locate/knosys A commonsense reasoning framework for explanatory emotion attribution, generation and re-classification Antonio Lieto a,b , , Gian Luca Pozzato a , Stefano Zoia a , Viviana Patti a , Rossana Damiano a ∗ a b University of Turin, Department of Computer Science, Turin, Italy ICAR-CNR, Palermo, Italy article info Article history: Received 25 December 2020 Received in revised form 14 May 2021 Accepted 21 May 2021 Available online xxxx Keywords: Explainable AI Commonsense reasoning Knowledge generation Concept combination Computational models of emotion 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 a b s t r a c t We present DEGARI (Dynamic Emotion Generator And ReclassIfier), an explainable system for emotion attribution and recommendation. This system relies on a recently introduced commonsense reasoning framework, the TCL logic, which is based on a human-like procedure for the automatic generation of novel concepts in a Description Logics knowledge base. Starting from an ontological formalization of emotions based on the Plutchik model, known as ArsEmotica, the system exploits the logic TCL to automatically generate novel commonsense semantic representations of compound emotions (e.g. Love as derived from the combination of Joy and Trust according to Plutchik). The generated emotions correspond to prototypes, i.e. commonsense representations of given concepts, and have been used to reclassify emotion-related contents in a variety of artistic domains, ranging from art datasets to the editorial contents available in RaiPlay, the online platform of RAI Radiotelevisione Italiana (the Italian public broadcasting company). We show how the reported results (evaluated in the light of the obtained reclassifications, the user ratings assigned to such reclassifications, and their explainability) are encouraging, and pave the way to many further research directions. © 2021 Elsevier B.V. All rights reserved. 1. Introduction and background Emotions have been acknowledged as a key part of the aesthetic experience through all ages and cultures, as witnessed by terms such as ‘‘sublime’’ [1] and ‘‘pathos’’ [2], associated with the experience of art since the ancient times. The advent of computational tools and methods for investigating the way we respond emotionally to objects and situations has paved the way to a deeper understanding of the intricate relationship between emotions and art. For example, Van Dongen et al. [3] have studied how art affects emotional regulation by measuring the brain response through EEG: their research shows that, in comparison with photographs depicting real events, artworks determine stronger electro-physiological responses; in parallel, Leder et al. [4] argue that the emotional response to art – measured through facial muscle movements – is attenuated in art critics, and stronger in non-experts, thus showing the universality and spontaneity of this response. The association between art and emotions is even stronger when the artistic expression is conveyed by non-textual media, as in music and, at least partly, in movies. For example, music has proven to be an effective tool for emotion regulation: as ∗ Corresponding author at: University of Turin, Department of Computer Science, Turin, Italy. E-mail address: antonio.lieto@unito.it (A. Lieto). demonstrated by Thoma et al. [5], music can induce specific emotional states in everyday situations, an effect which is sought for by the users and can be exploited to create effective affective recommender systems [6]. Finally, emotional engagement is of primary importance in narrative media, such as film and television, as extensively investigated by a line of research which draws from both film studies and emotion theories [7,8]. As a consequence of the multi-faceted, complex role played by emotions in the experience of art and media, the investigation of this phenomenon with computational tools has relied on a variety of models and methodologies, ranging from dimensional models, better suited to investigate physiological, continuous correlate of emotions [9–11], to categorical models, which lend themselves to inspecting the conscious level of emotional experience [12–14]. Dimensional models typically measure the emotional engagement along the arousal and hedonic axes, and are useful to study how the emotional response evolves over time. For example, Lopes et al. [15] rely on crowdsourced annotations of tension, arousal and variance in audio pieces to realize sound-based affective interaction in games. Categorical models are useful to collect the audience experience as discrete emotional labels, and are easily mapped to textual descriptions of emotions across languages. As exemplified by Mohammad and Kiritchenko [16], discrete emotional labels, merged from different categorical models [12, 17], can shed light on the reception of art, letting correlations emerge between attributed emotions, liking and subjects. https://doi.org/10.1016/j.knosys.2021.107166 0950-7051/© 2021 Elsevier B.V. All rights reserved. Please cite this article as: A. Lieto, G.L. Pozzato, S. Zoia et al., A commonsense reasoning framework for explanatory emotion attribution, generation and re-classification, Knowledge-Based Systems (2021) 107166, https://doi.org/10.1016/j.knosys.2021.107166. 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 Knowledge-Based Systems xxx (xxxx) xxx In many cases, the emotional response of the audience is conveyed through language, non only in textual media, but also in relation to art and other media: consider, for example, tags and social media comments concerning artworks and exhibitions. Automatically detecting affective states and emotions from text has gained considerable attention over recent years, leading to the development of several resources — such as annotated corpora, ontologies and lexicons within the Computational Linguistics community [18–20], also in response to the spread of supervised methods requiring a large amount of training data. Recently, deep neural network models have attracted increasing attention, and are being applied also to tasks related to the detection of affective states, obtaining promising results. Several neural architectures have been developed for a variety of tasks ranging from emotion detection [21], to dimensional sentiment analysis [22] and the most common sentiment polarity detection task, and have been evaluated against datasets on different types of social media texts, including long reviews, or, most of the time, short microblog messages, such as tweets. Interestingly, attention-based deep models turned out to be particularly effective, achieving state-of-the-art results on both long review and short tweet polarity classification. This is the case of the attention-based bidirectional CNN–RNN deep model for sentiment analysis in [23], showing that applying an attention layer to the outputs of the LSTM and GRU branches of the network makes the semantic representations more informative. Affective information expressed in texts is multi-faceted, and the wide variety of affective linguistic resources developed in the last years, mainly for English, but also for other languages, basically reflects such richness. When we speak about affective states in the context of natural language communication, we mean to refer to several aspects, which vary in their degree of stability, such as: emotion, sentiment, personality, mood, attitudes or interpersonal stance [24]. Given the wide variety of affective states, in recent years, research has focused on a finergrained investigation of the role of emotions, as well as on the importance of other affect dimensions such as sentiment and emotion intensity [25,26] or activation. On this line, recent efforts on predicting the degree of intensity for emotion and sentiment in different domains led to interesting experiments devoted to effectively combining deep learning and feature driven traditional models via an ensemble framework approach [26]. Depending on the specific research goals addressed, one could be interested in issuing a discrete label describing the affective state expressed (frustration, anger, joy, etc.) to address different contexts of interaction and tasks. Both basic emotion theories, in the Plutchik-Ekman [13] tradition, and dimensional models of emotions, have provided a precious theoretical grounding for the development of lexical resources [27–31] and computational models for emotion extraction. However, there is a general tendency to move towards richer, finer-grained models, possibly including complex emotions, especially in the context of datadriven and task-driven approaches, where restricting the automatic detection to a small set of basic emotions would fall short to achieve the objective. This is also our perspective. From a computational point of view, the choice of the model of affect to be used in order to give psychological grounding to the resource or the corpus to be developed is driven from, and highly intertwined with, the specific sentiment analysis task to be addressed, which, in turn, usually depends on the application domain to be tackled and on the final purpose of mining affective contents in texts. In this sense, evaluating the emotional responses of an audience in front of an artwork, with the purpose of monitoring the emotional impact of a cultural heritage exhibition on visitors [32], is different from monitoring political sentiment or mining the levels of anger in comments threads of software developers [33]. There are still few works and resources specifically developed to address emotion detection in the art and media domain. These include the work by Mohammad and Kiritchenko [16], where the authors describe the WikiArt Emotions Dataset, which includes emotion annotations for thousands of pieces of art from the WikiArt.org collection, and the work by Patti et al. [32,34] where the ArsEmotica framework1 is proposed, which relies on the combined use of NLP affect resources and an ontology of emotions to enable an emotion-driven exploration of online art collections. The diversity of computational models implied by the analysis of the emotional response to art and media, and of the applications that exploit this response to improve the user experience – from learning to entertainment – witnesses the complexity of the underlying processes (including, but not limited to, aesthetic, self-regulatory, social and cultural processes). This diversity, however, can be an obstacle to the development of models which work across domains and formats, preventing techniques from being transferred across similar tasks (e.g., emotion annotation and affective recommendation). In particular, the differences in emotion annotation between datasets can endanger the development and cross-validation of new techniques for analyzing and exploiting emotions in art and media. In this sense, techniques for merging and extending emotional categories can be useful to overcome these limitations. A notable example of such a comprehensive system is SenticNet [30], which relies on the Hourglass model [35]. The Hourglass model, recently revised and extended [36], is inspired by Plutchik’s model of emotions [37]. Such model, formalized in the ArsEmotica ontology and described in detail in Section 4, can be represented as a wheel of emotions, formed by: basic or primary emotions; opposite emotions; similarity between the emotions; compound emotions (or complex emotions) generated by the primary ones. Similarly to the SenticNet framework, our system also relies on the Plutchik model. The choice of this model is based on the fact that it provides a recipe for the generation of compound emotions that is compliant with the commonsense reasoning framework of the TCL logic. As such, we exploited the reasoning mechanisms of TCL to generate the compound emotions according to Plutchik’s theory. In this paper, we illustrate and validate this approach by means of the DEGARI system (Dynamic Emotion Generator And ReclassIfier) for emotion attribution and recommendation. In particular, we exploit the compound concepts generated by the system to automatically reclassify items in three datasets in the artistic and media domains. As a result of this reclassification process, an emotional enrichment is obtained and new emotional labels are associated with the items in the original datasets. To the best of our knowledge, DEGARI is the first emotion-oriented system employing a white box approach to emotion classification based on the human-like conceptual combination framework proposed in the TCL logic. DEGARI is available at http://di.unito.it/ DEGARI. The key contributions provided in this work are the following: 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 • an entirely explainable AI system for automatic emotion re- 120 121 122 123 124 125 126 127 128 129 classification and recommendation based on a well founded emotion theory (Plutchik model) and on a probabilistic logic framework modeling human-like for concept combination (i.e. the TCL logic); • the ontology of the Plutchik emotion model (made available as a free resource at http://130.192.212.225/fuseki/ ArsEmotica-core and queryable via a SPARQL endpoint at h ttp://130.192.212.225/fuseki/dataset.html?tab=query&ds=/A rsEmotica-core); 1 Available at http://130.192.212.225/fuseki/ArsEmotica-core. 2 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 4 5 • an empirical and replicable validation (i.e. data and software 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 The paper is organized as follows: after a brief overview of the rationale adopted by our commonsense reasoning framework (Section 2), we present in Section 3 – for the sake of self-containedness – a more detailed description of the TCL logic (by referring to [38] for a complete explanation). In Section 4 we present the ontological model ArsEmotica (enriched with an emotional lexicon) formalizing Plutchik’s theory of emotions and used as a standard representation to leverage the reasoning capabilities of the TCL within the system DEGARI. Sections 5 and 6 present the DEGARI system that, starting from the basic emotions represented in ArsEmotica (and according to Plutchik’s theory), generates compound emotions and uses these novel emotional categories for artistic content reclassification. Section 7 discusses how DEGARI can be considered as an explainable AI system. Finally, Section 8 shows the outcome of the automatic and explainable reclassification obtained with DEGARI (exploited in different settings and with different affective lexicons) and the results of a user study on 44 people showing the feasibility of using the obtained reclassifications as recommended emotion labels. Section 9 ends the paper. 26 27 2. Commonsense concept invention via dynamic knowledge combination 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 The overall rationale assumed in the TCL reasoning framework is that the process of automatic generation of novel concepts within a knowledge base (also known as knowledge invention process) can be obtained, as happens in humans [38,39], by exploiting a process of commonsense conceptual combination. This generative phenomenon highlights some crucial aspects of the knowledge capabilities in human cognition. Such ability, in fact, is associated to creative thinking and problem solving. Still, however, it represents an open challenge in the field of Artificial Intelligence (AI) [40]. Dealing with this problem, indeed, requires, from an AI perspective, the harmonization of two conflicting requirements that are hardly accommodated in symbolic systems [41]: the need for a syntactic and semantic compositionality (typical of logical systems) and the one concerning the exhibition of typicality effects. According to a well-known argument [42], in fact, prototypes (i.e. commonsense conceptual representations based on typical properties) are not compositional. The argument runs as follows: consider a concept like pet fish. It results from the composition of the concept pet and of the concept fish. However, the prototype of pet fish cannot result from the composition of the prototypes of a pet and a fish: e.g., a typical pet is furry and warm, a typical fish is grayish, but a typical pet fish is neither furry and warm nor grayish (typically, it is red). The pet fish phenomenon is a paradigmatic example of the difficulty to address when building formalisms and systems trying to imitate this combinatorial human ability. In this paper, we exploit the recently introduced nonmonotonic extension of Description Logics TCL (typicality-based compositional logic, introduced in [38]), which is able to account for this type of human-like concept combination.2 More specifically, we show how it can be used Knowledge-Based Systems xxx (xxxx) xxx are publicly available) of the proposed framework on three different datasets covering diverse artistic domains: multimedia, paintings and miscellaneous artistic items (poems, videos, pictures and music). 2 Other works have already shown how such logic can be used to model complex cognitive phenomena [38], goal-directed creative problem solving [43– 45] and to build intelligent applications for computational creativity [46]. Alternative approaches to the problem of commonsense conceptual combination are discussed in [47–49]. The advantages of TCL with respect to such approaches are detailed in [38]. 3 as a tool for the generation of novel compound emotions and, as a consequence, for the suggestion of novel emotion-related contents. In TCL , ‘‘typical’’ properties can be directly specified by means of a ‘‘typicality’’ operator T enriching the underlying Description Logic (from now on, DL for short), and a TBox can contain inclusions of the form T(C ) ⊑ D to represent that ‘‘typical C s are also Ds’’. As a difference with standard DLs, in the logic TCL one can consistently express exceptions and reason about defeasible inheritance as well. Typicality inclusions are also equipped by a real number p ∈ (0.5, 1] representing the probability/degree of belief in such a typical property: this allows us to define a semantics inspired to the DISPONTE semantics [50] characterizing probabilistic extensions of DLs, which in turn is used in order to describe different scenarios where only some typicality properties are considered. Given a KB containing the description of two concepts CH and CM occurring in it, we then consider only some scenarios in order to define a revised knowledge base, enriched by typical properties of the combined concept C ⊑ CH ⊓ CM by also implementing a heuristics coming from the cognitive semantics. By relying on TCL , this work introduces the system DEGARI which, first, automatically builds prototypes of existing compound emotions by extracting information about concepts or properties relying on the ArsEmotica ontology enriched with the NRC Emotion Intensity Lexicon [25] (this lexicon associates, in descending order of frequency, words to emotional concepts). In this setting, words with the highest frequencies of association to emotional concepts have been used as typical features of the basic emotions in the Plutchik model. Such prototypes of basic emotions have been formalized by means of a TCL knowledge base, whose TBox contains both rigid inclusions of the form 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 BasicEmotion ⊑ Concept, 89 in order to express essential desiderata but also constraints, as an example Joy ⊑ PositiveEmotion as well as prototypical properties of the form 90 91 92 p :: T(BasicEmotion) ⊑ TypicalConcept, 93 representing typical concepts of a given emotion, where p is a real number in the range (0.5, 1], expressing the frequency of such a concept in items belonging to that emotion: for instance, 0.72 :: T(Surprise) ⊑ Delight is used to express that the typical feature of being surprised contains/refers to the emotional concept Delight with a frequency/probability/degree of belief of the 72%. Given the ArsEmotica knowledge base (see Section 4) equipped with the prototypical descriptions of basic emotions, DEGARI exploits the reasoning capabilities of the logic TCL in order to generate new derived emotions as the result of the creative combination of two (or even more) basic or derived ones. DEGARI also reclassifies the artistic and multimedia contents taking the new, derived emotions into account. Intuitively, an item of the tested dataset belongs to the new generated emotion if its metadata (name, description, title) contain all the rigid properties as well as at least the 30% of the typical properties of such a derived emotion. In this respect, DEGARI can be seen as a ‘‘white box’’ recommender system, able to suggest to its users artistic contents belonging to new emotions by providing an explanation of such a recommendation. We have tested DEGARI by performing different kinds of evaluation that are reported and discussed in Section 8. In the following, in order to make the paper self-contained we recall in more detail the main features of the above described TCL logic. 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 3. The description logic TCL for concept combination The logic TCL [38] used by the system DEGARI as the basis for the generation of new compound emotions combines three main ingredients. The first one relies on the DL of typicality ALC + TR introduced in [51], which allows to describe the protoype of a concept. In this logic, ‘‘typical’’ properties can be directly specified by means of a ‘‘typicality’’ operator T enriching the underlying DL, and a TBox can contain inclusions of the form T(C ) ⊑ D to represent that ‘‘typical C s are also Ds’’. As a difference with standard DLs, in the logic ALC + TR one can consistently express exceptions and reason about defeasible inheritance as well. For instance, a knowledge base can consistently express that ‘‘normally, athletes are fit’’, whereas ‘‘sumo wrestlers usually are not fit’’ by T(Athlete) ⊑ Fit and T(SumoWrestler) ⊑ ¬Fit, given that SumoWrestler ⊑ Athlete. The semantics of the T operator is characterized by the properties of rational logic [52], recognized as the core properties of nonmonotonic reasoning. ALC + TR is characterized by a minimal model semantics corresponding to an extension to DLs of a notion of rational closure as defined in [52] for propositional logic: the idea is to adopt a preference relation over ALC + TR models, where intuitively a model is preferred to another one if it contains less exceptional elements, as well as a notion of minimal entailment restricted to models that are minimal with respect to such preference relation. As a consequence, T inherits well-established properties like specificity and irrelevance: in the example, the logic ALC + TR allows us to infer T(Athlete ⊓ Bald) ⊑ Fit (being bald is irrelevant with respect to being fit) and, if one knows that Hiroyuki is a typical sumo wrestler, to infer that he is not fit, giving preference to the most specific information. A second ingredient consists of a distributed semantics similar to the one of probabilistic DLs known as DISPONTE [53], which allows labeling inclusions T(C ) ⊑ D with a real number between 0.5 and 1, which represents its degree of belief/probability, under the assumption that each axiom is independent from each others. Degrees of belief in typicality inclusions allow defining a probability distribution over scenarios: roughly speaking, a scenario is obtained by choosing, for each typicality inclusion, whether it is considered as true or false. In a slight extension of the above example, we could have the need to represent that both the typicality inclusions about athletes and sumo wrestlers have a degree of belief of 80%, whereas we also believe that athletes are usually young with a higher degree of 95%, with the following KB: 44 (1) SumoWrestler ⊑ Athlete 45 (2) 0.8 :: T(Athlete) ⊑ Fit 46 (3) 0.8 :: T(SumoWrestler) ⊑ ¬Fit 47 (4) 0.95 :: T(Athlete) ⊑ YoungPerson 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 Knowledge-Based Systems xxx (xxxx) xxx given a KB and two concepts CH (HEAD) and CM (MODIFIER) occurring in it, we consider only some scenarios in order to define a revised knowledge base, enriched by typical properties of the combined concept C ⊑ CH ⊓ CM . Let us now present the logic TCL more precisely. The language of TCL extends the basic DL ALC by typicality inclusions of the form T(C ) ⊑ D equipped by a real number p ∈ (0.5, 1] – observe that the extreme 0.5 is not included – representing its degree of belief, whose meaning is that ‘‘we believe with degree/probability p that, normally, C s are also Ds’’3 64 65 66 67 68 69 70 71 72 73 Definition 3.1 (Language of TCL ). We consider an alphabet of concept names C, of role names R, and of individual constants O. Given A ∈ C and R ∈ R, we define: 74 75 76 C , D := A | ⊤ | ⊥ | ¬C | C ⊓ C | C ⊔ C | ∀R.C | ∃R.C We define a knowledge base K = ⟨R, T , A⟩ where: • R is a finite set of rigid properties of the form C ⊑ D; • T is a finite set of typicality properties of the form 81 where p ∈ (0.5, 1] ⊆ R is the degree of belief of the typicality inclusion; • A is the ABox, i.e. a finite set of formulas of the form either C (a) or R(a, b), where a, b ∈ O and R ∈ R. 82 83 84 85 A model M in the logic TCL extends standard ALC models by a preference relation among domain elements as in the logic of typicality [51]. In this respect, x < y means that x is ‘‘more normal’’ than y, and that the typical members of a concept C are the minimal elements of C with respect to this relation.4 An element x ∈ ∆I is a typical instance of some concept C if x ∈ C I and there is no C -element in ∆I more normal than x. Formally: 86 87 88 89 90 91 92 Definition 3.2 (Model of TCL ). A model M is any structure 93 where: 94 95 • ∆ is a non empty set of items called the domain; • < is an irreflexive, transitive, well-founded and modular (for all x, y, z in ∆I , if x < y then either x < z or z < y) relation over ∆I ; • .I is the extension function that maps each atomic concept C to C I ⊆ ∆I , and each role R to RI ⊆ ∆I × ∆I , and is I extended to complex concepts as follows: – – – – – – (¬C )I = ∆I \ C I (C ⊓ D)I = C I ∩ DI (C ⊔ D)I = C I ∪ DI (∃R.C )I = {x ∈ ∆I | ∃(x, y) ∈ RI such that y ∈ C I } (∀R.C )I = {x ∈ ∆I | ∀(x, y) ∈ RI we have y ∈ C I } (T(C ))I = Min< (C I ), where Min< (C I ) = {x ∈ C I | ∄y ∈ C I s.t. y < x}. 3 The reason why we only allow typicality inclusions equipped with probabilities p > 0.5 is due to our effort of integrating two different semantics: typicality based logic and DISPONTE. In particular, as detailed in [38] this choice seems to be the only one compliant with both formalisms. On the contrary, it would be misleading to also allow low degrees of belief for typicality inclusions, since typical knowledge is known to come with a low degree of uncertainty. 4 It could be possible to consider an alternative semantics whose models are equipped with multiple preference relations. However the approach based on a single preference relation in [51] ensures good computational properties (reasoning in the resulting nonmonotonic logic ALC + TR has the same complexity of the standard ALC ), whereas adopting multiple preference relations could lead to higher complexities. 4 78 79 80 p :: T(C ) ⊑ D ⟨∆I , <, .I ⟩ In this case, we consider eight different scenarios, representing all possible combinations of typicality inclusion: as an example, {((2), 1), ((3), 0), ((4), 1)} represents the scenario in which (2) and (4) hold, whereas (3) does not. Obviously, (1) holds in every scenario, since it represents a rigid property, not admitting exceptions. We equip each scenario with a probability depending on those of the involved inclusions: the scenario of the example has probability 0.8 × 0.95 (since 2 and 4 are involved) ×(1 − 0.8) (since 3 is not involved) = 0.152 = 15.2%. Such probabilities are then taken into account in order to choose the most adequate scenario describing the prototype of the combined concept. As a third element of the proposed formalization is a method inspired by cognitive semantics [54] for the identification of a dominance effect between the concepts to be combined: for every combination, we distinguish a HEAD, representing the stronger element of the combination, and a MODIFIER. The basic idea is: 77 96 97 98 99 100 101 102 103 104 105 106 107 108 109 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 4 5 6 7 A model M can be equivalently defined by postulating the existence of a function kM : ∆I ↦ −→ N, where kM assigns a finite rank to each domain element [51]: the rank of x is the length of the longest chain x0 < · · · < x from x to a minimal x0 , i.e. such that there is no x′ such that x′ < x0 . The rank function kM and < can be defined from each other by letting x < y if and only if kM (x) < kM (y). 8 9 10 11 Definition 3.3 (Model Satisfying a Knowledge Base in TCL ). Let K = ⟨R, T , A⟩ be a KB. Given a model M = ⟨∆I , <, .I ⟩, we assume that .I is extended to assign a domain element aI of ∆I to each individual constant a of O. We say that: 12 13 14 15 16 • M satisfies R if, for all C ⊑ D ∈ R, we have C I ⊆ DI ; • M satisfies T if, for all q :: T(C ) ⊑ D ∈ T , we have that5 T(C )I ⊆ DI , i.e. Min< (C I ) ⊆ DI ; • M satisfies A if, for each assertion F ∈ A, if F = C (a) then aI ∈ C I , otherwise if F = R(a, b) then (aI , bI ) ∈ RI . 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 Even if the typicality operator T itself is nonmonotonic (i.e. T(C ) ⊑ E does not imply T(C ⊓ D) ⊑ E), what is inferred from a KB can still be inferred from any KB’ with KB ⊆ KB’, i.e. the resulting logic is monotonic. As already mentioned, in order to perform useful nonmonotonic inferences, in [51] the authors have strengthened the above semantics by restricting entailment to a class of minimal models. Intuitively, the idea is to restrict entailment to models that minimize the atypical instances of a concept. The resulting logic corresponds to a notion of rational closure on top of ALC + TR . Such a notion is a natural extension of the rational closure construction provided in [52] for the propositional logic. This nonmonotonic semantics relies on minimal rational models that minimize the rank of domain elements. Informally, given two models of KB, one in which a given domain element x has rank 2 (because for instance z < y < x), and another in which it has rank 1 (because only y < x), we prefer the latter, as in this model the element x is assumed to be ‘‘more typical’’ than in the former. Query entailment is then restricted to minimal canonical models. The intuition is that a canonical model contains all the individuals that enjoy properties that are consistent with KB. This is needed when reasoning about the rank of the concepts: it is important to have them all represented. Given a KB K = ⟨R, T , A⟩ and given two concepts CH and CM occurring in K, the logic TCL allows defining a prototype of the combined concept C as the combination of the HEAD CH and the MODIFIER CM , where the typical properties of the form T(C ) ⊑ D (or, equivalently, T(CH ⊓ CM ) ⊑ D) to be ascribed to the concept C are obtained by considering blocks of scenarios with the same probability, in decreasing order starting from the highest one. We first discard all the inconsistent scenarios, then: 47 48 49 50 51 52 53 54 55 • we discard those scenarios considered as trivial, consistently Knowledge-Based Systems xxx (xxxx) xxx • among the remaining ones, we discard those inheriting inheriting all the properties from the HEAD from the starting concepts to be combined. This choice is motivated by the challenges provided by task of commonsense conceptual combination itself: in order to generate plausible and creative compounds, it is necessary to maintain a level of surprise in the combination. Thus both scenarios inheriting all the properties of the two concepts and all the properties of the HEAD are discarded, since they prevent this surprise; 5 It is worth noticing that here the degree q does not play any role. Indeed, a typicality inclusion T(C ) ⊑ D holds in a model only if it satisfies the semantic condition of the underlying DL of typicality, i.e. minimal (typical) elements of C are elements of D. The degree of belief q will have a crucial role in the application of the distributed semantics, allowing the definition of scenarios as well as the computation of their probabilities. 5 properties from the MODIFIER which are in conflict with properties that could be consistently inherited from the HEAD; • if the set of scenarios of the current block is empty, i.e. all the scenarios have been discarded either because trivial or because the MODIFIER is preferred, we repeat the procedure by considering the block of scenarios having the immediately lower probability. 56 57 58 59 60 61 62 63 64 Remaining scenarios are those selected by the logic TCL . The ultimate output of our mechanism is a knowledge base in the logic TCL whose set of typicality properties is enriched by those of the compound concept C . Given a scenario w satisfying the above properties, we define the properties of C as the set of inclusions p :: T(C ) ⊑ D, for all T(C ) ⊑ D that are entailed from w in the logic TCL . The probability p is such that: 65 66 67 68 69 70 71 • if T(CH ) ⊑ D is entailed from w, that is to say D is a property inherited either from the HEAD (or from both the HEAD and the MODIFIER), then p corresponds to the degree of belief of such inclusion of the HEAD in the initial knowledge base, i.e. p : T(CH ) ⊑ D ∈ T ; • otherwise, i.e. T(CM ) ⊑ D is entailed from w, then p corresponds to the degree of belief of such inclusion of a MODIFIER in the initial knowledge base, i.e. p : T(CM ) ⊑ D∈T. 72 73 74 75 76 77 78 79 80 The knowledge base obtained as the result of combining concepts CH and CM into the compound concept C is called C -revised knowledge base, and it is defined as follows: 81 82 83 KC = ⟨R, T ∪ {p : T(C ) ⊑ D}, A⟩, 84 for all D such that either T(CH ) ⊑ D is entailed in w or T(CM ) ⊑ D is entailed in w , and p is defined as above. As an example, consider the following version of the above mentioned Pet-Fish problem. Let KB contains the following inclusions: 85 86 87 88 89 Fish ⊑ LivesInWater (1) 90 0.6 :: T(Fish) ⊑ Greyish (2) 91 0.8 :: T(Fish) ⊑ Scaly (3) 92 0.8 :: T(Fish) ⊑ ¬Affectionate (4) 93 0.9 :: T(Pet) ⊑ ¬LivesInWater (5) 94 0.9 :: T(Pet) ⊑ LovedByKids (6) 95 0.9 :: T(Pet) ⊑ Affectionate (7) 96 representing that a typical fish is greyish (2), scaly (3) and not affectionate (4), whereas a typical pet does not live in water (5), is loved by kids (6) and is affectionate (7). Concerning rigid properties, we have that all fishes live in water (1). The logic TCL combines the concepts Pet and Fish, by using the latter as the HEAD and the former as the MODIFIER. The prototypical Pet-Fish inherits from the prototypical fish the fact that it is scaly and not affectionate, the last one by giving preference to the HEAD since such a property conflicts with the opposite one in the modifier (a typical pet is affectionate). The scenarios in which all the three typical properties of a typical fish are inherited by the combined concept are considered as trivial and, therefore, discarded, as a consequence the property having the lowest degree (Greyish with degree 0.6) is not inherited. The prototypical Pet-Fish inherits from the prototypical pet only property (6), since (5) conflicts with the rigid property (1), stating that all fishes (then, also pet fishes) live in water, whereas (7) is blocked, as already mentioned, by the HEAD/MODIFIER heuristics. Formally, the Pet ⊓ Fish-revised knowledge base contains, in addition to the above inclusions, the following ones: 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx 1 0.8 :: T(Pet ⊓ Fish) ⊑ Scaly (3’) 2 0.8 :: T(Pet ⊓ Fish) ⊑ ¬Affectionate (4’) 3 0.9 :: T(Pet ⊓ Fish) ⊑ LovedByKids (6’) 4 5 6 7 8 9 10 11 In [38] it has been also shown that reasoning in TCL remains in the same complexity class of standard ALC Description Logics. For the purposes of this paper, it is worth-noticing that, as mentioned, the TCL reasoning framework presented in this section has been applied, via the DEGARI system, to the generation of new compound emotions by starting from the affective ontological knowledge base named ArsEmotica. Such ontological model is described in the next section. 12 13 4. The ArsEmotica ontological model enriched with the NRC emotion intensity lexicon 14 15 16 17 18 19 20 21 22 23 24 The affective knowledge leveraged by the TCL logic via the DEGARI system is encoded in an ontology of emotional categories based on Plutchik’s psychological circumplex model [12], called ArsEmotica6 and includes also concepts from the Hourglass model [35]. The ontology structures emotional categories in a taxonomy, which currently includes 48 emotional concepts. The design of the taxonomic structure of emotional categories, of the disjunction axioms and of the object and data properties mirrors the main features of Plutchik’s circumplex model. As already mentioned, such model can be represented as a wheel of emotions (see Fig. 1) and encodes the following elements: 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 • Basic or primary emotions: Joy, Trust, Fear, Surprise, Sad- 45 46 47 48 49 50 51 52 53 54 We have chosen to encode the Plutchik model in the ontology for several reasons. First, it is well-grounded in psychology and general enough to guarantee a wide coverage of emotions. This is important for implementing successful strategies aimed at mapping tags to the emotional concepts of the ontology. In addition, it is easy to imagine that many different shades of emotions can be sought in the artistic domain. Literature on the psychology of art, indeed, suggests that the encoding of further complex emotions, such as Pride and Shame (belonging to the secondary and tertiary dyads, respectively, in the Plutchik model) • • • • could give further interesting results [55]. For this reason, we considered it appropriate to broaden the classification with respect to what has been done in [32,34], also including the compound emotions corresponding to the secondary and tertiary dyads of the Plutchik model. Second, the Plutchik wheel of emotions is perfectly compliant with the generative model underlying the TCL logic. Finally, it encodes interesting notions, e.g. emotional polar opposites, which can be exploited for finding novel, non obvious relations among artworks. Within the ArsEmotica ontology, the class Emotion is the root for all the emotional concepts. The Emotions hierarchy includes all the 48 emotional categories presented as distinguished labels in the model. In particular, the Emotion class has two disjoint subclasses: BasicEmotion and ComplexEmotion. The basic emotions of the Plutchik model are direct sub-classes of the BasicEmotion category. Each of them is specialized again into two subclasses representing the same emotion with weaker or stronger intensity (e.g. the basic emotion Joy has Ecstasy and Serenity as sub-classes). Therefore, we have 24 emotional concepts subsumed by the BasicEmotion concept. Instead, the class CompositeEmotion has 24 subclasses, corresponding to the primary (Love, Submission, Awe, Disapproval, Remorse, Contempt, Aggressiveness e Optimism), secondary (Hope, Guilt, Curiosity, Despair, Unbelief, Envy, Cynicism e Pride) and tertiary (Anxiety, Delight, Sentimentality, Shame, Outrage, Pessimism, Morbidness, Dominance) dyads. Other relations in the Plutchik model have been expressed in the ontology by means of object properties: the hasOpposite property encodes the notion of polar opposites; the hasSibling property encodes the notion of similarity and the isComposedOf property encodes the notion of composition of basic emotions. Moreover, a data type property hasScore was introduced to link each emotion with an intensity value i mapped into the above mentioned Hourglass model. The devised model allows attributing complex emotions from basic ones by exploiting simple SWRL rules (i.e. if-then clauses) allowing to infer, via the isComposedOf property connecting Basic and Composite Emotions, the fact that if an agent feels two emotions (suppose for a given item), and if these emotions jointly constitute a Composite Emotion, then the latter emotion will be automatically assigned to the agent in order to better describe his/her aesthetic experience. Due to the need of modeling the links between words in a language and the emotions they refer to, the ArsEmotica Ontology is also integrated with the ontology framework LExicon Model for ONtologies (LEMON) [56]. In particular, such integration allows differentiating explicitly between the language level (lexiconbased) and the conceptual one in representing the emotional concepts [34]. Within this enriched framework, it is possible to associate different emotional words, with the encoding of language information, to the corresponding emotional concepts. In this work, we have used the ArsEmotica model of emotional concepts with the NRC Emotion Intensity Lexicon mentioned above [25]. Such lexicon provides a list of English words, each with real-values representing intensity scores for the eight basic emotions of Plutchik’s theory. The lexicon includes close to 10,000 words including terms already known to be associated with emotions as well as terms that co-occur in Twitter posts that convey emotions. The intensity scores were obtained via crowdsourcing, using best-worst scaling annotation scheme. For our purposes, we considered the most frequent terms available in such lexicon (and associated to the basic emotions of the Plutchik wheel) as typical features of such emotions. In this way, once the prototypes of the basic emotional concepts were formed, the TCL reasoning framework was used to generate the compound emotions. ness, Disgust, Anger, Anticipation; in the color wheel, this is represented by differently colored sectors. Opposites: basic emotions can be conceptualized in terms of polar opposites: Joy versus Sadness, Anger versus Fear, Trust versus Disgust, Surprise versus Anticipation. Intensity: each emotion can exist in varying degrees of intensity; in the wheel, this is represented by the vertical dimension. Similarity: emotions vary in their degree of similarity to one another; in the wheel, this is represented by the radial dimension. Complex emotions: a complex emotion is a composition of two basic emotions; the pair of basic emotions involved in the composition is called a dyad. Looking at the Plutchik wheel, the eight emotions in the blank spaces are compositions of similar basic emotions, called primary dyads. Pairs of less similar emotions are called secondary dyads (if the radial distance between them is 2) or tertiary dyads (if the distance is 3), while opposites cannot be combined. 6 The ArsEmotica ontology is available here: http://130.192.212.225/fuseki/ ArsEmotica-core and queryable via SPARQL endpoint at: http://130.192.212.225/ fuseki/dataset.html?tab=query&ds=/ArsEmotica-core. 6 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx Fig. 1. The wheel of emotions of the Plutchik model. 1 and Anticipation. Typical properties are selected from the list of words characterizing each basic emotion in the NRC Emotion Intensity Lexicon where, as already mentioned, the probability p represents the intensity score for the emotion. In detail, for each basic emotion, we consider the six properties/words having the highest scores. As an example, consider the basic emotion Joy. The words having the highest scores are happiness (0.98), bliss (0.97), to celebrate (0.97), jubilant (0.97), ecstatic (0.95), and euphoria (0.94). Therefore, the knowledge base generated by DEGARI will contain, among others, the following inclusions: 5. DEGARI: Generating novel emotions from ArsEmotica 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 In this section, we describe DEGARI: the system exploiting the logic TCL on the ArsEmotica knowledge base in order to generate and suggest novel emotion-related contents and tested on the RaiPlay catalog,7 as well as on two artwork datasets: WikiArt Emotions8 and ArsMeteo [32,57]. DEGARI is implemented in Python and it makes use of the library owlready29 for relying on the services of efficient DL reasoners (like HermiT). DEGARI’s prototypes generation proceeds in two steps: in the first one, it builds a prototypical description of basic emotions in the language of the logic TCL , in order to describe their typical properties; as a second step, it exploits the above described reasoning mechanism of such a Description Logic in order to combine the prototypical descriptions of pairs of basic emotions, generating the prototypical description of compound emotions. As mentioned above, the obtained ontology is then tested by reclassifying the items belonging to RaiPlay, Wiki Art and ArsMeteo, keeping the generated compound emotions into account: this allows us to describe a novel and completely explainable recommending system, which is able to suggest items belonging also to compound emotions. Concerning the first step, DEGARI builds a knowledge base in the logic TCL characterized by typicality inclusions of the form 24 p :: T(BasicEmotion) ⊑ Property 25 26 where BasicEmotion is one of the eight basic emotions of the Plutchik model: Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger, 27 28 29 30 31 32 33 34 35 36 37 Joy ⊑ ¬Holocaust 0.98 :: T(Joy) ⊑ Happiness 0.97 :: T(Joy) ⊑ Bliss 0.97 :: T(Joy) ⊑ Celebrating 0.97 :: T(Joy) ⊑ Jubilant 0.95 :: T(Joy) ⊑ Ecstatic 0.94 :: T(Joy) ⊑ Elation DEGARI then computes novel compound emotions by combining existing ones (by using the same logical procedure of the pet-fish problem). As an example, let us consider the combination of the above basic emotion Joy with Fear, whose prototypical description is as follows: 7 https://www.raiplay.it. 8 http://saifmohammad.com/WebPages/wikiartemotions.html. 9 https://pythonhosted.org/Owlready2/. 0.96 :: T(Fear) ⊑ Kill 0.95 :: T(Fear) ⊑ Annihilate 7 38 39 40 41 42 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx 0.95 :: T(Fear) ⊑ Terror 0.98 :: T(Fear) ⊑ Torture 0.97 :: T(Fear) ⊑ Terrorist 0.97 :: T(Fear) ⊑ Horrific 1 2 3 4 5 6 7 8 9 10 11 12 13 In order to obtain a description of the compound emotion Guilt as the result of the combination of the two basic emotions (Joy ⊓ Fear) in the logic TCL , DEGARI combines the two basic emotions by implementing a variant of CoCoS [58], a Python implementation of reasoning services for the logic TCL in order to exploit efficient DLs reasoners for checking both the consistency of each generated scenario and the existence of conflicts among properties, following the line of the system DENOTER [59]. More in detail, DEGARI considers both the available choices for the HEAD and the MODIFIER, and it allows restricting its concern to a given and fixed number of inherited properties. The combined emotion Guilt has the following TCL description (concept Joy ⊓ Fear): 0.98 :: T(Joy ⊓ Fear) ⊑ Happiness 0.97 :: T(Joy ⊓ Fear) ⊑ Celebrating 0.97 :: T(Joy ⊓ Fear) ⊑ Bliss 0.98 :: T(Joy ⊓ Fear) ⊑ Torture 0.97 :: T(Joy ⊓ Fear) ⊑ Terrorist 0.97 :: T(Joy ⊓ Fear) ⊑ Horrific 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 Obviously, rigid properties of basic emotions (if any) are inherited by the compound emotion (in the example, Joy ⊓ Fear ⊑ ¬Holocaust), and this retain the system from considering any inconsistent typical properties even if they have the highest probability. It is worth noticing that the properties of the derived emotion are still expressed in the language of the logic TCL , therefore the combined emotion, Guilt in the example, can be further combined with another emotion, in order to iterate the procedure. As mentioned above, the DEGARI component devoted to compute the above described concept combination of prototypical descriptions relies on CoCoS [58], an implementation of a reasoning machinery for the logic TCL , generating scenarios and choosing the selected one(s) as described in Section 3. CoCoS is implemented in Python and exploits the translation of an ALC + TR knowledge base into standard ALC introduced in [51] and adopted by the system RAT-OWL [60]. CoCoS makes use of the above mentioned library owlready2, which allows relying on the services of efficient DL reasoners, e.g. the HermiT reasoner. CoCoS is embedded in DEGARI and allows one: (i) to include the logical descriptions of the concepts to be combined; (ii) to select which among the concepts has to be intended as HEAD and as MODIFIER(s); iii) to choose how many typical properties one wants to inherit in the scenarios that will be selected by TCL . In addition to presenting the selected scenario with typical properties of the combined concept, CoCoS also allows the users to select alternative scenarios, ranging from more trivial to more surprising ones. 42 43 6. Reclassification of emotion-related content based on DEGARI 44 45 46 47 By starting from the generated prototypes of the compound emotions in ArsEmotica, DEGARI is also able to perform an emotion-oriented reclassification of the items of the considered datasets. In particular, DEGARI employs two different strategies to extract metadata from the items to reclassify. In a first case (e.g., for the ArsMeteo and WikiArt datasets) the metadata are either stored in the provided resource (e.g., in WikiArt) or are the result of a social tagging activity based on the artistic community. In the second case (e.g., in the case of the RaiPlay dataset) the metadata associated to every and each item (title, name of the program/episode, description of the program/series, description of the episode) are extracted from a crawler. Such metadata are then used to generate the typical description of the items via the computation of the most frequent terms retrieved in their textual description (the assumption is that the most frequently used terms to describe an item are also the ones that are more typically associated to them). The frequencies are computed as the proportion of each property with respect to the set of all properties characterizing the item, in order to compare them with the properties of the derived emotion. If the item contains all the rigid properties and at least the 30% of the typical properties of the compound emotion under consideration, then the item is classified as belonging to it. Last, DEGARI suggests the set of classified contents, in a descending order of compatibility, where the rank of compatibility of a single item with respect to an emotion is intuitively obtained as the sum of the frequencies of ‘‘compatible’’ concepts, i.e. concepts belonging to both the item and the prototypical description of the genre. Formally: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 Definition 6.1. Given an item m, let DerivedEmotion be a compound emotion generated from the ArsEmotica mode as defined in Section 5 and let Sm be the set of words occurring in m. Given a knowledge base KB of compound emotions built by DEGARI, we say that m is compatible with DerivedEmotion if the following conditions hold: 73 74 75 76 77 78 1. m contains all rigid properties of DerivedEmotion, i.e. {C | DerivedEmotion ⊑ C ∈ KB} ⊆ Sm 2. m contains at least the 30% of typical properties of DerivedEmotion, i.e. 79 80 81 82 |Sm ∩SDerivedEmotion | |SDerivedEmotion | ≥ 0.3, 83 where SDerivedEmotion is the set of typical properties of DerivedEmotion. 84 85 As another example, consider the derived emotion Joy ⊓ Surprise, which in the Plutchik wheel corresponds to the combined emotion ‘‘delight’’. The knowledge base in the logic TCL describing such a compound emotion is as follows: 86 87 88 89 0.98 :: T(Joy ⊓ Surprise) ⊑ Happiness 0.97 :: T(Joy ⊓ Surprise) ⊑ Bliss 0.97 :: T(Joy ⊓ Surprise) ⊑ Celebrating 0.97 :: T(Joy ⊓ Surprise) ⊑ Jubilant 0.95 :: T(Joy ⊓ Surprise) ⊑ Estatic Joy ⊓ Surprise ⊑ ¬Holocaust Joy ⊓ Surprise ⊑ ¬Anticipation For instance, the multimedia item ‘‘È arrivata la felicità’’ (‘‘Happiness has arrived’’) (https://www.raiplay.it/programmi/earrivat alafelicita) from the RaiPlay dataset is reclassified in the novel, generated emotion Joy ⊓ Surprise, since: 90 91 92 93 • all rigid properties of both basic emotions are satisfied, that 94 95 96 is to say neither Holocaust nor Anticipation belong to the properties extracted for the item; 8 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 • more than the 30% of the typical properties10 of the com- 4 5 This item will be then recommended by DEGARI as shown in Fig. 2. 6 7. Explanation Knowledge-Based Systems xxx (xxxx) xxx pound emotion are satisfied; in particular, ‘‘È arrivata la felicità’’ has Happiness (0.98) and Surprise (0.93). 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Fig. 2 also shows how DEGARI can be considered as an explainable AI system: indeed, an explanation of the reasons why the multimedia item ‘‘È arrivata la felicità’’ (‘‘Happiness has arrived’’) has been reclassified in the compound concept is provided, in order to let the user be aware of the procedure of the system. As a difference with ‘‘black box’’ approaches, DEGARI explicitly reports that the ‘‘instance’s description has the following word(s) in common with category prototype’’, followed by the two matching properties Surprise and Happiness. The list of matching properties provided by DEGARI as a textual output is generated during the reclassification process. For each considered artwork, the properties of the artwork’s prototype’s are compared to the ones of the emotion’s prototype. The reclassification is based only on the matching properties (if present), therefore providing an explanation of the reclassification is as simple as storing the matches in an array and printing it as an output. Moreover, the whole procedure is completely transparent and could be used to further expand the feedback provided by system: from the axiom system (and the corresponding semantics of the Description Logic with typicality) to the DISPONTE semantics adopted by the logic TCL in order to compute the prototypical description of the compound emotion. 29 8. Evaluation 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 DEGARI evaluation has been carried out on three different datasets and evaluated in a fourfold way. The datasets considered (described in detail below) are: ArsMeteo, the RaiPlay catalog, and Wiki Art Emotion. The evaluation has concerned a first, completely automatic, test consisting in calculating the percentage of the reclassified items within the novel hybrid emotion classes generated by the system via TCL . In this case, the spread of the reclassified items along most of the concepts of the wheel of emotion has been considered as a positive indicator. This aspect, indeed, shows how the created prototypes of the compound emotions are mostly meaningful and able to reclassify the artistic content available in the three original datasets. A second evaluation has concerned the use of an ablation experiment testing the components of the TCL logic used by DEGARI in order to determine, if any, a difference in terms of the obtained output. A third evaluation has consisted in testing, within the pipeline provided by DEGARI, two different emotional lexicons: the above mentioned NRC lexicon and the lexicon of Shaver’s model of emotions [61]. The rationale of this experiment was to assess to what extent the combinatorial mechanisms used by DEGARI were dependent from the particular lexicons used. A fourth evaluation, finally, aimed at measuring the satisfaction of the potential users of the system when exposed to the contents of the novel categories suggested by DEGARI, consisted in a user study11 involving 44 subjects, who evaluated a total of 30 recommendations generated by the system. All the participants were recruited online 10 The 30% threshold was empirically determined: i.e., it is the percentage that provides the better trade-off between overcategorization and missed categorizations. 11 This is one of the most commonly used methodology for the evaluation of recommender systems based on controlled small groups analysis, see [62]. 9 using an availability sampling strategy. Participants were all naive to the experimental procedure and to the aims of the study. In the following, we briefly describe the adopted datasets: two of them are art-related ones (ArsMeteo and WikiArt Emotion), while the RaiPlay dataset contains all the multimedia items (e.g. movies, tv series, tv shows, documentaries etc.) available on the online multimedia platform or RAI, Radiotelevisione Italiana. 56 57 58 59 60 61 62 8.1. ArsMeteo dataset 63 ArsMeteo [57] is an art portal for sharing artworks and their emerging, connective meanings. Its development is led by a nonprofit cultural organization called Associazione Culturale ArsMeteo (AMA), based in Turin, Italy. It enables the collection of digital (or digitalized) artworks and performances, belonging to a variety of artistic forms including poems, videos, pictures and music. Meanings are given by the tagging activity of the community. All contents are accessible as ‘‘digital commons’’. Currently, the portal has collected over 350,000 visits and gathered a collection of 9171 artifacts produced by 307 artists; it has collected almost 38,000 tags. 64 65 66 67 68 69 70 71 72 73 74 8.2. RAIPLAY dataset 75 The RaiPlay dataset is composed of 4612 multimedia items extracted from RaiPlay https://www.raiplay.it/, the online platform of RAI, the national Italian broadcaster. Such dataset contains different types of multimedia content grouped in six main narrative categories: Movies, Fiction, Kids, TV Series, Drama, Comedy. As mentioned, each multimedia item/episode is equipped with metadata, including: title, name of the program/episode, description of the program/series, description of the episode. Such descriptions are used by DEGARI to extract the relevant information to associate to every item and to decide whether, given the extracted information, the item should be reclassified in one of the previously generated compound emotions. 76 77 78 79 80 81 82 83 84 85 86 87 8.3. WikiArt emotions 88 WikiArt Emotions is a dataset of 4105 artworks with annotations for the emotions evoked in the observer [16]. The artworks were selected from the online visual art encyclopedia WikiArt.org. Each piece of art is annotated for one or more of 20 emotion categories (including neutral). Annotations were obtained via crowdsourcing, asking annotators to include all the emotions evoked by the title of the artwork, the image of the artwork or the artwork as a whole. The annotators were also asked to specify if the artwork depicted a face or a human body (but not a face): this additional information is included in the dataset (if an artwork did not depict a face nor a body, it was marked as ‘‘none’’). In order to decide if an emotion applies or not to an artwork, the authors specified an aggregation threshold of 40%: if at least 40% of the responses indicated that a certain emotion applied, then the label was chosen. Other distributions on the dataset with different aggregation thresholds (30% and 50%) are available, but we chose to use the 40% threshold version, as recommended by the authors of the dataset [16]. 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 8.4. Automatic reclassification 107 The obtained results for what concerns the automatic evaluation are presented in Table 1. Overall, the figure shows that for two of the three datasets (ArsMeteo and RaiPlay) DEGARI is able to reclassify and spread the original items along the entire wheel of emotions assumed by the Plutchik model, thus allowing a more fine grained characterization. 108 109 110 111 112 113 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx Fig. 2. An example of the explanation provided by DEGARI about the reasons determining the reclassification of an item in the RaiPlay dataset. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 the adoption of such an experimental setting is not usual in symbolic approaches (since the causal connection between the output and the composing elements of a formalism or of an algorithms is – in such approaches – directly made explicit),13 the dissection imposed by such a setting allowed, nonetheless, a better understanding of what are the mandatory and the corollary elements of the formalism upon which our system is based on. In particular, as described in Section 3, TCL relies on three ingredients: a Description Logic of Tipicality ALC + TR , the DISPONTE semantics, and the Head/Modifier heuristics. The first two ingredients (i.e. the ALC + TR and the DISPONTE logics) are mandatory elements in order to obtain the commonsense conceptual combination proposed in TCL and therefore cannot be ‘‘ablated’’ (since no output would be provided in case of deleting either the typical component of the logic or the probabilistic one). On the other hand, the head/modifier heuristics is an additional element built on the top of the probabilistic commonsense framework and, as such, can be an element subject to an ablation study. We performed such a study and the results are available below. The Table 2 shows the evident advantage (in terms of reclassified items) of using this heuristics compared to the cases in which it is not used. In these two cases, the percentage of the reclassified items is of 5.49% and 1.81%, respectively. On the other hand, the WikiArt Emotions dataset contains orthogonal results since, in this case, 16 out of the 31 generated compound emotions are filled with reclassified items. In this case, however, a large part of the dataset (59.29%) is involved in such a reclassification. The main reason for these orthogonal results is in the kind of input considered by DEGARI. Indeed, while for ArsMeteo and RaiPlay the metadata associated to the items are either the result of a social tagging activity by a community of artists (like in ArsMeteo12 ) or the result of an information extraction pipeline (in RaiPlay); in WikiArt Emotions, all the metadata associated to the items are the result of a controlled crowdsourcing activity based on predefined emotion tags. While this fact, on one hand, creates – for the WikiArt Emotions dataset – cleaner metadata and allows the reclassifications of most of the items available in the dataset, on the other hand, it forces the user to use a predefined vocabulary for annotation that, as such, inhibits more free associations that could have led to a wider reclassification and redistribution of the items along the entire Plutchik wheel. In all the 3 cases, however, most of the compound emotions generated by DEGARI are filled with new items. In order to test the efficacy of the method employed by the DEGARI system (relying, as mentioned, on the TCL logic) we conducted an ablation experiment in order to determine the components of TCL shaping the above presented output. Although 13 This aspect represents an important difference with the currently in auge deep learning models that, on the other hand, suffer from the well known opacity issues and, as a consequence, require by default the adoption of such experimental technique in order to investigate which elements of such models are the relevant ones for producing and justifying a particular output. 12 The ArsMeteo dataset has the additional difficulty of being a heterogeneous dataset, touching different artistic genres (from poetry to literature, to paintings). 10 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx Table 1 Automatic evaluation overall data. Emotion ArsMeteo (9171 artworks) RaiPlay (4612 media items) Reclassified Reclassified Items aggressiveness (anger_anticipation) aggressiveness (anticipation_anger) anxiety (anticipation_fear) awe (fear_surprise) contempt (anger_disgust) curiosity (surprise_trust) cynicism (anticipation_disgust) delight (joy_surprise) despair (fear_sadness) disapproval (sadness_surprise) dominance (anger_trust) envy (anger_sadness) envy (sadness_anger) guilt (fear_joy) guilt (joy_fear) hope (anticipation_trust) hope (trust_anticipation) love (joy_trust) morbidness (disgust_joy) optimism (anticipation_joy) outrage (anger_surprise) pessimism (anticipation_sadness) pessimism (sadness_anticipation) pride (anger_joy) remorse (disgust_sadness) sentimentality (sadness_trust) sentimentality (trust_sadness) shame (disgust_fear) submission (fear_trust) unbelief (disgust_surprise) unbelief (surprise_disgust) OVERALLa a WikiArt emotions (4105 artworks) Reclassified Percentage Items Percentage Items Percentage 12 11 33 63 13 47 5 52 19 58 11 25 17 22 24 50 44 24 25 23 41 37 29 20 31 18 28 35 33 35 47 0.13% 0.12% 0.36% 0.69% 0.14% 0.51% 0.05% 0.57% 0.21% 0.63% 0.12% 0.27% 0.19% 0.24% 0.26% 0.55% 0.48% 0.26% 0.27% 0.25% 0.45% 0.40% 0.32% 0.22% 0.34% 0.20% 0.31% 0.38% 0.36% 0.38% 0.51% 7 2 88 132 2 49 0 117 20 64 2 21 20 73 73 51 52 72 70 24 46 20 20 72 20 20 20 88 88 44 47 0.15% 0.04% 1.91% 2.86% 0.04% 1.06% 0.00% 2.54% 0.43% 1.39% 0.04% 0.46% 0.43% 1.58% 1.58% 1.11% 1.13% 1.56% 1.52% 0.52% 1.00% 0.43% 0.43% 1.56% 0.43% 0.43% 0.43% 1.91% 1.91% 0.95% 1.02% 0 0 0 508 0 508 0 1672 0 404 0 0 0 1618 1618 661 661 1618 1618 1413 508 412 0 1618 0 0 0 0 0 508 508 0.00% 0.00% 0.00% 12.38% 0.00% 12.38% 0.00% 40.73% 0.00% 9.84% 0.00% 0.00% 0.00% 39.42% 39.42% 16.10% 16.10% 39.42% 39.42% 34.42% 12.38% 10.04% 0.00% 39.42% 0.00% 0.00% 0.00% 0.00% 0.00% 12.38% 12.38% 166 1.81% 235 5.49% 2434 59.29% This is NOT a sum: the overall count represents the total number of artworks classified for at least one emotion. typical features of each emotional concept, the most common words used to described them.14 Starting from a list of 135 words that human subjects rated as proper emotion words, Shaver and colleagues asked 100 participants to gather these words by similarity, then computed clusters of words that represent subordinate emotions categories. These clusters, detected based on co-occurrence relations, are characterized by heterogeneity and by the intrinsic fuzziness of the terms they contain: for example, one cluster contained, among others, related words such ‘‘arousal’’, ‘‘desire’’ and ‘‘lust’’. A higher level of clustering, roughly corresponding to the so called basic or primary emotions in the literature, was then detected: love, joy, surprise, anger, sadness, and fear were the resulting basic prototypical categories. The 6 basic emotion prototypes can be further identifiable as positive or negative emotions at a higher level and each prototype emotion subsumes a list of emotion words, obtained by merging the subordinate clusters they subsume. In this experiment, we have used the Shaver model of emotion as a lexical base for generating the prototypical emotional concepts with TCL (thus substituting, de facto, the NRC lexicon used in the previous experiment). As mentioned above, the Shaver model of emotion is particularly compliant with the overall assumptions of TCL since it provides the typical words associated to emotional concepts, collected from empirical data. It, however, differs from Pluthick’s model, since it does not provide any compositional procedure to generate compound emotions by design. As a consequence of this state of affairs, in order to test the generative Table 2 Ablation experiment of DEGARI showing the difference of considering (or not) the Head/Modifier (H/M) cognitive heuristics and its crucial effect on emotion reclassifications. Total (with H/M) Total (without H/M) Delta (n. of reclassifications) Delta (percentage of reclassifications) ArsMeteo WikiArt RaiPlay 607 212 −395 −65.07% 1428 570 −854 −59.97% 15 853 6531 −9322 −58.80% 1 2 3 4 In the Figs. 3, 4, 5, finally, are also reported, for each dataset, the generated compound emotions that have received more reclassifications (the horizontal histograms indicate the number of the reclassified items for each compound emotion). 5 6 7 8 9 10 11 12 13 14 15 16 17 18 8.4.1. Extended experiment with the Shaver lexicon In order to extend our experimental evaluation, we decided to test the well known lexicon provided by Shaver’s emotion model [61] (along with the Plutchik model and with the commonsense compositional mechanisms adopted by DEGARI over it) and compare the reclassifications/recommendations provided by our system with the first evaluation (using the NRC lexicon described above). Created with the goal of investigating the intuitions behind people’s conceptualization of emotions — as a way to reveal their nature, Shaver’s model has been created through a hierarchical clustering analysis of lexical data, having the Rosch’s theory of prototypes as Ref. [63]. In other words: the work by Shaver and colleagues investigates the role of prototypes in representing emotions in human conceptualization. It uses, as 14 We remind here that, according to Rosch’s theory [63], the prototypes are commonsense representations of a given concept. Here concepts are represented by means of typicality-based features, like in the TCL logic. 11 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx Fig. 3. Top reclassified content of Arsmeteo in the compound emotional classes generated by DEGARI. Fig. 4. Top reclassified content of RaiPlay in the compound emotional classes generated by DEGARI. 12 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx Fig. 5. Top reclassified content of WikiArt emotions in the compound emotional classes generated by DEGARI. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 capacity of TCL also in this setting, we re-used Plutchik’s model for determining the compositional rules of emotion combination. However, this time, the lexicon used was provided by the Shaver model (and not obtained from the NRC). Plutchik’s and Shaver’s emotional models present some differences. In particular: Shaver’s model considers only 6 emotional concepts (of which 5 are also considered as primary emotions in the Pluthcik model). As a consequence of this state of affairs, we employed the TCL mechanisms only on such 5 emotional concepts (anger, fear, joy, sadness and surprise). As the Shaver model provides many lexical terms for each emotion, we selected a subset of terms. To be consistent to what we did with the NRC Lexicon, we selected a maximum of 6 terms as typical properties. The probabilistic rating of the words selected form the Shaver’s model were obtained by the intensity ratings already provided in the NRC Lexicon. As a result, the typical properties selected to be part of an emotion prototype were the Shaver terms having a higher intensity in NRC. In Table 3 are the results of the different reclassifications. For the compound emotions generated starting from both Shaver’s lexicon and the NRC lexicon, it emerges that Shaver’s lexicon performs better on the ArsMeteo dataset, which is the most diversified one. For 4 out of 10 reclassifications with compound emotions, it also obtains better results on the RaiPlay dataset, while only in 3 cases obtains more reclassifications than the NRC dataset. This datum overall shows that there are minimal differences in the adoption of the different lexicons using the provided pipeline in terms of number of reclassifications and that, overall, the combinatorial mechanisms used by DEGARI for emotion reclassification are, for these two lexicons, relatively independent from the particular lexicon used. 13 8.5. User study 32 The goal of the user study was to assess the acceptance of the emotion categories suggested by DEGARI, with the ultimate goal of using the reclassifications produced by the system to improve the annotation of artworks and media, and, consequently, the applications which depend on it, such as personalization and recommendation. Methods and material. The user study consisted in an online questionnaire (in Italian). The questionnaire contained 10 items, each accompanied by an image, or, for the multimedia items, by the film poster, accompanied by the link to the online player for watching the content. For each item, the users received two questions: the first question (Question 1) asked them to rate the association of the item with the emotional category provided by DEGARI on 10-point scale; the second question (Question 2) asked them to associate the item to additional emotion categories, taken from the Plutchik model. Users were divided in 3 groups, each corresponding to a different set of 10 items. For each dataset, the selected items were the ones ranked higher by DEGARI for each generated compound emotional category. Participants and procedure. The study involved 44 users (23 females, 21 males). Concerning the age groups, 2 users were below 18; 17 users were in the 19–35 age range; 12 in the 36–50 range; 10 in the 51–70 range; 3 were older than 70. Users were randomly assigned to the questionnaires. The first questionnaire was filled out by 12 users; the second questionnaire was filled out by 20 users; the third questionnaire was assigned to 12 users. As a result, 440 ratings and 1065 emotion categories were collected. Results and analysis. Concerning Question 1, the average rating assigned by the users to the emotion category proposed by Degari was 6.32, with only slight differences between the datasets (see Table 4). The average rating was 6.47 for ArsMeteo, 6.42 for RaiPlay, and 6.12 for WikiArt. The standard deviation was 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. Knowledge-Based Systems xxx (xxxx) xxx Table 3 Reclassification results using Shaver lexicon, compared to the original NRC lexicon. Emotion awe (fear_surprise) delight (joy_surprise) delight (surprise_joy) despair (fear_sadness) disapproval (sadness_surprise) envy (anger_sadness) guilt (fear_joy) guilt (joy_fear) outrage (anger_surprise) pride (anger_joy) TOTAL ArsMeteo (9171 artworks) RaiPlay (4612 media items) WikiArt emotions (4105 artworks) NRC Shaver NRC Shaver NRC Shaver 63 52 52 19 58 25 22 24 41 20 117 71 57 49 83 30 44 45 49 27 132 117 117 20 64 21 73 73 46 72 7 2 88 132 2 49 0 117 20 64 508 1672 1672 0 404 0 1618 1618 508 1618 0 0 0 508 0 508 0 1672 0 404 324 572 618 481 7946 3092 emotion, compound emotional concepts starting from the basic ones. Such newly created categories, characterized by lexiconbased typical features, are then used in DEGARI to reclassify, in an emotional settings, the items of three different datasets. The novelty of this system relies on the fact that DEGARI is, to the best of our knowledge, the first emotion-oriented system employing a white box approach to emotion classification based on the human-like conceptual combination framework proposed in the TCL logic. The explainability requirement comes for free as a consequence of this logic-based approach, as shown in Fig. 2. Overall, the white box approach proposed by DEGARI for emotionally-driven content reclassification could be useful for addressing the very well known filter bubble effect [64] in recommender systems, by introducing seeds of serendipity in content discovery by users. One fundamental discussion about the applicability of DEGARI in practice is whether or not it represents a truly innovative technical solution for an emotion-based recommender system. According to Sohail et al. [65], recommender systems ‘‘try to identify the need and preferences of users, filter the huge collection of data accordingly and present the best suited option before the users by using some well-defined mechanism’’. Despite the huge amount of proposals, the main families of recommender systems can be identified as based on: (i) collaborative filtering; (ii) content-based filtering; (iii) hybrid filtering. At their core of functioning, collaborative filtering exploits similarities of usage patterns among mutually affine users, while content-based filtering exploits content similarity. DEGARI by definition falls into the latter category since in its current form it uses content description (obtained in different ways) as the input. From the technical point of view, however, it differs from the current mainstream approaches that are mostly based on the comparison and matching of visual and perceptual features of the content [66,67]. In practice, our approach adds a logic layer capable of mapping and representing – in a commonsense and cognitively compliant fashion [68] – new emotional categories which can be used to affect user preferences and content consumption in a way that cannot be derived from the pure statistical analysis of content and/or the comparison of similar users. Moreover, the proposed approach has been applied to a well-known model, the Plutchik circumplex model of emotions [12] and to two different emotional lexica (NRC and Shaver’s), but could in principle be applied to other models which organize emotions by similarity, opposition and composition, such as for example the extended version of the Hourglass model used in SenticNet [36]. Being independent from the specific application model and type of expression, this approach can work effectively in different domains, as shown by its use on the datasets of artworks and media illustrated in this paper. In this sense, it can promote the interoperability of affective annotations and the cross-domain reuse of techniques and methods. In the future work, we plan to extend the evaluation currently conducted in the form of a user study to a large scale Table 4 User ratings of the emotions proposed by DEGARI. Average rating Standard deviation Median ArsMeteo WikiArt RaiPlay All 6.47 2.48 7 6.12 2.47 7 6.42 2.1 7 6.32 2.41 7 Table 5 Overlapping of user tags with DEGARI emotions. User tags Proposed emotions Overlapping ArsMeteo WikiArt RaiPlay All 308 36 27.28% 308 50 20% 449 41 19.50% 1065 127 22.05% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 2.41 (2.48 for ArsMeteo, 2.47 for WikiArt, and 2.1 for RaiPlay), suggesting that the differences in ratings were limited. Also, the median rating is 7 for all data sets, with only 2 proposed emotion categories (1 from Arsmeteo and 1 from WikiArt) rated below 5. Concerning Question 2 (namely, the additional emotions attached by the users to the items), 308 were attached to the items in ArsMeteo, 308 to the items in WikiArt, and 449 to the items in RaiPlay, yielding 1065 user emotion categories. The average number of emotion categories per users was 24.2. In order to investigate the overlapping between the set of emotion categories proposed by DEGARI for each item (apart for the top ranked category tested through Question 1), we compared the emotion categories selected by the users with the ones proposed by DEGARI (Table 5). Data show that 22.05% of the emotion categories additionally proposed by DEGARI for each item matched those selected by the users, with a higher value for ArsMeteo (27.78%), and a lower value for WikiArt (20%) and RaiPlay (19.51%). This datum is a positive one since it concerns the non-top ranked emotional categories suggested by the system (for which the degree of acceptability by the users was always above 5 out of a 10-point scale except for 2 items, and with a median of 7 for every considered dataset). To conclude, the collected data suggest that the emotion categories proposed by DEGARI as a result of the reclassification process are generally accepted by the users, with few exceptions that deserve further investigation. The acceptance is clear for the top ranked emotion, but a satisfactory degree of acceptance can be inferred also for the remaining suggested categories, for which an overlapping of 20% and more with the user tags has been found in all datasets. 31 9. Discussion and conclusion 32 33 34 In this paper, we presented DEGARI: an explainable AI system relying on the TCL Description Logics and on the ArsEmotica knowledge base to generate, according to Plutchik’s theory of 14 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 Knowledge-Based Systems xxx (xxxx) xxx one to further validate the effectiveness of the proposed approach. We also plan to extend the applications of this system to different domains. A first extension will be in the field of the emotional-oriented recommendation of artworks within Museums and cultural heritage sites (this is a work currently under development within the H2020 European SPICE project15 ). In addition, also the field of music recommendation represents a current area of investigation. From a technical perspective, in future research, we aim at studying the application of optimization techniques in [69] in order to improve the efficiency of the DEGARI knowledge generation system. Secondly, we aim at considering more accurate and multimodal descriptions of artistic and media items, by exploiting Automatic Speech Recognition data and semantic visual categories extracted from video and audio channels of the content. Finally, as mentioned, we plan to improve the provided recommendations by justifying the content reclassification (and the derived recommendations) based on the probabilistic ranks assigned to the shared features between the generated emotion and the items being reclassified. [14] T. Bänziger, K.R. Scherer, Introducing the geneva multimodal emotion portrayal (gemep) corpus, in: Blueprint for Affective Computing: A Sourcebook, Vol. 2010, Oxford University Press, Oxford, UK, 2010, pp. 271–294. [15] P. Lopes, A. Liapis, G.N. Yannakakis, Modelling affect for horror soundscapes, IEEE Trans. Affect. Comput. 10 (2) (2017) 209–222. [16] S. Mohammad, S. Kiritchenko, Wikiart emotions: An annotated dataset of emotions evoked by art, in: N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan, 2018. [17] P. Noy, D. Noy-Sharav, Art and emotions, Int. J. Appl. Psychoanal. Stud. 10 (2) (2013) 100–107. [18] D. Jurafsky, J.H. Martin, Lexicons for sentiment, affect, and connotation, in: Speech and Language Processing: An Introduction To Natural Language Processing, Computational Linguistics, and Speech Recognition, 3nd Edition, in: Prentice Hall Series in artificial intelligence, Prentice Hall, Pearson Education International, Draft, 2020, Chapter 20. available online: https: //web.stanford.edu/~jurafsky/slp3/20.pdf. Accessed on December 2020. [19] M. Nissim, V. Patti, Chapter 3 - Semantic aspects in sentiment analysis, in: F.A. Pozzi, E. Fersini, E. Messina, B. Liu (Eds.), Sentiment Analysis in Social Networks, Morgan Kaufmann, Boston, 2017, pp. 31–48. [20] Z. Wang, S. Ho, E. Cambria, A review of emotion sensing: categorization models and algorithms, Multimedia Tools Appl. 79 (2020). Declaration of competing interest [21] A. Chatterjee, U. Gupta, M.K. Chinnakotla, R. Srikanth, M. Galley, P. Agrawal, Understanding emotions in text using deep learning and big data, Comput. Hum. Behav. 93 (2019) 309–317. The authors declare that they have no known competing finan[22] J. Wang, L.-C. Yu, K.R. Lai, X. Zhang, Dimensional sentiment analysis cial interests or personal relationships that could have appeared using a regional CNN-LSTM model, in: Proceedings of the 54th Annual to influence the work reported in this paper. Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Berlin, Germany, 2016, pp. 225–230. Acknowledgments [23] M.E. Basiri, S. Nemati, M. Abdar, E. Cambria, U.R. Acharya, ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis, The research leading to these results/this publication has been Future Gener. Comput. Syst. 115 (2021) 279–294. [24] S.M. Mohammad, Sentiment analysis: Detecting valence, emotions, and partially funded by the European Union’s Horizon 2020 research other affectual states from text, 2020, CoRR, abs/2005.11882. and innovation programme http://dx.doi.org/10.13039/501100007601 [25] S. Mohammad, Word affect intensities, in: N. Calzolari, K. Choukri, C. Cieri, under grant agreement SPICE 870811. The authors thank RAI for T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, the RaiPlay dataset and the Associazione Culturale ArsMeteo that A. Moreno, J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the provided the ArsMeteo dataset. Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018, European Language Resources Association (ELRA), 2018. References [26] M.S. Akhtar, A. Ekbal, E. Cambria, How intense are you? Predicting intensities of emotions and sentiments using stacked ensemble [application [1] D.A. Russell, et al., On the Sublime, Clarendon, 1964. notes], IEEE Comput. Intell. Mag. 15 (1) (2020) 64–75. [2] W. Ross, Ars Rhetorica. Aristotle, Clarendon Press, Oxford, 1959. [27] C. Strapparava, R. Mihalcea, Semeval-2007 task 14: Affective text, in: E. [3] N.N. Van Dongen, J.W. Van Strien, K. Dijkstra, Implicit emotion regulation Agirre, L.M. i Villodre, R. Wicentowski (Eds.), Proceedings of the 4th Interin the context of viewing artworks: ERP evidence in response to pleasant national Workshop on Semantic Evaluations, in: SemEval ’07, Association and unpleasant pictures, Brain Cogn. 107 (2016) 48–54. for Computational Linguistics, Stroudsburg, PA, USA, 2007, pp. 70–74. [4] H. Leder, G. Gerger, D. Brieber, N. Schwarz, What makes an art expert? [28] S.M. Mohammad, P.D. Turney, Crowdsourcing a word-emotion association Emotion and evaluation in art appreciation, Cogn. Emot. 28 (6) (2014) lexicon, Comput. Intell. 29 (3) (2013) 436–465. 1137–1147. [29] S. Mohammad, Obtaining reliable human ratings of valence, arousal, and [5] M.V. Thoma, S. Ryf, C. Mohiyeddini, U. Ehlert, U.M. Nater, Emotion dominance for 20,000 english words, in: Proceedings of the 56th Annual regulation through listening to music in everyday situations, Cogn. Emot. Meeting of the Association for Computational Linguistics (Volume 1: Long 26 (3) (2012) 550–560. Papers), Association for Computational Linguistics, Melbourne, Australia, [6] I. Andjelkovic, D. Parra, J. O’Donovan, Moodplay: Interactive music recom2018, pp. 174–184. mendation based on Artists’ mood similarity, Int. J. Hum.-Comput. Stud. [30] E. Cambria, Y. Li, F.Z. Xing, S. Poria, K. Kwok, SenticNet 6: Ensemble 121 (2019) 142–159. application of symbolic and subsymbolic AI for sentiment analysis, in: M. [7] G.M. Smith, Film Structure and the Emotion System, Cambridge University d’Aquin, S. Dietze, C. Hauff, E. Curry, P. Cudré-Mauroux (Eds.), CIKM ’20: Press, 2003. The 29th ACM International Conference on Information and Knowledge [8] E.S. Tan, Emotion and the Structure of Narrative Film: Film As an Emotion Management, Virtual Event, Ireland, October 19-23, 2020, ACM, 2020, pp. Machine, Routledge, 2013. 105–114. [9] J.A. Russell, A circumplex model of affect, J. Personal. Soc. Psychol. 39 (6) [31] C. Strapparava, A. Valitutti, Wordnet affect: an affective extension of (1980) 1161. wordnet, in: Proceedings of the Fourth International Conference on Lan[10] D. Watson, L.A. Clark, A. Tellegen, Development and validation of brief guage Resources and Evaluation (LREC’04), European Language Resources measures of positive and negative affect: the PANAS scales, J. Personal. Association (ELRA), Lisbon, Portugal, 2004. Soc. Psychol. 54 (6) (1988) 1063. [32] F. Bertola, V. Patti, Ontology-based affective models to organize artworks in the social semantic web, Inf. Process. Manage. 52 (1) (2016) 139–162. [11] A. Mehrabian, Pleasure-arousal-dominance: A general framework for de[33] D. Gachechiladze, F. Lanubile, N. Novielli, A. Serebrenik, Anger and its scribing and measuring individual differences in temperament, Curr. direction in collaborative software development, in: Proceedings of the Psychol. 14 (4) (1996) 261–292. 39th International Conference on Software Engineering: New Ideas and [12] R. Plutchik, A general psychoevolutionary theory of emotion, in: Theories Emerging Results Track, in: ICSE-NIER ’17, IEEE Press, Piscataway, NJ, USA, of Emotion, Elsevier, 1980, pp. 3–33. 2017, pp. 11–14. [13] P. Ekman, Basic emotions, in: M.J.P. Tim Dalgleish (Ed.), Handbook of [34] V. Patti, F. Bertola, A. Lieto, Arsemotica for arsmeteo. org: Emotion-driven Cognition and Emotion, Vol. 98, Wiley Online Library, 1999, pp. 45–60, exploration of online art collections, in: I. Russell, W. Eberle (Eds.), The chapter 3. Twenty-Eighth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2015), Association for the Advancement of Artificial 15 https://spice-h2020.eu/. Intelligence, AAAI Press, 2015, pp. 288–293. 15 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 KNOSYS: 107166 A. Lieto, G.L. Pozzato, S. Zoia et al. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 Knowledge-Based Systems xxx (xxxx) xxx [35] E. Cambria, A. Livingstone, A. Hussain, The hourglass of emotions, in: A. Esposito, A.M. Esposito, A. Vinciarelli, R. Hoffmann, V.C. Müller (Eds.), Cognitive Behavioural Systems - COST 2102 International Training School, Dresden, Germany, February 21-26, 2011, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 7403, Springer, 2012, pp. 144–157. [36] Y. Susanto, A.G. Livingstone, B.C. Ng, E. Cambria, The hourglass model revisited, IEEE Intell. Syst. 35 (5) (2020) 96–102. [37] R. Plutchik, The nature of emotions, Am. Sci. 89 (4) (2001) 344–350. [38] A. Lieto, G.L. Pozzato, A description logic framework for commonsense conceptual combination integrating typicality, probabilities and cognitive heuristics, J. Exp. Theor. Artif. Intell. 32 (5) (2020) 769–804. [39] A. Lieto, G.L. Pozzato, A description logic of typicality for conceptual combination, in: M. Ceci, N. Japkowicz, J. Liu, G.A. Papadopoulos, Z.W. Ras (Eds.), Foundations of Intelligent Systems - 24th International Symposium, ISMIS 2018, Limassol, Cyprus, October 29-31, 2018, Proceedings, in: Lecture Notes in Computer Science, vol. 11177, Springer, 2018, pp. 189–199. [40] M.A. Boden, Creativity and artificial intelligence, Artificial Intelligence 103 (1–2) (1998) 347–356. [41] M. Frixione, A. Lieto, Representing and reasoning on typicality in formal ontologies, in: C. Ghidini, A.N. Ngomo, S.N. Lindstaedt, T. Pellegrini (Eds.), Proceedings of the 7th International Conference on Semantic Systems, in: ACM International Conference Proceeding Series, ACM, 2011, pp. 119–125. [42] D.N. Osherson, E.E. Smith, On the adequacy of prototype theory as a theory of concepts, Cognition 9 (1) (1981) 35–58. [43] A. Lieto, G.L. Pozzato, F. Perrone, E. Chiodino, Knowledge capturing via conceptual reframing: A goal-oriented framework for knowledge invention, in: M. Kejriwal, P.A. Szekely, R. Troncy (Eds.), Proceedings of the 10th ACM Conference on Knowledge Capture, K-CAP 2019, Marina Del Rey, ACM, 2019, pp. 109–114. [44] A. Lieto, F. Perrone, G.L. Pozzato, E. Chiodino, Beyond subgoaling: A dynamic knowledge generation framework for creative problem solving in cognitive architectures, Cogn. Syst. Res. 58 (2019) 305–316. [45] E. Chiodino, A. Lieto, F. Perrone, G.L. Pozzato, A goal-oriented framework for knowledge invention and creative problem solving in cognitive architectures, in: G.D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A. Bugarín, J. Lang (Eds.), Proceedings of ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, 2020, in: Frontiers in Artificial Intelligence and Applications, vol. 325, IOS Press, 2020, pp. 2893–2894. [46] A. Lieto, G.L. Pozzato, Applying a description logic of typicality as a generative tool for concept combination in computational creativity, Intell. Artif. 13 (1) (2019) 93–106. [47] M. Eppe, E. Maclean, R. Confalonieri, O. Kutz, M. Schorlemmer, E. Plaza, K.-U. Kühnberger, A computational framework for conceptual blending, Artificial Intelligence 256 (2018) 105–129. [48] M. Lewis, J. Lawry, Hierarchical conceptual spaces for concept combination, Artificial Intelligence 237 (2016) 204–227. [49] R. Confalonieri, M. Schorlemmer, O. Kutz, R. Peñaloza, E. Plaza, M. Eppe, Conceptual blending in EL++ , in: Proceedings of the 29th International Workshop on Description Logics, Cape Town, South Africa, April 22-25, 2016, 2016. [50] F. Riguzzi, E. Bellodi, E. Lamma, R. Zese, Probabilistic description logics under the distribution semantics, Semant. Web 6 (5) (2015) 477–501. [51] L. Giordano, V. Gliozzi, N. Olivetti, G.L. Pozzato, Semantic characterization of rational closure: from propositional logic to description logics, Artificial Intelligence 226 (2015) 1–33. [52] D. Lehmann, M. Magidor, What does a conditional knowledge base entail? Artificial Intelligence 55 (1) (1992) 1–60. [53] F. Riguzzi, E. Bellodi, E. Lamma, R. Zese, Reasoning with probabilistic ontologies, in: Q. Yang, M. Wooldridge (Eds.), Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, AAAI Press, 2015, pp. 4310–4316. [54] J.A. Hampton, Inheritance of attributes in natural concept conjunctions, Mem. Cogn. 15 (1) (1987) 55–71. [55] P. Silvia, Looking past pleasure: Anger, confusion, disgust, pride, surprise, and other unusual aesthetic emotions, Psychol. Aesthet. Creat. Arts 3 (2009) 48–51. [56] J.P. McCrae, D. Spohr, P. Cimiano, Linking lexical resources and ontologies on the semantic web with lemon, in: G. Antoniou, M. Grobelnik, E.P.B. Simperl, B. Parsia, D. Plexousakis, P.D. Leenheer, J.Z. Pan (Eds.), The Semantic Web: Research and Applications - 8th Extended Semantic Web Conference, ESWC 2011, Heraklion, Crete, Greece, May 29-June 2, 2011, Proceedings, Part I, in: Lecture Notes in Computer Science, vol. 6643, Springer, 2011, pp. 245–259. [57] E. Acotto, M. Baldoni, C. Baroglio, V. Patti, F. Portis, G. Vaccarino, Arsmeteo: artworks and tags floating over the planet art, in: Proceedings of the 20th ACM Conference on Hypertext and Hypermedia, in: HT ’09, Association for Computing Machinery, New York, NY, USA, ISBN: 9781605584867, 2009, pp. 331–332, http://dx.doi.org/10.1145/1557914.1557973, https://doi.org/ 10.1145/1557914.1557973. [58] A. Lieto, G. Pozzato, A. Valese, COCOS: a typicality based COncept COmbination System, in: M. Montali and P. Felli (Eds.), Proceedings of the 33rd Italian Conference on Computational Logic (CILC 2018), CEUR Workshop Proceedings, Bozen, Italy, 2018, pp. 55–59. [59] E. Chiodino, D.D. Luccio, A. Lieto, A. Messina, G.L. Pozzato, D. Rubinetti, A knowledge-based system for the dynamic generation and classification of novel contents in multimedia broadcasting, in: G.D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A. Bugarín, J. Lang (Eds.), Proceedings of ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August8 September 2020, Santiago de Compostela, Spain, 2020, in: Frontiers in Artificial Intelligence and Applications, vol. 325, IOS Press, 2020, pp. 680–687. [60] L. Giordano, V. Gliozzi, G.L. Pozzato, R. Renzulli, An efficient reasoner for description logics of typicality and rational closure, in: A. Artale, B. Glimm, R. Kontchakov (Eds.), Proceedings of the 30th International Workshop on Description Logics, Montpellier, France, July 18-21, 2017, in: CEUR Workshop Proceedings, vol. 1879, CEUR-WS.org, 2017. [61] P. Shaver, J. Schwartz, D. Kirson, C. O’connor, Emotion knowledge: further exploration of a prototype approach, J. Personal. Soc. Psychol. 52 (6) (1987) 1061. [62] G. Shani, A. Gunawardana, Evaluating recommendation systems, in: Recommender Systems Handbook, Springer, 2011, pp. 257–297. [63] E. Rosch, Cognitive representations of semantic categories, J. Exp. Psychol. [Gen.] 104 (3) (1975) 192–233. [64] E. Parisier, The Filter Bubble: What the Internet is Hiding from You, Penguin Books Limited, 2012. [65] S.S. Sohail, J. Siddiqui, R. Ali, Classifications of recommender systems: A review, Eng. Sci. Technol. Rev. 10 (4) (2017) 132–153. [66] M.S. Sun, F. Li, J. Zhang, A multi-modality deep network for cold-start recommendation, Big Data Cogn. Comput. 2 (7) (2018). [67] Y. Deldjoo, M.G. Constantin, H. Eghbal-Zadeh, B. Ionescu, M. Schedl, P. Cremonesi, Audio-visual encoding of multimedia content for enhancing movie recommendations, in: S. Pera, M.D. Ekstrand, X. Amatriain, J. O’Donovan (Eds.), Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, ACM, 2018, pp. 455–459. [68] A. Lieto, Cognitive Design for Artificial Minds, Routledge, 2021. [69] M. Alberti, E. Bellodi, G. Cota, F. Riguzzi, R. Zese, Cplint on SWISH: Probabilistic logical inference with a web browser, Intell. Artif. 11 (1) (2017) 47–64. 16 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121

RELATED PAPERS

RELATED TOPICS

Log In

A Commonsense Reasoning Framework for Explanatory Emotion Attribution, Generation and Re-classification

A Commonsense Reasoning Framework for Explanatory Emotion Attribution, Generation and Re-classification

Related Papers

RELATED PAPERS

RELATED TOPICS