KNOSYS: 107166
Model 5G
pp. 1–16 (col. fig: NIL)
Knowledge-Based Systems xxx (xxxx) xxx
Contents lists available at ScienceDirect
Knowledge-Based Systems
journal homepage: www.elsevier.com/locate/knosys
A commonsense reasoning framework for explanatory emotion
attribution, generation and re-classification
Antonio Lieto a,b , , Gian Luca Pozzato a , Stefano Zoia a , Viviana Patti a , Rossana Damiano a
∗
a
b
University of Turin, Department of Computer Science, Turin, Italy
ICAR-CNR, Palermo, Italy
article
info
Article history:
Received 25 December 2020
Received in revised form 14 May 2021
Accepted 21 May 2021
Available online xxxx
Keywords:
Explainable AI
Commonsense reasoning
Knowledge generation
Concept combination
Computational models of emotion
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
a b s t r a c t
We present DEGARI (Dynamic Emotion Generator And ReclassIfier), an explainable system for emotion
attribution and recommendation. This system relies on a recently introduced commonsense reasoning
framework, the TCL logic, which is based on a human-like procedure for the automatic generation
of novel concepts in a Description Logics knowledge base. Starting from an ontological formalization
of emotions based on the Plutchik model, known as ArsEmotica, the system exploits the logic TCL
to automatically generate novel commonsense semantic representations of compound emotions (e.g.
Love as derived from the combination of Joy and Trust according to Plutchik). The generated emotions
correspond to prototypes, i.e. commonsense representations of given concepts, and have been used
to reclassify emotion-related contents in a variety of artistic domains, ranging from art datasets to
the editorial contents available in RaiPlay, the online platform of RAI Radiotelevisione Italiana (the
Italian public broadcasting company). We show how the reported results (evaluated in the light of the
obtained reclassifications, the user ratings assigned to such reclassifications, and their explainability)
are encouraging, and pave the way to many further research directions.
© 2021 Elsevier B.V. All rights reserved.
1. Introduction and background
Emotions have been acknowledged as a key part of the aesthetic experience through all ages and cultures, as witnessed by
terms such as ‘‘sublime’’ [1] and ‘‘pathos’’ [2], associated with the
experience of art since the ancient times. The advent of computational tools and methods for investigating the way we respond
emotionally to objects and situations has paved the way to a
deeper understanding of the intricate relationship between emotions and art. For example, Van Dongen et al. [3] have studied how
art affects emotional regulation by measuring the brain response
through EEG: their research shows that, in comparison with
photographs depicting real events, artworks determine stronger
electro-physiological responses; in parallel, Leder et al. [4] argue
that the emotional response to art – measured through facial
muscle movements – is attenuated in art critics, and stronger
in non-experts, thus showing the universality and spontaneity of
this response.
The association between art and emotions is even stronger
when the artistic expression is conveyed by non-textual media,
as in music and, at least partly, in movies. For example, music
has proven to be an effective tool for emotion regulation: as
∗ Corresponding author at: University of Turin, Department of Computer
Science, Turin, Italy.
E-mail address: antonio.lieto@unito.it (A. Lieto).
demonstrated by Thoma et al. [5], music can induce specific
emotional states in everyday situations, an effect which is sought
for by the users and can be exploited to create effective affective
recommender systems [6]. Finally, emotional engagement is of
primary importance in narrative media, such as film and television, as extensively investigated by a line of research which draws
from both film studies and emotion theories [7,8].
As a consequence of the multi-faceted, complex role played by
emotions in the experience of art and media, the investigation of
this phenomenon with computational tools has relied on a variety
of models and methodologies, ranging from dimensional models,
better suited to investigate physiological, continuous correlate of
emotions [9–11], to categorical models, which lend themselves to
inspecting the conscious level of emotional experience [12–14].
Dimensional models typically measure the emotional engagement along the arousal and hedonic axes, and are useful to
study how the emotional response evolves over time. For example, Lopes et al. [15] rely on crowdsourced annotations of tension,
arousal and variance in audio pieces to realize sound-based affective interaction in games. Categorical models are useful to collect
the audience experience as discrete emotional labels, and are easily mapped to textual descriptions of emotions across languages.
As exemplified by Mohammad and Kiritchenko [16], discrete
emotional labels, merged from different categorical models [12,
17], can shed light on the reception of art, letting correlations
emerge between attributed emotions, liking and subjects.
https://doi.org/10.1016/j.knosys.2021.107166
0950-7051/© 2021 Elsevier B.V. All rights reserved.
Please cite this article as: A. Lieto, G.L. Pozzato, S. Zoia et al., A commonsense reasoning framework for explanatory emotion attribution, generation and re-classification,
Knowledge-Based Systems (2021) 107166, https://doi.org/10.1016/j.knosys.2021.107166.
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
Knowledge-Based Systems xxx (xxxx) xxx
In many cases, the emotional response of the audience is
conveyed through language, non only in textual media, but also
in relation to art and other media: consider, for example, tags
and social media comments concerning artworks and exhibitions. Automatically detecting affective states and emotions from
text has gained considerable attention over recent years, leading
to the development of several resources — such as annotated
corpora, ontologies and lexicons within the Computational Linguistics community [18–20], also in response to the spread of
supervised methods requiring a large amount of training data.
Recently, deep neural network models have attracted increasing
attention, and are being applied also to tasks related to the
detection of affective states, obtaining promising results. Several
neural architectures have been developed for a variety of tasks
ranging from emotion detection [21], to dimensional sentiment
analysis [22] and the most common sentiment polarity detection
task, and have been evaluated against datasets on different types
of social media texts, including long reviews, or, most of the
time, short microblog messages, such as tweets. Interestingly,
attention-based deep models turned out to be particularly effective, achieving state-of-the-art results on both long review
and short tweet polarity classification. This is the case of the
attention-based bidirectional CNN–RNN deep model for sentiment analysis in [23], showing that applying an attention layer to
the outputs of the LSTM and GRU branches of the network makes
the semantic representations more informative.
Affective information expressed in texts is multi-faceted, and
the wide variety of affective linguistic resources developed in
the last years, mainly for English, but also for other languages,
basically reflects such richness. When we speak about affective
states in the context of natural language communication, we
mean to refer to several aspects, which vary in their degree of
stability, such as: emotion, sentiment, personality, mood, attitudes or interpersonal stance [24]. Given the wide variety of
affective states, in recent years, research has focused on a finergrained investigation of the role of emotions, as well as on the
importance of other affect dimensions such as sentiment and
emotion intensity [25,26] or activation. On this line, recent efforts
on predicting the degree of intensity for emotion and sentiment
in different domains led to interesting experiments devoted to
effectively combining deep learning and feature driven traditional
models via an ensemble framework approach [26].
Depending on the specific research goals addressed, one could
be interested in issuing a discrete label describing the affective
state expressed (frustration, anger, joy, etc.) to address different
contexts of interaction and tasks. Both basic emotion theories,
in the Plutchik-Ekman [13] tradition, and dimensional models
of emotions, have provided a precious theoretical grounding for
the development of lexical resources [27–31] and computational
models for emotion extraction. However, there is a general tendency to move towards richer, finer-grained models, possibly
including complex emotions, especially in the context of datadriven and task-driven approaches, where restricting the automatic detection to a small set of basic emotions would fall short
to achieve the objective. This is also our perspective.
From a computational point of view, the choice of the model
of affect to be used in order to give psychological grounding to
the resource or the corpus to be developed is driven from, and
highly intertwined with, the specific sentiment analysis task to
be addressed, which, in turn, usually depends on the application domain to be tackled and on the final purpose of mining
affective contents in texts. In this sense, evaluating the emotional
responses of an audience in front of an artwork, with the purpose of monitoring the emotional impact of a cultural heritage
exhibition on visitors [32], is different from monitoring political
sentiment or mining the levels of anger in comments threads of
software developers [33]. There are still few works and resources
specifically developed to address emotion detection in the art
and media domain. These include the work by Mohammad and
Kiritchenko [16], where the authors describe the WikiArt Emotions Dataset, which includes emotion annotations for thousands
of pieces of art from the WikiArt.org collection, and the work by
Patti et al. [32,34] where the ArsEmotica framework1 is proposed,
which relies on the combined use of NLP affect resources and an
ontology of emotions to enable an emotion-driven exploration of
online art collections.
The diversity of computational models implied by the analysis
of the emotional response to art and media, and of the applications that exploit this response to improve the user experience
– from learning to entertainment – witnesses the complexity of
the underlying processes (including, but not limited to, aesthetic,
self-regulatory, social and cultural processes). This diversity, however, can be an obstacle to the development of models which
work across domains and formats, preventing techniques from
being transferred across similar tasks (e.g., emotion annotation
and affective recommendation). In particular, the differences in
emotion annotation between datasets can endanger the development and cross-validation of new techniques for analyzing
and exploiting emotions in art and media. In this sense, techniques for merging and extending emotional categories can be
useful to overcome these limitations. A notable example of such
a comprehensive system is SenticNet [30], which relies on the
Hourglass model [35]. The Hourglass model, recently revised and
extended [36], is inspired by Plutchik’s model of emotions [37].
Such model, formalized in the ArsEmotica ontology and described
in detail in Section 4, can be represented as a wheel of emotions,
formed by: basic or primary emotions; opposite emotions; similarity between the emotions; compound emotions (or complex
emotions) generated by the primary ones. Similarly to the SenticNet framework, our system also relies on the Plutchik model. The
choice of this model is based on the fact that it provides a recipe
for the generation of compound emotions that is compliant with
the commonsense reasoning framework of the TCL logic. As such,
we exploited the reasoning mechanisms of TCL to generate the
compound emotions according to Plutchik’s theory.
In this paper, we illustrate and validate this approach by
means of the DEGARI system (Dynamic Emotion Generator And
ReclassIfier) for emotion attribution and recommendation. In particular, we exploit the compound concepts generated by the
system to automatically reclassify items in three datasets in the
artistic and media domains. As a result of this reclassification
process, an emotional enrichment is obtained and new emotional
labels are associated with the items in the original datasets. To
the best of our knowledge, DEGARI is the first emotion-oriented
system employing a white box approach to emotion classification based on the human-like conceptual combination framework
proposed in the TCL logic. DEGARI is available at http://di.unito.it/
DEGARI.
The key contributions provided in this work are the following:
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
• an entirely explainable AI system for automatic emotion re-
120
121
122
123
124
125
126
127
128
129
classification and recommendation based on a well founded
emotion theory (Plutchik model) and on a probabilistic logic
framework modeling human-like for concept combination
(i.e. the TCL logic);
• the ontology of the Plutchik emotion model (made available as a free resource at http://130.192.212.225/fuseki/
ArsEmotica-core and queryable via a SPARQL endpoint at h
ttp://130.192.212.225/fuseki/dataset.html?tab=query&ds=/A
rsEmotica-core);
1 Available at http://130.192.212.225/fuseki/ArsEmotica-core.
2
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
4
5
• an empirical and replicable validation (i.e. data and software
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
The paper is organized as follows: after a brief overview of
the rationale adopted by our commonsense reasoning framework (Section 2), we present in Section 3 – for the sake of
self-containedness – a more detailed description of the TCL logic
(by referring to [38] for a complete explanation). In Section 4
we present the ontological model ArsEmotica (enriched with
an emotional lexicon) formalizing Plutchik’s theory of emotions
and used as a standard representation to leverage the reasoning
capabilities of the TCL within the system DEGARI. Sections 5
and 6 present the DEGARI system that, starting from the basic
emotions represented in ArsEmotica (and according to Plutchik’s
theory), generates compound emotions and uses these novel
emotional categories for artistic content reclassification. Section 7
discusses how DEGARI can be considered as an explainable AI
system. Finally, Section 8 shows the outcome of the automatic
and explainable reclassification obtained with DEGARI (exploited
in different settings and with different affective lexicons) and
the results of a user study on 44 people showing the feasibility
of using the obtained reclassifications as recommended emotion
labels. Section 9 ends the paper.
26
27
2. Commonsense concept invention via dynamic knowledge
combination
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
The overall rationale assumed in the TCL reasoning framework
is that the process of automatic generation of novel concepts
within a knowledge base (also known as knowledge invention
process) can be obtained, as happens in humans [38,39], by
exploiting a process of commonsense conceptual combination.
This generative phenomenon highlights some crucial aspects of
the knowledge capabilities in human cognition. Such ability, in
fact, is associated to creative thinking and problem solving. Still,
however, it represents an open challenge in the field of Artificial
Intelligence (AI) [40]. Dealing with this problem, indeed, requires,
from an AI perspective, the harmonization of two conflicting
requirements that are hardly accommodated in symbolic systems [41]: the need for a syntactic and semantic compositionality
(typical of logical systems) and the one concerning the exhibition
of typicality effects. According to a well-known argument [42],
in fact, prototypes (i.e. commonsense conceptual representations
based on typical properties) are not compositional. The argument
runs as follows: consider a concept like pet fish. It results from the
composition of the concept pet and of the concept fish. However,
the prototype of pet fish cannot result from the composition of
the prototypes of a pet and a fish: e.g., a typical pet is furry
and warm, a typical fish is grayish, but a typical pet fish is
neither furry and warm nor grayish (typically, it is red). The pet
fish phenomenon is a paradigmatic example of the difficulty to
address when building formalisms and systems trying to imitate
this combinatorial human ability. In this paper, we exploit the
recently introduced nonmonotonic extension of Description Logics TCL (typicality-based compositional logic, introduced in [38]),
which is able to account for this type of human-like concept
combination.2 More specifically, we show how it can be used
Knowledge-Based Systems xxx (xxxx) xxx
are publicly available) of the proposed framework on three
different datasets covering diverse artistic domains: multimedia, paintings and miscellaneous artistic items (poems,
videos, pictures and music).
2 Other works have already shown how such logic can be used to model
complex cognitive phenomena [38], goal-directed creative problem solving [43–
45] and to build intelligent applications for computational creativity [46].
Alternative approaches to the problem of commonsense conceptual combination
are discussed in [47–49]. The advantages of TCL with respect to such approaches
are detailed in [38].
3
as a tool for the generation of novel compound emotions and,
as a consequence, for the suggestion of novel emotion-related
contents.
In TCL , ‘‘typical’’ properties can be directly specified by means
of a ‘‘typicality’’ operator T enriching the underlying Description
Logic (from now on, DL for short), and a TBox can contain inclusions of the form T(C ) ⊑ D to represent that ‘‘typical C s are
also Ds’’. As a difference with standard DLs, in the logic TCL one
can consistently express exceptions and reason about defeasible
inheritance as well. Typicality inclusions are also equipped by
a real number p ∈ (0.5, 1] representing the probability/degree
of belief in such a typical property: this allows us to define a
semantics inspired to the DISPONTE semantics [50] characterizing
probabilistic extensions of DLs, which in turn is used in order to
describe different scenarios where only some typicality properties
are considered. Given a KB containing the description of two
concepts CH and CM occurring in it, we then consider only some
scenarios in order to define a revised knowledge base, enriched
by typical properties of the combined concept C ⊑ CH ⊓ CM by also
implementing a heuristics coming from the cognitive semantics.
By relying on TCL , this work introduces the system DEGARI
which, first, automatically builds prototypes of existing compound
emotions by extracting information about concepts or properties
relying on the ArsEmotica ontology enriched with the NRC Emotion Intensity Lexicon [25] (this lexicon associates, in descending
order of frequency, words to emotional concepts). In this setting,
words with the highest frequencies of association to emotional
concepts have been used as typical features of the basic emotions
in the Plutchik model. Such prototypes of basic emotions have
been formalized by means of a TCL knowledge base, whose TBox
contains both rigid inclusions of the form
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
BasicEmotion ⊑ Concept,
89
in order to express essential desiderata but also constraints, as an
example Joy ⊑ PositiveEmotion as well as prototypical properties
of the form
90
91
92
p :: T(BasicEmotion) ⊑ TypicalConcept,
93
representing typical concepts of a given emotion, where p is a real
number in the range (0.5, 1], expressing the frequency of such a
concept in items belonging to that emotion: for instance, 0.72 ::
T(Surprise) ⊑ Delight is used to express that the typical feature of
being surprised contains/refers to the emotional concept Delight
with a frequency/probability/degree of belief of the 72%.
Given the ArsEmotica knowledge base (see Section 4) equipped
with the prototypical descriptions of basic emotions, DEGARI
exploits the reasoning capabilities of the logic TCL in order to
generate new derived emotions as the result of the creative combination of two (or even more) basic or derived ones. DEGARI also
reclassifies the artistic and multimedia contents taking the new,
derived emotions into account. Intuitively, an item of the tested
dataset belongs to the new generated emotion if its metadata
(name, description, title) contain all the rigid properties as well
as at least the 30% of the typical properties of such a derived
emotion. In this respect, DEGARI can be seen as a ‘‘white box’’
recommender system, able to suggest to its users artistic contents
belonging to new emotions by providing an explanation of such
a recommendation.
We have tested DEGARI by performing different kinds of evaluation that are reported and discussed in Section 8. In the following, in order to make the paper self-contained we recall in more
detail the main features of the above described TCL logic.
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
3. The description logic TCL for concept combination
The logic TCL [38] used by the system DEGARI as the basis for
the generation of new compound emotions combines three main
ingredients. The first one relies on the DL of typicality ALC + TR
introduced in [51], which allows to describe the protoype of a
concept. In this logic, ‘‘typical’’ properties can be directly specified
by means of a ‘‘typicality’’ operator T enriching the underlying
DL, and a TBox can contain inclusions of the form T(C ) ⊑ D
to represent that ‘‘typical C s are also Ds’’. As a difference with
standard DLs, in the logic ALC + TR one can consistently express
exceptions and reason about defeasible inheritance as well. For
instance, a knowledge base can consistently express that ‘‘normally, athletes are fit’’, whereas ‘‘sumo wrestlers usually are not
fit’’ by T(Athlete) ⊑ Fit and T(SumoWrestler) ⊑ ¬Fit, given
that SumoWrestler ⊑ Athlete. The semantics of the T operator is
characterized by the properties of rational logic [52], recognized
as the core properties of nonmonotonic reasoning. ALC + TR is
characterized by a minimal model semantics corresponding to
an extension to DLs of a notion of rational closure as defined
in [52] for propositional logic: the idea is to adopt a preference
relation over ALC + TR models, where intuitively a model is
preferred to another one if it contains less exceptional elements,
as well as a notion of minimal entailment restricted to models
that are minimal with respect to such preference relation. As a
consequence, T inherits well-established properties like specificity
and irrelevance: in the example, the logic ALC + TR allows us to
infer T(Athlete ⊓ Bald) ⊑ Fit (being bald is irrelevant with respect
to being fit) and, if one knows that Hiroyuki is a typical sumo
wrestler, to infer that he is not fit, giving preference to the most
specific information.
A second ingredient consists of a distributed semantics similar
to the one of probabilistic DLs known as DISPONTE [53], which
allows labeling inclusions T(C ) ⊑ D with a real number between
0.5 and 1, which represents its degree of belief/probability, under
the assumption that each axiom is independent from each others.
Degrees of belief in typicality inclusions allow defining a probability distribution over scenarios: roughly speaking, a scenario
is obtained by choosing, for each typicality inclusion, whether it
is considered as true or false. In a slight extension of the above
example, we could have the need to represent that both the
typicality inclusions about athletes and sumo wrestlers have a
degree of belief of 80%, whereas we also believe that athletes are
usually young with a higher degree of 95%, with the following KB:
44
(1) SumoWrestler ⊑ Athlete
45
(2) 0.8 :: T(Athlete) ⊑ Fit
46
(3) 0.8 :: T(SumoWrestler) ⊑ ¬Fit
47
(4) 0.95 :: T(Athlete) ⊑ YoungPerson
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Knowledge-Based Systems xxx (xxxx) xxx
given a KB and two concepts CH (HEAD) and CM (MODIFIER)
occurring in it, we consider only some scenarios in order to define
a revised knowledge base, enriched by typical properties of the
combined concept C ⊑ CH ⊓ CM .
Let us now present the logic TCL more precisely. The language
of TCL extends the basic DL ALC by typicality inclusions of the form
T(C ) ⊑ D equipped by a real number p ∈ (0.5, 1] – observe that
the extreme 0.5 is not included – representing its degree of belief,
whose meaning is that ‘‘we believe with degree/probability p that,
normally, C s are also Ds’’3
64
65
66
67
68
69
70
71
72
73
Definition 3.1 (Language of TCL ). We consider an alphabet of
concept names C, of role names R, and of individual constants O.
Given A ∈ C and R ∈ R, we define:
74
75
76
C , D := A | ⊤ | ⊥ | ¬C | C ⊓ C | C ⊔ C | ∀R.C | ∃R.C
We define a knowledge base K = ⟨R, T , A⟩ where:
• R is a finite set of rigid properties of the form C ⊑ D;
• T is a finite set of typicality properties of the form
81
where p ∈ (0.5, 1] ⊆ R is the degree of belief of the typicality
inclusion;
• A is the ABox, i.e. a finite set of formulas of the form either
C (a) or R(a, b), where a, b ∈ O and R ∈ R.
82
83
84
85
A model M in the logic TCL extends standard ALC models
by a preference relation among domain elements as in the logic
of typicality [51]. In this respect, x < y means that x is ‘‘more
normal’’ than y, and that the typical members of a concept C
are the minimal elements of C with respect to this relation.4 An
element x ∈ ∆I is a typical instance of some concept C if x ∈ C I
and there is no C -element in ∆I more normal than x. Formally:
86
87
88
89
90
91
92
Definition 3.2 (Model of TCL ). A model M is any structure
93
where:
94
95
• ∆ is a non empty set of items called the domain;
• < is an irreflexive, transitive, well-founded and modular (for
all x, y, z in ∆I , if x < y then either x < z or z < y) relation
over ∆I ;
• .I is the extension function that maps each atomic concept
C to C I ⊆ ∆I , and each role R to RI ⊆ ∆I × ∆I , and is
I
extended to complex concepts as follows:
–
–
–
–
–
–
(¬C )I = ∆I \ C I
(C ⊓ D)I = C I ∩ DI
(C ⊔ D)I = C I ∪ DI
(∃R.C )I = {x ∈ ∆I | ∃(x, y) ∈ RI such that y ∈ C I }
(∀R.C )I = {x ∈ ∆I | ∀(x, y) ∈ RI we have y ∈ C I }
(T(C ))I = Min< (C I ), where Min< (C I ) = {x ∈ C I |
∄y ∈ C I s.t. y < x}.
3 The reason why we only allow typicality inclusions equipped with probabilities p > 0.5 is due to our effort of integrating two different semantics: typicality
based logic and DISPONTE. In particular, as detailed in [38] this choice seems
to be the only one compliant with both formalisms. On the contrary, it would
be misleading to also allow low degrees of belief for typicality inclusions, since
typical knowledge is known to come with a low degree of uncertainty.
4 It could be possible to consider an alternative semantics whose models are
equipped with multiple preference relations. However the approach based on a
single preference relation in [51] ensures good computational properties (reasoning in the resulting nonmonotonic logic ALC + TR has the same complexity
of the standard ALC ), whereas adopting multiple preference relations could lead
to higher complexities.
4
78
79
80
p :: T(C ) ⊑ D
⟨∆I , <, .I ⟩
In this case, we consider eight different scenarios, representing
all possible combinations of typicality inclusion: as an example,
{((2), 1), ((3), 0), ((4), 1)} represents the scenario in which (2) and
(4) hold, whereas (3) does not. Obviously, (1) holds in every
scenario, since it represents a rigid property, not admitting exceptions. We equip each scenario with a probability depending
on those of the involved inclusions: the scenario of the example
has probability 0.8 × 0.95 (since 2 and 4 are involved) ×(1 − 0.8)
(since 3 is not involved) = 0.152 = 15.2%. Such probabilities are
then taken into account in order to choose the most adequate
scenario describing the prototype of the combined concept.
As a third element of the proposed formalization is a method
inspired by cognitive semantics [54] for the identification of a
dominance effect between the concepts to be combined: for every
combination, we distinguish a HEAD, representing the stronger
element of the combination, and a MODIFIER. The basic idea is:
77
96
97
98
99
100
101
102
103
104
105
106
107
108
109
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
4
5
6
7
A model M can be equivalently defined by postulating the
existence of a function kM : ∆I ↦ −→ N, where kM assigns a finite
rank to each domain element [51]: the rank of x is the length of
the longest chain x0 < · · · < x from x to a minimal x0 , i.e. such
that there is no x′ such that x′ < x0 . The rank function kM and
< can be defined from each other by letting x < y if and only if
kM (x) < kM (y).
8
9
10
11
Definition 3.3 (Model Satisfying a Knowledge Base in TCL ). Let K =
⟨R, T , A⟩ be a KB. Given a model M = ⟨∆I , <, .I ⟩, we assume
that .I is extended to assign a domain element aI of ∆I to each
individual constant a of O. We say that:
12
13
14
15
16
• M satisfies R if, for all C ⊑ D ∈ R, we have C I ⊆ DI ;
• M satisfies T if, for all q :: T(C ) ⊑ D ∈ T , we have that5
T(C )I ⊆ DI , i.e. Min< (C I ) ⊆ DI ;
• M satisfies A if, for each assertion F ∈ A, if F = C (a) then
aI ∈ C I , otherwise if F = R(a, b) then (aI , bI ) ∈ RI .
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Even if the typicality operator T itself is nonmonotonic
(i.e. T(C ) ⊑ E does not imply T(C ⊓ D) ⊑ E), what is inferred
from a KB can still be inferred from any KB’ with KB ⊆ KB’,
i.e. the resulting logic is monotonic. As already mentioned, in
order to perform useful nonmonotonic inferences, in [51] the
authors have strengthened the above semantics by restricting
entailment to a class of minimal models. Intuitively, the idea
is to restrict entailment to models that minimize the atypical
instances of a concept. The resulting logic corresponds to a notion
of rational closure on top of ALC + TR . Such a notion is a natural
extension of the rational closure construction provided in [52]
for the propositional logic. This nonmonotonic semantics relies
on minimal rational models that minimize the rank of domain elements. Informally, given two models of KB, one in which a given
domain element x has rank 2 (because for instance z < y < x),
and another in which it has rank 1 (because only y < x), we prefer
the latter, as in this model the element x is assumed to be ‘‘more
typical’’ than in the former. Query entailment is then restricted
to minimal canonical models. The intuition is that a canonical
model contains all the individuals that enjoy properties that are
consistent with KB. This is needed when reasoning about the rank
of the concepts: it is important to have them all represented.
Given a KB K = ⟨R, T , A⟩ and given two concepts CH and CM
occurring in K, the logic TCL allows defining a prototype of the
combined concept C as the combination of the HEAD CH and the
MODIFIER CM , where the typical properties of the form T(C ) ⊑ D
(or, equivalently, T(CH ⊓ CM ) ⊑ D) to be ascribed to the concept
C are obtained by considering blocks of scenarios with the same
probability, in decreasing order starting from the highest one. We
first discard all the inconsistent scenarios, then:
47
48
49
50
51
52
53
54
55
• we discard those scenarios considered as trivial, consistently
Knowledge-Based Systems xxx (xxxx) xxx
• among the remaining ones, we discard those inheriting
inheriting all the properties from the HEAD from the starting concepts to be combined. This choice is motivated by
the challenges provided by task of commonsense conceptual combination itself: in order to generate plausible and
creative compounds, it is necessary to maintain a level of
surprise in the combination. Thus both scenarios inheriting
all the properties of the two concepts and all the properties
of the HEAD are discarded, since they prevent this surprise;
5 It is worth noticing that here the degree q does not play any role. Indeed,
a typicality inclusion T(C ) ⊑ D holds in a model only if it satisfies the semantic
condition of the underlying DL of typicality, i.e. minimal (typical) elements of
C are elements of D. The degree of belief q will have a crucial role in the
application of the distributed semantics, allowing the definition of scenarios as
well as the computation of their probabilities.
5
properties from the MODIFIER which are in conflict with
properties that could be consistently inherited from the
HEAD;
• if the set of scenarios of the current block is empty, i.e. all
the scenarios have been discarded either because trivial
or because the MODIFIER is preferred, we repeat the procedure by considering the block of scenarios having the
immediately lower probability.
56
57
58
59
60
61
62
63
64
Remaining scenarios are those selected by the logic TCL . The
ultimate output of our mechanism is a knowledge base in the
logic TCL whose set of typicality properties is enriched by those of
the compound concept C . Given a scenario w satisfying the above
properties, we define the properties of C as the set of inclusions
p :: T(C ) ⊑ D, for all T(C ) ⊑ D that are entailed from w in the
logic TCL . The probability p is such that:
65
66
67
68
69
70
71
• if T(CH ) ⊑ D is entailed from w, that is to say D is a property
inherited either from the HEAD (or from both the HEAD and
the MODIFIER), then p corresponds to the degree of belief
of such inclusion of the HEAD in the initial knowledge base,
i.e. p : T(CH ) ⊑ D ∈ T ;
• otherwise, i.e. T(CM ) ⊑ D is entailed from w, then p corresponds to the degree of belief of such inclusion of a
MODIFIER in the initial knowledge base, i.e. p : T(CM ) ⊑
D∈T.
72
73
74
75
76
77
78
79
80
The knowledge base obtained as the result of combining concepts CH and CM into the compound concept C is called C -revised
knowledge base, and it is defined as follows:
81
82
83
KC = ⟨R, T ∪ {p : T(C ) ⊑ D}, A⟩,
84
for all D such that either T(CH ) ⊑ D is entailed in w or T(CM ) ⊑ D
is entailed in w , and p is defined as above.
As an example, consider the following version of the above
mentioned Pet-Fish problem. Let KB contains the following inclusions:
85
86
87
88
89
Fish ⊑ LivesInWater
(1)
90
0.6 :: T(Fish) ⊑ Greyish
(2)
91
0.8 :: T(Fish) ⊑ Scaly
(3)
92
0.8 :: T(Fish) ⊑ ¬Affectionate
(4)
93
0.9 :: T(Pet) ⊑ ¬LivesInWater
(5)
94
0.9 :: T(Pet) ⊑ LovedByKids
(6)
95
0.9 :: T(Pet) ⊑ Affectionate
(7)
96
representing that a typical fish is greyish (2), scaly (3) and not
affectionate (4), whereas a typical pet does not live in water
(5), is loved by kids (6) and is affectionate (7). Concerning rigid
properties, we have that all fishes live in water (1). The logic TCL
combines the concepts Pet and Fish, by using the latter as the
HEAD and the former as the MODIFIER. The prototypical Pet-Fish
inherits from the prototypical fish the fact that it is scaly and not
affectionate, the last one by giving preference to the HEAD since
such a property conflicts with the opposite one in the modifier
(a typical pet is affectionate). The scenarios in which all the three
typical properties of a typical fish are inherited by the combined
concept are considered as trivial and, therefore, discarded, as a
consequence the property having the lowest degree (Greyish with
degree 0.6) is not inherited. The prototypical Pet-Fish inherits
from the prototypical pet only property (6), since (5) conflicts
with the rigid property (1), stating that all fishes (then, also pet
fishes) live in water, whereas (7) is blocked, as already mentioned,
by the HEAD/MODIFIER heuristics. Formally, the Pet ⊓ Fish-revised
knowledge base contains, in addition to the above inclusions, the
following ones:
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
1
0.8 :: T(Pet ⊓ Fish) ⊑ Scaly
(3’)
2
0.8 :: T(Pet ⊓ Fish) ⊑ ¬Affectionate
(4’)
3
0.9 :: T(Pet ⊓ Fish) ⊑ LovedByKids
(6’)
4
5
6
7
8
9
10
11
In [38] it has been also shown that reasoning in TCL remains in
the same complexity class of standard ALC Description Logics.
For the purposes of this paper, it is worth-noticing that, as
mentioned, the TCL reasoning framework presented in this section
has been applied, via the DEGARI system, to the generation of
new compound emotions by starting from the affective ontological knowledge base named ArsEmotica. Such ontological model
is described in the next section.
12
13
4. The ArsEmotica ontological model enriched with the NRC
emotion intensity lexicon
14
15
16
17
18
19
20
21
22
23
24
The affective knowledge leveraged by the TCL logic via the
DEGARI system is encoded in an ontology of emotional categories based on Plutchik’s psychological circumplex model [12],
called ArsEmotica6 and includes also concepts from the Hourglass
model [35]. The ontology structures emotional categories in a
taxonomy, which currently includes 48 emotional concepts. The
design of the taxonomic structure of emotional categories, of
the disjunction axioms and of the object and data properties
mirrors the main features of Plutchik’s circumplex model. As
already mentioned, such model can be represented as a wheel
of emotions (see Fig. 1) and encodes the following elements:
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
• Basic or primary emotions: Joy, Trust, Fear, Surprise, Sad-
45
46
47
48
49
50
51
52
53
54
We have chosen to encode the Plutchik model in the ontology for several reasons. First, it is well-grounded in psychology
and general enough to guarantee a wide coverage of emotions.
This is important for implementing successful strategies aimed
at mapping tags to the emotional concepts of the ontology. In
addition, it is easy to imagine that many different shades of
emotions can be sought in the artistic domain. Literature on the
psychology of art, indeed, suggests that the encoding of further
complex emotions, such as Pride and Shame (belonging to the
secondary and tertiary dyads, respectively, in the Plutchik model)
•
•
•
•
could give further interesting results [55]. For this reason, we considered it appropriate to broaden the classification with respect
to what has been done in [32,34], also including the compound
emotions corresponding to the secondary and tertiary dyads of
the Plutchik model. Second, the Plutchik wheel of emotions is
perfectly compliant with the generative model underlying the TCL
logic. Finally, it encodes interesting notions, e.g. emotional polar
opposites, which can be exploited for finding novel, non obvious
relations among artworks.
Within the ArsEmotica ontology, the class Emotion is the root
for all the emotional concepts. The Emotions hierarchy includes
all the 48 emotional categories presented as distinguished labels
in the model. In particular, the Emotion class has two disjoint subclasses: BasicEmotion and ComplexEmotion. The basic emotions
of the Plutchik model are direct sub-classes of the BasicEmotion
category. Each of them is specialized again into two subclasses
representing the same emotion with weaker or stronger intensity
(e.g. the basic emotion Joy has Ecstasy and Serenity as sub-classes).
Therefore, we have 24 emotional concepts subsumed by the
BasicEmotion concept. Instead, the class CompositeEmotion has
24 subclasses, corresponding to the primary (Love, Submission,
Awe, Disapproval, Remorse, Contempt, Aggressiveness e Optimism),
secondary (Hope, Guilt, Curiosity, Despair, Unbelief, Envy, Cynicism
e Pride) and tertiary (Anxiety, Delight, Sentimentality, Shame, Outrage, Pessimism, Morbidness, Dominance) dyads. Other relations in
the Plutchik model have been expressed in the ontology by means
of object properties: the hasOpposite property encodes the notion
of polar opposites; the hasSibling property encodes the notion of
similarity and the isComposedOf property encodes the notion of
composition of basic emotions. Moreover, a data type property
hasScore was introduced to link each emotion with an intensity
value i mapped into the above mentioned Hourglass model.
The devised model allows attributing complex emotions from
basic ones by exploiting simple SWRL rules (i.e. if-then clauses)
allowing to infer, via the isComposedOf property connecting
Basic and Composite Emotions, the fact that if an agent feels two
emotions (suppose for a given item), and if these emotions jointly
constitute a Composite Emotion, then the latter emotion will be
automatically assigned to the agent in order to better describe
his/her aesthetic experience.
Due to the need of modeling the links between words in a
language and the emotions they refer to, the ArsEmotica Ontology
is also integrated with the ontology framework LExicon Model for
ONtologies (LEMON) [56]. In particular, such integration allows
differentiating explicitly between the language level (lexiconbased) and the conceptual one in representing the emotional
concepts [34]. Within this enriched framework, it is possible
to associate different emotional words, with the encoding of
language information, to the corresponding emotional concepts.
In this work, we have used the ArsEmotica model of emotional
concepts with the NRC Emotion Intensity Lexicon mentioned
above [25]. Such lexicon provides a list of English words, each
with real-values representing intensity scores for the eight basic emotions of Plutchik’s theory. The lexicon includes close to
10,000 words including terms already known to be associated
with emotions as well as terms that co-occur in Twitter posts
that convey emotions. The intensity scores were obtained via
crowdsourcing, using best-worst scaling annotation scheme. For
our purposes, we considered the most frequent terms available in
such lexicon (and associated to the basic emotions of the Plutchik
wheel) as typical features of such emotions. In this way, once
the prototypes of the basic emotional concepts were formed, the
TCL reasoning framework was used to generate the compound
emotions.
ness, Disgust, Anger, Anticipation; in the color wheel, this is
represented by differently colored sectors.
Opposites: basic emotions can be conceptualized in terms of
polar opposites: Joy versus Sadness, Anger versus Fear, Trust
versus Disgust, Surprise versus Anticipation.
Intensity: each emotion can exist in varying degrees of
intensity; in the wheel, this is represented by the vertical
dimension.
Similarity: emotions vary in their degree of similarity to
one another; in the wheel, this is represented by the radial
dimension.
Complex emotions: a complex emotion is a composition of
two basic emotions; the pair of basic emotions involved in
the composition is called a dyad. Looking at the Plutchik
wheel, the eight emotions in the blank spaces are compositions of similar basic emotions, called primary dyads. Pairs
of less similar emotions are called secondary dyads (if the
radial distance between them is 2) or tertiary dyads (if the
distance is 3), while opposites cannot be combined.
6 The ArsEmotica ontology is available here: http://130.192.212.225/fuseki/
ArsEmotica-core and queryable via SPARQL endpoint at: http://130.192.212.225/
fuseki/dataset.html?tab=query&ds=/ArsEmotica-core.
6
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
Fig. 1. The wheel of emotions of the Plutchik model.
1
and Anticipation. Typical properties are selected from the list
of words characterizing each basic emotion in the NRC Emotion
Intensity Lexicon where, as already mentioned, the probability p
represents the intensity score for the emotion. In detail, for each
basic emotion, we consider the six properties/words having the
highest scores.
As an example, consider the basic emotion Joy. The words
having the highest scores are happiness (0.98), bliss (0.97), to celebrate (0.97), jubilant (0.97), ecstatic (0.95), and euphoria (0.94).
Therefore, the knowledge base generated by DEGARI will contain,
among others, the following inclusions:
5. DEGARI: Generating novel emotions from ArsEmotica
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
In this section, we describe DEGARI: the system exploiting the
logic TCL on the ArsEmotica knowledge base in order to generate
and suggest novel emotion-related contents and tested on the
RaiPlay catalog,7 as well as on two artwork datasets: WikiArt
Emotions8 and ArsMeteo [32,57]. DEGARI is implemented in
Python and it makes use of the library owlready29 for relying
on the services of efficient DL reasoners (like HermiT).
DEGARI’s prototypes generation proceeds in two steps: in the
first one, it builds a prototypical description of basic emotions in
the language of the logic TCL , in order to describe their typical
properties; as a second step, it exploits the above described
reasoning mechanism of such a Description Logic in order to
combine the prototypical descriptions of pairs of basic emotions,
generating the prototypical description of compound emotions.
As mentioned above, the obtained ontology is then tested by reclassifying the items belonging to RaiPlay, Wiki Art and ArsMeteo,
keeping the generated compound emotions into account: this
allows us to describe a novel and completely explainable recommending system, which is able to suggest items belonging also to
compound emotions.
Concerning the first step, DEGARI builds a knowledge base in
the logic TCL characterized by typicality inclusions of the form
24
p :: T(BasicEmotion) ⊑ Property
25
26
where BasicEmotion is one of the eight basic emotions of the
Plutchik model: Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger,
27
28
29
30
31
32
33
34
35
36
37
Joy ⊑ ¬Holocaust
0.98 :: T(Joy) ⊑ Happiness
0.97 :: T(Joy) ⊑ Bliss
0.97 :: T(Joy) ⊑ Celebrating
0.97 :: T(Joy) ⊑ Jubilant
0.95 :: T(Joy) ⊑ Ecstatic
0.94 :: T(Joy) ⊑ Elation
DEGARI then computes novel compound emotions by combining existing ones (by using the same logical procedure of the
pet-fish problem). As an example, let us consider the combination
of the above basic emotion Joy with Fear, whose prototypical
description is as follows:
7 https://www.raiplay.it.
8 http://saifmohammad.com/WebPages/wikiartemotions.html.
9 https://pythonhosted.org/Owlready2/.
0.96 :: T(Fear) ⊑ Kill
0.95 :: T(Fear) ⊑ Annihilate
7
38
39
40
41
42
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
0.95 :: T(Fear) ⊑ Terror
0.98 :: T(Fear) ⊑ Torture
0.97 :: T(Fear) ⊑ Terrorist
0.97 :: T(Fear) ⊑ Horrific
1
2
3
4
5
6
7
8
9
10
11
12
13
In order to obtain a description of the compound emotion
Guilt as the result of the combination of the two basic emotions
(Joy ⊓ Fear) in the logic TCL , DEGARI combines the two basic
emotions by implementing a variant of CoCoS [58], a Python
implementation of reasoning services for the logic TCL in order to
exploit efficient DLs reasoners for checking both the consistency
of each generated scenario and the existence of conflicts among
properties, following the line of the system DENOTER [59]. More
in detail, DEGARI considers both the available choices for the
HEAD and the MODIFIER, and it allows restricting its concern to
a given and fixed number of inherited properties. The combined
emotion Guilt has the following TCL description (concept Joy ⊓
Fear):
0.98 :: T(Joy ⊓ Fear) ⊑ Happiness
0.97 :: T(Joy ⊓ Fear) ⊑ Celebrating
0.97 :: T(Joy ⊓ Fear) ⊑ Bliss
0.98 :: T(Joy ⊓ Fear) ⊑ Torture
0.97 :: T(Joy ⊓ Fear) ⊑ Terrorist
0.97 :: T(Joy ⊓ Fear) ⊑ Horrific
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
Obviously, rigid properties of basic emotions (if any) are inherited by the compound emotion (in the example, Joy ⊓ Fear ⊑
¬Holocaust), and this retain the system from considering any
inconsistent typical properties even if they have the highest probability.
It is worth noticing that the properties of the derived emotion
are still expressed in the language of the logic TCL , therefore the
combined emotion, Guilt in the example, can be further combined
with another emotion, in order to iterate the procedure.
As mentioned above, the DEGARI component devoted to compute the above described concept combination of prototypical
descriptions relies on CoCoS [58], an implementation of a reasoning machinery for the logic TCL , generating scenarios and
choosing the selected one(s) as described in Section 3. CoCoS is
implemented in Python and exploits the translation of an ALC +
TR knowledge base into standard ALC introduced in [51] and
adopted by the system RAT-OWL [60]. CoCoS makes use of the
above mentioned library owlready2, which allows relying on the
services of efficient DL reasoners, e.g. the HermiT reasoner.
CoCoS is embedded in DEGARI and allows one: (i) to include
the logical descriptions of the concepts to be combined; (ii) to
select which among the concepts has to be intended as HEAD
and as MODIFIER(s); iii) to choose how many typical properties
one wants to inherit in the scenarios that will be selected by
TCL . In addition to presenting the selected scenario with typical
properties of the combined concept, CoCoS also allows the users
to select alternative scenarios, ranging from more trivial to more
surprising ones.
42
43
6. Reclassification of emotion-related content based on DEGARI
44
45
46
47
By starting from the generated prototypes of the compound
emotions in ArsEmotica, DEGARI is also able to perform an
emotion-oriented reclassification of the items of the considered
datasets.
In particular, DEGARI employs two different strategies to extract metadata from the items to reclassify. In a first case (e.g., for
the ArsMeteo and WikiArt datasets) the metadata are either
stored in the provided resource (e.g., in WikiArt) or are the result
of a social tagging activity based on the artistic community. In
the second case (e.g., in the case of the RaiPlay dataset) the
metadata associated to every and each item (title, name of the
program/episode, description of the program/series, description
of the episode) are extracted from a crawler. Such metadata are
then used to generate the typical description of the items via
the computation of the most frequent terms retrieved in their
textual description (the assumption is that the most frequently
used terms to describe an item are also the ones that are more
typically associated to them). The frequencies are computed as
the proportion of each property with respect to the set of all
properties characterizing the item, in order to compare them with
the properties of the derived emotion. If the item contains all
the rigid properties and at least the 30% of the typical properties
of the compound emotion under consideration, then the item
is classified as belonging to it. Last, DEGARI suggests the set of
classified contents, in a descending order of compatibility, where
the rank of compatibility of a single item with respect to an
emotion is intuitively obtained as the sum of the frequencies of
‘‘compatible’’ concepts, i.e. concepts belonging to both the item
and the prototypical description of the genre. Formally:
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Definition 6.1. Given an item m, let DerivedEmotion be a compound emotion generated from the ArsEmotica mode as defined
in Section 5 and let Sm be the set of words occurring in m. Given
a knowledge base KB of compound emotions built by DEGARI,
we say that m is compatible with DerivedEmotion if the following
conditions hold:
73
74
75
76
77
78
1. m contains all rigid properties of DerivedEmotion, i.e. {C |
DerivedEmotion ⊑ C ∈ KB} ⊆ Sm
2. m contains at least the 30% of typical properties of
DerivedEmotion, i.e.
79
80
81
82
|Sm ∩SDerivedEmotion |
|SDerivedEmotion |
≥ 0.3,
83
where SDerivedEmotion is the set of typical properties of
DerivedEmotion.
84
85
As another example, consider the derived emotion Joy ⊓
Surprise, which in the Plutchik wheel corresponds to the combined emotion ‘‘delight’’. The knowledge base in the logic TCL
describing such a compound emotion is as follows:
86
87
88
89
0.98 :: T(Joy ⊓ Surprise) ⊑ Happiness
0.97 :: T(Joy ⊓ Surprise) ⊑ Bliss
0.97 :: T(Joy ⊓ Surprise) ⊑ Celebrating
0.97 :: T(Joy ⊓ Surprise) ⊑ Jubilant
0.95 :: T(Joy ⊓ Surprise) ⊑ Estatic
Joy ⊓ Surprise ⊑ ¬Holocaust
Joy ⊓ Surprise ⊑ ¬Anticipation
For instance, the multimedia item ‘‘È arrivata la felicità’’ (‘‘Happiness has arrived’’) (https://www.raiplay.it/programmi/earrivat
alafelicita) from the RaiPlay dataset is reclassified in the novel,
generated emotion Joy ⊓ Surprise, since:
90
91
92
93
• all rigid properties of both basic emotions are satisfied, that
94
95
96
is to say neither Holocaust nor Anticipation belong to the
properties extracted for the item;
8
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
• more than the 30% of the typical properties10 of the com-
4
5
This item will be then recommended by DEGARI as shown in
Fig. 2.
6
7. Explanation
Knowledge-Based Systems xxx (xxxx) xxx
pound emotion are satisfied; in particular, ‘‘È arrivata la
felicità’’ has Happiness (0.98) and Surprise (0.93).
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Fig. 2 also shows how DEGARI can be considered as an explainable AI system: indeed, an explanation of the reasons why the
multimedia item ‘‘È arrivata la felicità’’ (‘‘Happiness has arrived’’)
has been reclassified in the compound concept is provided, in
order to let the user be aware of the procedure of the system. As a
difference with ‘‘black box’’ approaches, DEGARI explicitly reports
that the ‘‘instance’s description has the following word(s) in
common with category prototype’’, followed by the two matching
properties Surprise and Happiness. The list of matching properties
provided by DEGARI as a textual output is generated during the
reclassification process. For each considered artwork, the properties of the artwork’s prototype’s are compared to the ones of
the emotion’s prototype. The reclassification is based only on the
matching properties (if present), therefore providing an explanation of the reclassification is as simple as storing the matches
in an array and printing it as an output. Moreover, the whole
procedure is completely transparent and could be used to further
expand the feedback provided by system: from the axiom system
(and the corresponding semantics of the Description Logic with
typicality) to the DISPONTE semantics adopted by the logic TCL in
order to compute the prototypical description of the compound
emotion.
29
8. Evaluation
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
DEGARI evaluation has been carried out on three different
datasets and evaluated in a fourfold way. The datasets considered
(described in detail below) are: ArsMeteo, the RaiPlay catalog,
and Wiki Art Emotion. The evaluation has concerned a first, completely automatic, test consisting in calculating the percentage
of the reclassified items within the novel hybrid emotion classes
generated by the system via TCL . In this case, the spread of the
reclassified items along most of the concepts of the wheel of
emotion has been considered as a positive indicator. This aspect,
indeed, shows how the created prototypes of the compound
emotions are mostly meaningful and able to reclassify the artistic
content available in the three original datasets. A second evaluation has concerned the use of an ablation experiment testing
the components of the TCL logic used by DEGARI in order to
determine, if any, a difference in terms of the obtained output.
A third evaluation has consisted in testing, within the pipeline
provided by DEGARI, two different emotional lexicons: the above
mentioned NRC lexicon and the lexicon of Shaver’s model of
emotions [61]. The rationale of this experiment was to assess to
what extent the combinatorial mechanisms used by DEGARI were
dependent from the particular lexicons used. A fourth evaluation,
finally, aimed at measuring the satisfaction of the potential users
of the system when exposed to the contents of the novel categories suggested by DEGARI, consisted in a user study11 involving
44 subjects, who evaluated a total of 30 recommendations generated by the system. All the participants were recruited online
10 The 30% threshold was empirically determined: i.e., it is the percentage that provides the better trade-off between overcategorization and missed
categorizations.
11 This is one of the most commonly used methodology for the evaluation of
recommender systems based on controlled small groups analysis, see [62].
9
using an availability sampling strategy. Participants were all naive
to the experimental procedure and to the aims of the study.
In the following, we briefly describe the adopted datasets:
two of them are art-related ones (ArsMeteo and WikiArt Emotion), while the RaiPlay dataset contains all the multimedia items
(e.g. movies, tv series, tv shows, documentaries etc.) available on
the online multimedia platform or RAI, Radiotelevisione Italiana.
56
57
58
59
60
61
62
8.1. ArsMeteo dataset
63
ArsMeteo [57] is an art portal for sharing artworks and their
emerging, connective meanings. Its development is led by a nonprofit cultural organization called Associazione Culturale ArsMeteo (AMA), based in Turin, Italy. It enables the collection of digital
(or digitalized) artworks and performances, belonging to a variety
of artistic forms including poems, videos, pictures and music.
Meanings are given by the tagging activity of the community.
All contents are accessible as ‘‘digital commons’’. Currently, the
portal has collected over 350,000 visits and gathered a collection
of 9171 artifacts produced by 307 artists; it has collected almost
38,000 tags.
64
65
66
67
68
69
70
71
72
73
74
8.2. RAIPLAY dataset
75
The RaiPlay dataset is composed of 4612 multimedia items
extracted from RaiPlay https://www.raiplay.it/, the online platform of RAI, the national Italian broadcaster. Such dataset contains different types of multimedia content grouped in six main
narrative categories: Movies, Fiction, Kids, TV Series, Drama, Comedy. As mentioned, each multimedia item/episode is equipped
with metadata, including: title, name of the program/episode,
description of the program/series, description of the episode.
Such descriptions are used by DEGARI to extract the relevant
information to associate to every item and to decide whether,
given the extracted information, the item should be reclassified
in one of the previously generated compound emotions.
76
77
78
79
80
81
82
83
84
85
86
87
8.3. WikiArt emotions
88
WikiArt Emotions is a dataset of 4105 artworks with annotations for the emotions evoked in the observer [16]. The artworks
were selected from the online visual art encyclopedia WikiArt.org.
Each piece of art is annotated for one or more of 20 emotion categories (including neutral). Annotations were obtained
via crowdsourcing, asking annotators to include all the emotions
evoked by the title of the artwork, the image of the artwork or the
artwork as a whole. The annotators were also asked to specify if
the artwork depicted a face or a human body (but not a face): this
additional information is included in the dataset (if an artwork
did not depict a face nor a body, it was marked as ‘‘none’’). In
order to decide if an emotion applies or not to an artwork, the
authors specified an aggregation threshold of 40%: if at least 40%
of the responses indicated that a certain emotion applied, then
the label was chosen. Other distributions on the dataset with
different aggregation thresholds (30% and 50%) are available, but
we chose to use the 40% threshold version, as recommended by
the authors of the dataset [16].
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
8.4. Automatic reclassification
107
The obtained results for what concerns the automatic evaluation are presented in Table 1. Overall, the figure shows that for
two of the three datasets (ArsMeteo and RaiPlay) DEGARI is able
to reclassify and spread the original items along the entire wheel
of emotions assumed by the Plutchik model, thus allowing a more
fine grained characterization.
108
109
110
111
112
113
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
Fig. 2. An example of the explanation provided by DEGARI about the reasons determining the reclassification of an item in the RaiPlay dataset.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
the adoption of such an experimental setting is not usual in
symbolic approaches (since the causal connection between the
output and the composing elements of a formalism or of an
algorithms is – in such approaches – directly made explicit),13
the dissection imposed by such a setting allowed, nonetheless, a
better understanding of what are the mandatory and the corollary
elements of the formalism upon which our system is based on. In
particular, as described in Section 3, TCL relies on three ingredients: a Description Logic of Tipicality ALC + TR , the DISPONTE
semantics, and the Head/Modifier heuristics. The first two ingredients (i.e. the ALC + TR and the DISPONTE logics) are mandatory
elements in order to obtain the commonsense conceptual combination proposed in TCL and therefore cannot be ‘‘ablated’’ (since
no output would be provided in case of deleting either the typical
component of the logic or the probabilistic one). On the other
hand, the head/modifier heuristics is an additional element built
on the top of the probabilistic commonsense framework and,
as such, can be an element subject to an ablation study. We
performed such a study and the results are available below. The
Table 2 shows the evident advantage (in terms of reclassified
items) of using this heuristics compared to the cases in which
it is not used.
In these two cases, the percentage of the reclassified items is
of 5.49% and 1.81%, respectively. On the other hand, the WikiArt
Emotions dataset contains orthogonal results since, in this case,
16 out of the 31 generated compound emotions are filled with
reclassified items. In this case, however, a large part of the dataset
(59.29%) is involved in such a reclassification.
The main reason for these orthogonal results is in the kind
of input considered by DEGARI. Indeed, while for ArsMeteo and
RaiPlay the metadata associated to the items are either the result
of a social tagging activity by a community of artists (like in
ArsMeteo12 ) or the result of an information extraction pipeline
(in RaiPlay); in WikiArt Emotions, all the metadata associated to
the items are the result of a controlled crowdsourcing activity
based on predefined emotion tags.
While this fact, on one hand, creates – for the WikiArt Emotions dataset – cleaner metadata and allows the reclassifications
of most of the items available in the dataset, on the other hand,
it forces the user to use a predefined vocabulary for annotation
that, as such, inhibits more free associations that could have led
to a wider reclassification and redistribution of the items along
the entire Plutchik wheel. In all the 3 cases, however, most of
the compound emotions generated by DEGARI are filled with new
items.
In order to test the efficacy of the method employed by the
DEGARI system (relying, as mentioned, on the TCL logic) we
conducted an ablation experiment in order to determine the
components of TCL shaping the above presented output. Although
13 This aspect represents an important difference with the currently in auge
deep learning models that, on the other hand, suffer from the well known
opacity issues and, as a consequence, require by default the adoption of such
experimental technique in order to investigate which elements of such models
are the relevant ones for producing and justifying a particular output.
12 The ArsMeteo dataset has the additional difficulty of being a heterogeneous
dataset, touching different artistic genres (from poetry to literature, to paintings).
10
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
Table 1
Automatic evaluation overall data.
Emotion
ArsMeteo (9171 artworks)
RaiPlay (4612 media items)
Reclassified
Reclassified
Items
aggressiveness (anger_anticipation)
aggressiveness (anticipation_anger)
anxiety (anticipation_fear)
awe (fear_surprise)
contempt (anger_disgust)
curiosity (surprise_trust)
cynicism (anticipation_disgust)
delight (joy_surprise)
despair (fear_sadness)
disapproval (sadness_surprise)
dominance (anger_trust)
envy (anger_sadness)
envy (sadness_anger)
guilt (fear_joy)
guilt (joy_fear)
hope (anticipation_trust)
hope (trust_anticipation)
love (joy_trust)
morbidness (disgust_joy)
optimism (anticipation_joy)
outrage (anger_surprise)
pessimism (anticipation_sadness)
pessimism (sadness_anticipation)
pride (anger_joy)
remorse (disgust_sadness)
sentimentality (sadness_trust)
sentimentality (trust_sadness)
shame (disgust_fear)
submission (fear_trust)
unbelief (disgust_surprise)
unbelief (surprise_disgust)
OVERALLa
a
WikiArt emotions (4105 artworks)
Reclassified
Percentage
Items
Percentage
Items
Percentage
12
11
33
63
13
47
5
52
19
58
11
25
17
22
24
50
44
24
25
23
41
37
29
20
31
18
28
35
33
35
47
0.13%
0.12%
0.36%
0.69%
0.14%
0.51%
0.05%
0.57%
0.21%
0.63%
0.12%
0.27%
0.19%
0.24%
0.26%
0.55%
0.48%
0.26%
0.27%
0.25%
0.45%
0.40%
0.32%
0.22%
0.34%
0.20%
0.31%
0.38%
0.36%
0.38%
0.51%
7
2
88
132
2
49
0
117
20
64
2
21
20
73
73
51
52
72
70
24
46
20
20
72
20
20
20
88
88
44
47
0.15%
0.04%
1.91%
2.86%
0.04%
1.06%
0.00%
2.54%
0.43%
1.39%
0.04%
0.46%
0.43%
1.58%
1.58%
1.11%
1.13%
1.56%
1.52%
0.52%
1.00%
0.43%
0.43%
1.56%
0.43%
0.43%
0.43%
1.91%
1.91%
0.95%
1.02%
0
0
0
508
0
508
0
1672
0
404
0
0
0
1618
1618
661
661
1618
1618
1413
508
412
0
1618
0
0
0
0
0
508
508
0.00%
0.00%
0.00%
12.38%
0.00%
12.38%
0.00%
40.73%
0.00%
9.84%
0.00%
0.00%
0.00%
39.42%
39.42%
16.10%
16.10%
39.42%
39.42%
34.42%
12.38%
10.04%
0.00%
39.42%
0.00%
0.00%
0.00%
0.00%
0.00%
12.38%
12.38%
166
1.81%
235
5.49%
2434
59.29%
This is NOT a sum: the overall count represents the total number of artworks classified for at least one emotion.
typical features of each emotional concept, the most common
words used to described them.14
Starting from a list of 135 words that human subjects rated as
proper emotion words, Shaver and colleagues asked 100 participants to gather these words by similarity, then computed clusters
of words that represent subordinate emotions categories. These
clusters, detected based on co-occurrence relations, are characterized by heterogeneity and by the intrinsic fuzziness of the
terms they contain: for example, one cluster contained, among
others, related words such ‘‘arousal’’, ‘‘desire’’ and ‘‘lust’’. A higher
level of clustering, roughly corresponding to the so called basic or
primary emotions in the literature, was then detected: love, joy,
surprise, anger, sadness, and fear were the resulting basic prototypical categories. The 6 basic emotion prototypes can be further
identifiable as positive or negative emotions at a higher level
and each prototype emotion subsumes a list of emotion words,
obtained by merging the subordinate clusters they subsume.
In this experiment, we have used the Shaver model of emotion
as a lexical base for generating the prototypical emotional concepts with TCL (thus substituting, de facto, the NRC lexicon used in
the previous experiment). As mentioned above, the Shaver model
of emotion is particularly compliant with the overall assumptions
of TCL since it provides the typical words associated to emotional
concepts, collected from empirical data. It, however, differs from
Pluthick’s model, since it does not provide any compositional
procedure to generate compound emotions by design. As a consequence of this state of affairs, in order to test the generative
Table 2
Ablation experiment of DEGARI showing the difference of considering (or not)
the Head/Modifier (H/M) cognitive heuristics and its crucial effect on emotion
reclassifications.
Total (with H/M)
Total (without H/M)
Delta (n. of reclassifications)
Delta (percentage of reclassifications)
ArsMeteo
WikiArt
RaiPlay
607
212
−395
−65.07%
1428
570
−854
−59.97%
15 853
6531
−9322
−58.80%
1
2
3
4
In the Figs. 3, 4, 5, finally, are also reported, for each dataset,
the generated compound emotions that have received more reclassifications (the horizontal histograms indicate the number of
the reclassified items for each compound emotion).
5
6
7
8
9
10
11
12
13
14
15
16
17
18
8.4.1. Extended experiment with the Shaver lexicon
In order to extend our experimental evaluation, we decided
to test the well known lexicon provided by Shaver’s emotion
model [61] (along with the Plutchik model and with the commonsense compositional mechanisms adopted by DEGARI over
it) and compare the reclassifications/recommendations provided
by our system with the first evaluation (using the NRC lexicon described above). Created with the goal of investigating the
intuitions behind people’s conceptualization of emotions — as
a way to reveal their nature, Shaver’s model has been created
through a hierarchical clustering analysis of lexical data, having
the Rosch’s theory of prototypes as Ref. [63]. In other words: the
work by Shaver and colleagues investigates the role of prototypes
in representing emotions in human conceptualization. It uses, as
14 We remind here that, according to Rosch’s theory [63], the prototypes are
commonsense representations of a given concept. Here concepts are represented
by means of typicality-based features, like in the TCL logic.
11
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
Fig. 3. Top reclassified content of Arsmeteo in the compound emotional classes generated by DEGARI.
Fig. 4. Top reclassified content of RaiPlay in the compound emotional classes generated by DEGARI.
12
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
Fig. 5. Top reclassified content of WikiArt emotions in the compound emotional classes generated by DEGARI.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
capacity of TCL also in this setting, we re-used Plutchik’s model
for determining the compositional rules of emotion combination.
However, this time, the lexicon used was provided by the Shaver
model (and not obtained from the NRC).
Plutchik’s and Shaver’s emotional models present some differences. In particular: Shaver’s model considers only 6 emotional
concepts (of which 5 are also considered as primary emotions
in the Pluthcik model). As a consequence of this state of affairs,
we employed the TCL mechanisms only on such 5 emotional
concepts (anger, fear, joy, sadness and surprise). As the Shaver
model provides many lexical terms for each emotion, we selected
a subset of terms. To be consistent to what we did with the NRC
Lexicon, we selected a maximum of 6 terms as typical properties.
The probabilistic rating of the words selected form the Shaver’s
model were obtained by the intensity ratings already provided
in the NRC Lexicon. As a result, the typical properties selected to
be part of an emotion prototype were the Shaver terms having a
higher intensity in NRC.
In Table 3 are the results of the different reclassifications. For
the compound emotions generated starting from both Shaver’s
lexicon and the NRC lexicon, it emerges that Shaver’s lexicon
performs better on the ArsMeteo dataset, which is the most
diversified one. For 4 out of 10 reclassifications with compound
emotions, it also obtains better results on the RaiPlay dataset,
while only in 3 cases obtains more reclassifications than the
NRC dataset. This datum overall shows that there are minimal
differences in the adoption of the different lexicons using the
provided pipeline in terms of number of reclassifications and
that, overall, the combinatorial mechanisms used by DEGARI for
emotion reclassification are, for these two lexicons, relatively
independent from the particular lexicon used.
13
8.5. User study
32
The goal of the user study was to assess the acceptance of the
emotion categories suggested by DEGARI, with the ultimate goal
of using the reclassifications produced by the system to improve
the annotation of artworks and media, and, consequently, the
applications which depend on it, such as personalization and
recommendation.
Methods and material. The user study consisted in an online
questionnaire (in Italian). The questionnaire contained 10 items,
each accompanied by an image, or, for the multimedia items, by
the film poster, accompanied by the link to the online player
for watching the content. For each item, the users received two
questions: the first question (Question 1) asked them to rate the
association of the item with the emotional category provided
by DEGARI on 10-point scale; the second question (Question 2)
asked them to associate the item to additional emotion categories, taken from the Plutchik model. Users were divided in 3
groups, each corresponding to a different set of 10 items. For
each dataset, the selected items were the ones ranked higher by
DEGARI for each generated compound emotional category.
Participants and procedure. The study involved 44 users (23
females, 21 males). Concerning the age groups, 2 users were
below 18; 17 users were in the 19–35 age range; 12 in the 36–50
range; 10 in the 51–70 range; 3 were older than 70. Users were
randomly assigned to the questionnaires. The first questionnaire
was filled out by 12 users; the second questionnaire was filled out
by 20 users; the third questionnaire was assigned to 12 users. As
a result, 440 ratings and 1065 emotion categories were collected.
Results and analysis. Concerning Question 1, the average rating
assigned by the users to the emotion category proposed by Degari was 6.32, with only slight differences between the datasets
(see Table 4). The average rating was 6.47 for ArsMeteo, 6.42
for RaiPlay, and 6.12 for WikiArt. The standard deviation was
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
Knowledge-Based Systems xxx (xxxx) xxx
Table 3
Reclassification results using Shaver lexicon, compared to the original NRC lexicon.
Emotion
awe (fear_surprise)
delight (joy_surprise)
delight (surprise_joy)
despair (fear_sadness)
disapproval (sadness_surprise)
envy (anger_sadness)
guilt (fear_joy)
guilt (joy_fear)
outrage (anger_surprise)
pride (anger_joy)
TOTAL
ArsMeteo (9171 artworks)
RaiPlay (4612 media items)
WikiArt emotions (4105 artworks)
NRC
Shaver
NRC
Shaver
NRC
Shaver
63
52
52
19
58
25
22
24
41
20
117
71
57
49
83
30
44
45
49
27
132
117
117
20
64
21
73
73
46
72
7
2
88
132
2
49
0
117
20
64
508
1672
1672
0
404
0
1618
1618
508
1618
0
0
0
508
0
508
0
1672
0
404
324
572
618
481
7946
3092
emotion, compound emotional concepts starting from the basic
ones. Such newly created categories, characterized by lexiconbased typical features, are then used in DEGARI to reclassify,
in an emotional settings, the items of three different datasets.
The novelty of this system relies on the fact that DEGARI is, to
the best of our knowledge, the first emotion-oriented system
employing a white box approach to emotion classification based
on the human-like conceptual combination framework proposed
in the TCL logic. The explainability requirement comes for free as
a consequence of this logic-based approach, as shown in Fig. 2.
Overall, the white box approach proposed by DEGARI for
emotionally-driven content reclassification could be useful for
addressing the very well known filter bubble effect [64] in recommender systems, by introducing seeds of serendipity in content
discovery by users. One fundamental discussion about the applicability of DEGARI in practice is whether or not it represents
a truly innovative technical solution for an emotion-based recommender system. According to Sohail et al. [65], recommender
systems ‘‘try to identify the need and preferences of users, filter
the huge collection of data accordingly and present the best
suited option before the users by using some well-defined mechanism’’. Despite the huge amount of proposals, the main families
of recommender systems can be identified as based on: (i) collaborative filtering; (ii) content-based filtering; (iii) hybrid filtering.
At their core of functioning, collaborative filtering exploits similarities of usage patterns among mutually affine users, while
content-based filtering exploits content similarity. DEGARI by
definition falls into the latter category since in its current form
it uses content description (obtained in different ways) as the
input. From the technical point of view, however, it differs from
the current mainstream approaches that are mostly based on the
comparison and matching of visual and perceptual features of
the content [66,67]. In practice, our approach adds a logic layer
capable of mapping and representing – in a commonsense and
cognitively compliant fashion [68] – new emotional categories
which can be used to affect user preferences and content consumption in a way that cannot be derived from the pure statistical
analysis of content and/or the comparison of similar users. Moreover, the proposed approach has been applied to a well-known
model, the Plutchik circumplex model of emotions [12] and to
two different emotional lexica (NRC and Shaver’s), but could in
principle be applied to other models which organize emotions by
similarity, opposition and composition, such as for example the
extended version of the Hourglass model used in SenticNet [36].
Being independent from the specific application model and type
of expression, this approach can work effectively in different
domains, as shown by its use on the datasets of artworks and
media illustrated in this paper. In this sense, it can promote the
interoperability of affective annotations and the cross-domain
reuse of techniques and methods.
In the future work, we plan to extend the evaluation currently conducted in the form of a user study to a large scale
Table 4
User ratings of the emotions proposed by DEGARI.
Average rating
Standard deviation
Median
ArsMeteo
WikiArt
RaiPlay
All
6.47
2.48
7
6.12
2.47
7
6.42
2.1
7
6.32
2.41
7
Table 5
Overlapping of user tags with DEGARI emotions.
User tags
Proposed emotions
Overlapping
ArsMeteo
WikiArt
RaiPlay
All
308
36
27.28%
308
50
20%
449
41
19.50%
1065
127
22.05%
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2.41 (2.48 for ArsMeteo, 2.47 for WikiArt, and 2.1 for RaiPlay),
suggesting that the differences in ratings were limited. Also, the
median rating is 7 for all data sets, with only 2 proposed emotion
categories (1 from Arsmeteo and 1 from WikiArt) rated below 5.
Concerning Question 2 (namely, the additional emotions attached by the users to the items), 308 were attached to the items
in ArsMeteo, 308 to the items in WikiArt, and 449 to the items
in RaiPlay, yielding 1065 user emotion categories. The average
number of emotion categories per users was 24.2. In order to
investigate the overlapping between the set of emotion categories
proposed by DEGARI for each item (apart for the top ranked
category tested through Question 1), we compared the emotion
categories selected by the users with the ones proposed by DEGARI (Table 5). Data show that 22.05% of the emotion categories
additionally proposed by DEGARI for each item matched those
selected by the users, with a higher value for ArsMeteo (27.78%),
and a lower value for WikiArt (20%) and RaiPlay (19.51%). This
datum is a positive one since it concerns the non-top ranked
emotional categories suggested by the system (for which the
degree of acceptability by the users was always above 5 out of
a 10-point scale except for 2 items, and with a median of 7 for
every considered dataset).
To conclude, the collected data suggest that the emotion categories proposed by DEGARI as a result of the reclassification
process are generally accepted by the users, with few exceptions
that deserve further investigation. The acceptance is clear for the
top ranked emotion, but a satisfactory degree of acceptance can
be inferred also for the remaining suggested categories, for which
an overlapping of 20% and more with the user tags has been found
in all datasets.
31
9. Discussion and conclusion
32
33
34
In this paper, we presented DEGARI: an explainable AI system
relying on the TCL Description Logics and on the ArsEmotica
knowledge base to generate, according to Plutchik’s theory of
14
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Knowledge-Based Systems xxx (xxxx) xxx
one to further validate the effectiveness of the proposed approach. We also plan to extend the applications of this system
to different domains. A first extension will be in the field of the
emotional-oriented recommendation of artworks within Museums and cultural heritage sites (this is a work currently under
development within the H2020 European SPICE project15 ). In
addition, also the field of music recommendation represents a
current area of investigation.
From a technical perspective, in future research, we aim at
studying the application of optimization techniques in [69] in
order to improve the efficiency of the DEGARI knowledge generation system. Secondly, we aim at considering more accurate
and multimodal descriptions of artistic and media items, by exploiting Automatic Speech Recognition data and semantic visual
categories extracted from video and audio channels of the content. Finally, as mentioned, we plan to improve the provided
recommendations by justifying the content reclassification (and
the derived recommendations) based on the probabilistic ranks
assigned to the shared features between the generated emotion
and the items being reclassified.
[14] T. Bänziger, K.R. Scherer, Introducing the geneva multimodal emotion
portrayal (gemep) corpus, in: Blueprint for Affective Computing: A
Sourcebook, Vol. 2010, Oxford University Press, Oxford, UK, 2010, pp.
271–294.
[15] P. Lopes, A. Liapis, G.N. Yannakakis, Modelling affect for horror
soundscapes, IEEE Trans. Affect. Comput. 10 (2) (2017) 209–222.
[16] S. Mohammad, S. Kiritchenko, Wikiart emotions: An annotated dataset of
emotions evoked by art, in: N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S.
Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno,
J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the Eleventh
International Conference on Language Resources and Evaluation (LREC
2018), European Language Resources Association (ELRA), Miyazaki, Japan,
2018.
[17] P. Noy, D. Noy-Sharav, Art and emotions, Int. J. Appl. Psychoanal. Stud. 10
(2) (2013) 100–107.
[18] D. Jurafsky, J.H. Martin, Lexicons for sentiment, affect, and connotation,
in: Speech and Language Processing: An Introduction To Natural Language
Processing, Computational Linguistics, and Speech Recognition, 3nd Edition, in: Prentice Hall Series in artificial intelligence, Prentice Hall, Pearson
Education International, Draft, 2020, Chapter 20. available online: https:
//web.stanford.edu/~jurafsky/slp3/20.pdf. Accessed on December 2020.
[19] M. Nissim, V. Patti, Chapter 3 - Semantic aspects in sentiment analysis, in:
F.A. Pozzi, E. Fersini, E. Messina, B. Liu (Eds.), Sentiment Analysis in Social
Networks, Morgan Kaufmann, Boston, 2017, pp. 31–48.
[20] Z. Wang, S. Ho, E. Cambria, A review of emotion sensing: categorization
models and algorithms, Multimedia Tools Appl. 79 (2020).
Declaration of competing interest
[21] A. Chatterjee, U. Gupta, M.K. Chinnakotla, R. Srikanth, M. Galley, P. Agrawal,
Understanding emotions in text using deep learning and big data, Comput.
Hum. Behav. 93 (2019) 309–317.
The authors declare that they have no known competing finan[22] J. Wang, L.-C. Yu, K.R. Lai, X. Zhang, Dimensional sentiment analysis
cial interests or personal relationships that could have appeared
using a regional CNN-LSTM model, in: Proceedings of the 54th Annual
to influence the work reported in this paper.
Meeting of the Association for Computational Linguistics (Volume 2: Short
Papers), Association for Computational Linguistics, Berlin, Germany, 2016,
pp. 225–230.
Acknowledgments
[23] M.E. Basiri, S. Nemati, M. Abdar, E. Cambria, U.R. Acharya, ABCDM: An
attention-based bidirectional CNN-RNN deep model for sentiment analysis,
The research leading to these results/this publication has been
Future Gener. Comput. Syst. 115 (2021) 279–294.
[24] S.M. Mohammad, Sentiment analysis: Detecting valence, emotions, and
partially funded by the European Union’s Horizon 2020 research
other affectual states from text, 2020, CoRR, abs/2005.11882.
and innovation programme http://dx.doi.org/10.13039/501100007601
[25] S. Mohammad, Word affect intensities, in: N. Calzolari, K. Choukri, C. Cieri,
under grant agreement SPICE 870811. The authors thank RAI for
T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo,
the RaiPlay dataset and the Associazione Culturale ArsMeteo that
A. Moreno, J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the
provided the ArsMeteo dataset.
Eleventh International Conference on Language Resources and Evaluation,
LREC 2018, Miyazaki, Japan, May 7-12, 2018, European Language Resources
Association (ELRA), 2018.
References
[26] M.S. Akhtar, A. Ekbal, E. Cambria, How intense are you? Predicting intensities of emotions and sentiments using stacked ensemble [application
[1] D.A. Russell, et al., On the Sublime, Clarendon, 1964.
notes], IEEE Comput. Intell. Mag. 15 (1) (2020) 64–75.
[2] W. Ross, Ars Rhetorica. Aristotle, Clarendon Press, Oxford, 1959.
[27] C. Strapparava, R. Mihalcea, Semeval-2007 task 14: Affective text, in: E.
[3] N.N. Van Dongen, J.W. Van Strien, K. Dijkstra, Implicit emotion regulation
Agirre, L.M. i Villodre, R. Wicentowski (Eds.), Proceedings of the 4th Interin the context of viewing artworks: ERP evidence in response to pleasant
national Workshop on Semantic Evaluations, in: SemEval ’07, Association
and unpleasant pictures, Brain Cogn. 107 (2016) 48–54.
for Computational Linguistics, Stroudsburg, PA, USA, 2007, pp. 70–74.
[4] H. Leder, G. Gerger, D. Brieber, N. Schwarz, What makes an art expert?
[28] S.M. Mohammad, P.D. Turney, Crowdsourcing a word-emotion association
Emotion and evaluation in art appreciation, Cogn. Emot. 28 (6) (2014)
lexicon, Comput. Intell. 29 (3) (2013) 436–465.
1137–1147.
[29] S. Mohammad, Obtaining reliable human ratings of valence, arousal, and
[5] M.V. Thoma, S. Ryf, C. Mohiyeddini, U. Ehlert, U.M. Nater, Emotion
dominance for 20,000 english words, in: Proceedings of the 56th Annual
regulation through listening to music in everyday situations, Cogn. Emot.
Meeting of the Association for Computational Linguistics (Volume 1: Long
26 (3) (2012) 550–560.
Papers), Association for Computational Linguistics, Melbourne, Australia,
[6] I. Andjelkovic, D. Parra, J. O’Donovan, Moodplay: Interactive music recom2018, pp. 174–184.
mendation based on Artists’ mood similarity, Int. J. Hum.-Comput. Stud.
[30] E. Cambria, Y. Li, F.Z. Xing, S. Poria, K. Kwok, SenticNet 6: Ensemble
121 (2019) 142–159.
application of symbolic and subsymbolic AI for sentiment analysis, in: M.
[7] G.M. Smith, Film Structure and the Emotion System, Cambridge University
d’Aquin, S. Dietze, C. Hauff, E. Curry, P. Cudré-Mauroux (Eds.), CIKM ’20:
Press, 2003.
The 29th ACM International Conference on Information and Knowledge
[8] E.S. Tan, Emotion and the Structure of Narrative Film: Film As an Emotion
Management, Virtual Event, Ireland, October 19-23, 2020, ACM, 2020, pp.
Machine, Routledge, 2013.
105–114.
[9] J.A. Russell, A circumplex model of affect, J. Personal. Soc. Psychol. 39 (6)
[31] C. Strapparava, A. Valitutti, Wordnet affect: an affective extension of
(1980) 1161.
wordnet, in: Proceedings of the Fourth International Conference on Lan[10] D. Watson, L.A. Clark, A. Tellegen, Development and validation of brief
guage Resources and Evaluation (LREC’04), European Language Resources
measures of positive and negative affect: the PANAS scales, J. Personal.
Association (ELRA), Lisbon, Portugal, 2004.
Soc. Psychol. 54 (6) (1988) 1063.
[32] F. Bertola, V. Patti, Ontology-based affective models to organize artworks
in the social semantic web, Inf. Process. Manage. 52 (1) (2016) 139–162.
[11] A. Mehrabian, Pleasure-arousal-dominance: A general framework for de[33] D. Gachechiladze, F. Lanubile, N. Novielli, A. Serebrenik, Anger and its
scribing and measuring individual differences in temperament, Curr.
direction in collaborative software development, in: Proceedings of the
Psychol. 14 (4) (1996) 261–292.
39th International Conference on Software Engineering: New Ideas and
[12] R. Plutchik, A general psychoevolutionary theory of emotion, in: Theories
Emerging Results Track, in: ICSE-NIER ’17, IEEE Press, Piscataway, NJ, USA,
of Emotion, Elsevier, 1980, pp. 3–33.
2017, pp. 11–14.
[13] P. Ekman, Basic emotions, in: M.J.P. Tim Dalgleish (Ed.), Handbook of
[34] V. Patti, F. Bertola, A. Lieto, Arsemotica for arsmeteo. org: Emotion-driven
Cognition and Emotion, Vol. 98, Wiley Online Library, 1999, pp. 45–60,
exploration of online art collections, in: I. Russell, W. Eberle (Eds.), The
chapter 3.
Twenty-Eighth International Florida Artificial Intelligence Research Society
Conference (FLAIRS 2015), Association for the Advancement of Artificial
15 https://spice-h2020.eu/.
Intelligence, AAAI Press, 2015, pp. 288–293.
15
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
KNOSYS: 107166
A. Lieto, G.L. Pozzato, S. Zoia et al.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
Knowledge-Based Systems xxx (xxxx) xxx
[35] E. Cambria, A. Livingstone, A. Hussain, The hourglass of emotions, in:
A. Esposito, A.M. Esposito, A. Vinciarelli, R. Hoffmann, V.C. Müller (Eds.),
Cognitive Behavioural Systems - COST 2102 International Training School,
Dresden, Germany, February 21-26, 2011, Revised Selected Papers, in:
Lecture Notes in Computer Science, vol. 7403, Springer, 2012, pp. 144–157.
[36] Y. Susanto, A.G. Livingstone, B.C. Ng, E. Cambria, The hourglass model
revisited, IEEE Intell. Syst. 35 (5) (2020) 96–102.
[37] R. Plutchik, The nature of emotions, Am. Sci. 89 (4) (2001) 344–350.
[38] A. Lieto, G.L. Pozzato, A description logic framework for commonsense
conceptual combination integrating typicality, probabilities and cognitive
heuristics, J. Exp. Theor. Artif. Intell. 32 (5) (2020) 769–804.
[39] A. Lieto, G.L. Pozzato, A description logic of typicality for conceptual
combination, in: M. Ceci, N. Japkowicz, J. Liu, G.A. Papadopoulos, Z.W. Ras
(Eds.), Foundations of Intelligent Systems - 24th International Symposium,
ISMIS 2018, Limassol, Cyprus, October 29-31, 2018, Proceedings, in: Lecture
Notes in Computer Science, vol. 11177, Springer, 2018, pp. 189–199.
[40] M.A. Boden, Creativity and artificial intelligence, Artificial Intelligence 103
(1–2) (1998) 347–356.
[41] M. Frixione, A. Lieto, Representing and reasoning on typicality in formal
ontologies, in: C. Ghidini, A.N. Ngomo, S.N. Lindstaedt, T. Pellegrini (Eds.),
Proceedings of the 7th International Conference on Semantic Systems, in:
ACM International Conference Proceeding Series, ACM, 2011, pp. 119–125.
[42] D.N. Osherson, E.E. Smith, On the adequacy of prototype theory as a theory
of concepts, Cognition 9 (1) (1981) 35–58.
[43] A. Lieto, G.L. Pozzato, F. Perrone, E. Chiodino, Knowledge capturing via
conceptual reframing: A goal-oriented framework for knowledge invention,
in: M. Kejriwal, P.A. Szekely, R. Troncy (Eds.), Proceedings of the 10th ACM
Conference on Knowledge Capture, K-CAP 2019, Marina Del Rey, ACM,
2019, pp. 109–114.
[44] A. Lieto, F. Perrone, G.L. Pozzato, E. Chiodino, Beyond subgoaling: A
dynamic knowledge generation framework for creative problem solving
in cognitive architectures, Cogn. Syst. Res. 58 (2019) 305–316.
[45] E. Chiodino, A. Lieto, F. Perrone, G.L. Pozzato, A goal-oriented framework
for knowledge invention and creative problem solving in cognitive architectures, in: G.D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A.
Bugarín, J. Lang (Eds.), Proceedings of ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago
de Compostela, Spain, 2020, in: Frontiers in Artificial Intelligence and
Applications, vol. 325, IOS Press, 2020, pp. 2893–2894.
[46] A. Lieto, G.L. Pozzato, Applying a description logic of typicality as a
generative tool for concept combination in computational creativity, Intell.
Artif. 13 (1) (2019) 93–106.
[47] M. Eppe, E. Maclean, R. Confalonieri, O. Kutz, M. Schorlemmer, E. Plaza,
K.-U. Kühnberger, A computational framework for conceptual blending,
Artificial Intelligence 256 (2018) 105–129.
[48] M. Lewis, J. Lawry, Hierarchical conceptual spaces for concept combination,
Artificial Intelligence 237 (2016) 204–227.
[49] R. Confalonieri, M. Schorlemmer, O. Kutz, R. Peñaloza, E. Plaza, M. Eppe,
Conceptual blending in EL++ , in: Proceedings of the 29th International
Workshop on Description Logics, Cape Town, South Africa, April 22-25,
2016, 2016.
[50] F. Riguzzi, E. Bellodi, E. Lamma, R. Zese, Probabilistic description logics
under the distribution semantics, Semant. Web 6 (5) (2015) 477–501.
[51] L. Giordano, V. Gliozzi, N. Olivetti, G.L. Pozzato, Semantic characterization
of rational closure: from propositional logic to description logics, Artificial
Intelligence 226 (2015) 1–33.
[52] D. Lehmann, M. Magidor, What does a conditional knowledge base
entail? Artificial Intelligence 55 (1) (1992) 1–60.
[53] F. Riguzzi, E. Bellodi, E. Lamma, R. Zese, Reasoning with probabilistic ontologies, in: Q. Yang, M. Wooldridge (Eds.), Proceedings of the
Twenty-Fourth International Joint Conference on Artificial Intelligence,
IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, AAAI Press, 2015,
pp. 4310–4316.
[54] J.A. Hampton, Inheritance of attributes in natural concept conjunctions,
Mem. Cogn. 15 (1) (1987) 55–71.
[55] P. Silvia, Looking past pleasure: Anger, confusion, disgust, pride, surprise,
and other unusual aesthetic emotions, Psychol. Aesthet. Creat. Arts 3
(2009) 48–51.
[56] J.P. McCrae, D. Spohr, P. Cimiano, Linking lexical resources and ontologies
on the semantic web with lemon, in: G. Antoniou, M. Grobelnik, E.P.B.
Simperl, B. Parsia, D. Plexousakis, P.D. Leenheer, J.Z. Pan (Eds.), The
Semantic Web: Research and Applications - 8th Extended Semantic Web
Conference, ESWC 2011, Heraklion, Crete, Greece, May 29-June 2, 2011,
Proceedings, Part I, in: Lecture Notes in Computer Science, vol. 6643,
Springer, 2011, pp. 245–259.
[57] E. Acotto, M. Baldoni, C. Baroglio, V. Patti, F. Portis, G. Vaccarino, Arsmeteo:
artworks and tags floating over the planet art, in: Proceedings of the 20th
ACM Conference on Hypertext and Hypermedia, in: HT ’09, Association for
Computing Machinery, New York, NY, USA, ISBN: 9781605584867, 2009,
pp. 331–332, http://dx.doi.org/10.1145/1557914.1557973, https://doi.org/
10.1145/1557914.1557973.
[58] A. Lieto, G. Pozzato, A. Valese, COCOS: a typicality based COncept COmbination System, in: M. Montali and P. Felli (Eds.), Proceedings of the 33rd
Italian Conference on Computational Logic (CILC 2018), CEUR Workshop
Proceedings, Bozen, Italy, 2018, pp. 55–59.
[59] E. Chiodino, D.D. Luccio, A. Lieto, A. Messina, G.L. Pozzato, D. Rubinetti, A
knowledge-based system for the dynamic generation and classification of
novel contents in multimedia broadcasting, in: G.D. Giacomo, A. Catalá, B.
Dilkina, M. Milano, S. Barro, A. Bugarín, J. Lang (Eds.), Proceedings of ECAI
2020 - 24th European Conference on Artificial Intelligence, 29 August8 September 2020, Santiago de Compostela, Spain, 2020, in: Frontiers
in Artificial Intelligence and Applications, vol. 325, IOS Press, 2020, pp.
680–687.
[60] L. Giordano, V. Gliozzi, G.L. Pozzato, R. Renzulli, An efficient reasoner for
description logics of typicality and rational closure, in: A. Artale, B. Glimm,
R. Kontchakov (Eds.), Proceedings of the 30th International Workshop
on Description Logics, Montpellier, France, July 18-21, 2017, in: CEUR
Workshop Proceedings, vol. 1879, CEUR-WS.org, 2017.
[61] P. Shaver, J. Schwartz, D. Kirson, C. O’connor, Emotion knowledge: further
exploration of a prototype approach, J. Personal. Soc. Psychol. 52 (6) (1987)
1061.
[62] G. Shani, A. Gunawardana, Evaluating recommendation systems, in:
Recommender Systems Handbook, Springer, 2011, pp. 257–297.
[63] E. Rosch, Cognitive representations of semantic categories, J. Exp. Psychol.
[Gen.] 104 (3) (1975) 192–233.
[64] E. Parisier, The Filter Bubble: What the Internet is Hiding from You,
Penguin Books Limited, 2012.
[65] S.S. Sohail, J. Siddiqui, R. Ali, Classifications of recommender systems: A
review, Eng. Sci. Technol. Rev. 10 (4) (2017) 132–153.
[66] M.S. Sun, F. Li, J. Zhang, A multi-modality deep network for cold-start
recommendation, Big Data Cogn. Comput. 2 (7) (2018).
[67] Y. Deldjoo, M.G. Constantin, H. Eghbal-Zadeh, B. Ionescu, M. Schedl, P. Cremonesi, Audio-visual encoding of multimedia content for enhancing movie
recommendations, in: S. Pera, M.D. Ekstrand, X. Amatriain, J. O’Donovan
(Eds.), Proceedings of the 12th ACM Conference on Recommender Systems,
RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, ACM, 2018, pp.
455–459.
[68] A. Lieto, Cognitive Design for Artificial Minds, Routledge, 2021.
[69] M. Alberti, E. Bellodi, G. Cota, F. Riguzzi, R. Zese, Cplint on SWISH:
Probabilistic logical inference with a web browser, Intell. Artif. 11 (1)
(2017) 47–64.
16
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121