ORIGINAL RESEARCH
published: 12 October 2021
doi: 10.3389/fpsyg.2021.674159
Applying Implicit Association Test
Techniques and Facial Expression
Analyses in the Comparative
Evaluation of Website User
Experience
Maurizio Mauri 1,2*†, Gaia Rancati 3‡, Andrea Gaggioli 1,4$ and Giuseppe Riva 1,4$
1 Department of Psychology, Catholic University of Milan, Milan, Italy, 2 Department of User Experience and Marketing
3
Edited by: Research, SR LABS, Milan, Italy, Department of Business and Economics, Allegheny College, Meadville, PA, United States,
4
Margherita Zito, Applied Technology for Neuro-Psychology Lab, I.R.C.C.S. Istituto Auxologico Italiano, Milan, Italy
Università IULM, Italy
Reviewed by: This research project has the goal to verify whether the application of neuromarketing
Ubaldo Cuesta,
techniques, such as implicit association test (IAT) techniques and emotional facial
Complutense University of
Madrid, Spain expressions analyses may contribute to the assessment of user experience (UX) during
Denis Helic, and after website navigation. These techniques have been widely and positively applied
Graz University of Technology, Austria
Giampaolo Viglia, in assessing customer experience (CX); however, little is known about their simultaneous
University of Portsmouth, application in the field of UX. As a specific context, the experience raised by different
United Kingdom
websites from two well-known automotive brands was compared. About 160 Italian
*Correspondence:
university students were enrolled in an online experimental study. Participants performed
Maurizio Mauri
maurizio.mauri@unicatt.it a Brand Association Reaction Time Test (BARTT) version of the IAT where the two
† brands were compared according to different semantic dimensions already used inFirst author
‡ the automotive field. After completing the BARTT test, the participants navigated theSenior author
§ target website: 80 participants navigated the first brand website, while the other halfThese authors share last authorship
navigated the second brand website (between-subject design). During the first 3min
Specialty section: of website navigation, emotional facial expressions were recorded. The participants
This article was submitted to
were asked to freely navigate the website home page, look for a car model and its
Organizational Psychology,
a section of the journal characteristics and price, use the customising tool, and in the end, look for assistance.
Frontiers in Psychology After the website navigation, all the participants performed, a second time, the BARTT
Received: 28 February 2021 version of the IAT, where the two brands were compared again, this time to assess
Accepted: 07 September 2021
Published: 12 October 2021 whether the website navigation may impact the Implicit Associations previously detected.
Citation: A traditional evaluation of the two websites was carried on by means of the classic
Mauri M, Rancati G, Gaggioli A and heuristic evaluation. Findings from this study show, first of all, the significant results
Riva G (2021) Applying Implicit
provided by neuromarketing techniques in the field of UX, as IAT can provide a positive
Association Test Techniques and
Facial Expression Analyses in the application for assessing UX played by brand websites, thanks to the comparison of
Comparative Evaluation of Website eventual changes in time reaction between the test performed before and after website
User Experience.
Front. Psychol. 12:674159. navigation exposure. Secondly, results from emotional facial expression analyses during
doi: 10.3389/fpsyg.2021.674159 the navigation of both brand websites showed significant differences between the two
Frontiers in Psychology | www.frontiersin.org 1 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
brands, allowing the researchers to predict the emotional impact raised by each website.
Finally, the positive correlation with heuristic evaluation shows that neuromarketing can
be successfully applied in UX.
Keywords: facial expression, emotions, user experience (UX), brand association, online experiment
INTRODUCTION websites. However, in this set of usability principles, none of them
was addressing attention to the emotional impact played by all
Advances in technology, digital transformation, cost pressure, these kinds of experiences. This lack was filled in a few years
and the emergence of new channels have considerably changed later, when Nielsen updated his list of nine “usability principles,”
the way customers shop and interact with brands (Gauri et al., adding a 10th one, labelled as “aesthetic design” (Nielsen, 1994b),
2016; Lemon and Verhoef, 2016; Bolton et al., 2018; Grewal thus recognising the importance of the emotional impact played
et al., 2018; Lee, 2020). Furthermore, the challenges faced by the digital experience. Nevertheless, the way to investigate this
during the COVID-19 pandemic have brought companies to additional principle is still mainly based on the opinion of expert
rethink their business models (Boudet et al., 2020). Today, evaluators as deeper described later in this article, mentioning
customers are following a cross-channel customer journey the heuristic evaluation procedure. Almost two decades ago, Don
rather than a linear path to purchase (Harris et al., 2020), Norman, another of the most quoted researchers in the field of
and this big shift in consumer behaviour transforms them UX, highlighted, and explained the importance of “emotional
from buyers to users, moving the focus on the customer, and design” in products and services (Norman, 2004). Although
user experience (UX) (Sheth, 2021). Therefore, classic research famous scientists highlighted the importance of this factor, there
techniques primarily based on qualitative methods such as is a lack of scientific procedures to characterise and measure this
self-report measures and interviews, largely predominating in specific domain in UX. For this reason, as already stated, there is
UX research (Pettersson et al., 2018), require development the need of developing newmethods allowing to assess, according
for new UX evaluation methods. This development can to empirical procedures, the effects of emotional impact played
improve practicability and scientific quality (Vermeeren et al., by UX. The present research brings light to this specific topic,
2010), leading to multidisciplinary research methods based showing how the application of two neuromarketing techniques
on more objective data (Verhulst et al., 2019). Among allows researchers to assess and rank different websites in terms
them, neuromarketing represents an evolving field of scientific of emotional responses. One technique is based on an implicit
investigations that have shown valuable understanding of association test (IAT) in relation to emotional items presented
consumer behaviour and its links with emotions in perception before and after website navigation to verify whether navigation
and decision-making processes. This area combines theories can change short-term associations, as no previous research
and practises from fields of behavioural sciences, including tried to apply this method to evaluate website experiences. The
neuroscience, psychology, and sociology, to determine the other technique is related to affective responses in terms of
reasoning and patterns of choices of consumers. As defined by facial expression analyses during navigation, as little is shown
Ale Smidts, as the first definition of the term, neuromarketing is, by scientific literature in the field of UX and website design.
“the study of the cerebral mechanism to understand the consumer’s The simultaneous application of both techniques, together with a
behaviour in order to improve themarketing strategies” (Stasi et al., traditional one relying on heuristic evaluation, allows researchers
2018). to explore whether the use of neuromarketing methods can
In the field of neuromarketing, themost significant techniques improve the scientific measurement of the emotional design of
are based on eye tracking, electroencephalography (EEG), websites, widening the application of neuromarketing techniques
fMRI, psychophysiology, analysis of facial expressions, and from CX to UX.
reaction times (Gacioppo and Petty, 1985; Stasi et al., 2018).
Although neuromarketing techniques have been largely applied
to customer experience (CX) (Gacioppo and Petty, 1985; RELATED RESEARCH WORK
Klinčeková, 2016), none of these techniques are commonly
applied to UX, except for eye-tracking and a few pioneering The IAT helps researchers identify biases through reaction
studies relying on neuro and psychophysiological measurements times and emotions and has been developed into a marketing-
(Bender and Sung, 2021). Emotions are considered a key point in oriented variation known as the Brand Association Reaction
UX, as mentioned by Marc Hassenzahl, one of the most quoted Time Test (BARTT) to expand on the understanding of
researchers in UX and its hedonic impact: “Reformers of Human consumer behaviour. This technique can clarify how consumers
Computer Interaction (HCI), often stress that the old HCI is, in value the brand by distinguishing biases among participants
essence, cognitive (i.e., focused on memory, task, etc.), and that through their intentional efforts to conceal attitudes towards
the future lies in emotions” (Hassenzahl, 2004). Jakob Nielsen, concepts. When using IAT for evaluating brand associations,
one of the most well-known researchers in UX, in 1990, provided the data collected can indicate the subject biases and determine
a set of nine “usability principles,” enabling the identification hierarchies of products by analysing reaction time as well as
of all main problems when using HCI interfaces, software, and latencies in the way participants associate concepts (Bercea,
Frontiers in Psychology | www.frontiersin.org 2 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
2012; Gregg and Klymowsky, 2013). Greenwald showed the 2017). Many methods from neuroscience research have been
reliability of this technique in his early IAT experiments, adopted to be used in marketing research (Clement et al., 2013;
regarding pleasant/unpleasant associations, showing that the Missaglia et al., 2017; Songa et al., 2019), such as the IAT, but little
delays of participants, or lack thereof, indicate bias in addition is known about the possibilities these methods have in the area of
to the choices selected (Greenwald et al., 1998). In this regard, UX with the exeption of few pioneering studies that show how to
the importance of this method can be argued for marketing derive emotions from user mouse behaviour (Yi et al., 2020). User
research as a means of determining the biases of the customers experience encompasses how people experience things around
through their conscious choices to avoid displaying biases. These them, including products, websites, and services (Bojko, 2013).
biases are shown distinctly through the response latencies in The Nielsen Norman Group defines UX as providing what the
the association of the concepts. Beginning with the seminal customer needs without hassle, crafting products that are a joy
work of Keller (1993) and Aaker (2009), the brand association to own or use, and the “seamless merging of the services of
became an important topic, supported by the demand of multiple disciplines, including engineering, marketing, graphical
marketers to have clear guidelines of brands in their business and industrial design, and interface design” (Nielsen andNorman,
and managerial decisions. However, the literature shows a lack 2021). Inherently, UX is multifaceted and touches on various
of research, enabling to highlight how brand associations may be parts of the use of a service, system, or product (Quaglini, 2020).
modified by communication activities, in particular to websites To aid in the understanding of UX, researchers have relied
communication strategies. Some research experiments already greatly on a neuromarketing technique, such as eye tracking.
presented significant results about methods to measure brand It is especially relevant in evaluating the UX of websites and
image based on the constellation of associations (Till et al., interfaces because it grants researchers the visual perspective of
2011; Schnittka et al., 2012; Camarrone and Van Hulle, 2019). a user and allows to establish the findability of specific calls to
Some other research projects showed how brand associations action (Mele and Federici, 2012; Fu et al., 2017). Two primary
can be efficiently applied to advertising assessments (Janakiraman reasons researchers use eye tracking are that it is non-invasive
et al., 2009; Anderson and Simester, 2013; Caldato et al., 2020). and can help determine how a consumer reacts during his or
However, no previous research tried to investigate how brand her interactions with a web product or service. Eye movements
associations vary depending on stimuli represented by a website. from the user can be fixations and saccades, and the movements
Brand associations are frequently characterised by a static mental of the eyes can indicate emotions such as confusion when the eyes
map; however, what happens when users are exposed to a return to a previous point (Bergstrom and Schall, 2014, p. 55–57).
website? Does the mental map vary accordingly? Furthermore, to This tool is significant to researchers pursuing information about
generate strong brands, firms have to implement a set of positive how consumers experience websites because the eyes of a person
associations around them (Till et al., 2011; Flight and Coker, are drawn to and remain in places that result in further thinking,
2016). Throughmarketing actions, firms can identify, strengthen, and the action of looking at a subject directly requires little
or alter the associations linked to their brands (Keller, 1993), conscious effort as it is a more reflexive process (Bojko, 2013).
changing their competitive placement. The experience of a brand Points the eyes of users are drawn to on a web page contribute
and consequently, its associations can be directly shaped by firms to fixation patterns that eye tracking technology can record, and
or can be even transferred by other brands or factors (Keller, these data can be converted into gaze plots and heat maps to
2003). Quite well-known examples are brand endorsement, co- determine points of significant focus on a web page (Djamasbi,
branding, and brand extensions of celebrities (Martini et al., 2014). Understanding the user behaviour on websites informs us
2016). In this vision, the experience of a brandmay be transferred about the decisions he or she makes, including ones to navigate
to another one, if there are some bonds linking them. These away from the web page due to clutter or disorganisation. If
bonds can also be retrieved in the brands of competitors as a website suffers from these and other problems, it may result
some associations are shared among different brands operating in an exhaustive review, which often frustrates the user since it
in the same market business. Based on the “transfer property,” is a product of a website that is not user-friendly (Nielsen and
what happens when competitors attempt to strengthen brand Pernice, 2010, p. 376). By using eye tracking, researchers can
associations shared by the company? All these questions are still determine how a user views a site and navigates good or bad
awaiting a proper and empirical answer. One of the aims of this website design to improve the design for better UX. Therefore,
research project is to fulfil the lack of scientific literature in the eye tracking is a common and useful tool for researchers of
field of website communication and UX, exploring the dynamics UX. However, eye tracking does not help researchers understand
of brand associations after being exposed to a website experience. a user’s comprehension of a subject nor does it help indicate
In this way, it is possible to verify whether such exposure may how a participant emotionally engages with the material in
raise short-term variations in the power of brand associations, question (Bojko, 2013). Even though eye tracking is a useful tool
not only on a specific brand but also on a competing one that for neuromarketing research and UX research, it, alone, does
may share some similar associations. Last but not least, as the not provide the entirety of data needed to create an effective
impact of webpages on brand perception can be classified as website. The IAT helps to understand the comprehension and
an application of reaction time techniques in testing marketing opinions of a user towards a brand/subject/service or product,
stimuli, some positive evidence addressing the feasibility of such such as a website. Understanding brand association in consumers
an application about the digital experience is supported by few is integral to determining brand equity. Brand associations,
pioneering studies (Matukin et al., 2016; Matukin and Ohme, as explained by Keller (1993), can be partitioned into three
Frontiers in Psychology | www.frontiersin.org 3 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
categories: associations of positive or negative favorability, possibility to take advantage of face orientation aside from facial
uniqueness, and strength of associations (Gattol et al., 2011). expression to predict the hedonic impact of the face presentation
In addition to these three elements, there is the relevance of models, as the facial orientation to the right-side significantly
of the association and how this connexion may or may not predicts with a more negative evaluation, while on the opposite,
present as a motivating factor, and the number of associations face orientation towards left side significantly correlates with
the consumer has (Gattol et al., 2011). A consumer may have a positive evaluation of the models’ face presentation (Park
significant associations for a brand, influencing the likelihood of et al., 2021). Facial expressions reveal affective states defined,
a purchase from that brand as well as potential brand loyalty. for instance, in EMFACS-7 (Friesen and Ekman, 1978) and thus
A study analysing brand association in relation to a focus of possibly predict related behaviour and attitude modification
prediction showed that participants had a significant association (Kulczynski et al., 2016). Facial expressions of emotions are
of brand names with cake flavour and quality (Van Osselaer universal sequences of facial muscle contractions associated
and Janiszewski, 2001). Analysis of brand associations clarifies with the emotional state of the person. The neuro-cultural
brand equity, and the aforementioned study establishes brand theory of emotion, developed by Paul Ekman (e.g., Ekman, 1972;
associations can lead to the positive favorability of a brand. Ekman and Cordaro, 2011), defines facial expressions of emotion
However, previous studies of brand association have not utilised as discrete, innate, and culturally independent. According to
the technique of the IAT to evaluate the website experience of a other studies, there is a two-way connexion between facial
user. The present project applied, for the first time, the BARTT to expressions and emotion regulation (Cole, 1986; Izard, 1990;
assess the effects of the experiences of two websites and to verify Gross and Thompson, 2007; Gross, 2014). Therefore, in studying
whether this technique can provide significant results enabling to facial expressions, it is difficult to establish causal relationships
measure and compare the effects of different website designs in between facial non-verbal behaviour and interpretations assigned
relation to both emotional and cognitive items. The appliance of to them—emotions. Emotions do cause facial expressions (“I feel
this neuromarketing technique to UX and website design widens happy, so I smile”), but facial expressions also cause emotions (“I
the range of neuromarketing from customer to UX, verifying smile and it makes me happy”). Any causal relationship between
whether digital experiences can change short-term associations. smiling and perception of the website has not been established
Additionally, the understanding of facial expressions helps in the UX context. Smiling or laughing may indicate liking for
researchers comprehend the emotional engagement and the website and, therefore, greater effectiveness of the website.
experience of a user towards the stimuli he/she has been Analysing facial expressions and user reactions to website
exposed to during the navigation of a website. Automatic interfaces identifies potential frustrations that can be improved
facial expressions analyses, efficiently used in neuromarketing for future users (Branco, 2006). Methods of facial expressions
to evaluate optimal advertising spots (Lewinski et al., 2014b; evaluation based on automatic software analyses further the
Lewinski, 2015; Hamelin et al., 2017; Cherubino et al., 2019) understanding of the interaction of a user with one interface over
or the level of engagement during social media interactions another (Andersen et al., 2014), as well as the overall experience
(Schreiner et al., 2019), could lead to additional insights into of the user with digital tools and resources (Liu and Lee, 2018).
UX research to improve the effects in terms of emotional design To understand UX based on emotions and facial expressions, the
(Small and Verrochi, 2009; Norman, 2013; Hamelin et al., 2017; participants completed a series of tasks while sitting in front of
Danner and Duerrschmid, 2018). Facial expressions are part a traditional PC equipped with a camera, allowing the software
of non-verbal communication, which has been highlighted to measure the emotional reactions they had while interacting
for a long time in the scientific literature as enabling to bring with a website, as this approach has been previously explored
important information aside from verbal expressions (Stewart with positive results from pioneering studies (Hazlett, 2003).
et al., 1987; Puccinelli et al., 2010). In this study, the affective The technique of facial expression analysis has been used little
reactions to websites will be measured in a quantitative way by by researchers of UX (Branco, 2006; Munim et al., 2017) despite
means of autonomic responses, namely facial expressions. This its value in clarifying the frustration and joy of users during
allows the researchers to overcome some of the limitations of their interactions. According to Hancock et al., “Hedonomics,”
the most used tools in UX research, where the evaluation of (Hancock et al., 2005) defined as “the promotion of pleasurable
emotional impact is mainly based on qualitative methods such human-machine interaction” by its creators, it is possible to
as interviews. The feasibility of this approach is widened when highlight the key role of the so-called “emotional design”
an automated tool is utilised, engaging commercially available, (Norman, 2004) as a fundamental factor in UX. The present
advanced, and unobtrusive software that catches and analyses research aims to explore whether the automatic facial expressions
facial expressions of emotions. This solution has been already analyses may provide useful information related to the emotional
used in many different contexts related to experimental research reaction raised by website experiences. This research can expand
in consumer behaviour. There are already several scientific the use of automatic facial expressions, helping professionals in
studies showing how the use of automated facial analysis of measuring the effects of website emotional design according to
expressions provides positive results in assessing CX (de Wijk more empirical procedures.
et al., 2012; He et al., 2012; Terzis et al., 2013; Danner et al., 2014; In conjunction with the two above techniques described, we
El Haj et al., 2017; Noordewier and van Dijk, 2019; Riem and integrated traditional heuristic evaluation (Nielsen and Molich,
Karreman, 2019; Meng et al., 2020). Recently, new pioneering 1990; Nielsen, 1992) performed by five experts from the UX
studies presented by the scientific literature have shown the field. Combining traditional heuristic evaluation with innovative
Frontiers in Psychology | www.frontiersin.org 4 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
techniques based on reaction time and facial expression analyses may need an update to remain consistent with modern usability
can allow to explore whether the results from classic qualitative problems” (Gonzalez-Holland et al., 2017). The present research
method based on heuristic evaluation converge or contrast study used a version of a heuristic evaluation set with 247
with findings emerging from the use of quantitative methods heuristics related to usability problems identified by Nielsen and
based on facial expression analyses and reaction time measures. revised specifically for website experience in the modern context,
In the case of convergence, it may be possible to envisage used in the professional field (Travis, 2017). The heuristic
a further integration of these innovative quantitative methods evaluation has been provided by five different professionals to
in UX research. On the opposite, in the case of divergence establish whether a traditional andmost-usedmethod in UXmay
or contrast, it may be possible to understand whether these support findings from facial expressions analyses and reaction
two different approaches are measuring different phenomena of times techniques.
UX. Jakob Nielsen developed the heuristic evaluation method The inclusion of classic methods like heuristic evaluation with
together with usability consultant Rolf Molich in 1990 due innovative techniques from the Neuromarketing field based on
to their many years of experience in teaching and consulting facial expression analysis and IAT helps to understand whether or
about usability and UX. As defined by the two authors, not the combination of these different approaches may widen the
“there are four main methods to evaluate a user interface: insights on how UX is affected by the emotional design shaping
formally, by some analysis techniques; automatically, by a websites contents and interactions.
computerised procedure; empirically, by experiments with test Finally, both the BARTT/IAT and facial expression analysis
users; and heuristically, by simply looking at the interface and have unique benefits in the current COVID-19 pandemic as
passing judgment according to one’s own opinion” (Nielsen and they are reliable methods of obtaining information that can be
Molich, 1990). In particular, the authors reported that “most collected and recorded without in-person interactions, taking
user interface evaluations are heuristic evaluations, but almost complete advantage of a remote setting. The participants used
nothing is known about this kind of evaluation since it has a personal computer equipped with its camera to provide their
been seen as inferior by most researchers.” For this reason, they facial recordings during the tasks assigned, releasing the needed
presented four experiments, enabling to derive a small set of nine data, enabling them to perform an automatic facial expression
“basic usability principles,” performed by at least three different analysis. The IAT also only requires the use of a personal
professionals, enabling to identify all main problems. Few years computer to be accomplished. Both parts of the experiment can
later, Nielsen refined the heuristics based on a factor analysis be administered by the researcher through a video call or even
of 249 usability problems (Nielsen, 1994a) that allowed the an audio call, eliminating any need to meet all the participants
definition of a set of heuristics withmaximum explanatory power, in person. Due to the global pandemic, the need for health and
resulting in this revised set of heuristics that are used today by safety of all those involved in the study was a high priority, so
most professionals and organisations for user interface design we relied on technology and internet connexion to acquire both
(Nielsen, 1994b): visibility of system status; a match between accurate and safe data.
system and the real world; user control and freedom; consistency Regarding the subject of our experiment, we chose automotive
and standards; error prevention; recognition rather than recall; sites from two American brands due to the impact of the
flexibility and efficiency of use; aesthetic and minimalist design; pandemic on this industry. This research is intended to
help users recognise, diagnose, and recover from errors; help investigate how the brands might take advantage of innovative
and documentation. Before this work, the guidelines were so insights for developing new digital strategies to overcome
many that a professional could need a lot of time before the crises raised by the COVID-19 pandemic and improve
accomplishing it. For instance, Smith and Mosier’s guidelines automotive sales through their websites.
for designing user interface software have 944 items and remain
one of the largest collections of publicly available user interface
guidelines (Smith and Mosier, 1986). Another set of research- MATERIALS AND METHODS
based heuristics has been proposed by Gerhardt-Powals (1996) to
provide an alternative to Nielsen and Molich’s list. Theoretically, The study was conducted between October 2020 and December
all heuristics proposed to share the same purpose to established 2020, and a sample consisting of 160 students (80 men, 80
usability standards that, if enhanced, can provide a better UX women; mean age, 23 ± 4) was recruited from the Catholic
about products or services. Unfortunately, “usability problems” University of Milan. One criterion was established to qualify
are often identified by means of qualitative methods, relying on the sample: The participants had to be in-market for a car
the opinion of expert evaluators (Catani and Biers, 1998). On and intended to purchase it within an appropriate time frame
one side, part of the problem could be explained considering of 2–3 months. In the event that the website proposed cars
that usability professionals have their own favourite sets of beyond their budget, we asked them to identify themselves
heuristics; on the other side, the problem is that there is with a potential buyer. The participants who had already made
not a research-based set of heuristics shared by the scientific their minds about exactly which car they were going to buy
community and based on international consensus. Moreover, the were removed from the sample to exclude the possibility that
scientific literature addressed the need to update the heuristics the participants might have already exhausted their capacity
provided many years ago: “with the rapid expansion and growth for exploration and evaluation of the website. The fact that all
of technology in the last 20 years, Nielsen’s 10 usability heuristics the participants are university students provides the limit that
Frontiers in Psychology | www.frontiersin.org 5 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
all results are representative of this specific population, and affective attitudes, namely: interest, boredom, and confusion has
further research with broader samples in terms of age range been introduced. Unlike regular facial expressions, these affective
and low/high skills in information technology may establish attitudes are computed over a time window (typically from 2
whether the results here presented can be representative of the to 5 s), rather than a frame-by-frame method. Therefore, the
whole general population. All the participants were required to intensity of the affective attitude at any point in time of analysis
have an internet connexion and a personal computer equipped does not just depend upon the current analysis of the face but
with a webcam. The minimum definition resolution required to also on the last 2 pr 5-s history of facial analysis. In addition,
participate in the test is a standard high-definition of 1,280 × some of these affective attitudes also take into account certain
720 pixels (HD Ready or 720 pixels). Two websites from the additional facial cues like nodding or head shaking, which are
automotive field have been selected to perform a comparative also internally computed over the analysis history. The literature
test: Ford and Tesla (version exposed in 2020). This study was on the affective attitudes is still exploring the accuracy of these
performed remotely by utilising software, including iCode, for additional metrics (Borges et al., 2019; Hirt et al., 2019); we
online IAT provided byNEUROHMand FaceReader 8.1 software provide here results related to confusion, as particularly useful
from Noldus for emotional facial expression analysis. All of the to evaluate the impact of website experience here considered, as
participants completed an online, pre-test survey that related to previous research showed positive results in considering subtle
the application of IAT. expressions (Salgado-Montejo et al., 2015).
iCode, an online platform in the field of reaction time First, FaceReader detects a face using the so-called “Active
recording, was used to assess the speed in providing their answers Template Method.” Second, the software builds a virtual, super-
from all the participants in this project. iCode accurately reflects imposed 3D “Active Appearance Model” of the face, featuring
the attitudes of the participants by using a two-part calibration nearly 500 distinctive landmarks. The third step measures the
process to analyse response time (iCode., 2019). Part one of intensity and probability of facial expressions, enabling basic
the calibration process of iCode uses motoric tasks to establish emotions to be computed (Van Kuilenburg et al., 2005). The
the movement speed and familiarity with the device of each neural network of the system has been trained, taking advantage
participant (iCode., 2019). The calibration of iCode consists of of a high-quality correlation of approximately 10,000 images
pressing the answer buttons without any cognitive load. It also that were manually annotated by real human expert coders.
serves as a tutorial for respondents as it makes them familiar The average scores of performances reported are 89% (Den Uyl
with using the scale. Part two of the calibration process of and Van Kuilenburg, 2005; Van Kuilenburg et al., 2005) and
iCode tests how fast the participants read statements of different 87% (Terzis et al., 2013). We consider in this study in the
lengths. Each participant was given a statement and one answer present project results only about “happiness,” as the accuracy
button to press when he/she finished reading the statement. of this specific emotion is the highest in comparison to all
The influence of statement length on corresponding response other emotions according to the scientific literature (Lewinski
times is minimised, allowing statements of different lengths to et al., 2014a,b; Stöckli et al., 2018; Dupré et al., 2020). Although
be compared (iCode., 2019). iCode uses Neurohm’s Confidence FaceReader can analyse offline videos, our study required the
Index to ensure accuracy and helps researchers determine the participants to have a live webcam to classify facial expressions
emotional certainty of participant opinions. However, to perform in real time. FaceReader contains five different face models that
a statistical analysis according to indications shared in the are used to find the best fit for the face that is going to be analysed.
scientific literature, results from this project rely on raw data The models include: (1) “General,” the default face model; (2)
expressed in milliseconds recorded by the iCode online platform. “Children,” the face model for children between ages 3 and 10
Facial expressions of the participants were recorded during years old; (3) “East Asian,” the face model for people of East Asian
web page navigation and processed in post-test using FaceReader, descent (Zhi et al., 2017) e.g., Korean, Japanese, and/or Chinese;
version 8.1, fromNoldus (Noldus, 2014; Loijens and Krips, 2019). (4) “Elderly,” a model for participants 60 years of age and older;
Objective facial measurements were used to capture reactions (5) “Baby FaceReader,” different software for infants between ages
to website exposure (Den Uyl and Van Kuilenburg, 2005). This 6 and 24 months old. We set FaceReader to “General” for this
system uses a three-layer neural network that automatically study to account for the mean age (23) and nationality (Italian)
identifies and examines facial expressions of emotions in human of the participants.
beings (Den Uyl and Van Kuilenburg, 2005). It detects and We did not use FaceReader’s Participation or Group
classifies facial expressions both from pictures and videos into calibration. Instead, we used “in the wild,” or spontaneous,
one of the following basic emotions: happy, sad, angry, surprised, facial expression data to predict real-world consumer responses.
scared, disgusted, contempt, and neutral (Ekman, 1972). Facial Facial expressions, often caused by a mixture of emotions, can
expressions, like happiness, sadness, etc., are examined in occur simultaneously at high intensities (Loijens and Krips,
FaceReader on a frame-by-frame method. This is since basic 2019). Spontaneous facial expressions are, therefore, processed
emotions can usually be expressed in full within a single frame immediately after being recorded. This process works well for
(snapshot) of the face. However, there exist many more complex larger samples; thus, spontaneous facial expressions were ideal
affects, which are not completely expressed with a single instance for our study of 160 participants. Spontaneous expressions can
but rather, over a longer amount of time. These longer temporal also provide a benchmark for comparisons between different
facial affects are called “affective attitudes.” With the release algorithms (Küster et al., 2020). We relied on a minimum of
of FaceReader 7.1, the analysis of three commonly occurring 90% of accurate facial analysis through all FaceReader analyses
Frontiers in Psychology | www.frontiersin.org 6 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
detected for each participant: each participant has been exposed respond “yes” or “no” to each brand/association pair. According
to the website for a total of 3min, and his/her facial expressions to the model of Till et al. (2011), we consider the speed of
have been recorded for a total of 180 s. Only recordings that response as an implicit measure of the association strength:
FaceReader processed properly for at least 162 s were considered. the faster the response to the association, the stronger the
However, results from the 16 participants (seven from the group association. We also recorded the number of explicit responses
assigned to the Tesla website and nine from the group assigned to in terms of “yes” or “no,” as well as the speed of their responses,
the Ford website) were discarded due to the participants leaning which are defined as “response latency.” Our procedure was
into the camera, covering their faces, or otherwise interfering based on the “Brand Association Reaction Time Task” (BARTT)
with the tracking and expression analysis of the FaceReader. script provided by iCode, which enables measurement of the
We enrolled 16 additional participants, according to gender frequencies and reaction times of opinions of the participants as
characteristics and willingness to purchase a car within the next to whether or not words are associated with the target brands, as
trimester. In this way, the researchers have been able to make described in Till et al. (2011). Based on the theoretical perspective
up for those discarded and checked in real time the quality of described in the method section, our procedure was designed to
facial expression analysis to collect datasets of 80 participants for find out the associations that are part of the immediate network
each group. of a brand and to provide an analysis of those associations in
All the participants were guided by a researcher to complete terms of their frequency and strength; regarding the “frequency,”
the protocol steps of the study. Prior to completing the study, it is defined as “the number of mentions over the associations to
consent forms were sent to the participants via email. The the brand”: as shown by Teichert and Schöntag (2010), the more
implicit association pre-test was sent to the participants through respondents have similar associations, the higher the average
an email link. The following statements were used in the node strength. Relating to the “strength,” it is defined as follows:
pre-test: reliable (affidabile), passion (passione), I would like “the latency of response to the brand associations” (Sanbonmatsu
to own it (mi piacerebbe averla), comfortable (confortevole), and Fazio, 1990). The faster the subjects responded to the
innovative (innovativa), I would like to have it (mi piacerebbe target investigation, the stronger the association. For each brand
averla), electric (elettrica), safe (sicura), traditional (tradizionale), (TESLA and FORD), we first calculated the “Frequency of
affordable (accessibile). All statements were presented in Associations” (FoA) and, secondly, the “Strength of Associations”
association with the logo of the two brands: eighty participants (SoA). Only the “yes” answers were considered both for FoA and
undertook a “response latency” task, in which they were asked to SoA (Till et al., 2011).
FIGURE 1 | The graph shows continuous values of Happiness for both Ford and Tesla during Task 1 (exploration of the homepage for 30 s). The average levels of
happiness are higher during the exploration of the Tesla website compared with Ford. On the X axis, time is expressed in 30 samples per second for a total of 900
samples, corresponding to 30 s as the total amount of time exposure. On the Y axis, values of happiness are expressed between 0 (no happiness expressed by the
face) and 1 (highest intensity of happiness expressed by the face).
Frontiers in Psychology | www.frontiersin.org 7 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
FIGURE 2 | The graph shows values for Happiness for both Tesla and Ford during Task 2 (subjects were asked to look for characteristics of a specific car model). On
the X axis, time is expressed in 30 samples per second for a total of 1,800 samples, corresponding to 60 s as the total maximum amount of time exposure to the
website during this task. On the Y axes, values of happiness are expressed between 0 (no happiness expressed by the face) and 1 (highest intensity of happiness
expressed by the face).
The 160 participants were randomly assigned to either Ford customise a specified vehicle model. The participants randomly
or Tesla. About 80 participants navigated Ford’s website, and the assigned to Tesla designed a “Model 3,” while the participants
other 80 participants navigated Tesla’s website. randomly assigned to Ford customised a “Mustang Bullitt2”
The web page navigation process required the participants to (as shown in Figure 3). Task 4 asked both groups to envisage
complete four navigation steps. If a participant was unable to the need for assistance to find information on electric battery
complete a task within the allotted time, the task was marked as packs (as shown in Figure 4). For the task to be successful, the
failed. However, the participants were not penalised for failure participants were required to utilise the search bar. Thus, the
and could proceed to the next step. The participants would participants who found the information without the search bar
signal to the researcher that they are ready to begin the task by failed the task. All the participants from both groups were given
showing a thumbs-up or waving. The participants would then 30 s to complete Task 4.
use a webcam to record themselves completing the task. After After completing the four tasks, both groups were instructed
the participants completed the four navigation tasks, all video to complete the implicit association post-test. All 160 participants
recordings would be sent to the researcher for analysis. completed the same implicit association post-test, regardless of
Task 1, the first impression test, exposed each group to its their assigned company. During the post-test, the participants
randomly assigned website homepage for 30 s. The participants were asked to apply their perception of the two homepages after
were asked to only scroll and avoid clicking when interacting UX. The participants were asked if they associate the homepage of
with the homepage of the website (as shown in Figure 1). Task Tesla and/or Ford with the following perception characteristics:
2 gave participants 1min to look for a specific model and its Trust and Credibility, Easy Navigation, Pleasant Visual Design,
functional characteristics, such as acceleration, maximum speed, Promotion, Clear Information, and Assistance. All these items
efficiency, and price (as shown in Figure 2). The participants were chosen from the dimensions used in heuristic evaluation
randomly assigned to Tesla were asked to find the “Model X,” in order to compare results from both techniques, with the
and the participants randomly assigned to Ford were asked to exception of “Promotion,” which was selected as there is a strong
find the “New Explorer.1” The “Model X” and “New Explorer” difference between the FORD website, rich of promotions, and
are comparable in price. Task 3 gave the participants 1min to TESLA website, where there are no promotions. Screenshots of
1The “New Explorer” sold on the Ford Italy site is equivalent to the “2021 Explorer” 2 The Ford “Mustang Bullitt” is no longer in production as of January 31, 2021
sold on the U.S. site. (Foote, 2021).
Frontiers in Psychology | www.frontiersin.org 8 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
FIGURE 3 | The graph shows values for Happiness both for Tesla and Ford during Task 3 (subjects were asked to use the customising tool). On the X axis, time is
expressed in 30 samples per second for a total of 1,800 samples, corresponding to 60 s as the total maximum amount of time exposure to the websites for this task.
On the Y axes values of happiness are expressed between 0 (no happiness expressed by the face) and 1 (highest intensity of happiness expressed by the face).
FIGURE 4 | The graph shows values for Happiness for both Tesla and Ford during Task 3 (subjects were asked to use the customising tool). On the X axis, time is
expressed in 30 samples per second for a total of 900 samples, corresponding to 30 s as total maximum amount of time exposure to the websites for this task. On
the Y axis, values of happiness are expressed between 0 (no happiness expressed by the face) and 1 (highest intensity of happiness expressed by the face).
Frontiers in Psychology | www.frontiersin.org 9 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
FIGURE 5 | Results from the heuristic evaluation analysis performed by five judges as professional experts in the field of User Experience (UX). The heuristic evaluation
was performed in order to provide a qualitative evaluation about a traditional technique applied to assess UX; no statistical analysis was applied to the collected data.
The overall scores show that the Tesla website provides a better UX in comparison with the Ford website (Tesla scored 79%, while Ford 66%). In particular, the
dimensions of: “Home Page,” “Writing and Content Quality,” “Page Layout and Visual Design,” and “Task orientation” show the highest gap between the two websites.
the homepages were used to provide the participants with a visual prevention”), Navigation and Information Architecture (29
aid, and once again, they had to choose between a “yes” or “no” questions aimed to evaluate user navigation, correlating with
response in associating each item. the third principle from Nielsen’s heuristics, “user control and
The participants were then asked to complete a five-item freedom”), Forms and Data Entry (23 items, partially covering
questionnaire with a Likert scale of 9 points. The following the fourth Nielsen principle “consistency and standard” and
questions were asked: How did you evaluate the website that partially covering the fifth Nielsen principle “error prevention”);
you navigated (from ≪0≫ = negative evaluation; to ≪9≫ = Trust and Credibility (13 items, partially covering the first
positive evaluation)? To what extent did you like the homepage Nielsen principle “visibility of status”), Writing and Content
of the website (from≪0≫= I did not like it at all; to≪9≫= I Quality (23 items, partially covering the second Nielsen principle
liked it a lot)? In your opinion, was it easy to find characteristics “a match between system and real world”), Page Layout and
(such as max speed, acceleration, efficiency, etc.) and price (from Visual Design (38 items, covering the eight Nielsen principle
≪0≫ = very hard; to ≪9≫ = very easy)? Was it easy to “aesthetic and minimalist design”), Search (20 items, partially
use the car customisation tool (from ≪0≫ = very hard; to covering the seventh Nielsen principle “flexibility and efficiency
≪9≫ = very easy)? Was it easy to find the search bar for of use”), Help, Feedback, and Error Tolerance (37 items, covering
customer service/assistance (from ≪0≫ = very hard; to ≪9≫ the ninth Nielsen principle “helping users recognise, diagnose,
= very easy)? and recover from errors”). The heuristic evaluation portrays
Separate from the questionnaire, five expert professionals the qualitative assessment of UX by means of a well-established
in the field of ergonomics and UX performed the heuristic procedure (Nielsen and Molich, 1990) where each score is
evaluation from both websites (see Figure 5). This evaluation derived by a standardised procedure based on the answers to
helps to identify usability scores in the following dimensions: 247 questions, covering all the dimensions mentioned above,
Home Page (20 heuristics to evaluate the usability of the where professionals can choose one of the following “answers”:
homepage, partially covering the sixth Nielsen principle “+1” (that means the website respects the guidelines), “−1”
“recognition rather than recall” and partially covering the fourth (the website does not respect the guidelines), and “0” (The
Nielsen principle “consistency and standard”), Task Orientation website respects the guidelines in part only). These five expert
(44 items aimed to assess the ability of the website in supporting professionals did not participate in the IAT pre- and post-test,
the tasks of users, covering the fifth Nielsen principle “Error and their facial expressions were not recorded.
Frontiers in Psychology | www.frontiersin.org 10 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
TABLE 1 | The average values for Happiness and Confusion for both groups (Ford and Tesla) during the four navigation tasks.
Task 1 Task 2 Task 3 Task 4
Emotion Ford Tesla Ford Tesla Ford Tesla Ford Tesla
Happiness 0.0112* 0.0451* 0.0117* 0.0232* 0.0114* 0.0381* 0.0104* 0.0310*
Confusion 0.0098 0.0043 0.0088 0.0106 0.0139 0.0111 0.0039* 0.1001*
Values bolded with an asterisk indicate significant differences.
RESULTS
TABLE 2 | Frequency of Associations (FoA) expressed in percentage of the
number of “yes” answers over the total sample (80 subjects exposed to the Tesla
The main output of FaceReader classifies facial expressions from
website).
the participants according to intensity. Facial expressions are
valued between 0 and 1, where 0 denotes an absent expression, Ford Tesla Ford Tesla
and 1 indicates a fully present expression. FaceReader also (Pre-test) (Pre-test) (Post-test) (Post-test)
calculates valence, which indicates whether the emotional state (%) (%) (%) (%)
of each participant is positive (happy) or negative (sad, angry,
Reliable 96 97 98 88
or disgusted). Valence is equivalent to the intensity of positive
Passion 48 48 51 76
expression minus the highest intensity of the three negative
It makes me free 35 56 39 69
expressions. FaceReader calculated arousal, indicating whether
Comfortable 92 91 90 89
the participant is active (+1) or not active (0). Arousal is based
Innovative 35 96 38 99
on the activation of 20 Action Units (AUs) of the Facial Coding
I would like to have it 41 58 48 84
System (FACS).
Electric 35 95 37 99
First, the t-test (two-tailed) on results related to the automatic
detection of the facial expression of happiness as an emotional Safe 96 97 95 91
reaction during the navigation of the two websites showed Traditional 91 13 95 9
significant differences between the two groups (see Table 1): Accessible 97 25 97 26
for Task 1 (statistic = −2.50, p = 0.015), where Tesla
elicited significant higher emotional expressions of happiness
TABLE 3 | FoA expressed in percentage of the number of “yes” answers over the
in comparison to Ford website during the exploration of the
total sample (80 subjects exposed to the Ford website).
home page for the first 30 s; for Task 2 (statistic = −2.51,
p = 0.014), where Tesla website showed higher induction of Ford Tesla Ford Tesla
happier facial expressions in comparison to a website from Ford (Pre-test) (Pre-test) (Post-test) (Post-test)
while users explored the characteristics of cars models, such as (%) (%) (%) (%)
speed, acceleration, price, and so forth; for Task 3 (statistic =
Reliable 94 94 97 90
−2.04, p = 0.046), where Tesla website elicited higher emotional
Passion 51 79 48 80
facial expressions of happiness in comparison to Ford website
It makes me free 36 57 44 59
while user used the car-customising tool; for Task 4 (statistic =
Comfortable 90 90 96 92
−3.23, p = 0.002), where Tesla website induced increased facial
expressions of happiness in comparison to Ford website, while Innovative 34 93 55 94
users searched for the information related to electric recharge I would like to have it 46 61 47 60
of cars equipped with an electric battery pack. Additionally, the Electric 36 94 73 95
applied results show confusion as an emotional reaction during Safe 94 95 88 96
the exploration of both websites (Tesla vs. Ford). Concerning Traditional 95 11 87 10
the automatic detection of facial expressions, the t-test showed Accessible 96 23 88 25
significant differences between Tesla and Ford only for Task 4 No differences between the group made by Tesla and Ford users were observed.
(statistic=−2.81, p= 0.008).
Statistical analyses on reaction times were performed, as the innovative, I would like to have it, electric, safe, traditional, and
first step, regarding the FoA. Descriptive statistics for both accessible), and two brands (i.e., Tesla and Ford) recorded before
brands (Tesla vs. Ford) and relative associations (i.e., reliable, website experience and after website experience. No significant
passion, comfortable, innovative, I would like to have it, electric, results emerged from FoA dataset analyses.
safe, traditional, and accessible) were calculated and reported As a second step, statistical analyses were performed in
in Table 2 (for 80 subjects exposed to the Tesla website) and the SoA: in this case, only the “Yes” answers were considered
Table 3 (for 80 subjects exposed to the Ford website). The (when subjects choose “Yes” to express a positive association
dataset consisted of “Yes” answers from 160 participants,10 brand between a brand, either Tesla or Ford, and dimensions, namely:
associations (i.e., reliable, passion, it makes me free, comfortable, reliable, passion, it makes me free, comfortable, innovative, I
Frontiers in Psychology | www.frontiersin.org 11 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
TABLE 4 | Strength of Associations (SoA) expressed in milliseconds of the number TABLE 5 | SoA expressed in milliseconds of the number of “yes” answers over
of “yes” answers over the total sample (80 subjects exposed to the Ford website). the total sample (80 subjects exposed to the Tesla website).
Ford Tesla Ford Tesla Ford Tesla Ford Tesla
(Pre-test) (Pre-test) (Post-test) (Post-test) (Pre-test) (Pre-test) (Post-test) (Post-test)
Reliable 2,269 2,323 1,982 2,099 Reliable 2,201 2,361 2,039 2,157
(p = 0.046) (p = 0.014) (p = 0.043) (p = 0.049)
Passion 2,361 2,314 2,166 2,182 Passion 2,269 2,591 2,061 2,105
It makes me free 2,327 2,459 1,952 2,307 (p = 0.029)
(p = 0.039) It makes me free 2,477 2,548 2,186 2,218
Comfortable 2,379 2,351 2,120 2,241 (p = 0.026)
Innovative 2,358 2,139 2,109 2,113 Comfortable 2,282 2,351 2,187 2,143
(p = 0.031) (p = 0.032)
I would like to have it 2,419 2,297 2,198 2,167 Innovative 2,279 2,322 2,245 2,130
Electric 2,534 2,187 2,156 2,016 I would like to have it 2,239 2,521 2,113 2,124
(p 0.021) (p < 0.001)=
Safe 2,269 2,318 2,113 2,155 Electric 2,083 2,105 2,264 2,007
(p = 0.019) Safe 2,247 2,349 2,097 2,098
Traditional 2,369 2,041 2,165 1,943 Traditional 2,233 2,367 2,209 2,101
(p = 0.012) Accessible 2,196 2,313 2,309 2,109
Accessible 2,304 2,116 2,211 1,956
(p 0.047) Values in bold revealed a significant difference between pre- and post-tests.=
Values in bold designate a significant difference between pre- and post-tests.
deal with emotional reactions: “passion,” as a powerful feeling
barely controllable by rational thinking; “comfortable,” the Tesla
would like to have it, electric, safe, traditional, and accessible). website has been able to convey information related to a car that
Before proceeding with the analysis, we removed outliers that is more prone to providing physical ease and pleasant relaxation
were defined as response latencies below 300ms and above while using it; “I would like to have it” deals with the desire
3,000ms (Greenwald et al., 1998). No differences between the of owning that car, once again highlighting the feeling, worthy,
group made by Tesla or Ford users were observed. Outliers or unworthy, that impels to the attainment or possession of
were identified and removed according to the threshold, which something that is, in reality or in imagination, able to bring
is typically employed with analysis involving reaction times. satisfaction and/or enjoyment).
Table 4 shows the SoA for the 80 subjects exposed to the Ford Statistical analyses were performed on the collected data
website, while in Table 5, the 80 subjects were exposed to the on the short survey exposed after the website navigation, in
Tesla website. As a third step, t-test statistical analyses performed relation to the perception of both Tesla and Ford websites in
on SoA data from the Ford dataset for each association were terms of reaction time (the six items exposed were: “Trust
examined; analyses revealed significant differences between pre- and Credibility”; “Easy Navigation”; “Pleasant Visual Design”;
and post-test for the following associations: reliable, it makes me “Promotion”; “Clear Information”; “Assistance”). The t-test
free, innovative, electric, traditional, and accessible (as shown showed a significant difference between the two groups (Tesla
in Table 4). As a fourth step, a t-test performed on SoA data vs. Ford) for one item only: “Pleasant Visual Design,” where
from the Tesla dataset for each association was considered; the reaction time is faster for subjects who navigated the Tesla
results revealed significant differences between pre- and post- website in comparison to Ford (see Table 6). Finally, statistical
test for the following associations: reliable, passion, it makes analyses were performed in the last data collected concerning
me free, comfortable, I would like to have it (as shown in the short survey, exploring the judgments expressed by each
Table 5). As a fifth step of the analysis, a comparison has been participant who navigated the website about the navigation [the
considered between results from Ford and results from Tesla: five items investigated were: “Do you like the website?”; “Do you
the comparison shows that there are two associations shared by like the Homepage?”; “Was it easy to find car characteristics and
both brands: “reliable” and “it makes me free.” However, the two price?”; “Was it easy to use the customisation tool?”; “Was it easy
brands differ regarding all other associations. The experience on to find assistance (use of the search bar)?”]; The t-test showed
the Ford website has been able to increase the associations of: a significant difference between the two groups (Tesla vs. Ford)
“innovative,” “electric” (these two are related to technological for all items (see Table 7). Results from heuristic evaluations
issues), “traditional” (related to the perception of a brand performed by five different expert professionals show that, except
considered as a long-established presence in the automotive for the dimension of “Help, Feedback, and Error Tolerance,”
market), and “accessible” (perception of the Ford website as an where the two websites scored very similar values (74% for Ford
experience enabling to convey information, allowing to evaluate and 72% for Tesla), and except for the dimension of “Search,”
the brand as more affordable); while the experience on the Tesla where the Ford website scored on average a greater value in
website has been able to increase the associations of: “passion,” comparison to Tesla (69% for Ford and 53% for Tesla), all
“comfortable,” “I would like to have it” (all these three dimensions the other dimensions are showing, on average, a higher score
Frontiers in Psychology | www.frontiersin.org 12 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
TABLE 6 | The final survey values (expressed in milliseconds) about reaction time TABLE 8 | Results from heuristic evaluations expressed by means of average
expressed in milliseconds. percentage scores for each dimension (in bold, the highest differences between
the two website average scores).
Ford HP (Post-test) Tesla HP (Post-test)
Ford (%) Tesla (%) Difference (%)
Trust and credibility 2,362 2,430
Easy navigation 2,314 2,352 Home page 58 81 23
Pleasant visual design 2,441 2,171 (p 0.047) Task orientation 59 78 19=
Promotion 2,250 2,352 Navigation and information architecture (IA) 74 81 07
Clear information 2,263 2,198 Forms and data entry 65 79 14
Assistance 2,054 2,248 Trust and credibility 75 82 07
Writing and content quality 74 92 18
The reaction time is compared and analysed for each item. Values in bold indicate a
Page layout and visual design 61 92 31
significant difference.
Search 69 53 16
Help, feedback and error tolerance 74 72 02
TABLE 7 | Results from the final survey expressed by means of average scores Overall score 66 79 13
for each item (from 1 to 9).
Ford HP (Post-test) Tesla HP (Post-test) two groups, each exposed to one of the two websites of well-
known American brands in the automotive industry, reacted
Do you like the website? 6.5 7.3 (p < 0.001)
in a significantly different way for all the methods considered.
Do you like the Homepage? 6.4 7.9 (p < 0.001)
The Tesla website has been able to induce a stronger emotional
Was it easy to find car 6.0 6.9 (p = 0.022)
reaction, according to all results. In terms of facial expressions,
characteristics and price?
it elicited much higher expressions of happiness in all the
Was it easy to use the 6.1 7.2 (p < p < 0.001)
customisation tool? tasks performed. Taking into account the results from heuristic
Was it easy to find assistance 7.4 3.7 (p < 0.001) evaluation where average scores for “web layout and visual
(use of the Search bar)? design” and “homepage” are higher for the Tesla website in
comparison to the Ford website, and taking into account results
Values in bold revealed a significant difference between the two groups (80 subjects
from the self-reports from all the participants enrolled in the
navigated Ford website and 80 subjects Tesla website). At the end of website navigation,
all subjects were asked to fill in a brief online questionnaire, providing their responses by research projects, showing significant differences in favour of
means of a 9 point Likert scale (from 1 = “not at all”; to 9 = “A lot”). For instance, the first the Tesla website in comparison to the Ford website, together
question is: “Do you like the website?”. Subjects who navigated the Ford website scored with time reaction analyses for the item “Pleasant Visual Design”
on average 6.5 on a 9 points Likert scale, while the 80 subjects who navigated the Tesla
website scored on average 7.3 on the same 9 points Likert scale: there is almost one from the survey that displays significant faster response for
point of difference, revealing a significantly higher level of satisfaction for Tesla website in the Tesla website, it is possible to claim a greater emotional
comparison to Ford website. impact played by the Tesla website in comparison with Ford.
This pattern of better emotional performance is also supported
for Tesla in comparison to the Ford website (see Table 8); in by semantic dimensions investigated through reaction time
particular, the highest difference is for the dimension of “page technique too: they show that respondents perceived the Tesla
layout and visual design” (where Tesla scored, on average, 92% website as conveying information, enabling to change implicit
while Ford 61%); “Home Page” (where Tesla scored, on average, attitudes for “reliable,” “passion,” “freedom,” “comfortable,” and
81% while Ford 58%); “writing and content quality”(where Tesla “desire to own it.” Taken altogether, all these dimensions aremore
scored, on average, 92% while Ford 64%); “task orientation” related to emotion rather than functions or information about
(where Tesla scored, on average, 78% while Ford 59%). The car performances and prices. At the same time, results show
overall scores from heuristic evaluations indicate that the Tesla how participants are convinced Tesla is not a traditional brand
website seems to provide an overall better UX in comparison to and they do not believe it is an accessible car (as there are no
the Ford website (Tesla scored 79% while Ford 66%). significant differences for those two dimensions).
On the opposite, results from FORD show a less important
emotional impact, not only in terms of facial expressions related
DISCUSSION to happiness, always at a lower level in all tasks accomplished on
the FORD website but also for all other techniques considered.
The aim of this study was to examine whether the use of Heuristic evaluation from five expert professionals in the field or
automatic facial emotional expression analyses and reaction time UX showed, on average, a decreased score (with the exception of
methods may broaden the assessment of UX in young adults the items of “Search” and “Help, Feedback, and Error Tolerance”)
by using novel integration of techniques that combine a variety for the FORD website in comparison with the TESLA website.
of approaches based on self-report and heuristic evaluation The final survey showed significantly decreased Likert scale
coupled with software both for emotional facial detection and scores for all items in comparison with FORD, except for the
reaction time measurements recorded by means of an online “Search bar” (we will consider that specific issue later here in
quantitative procedure only. Data analysed indicated that the this section). Finally, the dimensions investigated by means of
Frontiers in Psychology | www.frontiersin.org 13 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
reaction time analyses reveal that FORD websites have been able love, such as brand trust, CX, psychological attachment, and
to convey information enabling to change implicit attitudes for hedonic value of the brand, identifying how brand love is a
“reliable,” “freedom,” “innovative,” “electric,” “traditional,” and strong indicator of a customer’s affective response to the brand
“accessible.” Except for the dimension of “freedom,” all other during the CX (Roy et al., 2013; Trivedi, 2019; Trivedi and
items are more related to information cognitively conveyed by Sama, 2020). Therefore, a possible interpretation of our results
the website: innovation, and electric are the best examples, as can rely on the moderation effects of brand love for TESLA,
the FORD website shows the latest innovation regarding the as shown by the ranking provided by Interbrand (https://www.
technology implemented in some models and the “electrification rankingthebrands.com) where TESLA has been able to gain 59
process” started by the company developed few hybrid models; positions in 2020 in comparison to FORD, whose position raised
in addition, in Task 3, the participants had to look for a model, only 20 points. Overall, the design of the two websites seems
the “Mustang Bullitt,” which also presented a version of a car to raise different emotional impacts: the TESLA website takes
model that is completely electric (the only one from FORD advantage of much more pictures and visual elements, as well
panorama of car models). The dimensions of “Traditional” and as of colours and “3D virtual tours” that may represent one of
“Accessible” are more related to the general brand perception the key elements, enabling a general greater emotional impact.
of an automotive organisation that appeared in the market a For instance, considering now Task 2, subjects were instructed
long time ago and to provide much more affordable models in to look for the Tesla “Model X.” We choose, by purpose, this
comparison to Tesla, even if the models selected within tasks model, as the landing page of this model, once loaded by the
accomplished by experimental subjects were chosen according to internet browser, showed in the upper part of the page a “3D
a similar placement (a similar price range). virtual tour” of the car from the front to the rear, with the
Considering the specific case represented by Task 4, it is peculiar doors opening like two “wings” of a seagull: the “3D
possible to evaluate the emotional impact played by two different virtual tour,” lasting 5 or 6 s and automatically starting once
design choices more related to “information architecture.” the webpage was opened, raised quite a big effect in terms of
TESLA shows the button “Assistance” as the 11th label of a emotional reactions (see graph in picture 3 or the Results section,
vertical menu completely hidden in a hamburger menu located where the level of happiness is much higher in comparison
on the top-right side of the homepage: a user has to identify it to FORD, especially in the beginning part of the task, when
(he/she has to know or understand that the three small horizontal actually, the participants were exposed to the “3D virtual tour”
lines on the top-right of the homepage are a sort of a small described). For the FORD website, where the model was asked
icon that represents a so-called “hamburger menu,” enabling to look for the “New Explorer,” this car model was presented
to explode a menu only once requested) and click to open it by means of a landing page with classic pictures, videos (that
on the right side of the screen. FORD shows the same call could start only after clicking; thus, they could be considered
to action directly in the upper side of the homepage, as the as additional pictures with the “play icon” in the middle, as
fourth label of a horizontal menu composed of four labels in none of the participants decided to start a video) and longer text
total, where the label “Assistance” is available at a first look. sections in comparison to the TESLA landing page. All emotional
Data collected show which one of the two design solutions is effects from these distinct elements and layouts are detected by
preferable for users; this time, FORD seems to perform much the different levels of happiness showing up on the faces of
better in comparison with TESLA. The heuristic evaluation the participants.
average scores show better results for this specific function, and Aside from the specific web contents and “information
the survey brings a significant positive preference for FORD in architecture” styles and designs, the aim of the present
comparison to TESLA. Facial expressions are presenting mixed research project was to show how emotional impact
findings: on one side, facial expressions in terms of happiness played by websites can be assessed by neuromarketing
are always much higher for TESLA, also for Task 4. On the techniques such as automatic facial emotion detection,
other side, confusion, one of the three new affective attitudes coupled with reaction time methods, which no previous
released by FaceReader 8.1, is showing significantly higher values research tried to investigate. With this work, it is possible
for TESLA in comparison to FORD, detecting the negative to show how these techniques can be efficiently applied to
impact raised by the seek for the “Assistance” label and the website evaluation and widening insights to understand and
mental efforts to find it. It may be possible to explain the gap assess UX.
between these two outputs from automatic facial expressions
analysis because happiness is a more general emotional reaction
in comparison to confusion (Rozin and Cohen, 2003; Grafsgaard CONCLUSION
et al., 2011): happiness enrols a greater number of AUs and
lasts for a shorter time in comparison to confusion, an affective In our study, the data collected by means of automatic
state that shows up for 2 up to 5 s. These findings can also be facial emotional expressions during website exploration
explained through the strong customer-brand relationship that and implicit association techniques applied before and after
follows under the concept of brand love (Huber et al., 2016) that web navigation evidenced how different design solutions
is “the degree of passionate, emotional attachment a consumer to shape UX. Moreover, it shows how the integration of
has for a particular trade name” (Carroll and Ahuvia, 2006). neuromarketing techniques with traditional ones may enhance
Trivedi and Sama (2020) categorised several antecedents of brand the understanding and evaluation of UX. These findings may
Frontiers in Psychology | www.frontiersin.org 14 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
have implications for developing new protocols for the user and AUTHOR CONTRIBUTIONS
usability testing.
All authors listed have made a substantial, direct and intellectual
DATA AVAILABILITY STATEMENT contribution to the work, and approved it for publication.
The raw data supporting the conclusions of this article will ACKNOWLEDGMENTS
be made available by the authors, with the permission of the
companies (SR LABS and NEUROHM) that contributed to the We thank all the participants in our study, as well as SR
research project realisation. LABS (https://www.srlabs.it), NEUROHM (https://neurohm.
com), Allegheny College, and the Catholic University of Milan.
ETHICS STATEMENT We finally want to thank Allegheny College students Victoria
Vradenburg and Danae Fowler for the literature and research
Ethical review and approval was not required for the study on support, and the students from the Catholic University of Milan
human participants in accordance with the local legislation and (Elisa Di Pietro, Giordana Iannuzzi, Giulia Bragaglio, Giulio
institutional requirements. The patients/participants provided Roberto Varrà, Laura Cappello, and Marta Fortusini) that helped
their written informed consent to participate in this study. with the gathering of all data collected.
REFERENCES Catani, M. B., and Biers, D. W. (1998). “Usability evaluation and prototype
fidelity: users and usability professionals,” in Proceedings of the Human Factors
Aaker, D. A. (2009).Managing Brand Equity. New York, NY: Simon and Schuster. and Ergonomics Society Annual Meeting, Vol. 42 (Los Angeles, CA: Sage
Andersen, R. A., Nasrollahi, K., Moeslund, T. B., and Haque, M. A. Publications), 1331–1335.
(2014). “Interfacing assessment using facial expression recognition,” in Cherubino, P., Martinez-Levy, A. C., Caratù, M., Cartocci, G., Di Flumeri,
2014 International Conference on Computer Vision Theory and Applications G., Modica, E., et al. (2019). Consumer behaviour through the eyes of
(VISAPP), Vol. 3, (Lisbon: IEEE), 186–193. neurophysiological measures: state-of-the-art and future trends.Comput. Intell.
Anderson, E. T., and Simester, D. (2013). Advertising in a competitive market: the Neurosci. 2019, 1976847. doi: 10.1155/2019/1976847
role of product standards, customer learning, and switching costs. J. Market. Clement, J., Kristensen, T., and Grønhaug, K. (2013). Understanding
Res. 50, 489–504. doi: 10.1509/jmr.11.0538 consumers’ in-store visual perception: the influence of package design
Bender, S. M., and Sung, B. (2021). Fright, attention, and joy while killing zombies features on visual attention. J. Retail. Consum. Serv. 20, 234–239.
in Virtual Reality: A psychophysiological analysis of VR user experience. doi: 10.1016/j.jretconser.2013.01.003
Psychol. Mark. 38, 937–947. doi: 10.1002/mar.21444 Cole, P. M. (1986). Children’s spontaneous control of facial expression. Child Dev.
Bercea, M. D. (2012). “Anatomy of methodologies for measuring consumer 57, 1309–1321. doi: 10.2307/1130411
behavior in neuromarketing research,” in Proceedings of the Lupcon Center for Danner, L., and Duerrschmid, K. (2018). “Automatic facial expressions
Business Research (LCBR) European Marketing Conference (Ebermannstadt). analysis in consumer science,” in Methods in Consumer Research, Vol. 2,
Bergstrom, J. R., and Schall, A. (eds.). (2014). Eye Tracking in User Experience eds G. Ares and P. Verela (Sawston: Woodhead Publishing), 231–252.
Design. San Francisco, CA: Elsevier. doi: 10.1016/B978-0-08-101743-2.00010-8
Bojko, A. (2013). Eye Tracking the User Experience: A Practical Guide to Research. Danner, L., Haindl, S., Joechl, M., and Duerrschmid, K. (2014). Facial
New York, NY: Rosenfeld Media. expressions and autonomous nervous system responses elicited by tasting
Bolton, R. N., McColl-Kennedy, J. R., Cheung, L., Gallan, A., Orsingher, different juices. Food Res. Int. 64, 81–90. doi: 10.1016/j.foodres.2014.
C., Witell, L., et al. (2018). Customer experience challenges: bringing 06.003
together digital, physical and social realms. J. Serv. Manage. 29, 776–808. de Wijk, R. A., Kooijman, V., Verhoeven, R. H., Holthuysen, N. T., and de Graaf,
doi: 10.1108/JOSM-04-2018-0113 C. (2012). Autonomic nervous system responses on and facial expressions to
Borges, N., Lindblom, L., Clarke, B., Gander, A., and Lowe, R. (2019). “Classifying the sight, smell, and taste of liked and disliked foods. Food Qual. Prefer. 26,
confusion: autodetection of communicative misunderstandings using facial 196–203. doi: 10.1016/j.foodqual.2012.04.015
action units,” in 2019 8th International Conference on Affective Computing Den Uyl, M. J., and Van Kuilenburg, H. (2005). “The FaceReader: online
and Intelligent Interaction Workshops and Demos (ACIIW) (Cambridge: IEEE), facial expression recognition,” in Proceedings of Measuring Behavior, Vol. 30
401–406. doi: 10.1109/ACIIW.2019.8925037 (Wageningen), 589–590.
Boudet, J., Gordon, J., Gregg, B., Perrey, J., and Robinson, K. (2020). How Djamasbi, S. (2014). Eye tracking and web experience. AIS Trans. Hum. Comput.
marketing leaders can both manage the coronavirus crisis and plan for the Interact. 6, 37–54. doi: 10.17705/1thci.00060
future. McKinsey. Available online at: https://www.mckinsey.com/business- Dupré, D., Krumhuber, E. G., Küster, D., and McKeown, G. J. (2020). A
functions/marketing-and-sales/our-insights/how-marketing-leaders-can- performance comparison of eight commercially available automatic
both-manage-the-coronavirus-crisis-and-plan-for-the-future (accessed classifiers for facial affect recognition. PLoS ONE 15, e0231968.
February 1, 2021). doi: 10.1371/journal.pone.0231968
Branco, P. S. O. (2006). Computer-Based Facial Expression Analysis for Assessing Ekman, P. (1972). “Universal and cultural differences in facial expression of
User Experience. Universidade do Minho, Braga. emotion,” in Nebraska Symposium on Motivation, Vol. 19 (Lincoln: University
Caldato, C., Benedetto, S., and Checchinato, F. (2020). Measuring the effects of of Nebraska Press), 207–284. doi: 10.1177/0022022196271003
video advertising on brand associations. Micro Macro Market. 20, 395–415. Ekman, P., and Cordaro, D. (2011).What is meant by calling emotions basic. Emot.
doi: 10.1431/96472 Rev. 3, 364–370. doi: 10.1177%2F1754073911410740
Camarrone, F., and Van Hulle, M. M. (2019). Measuring brand association El Haj, M., Antoine, P., and Nandrino, J. L. (2017). Facial expressions triggered by
strength with EEG: a single-trial N400 ERP study. PLoS ONE 14, e0217125. imagining the future. J. Integr. Neurosci. 16, 483–492. doi: 10.3233/JIN-170030
doi: 10.1371/journal.pone.0217125 Flight, R. L., and Coker, K. K. (2016). Brand constellations: reflections
Carroll, B. A., and Ahuvia, A. C. (2006). Some antecedents and outcomes of brand of the emotional self. J. Prod. Brand Manage. 25, 134–147.
love.Mark. Lett. 17, 79–89. doi: 10.1007/s11002-006-4219-2 doi: 10.1108/JPBM-02-2015-0806
Frontiers in Psychology | www.frontiersin.org 15 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
Foote, B. (2021). Last Ford Mustang Bullitt has Rolled Off the Assembly Line at and relationship duration,” in Let’s Get Engaged! Crossing the Threshold of
Flat Rock. Ford Authority. Available online at: https://fordauthority.com/2021/ Marketing’s Engagement Era, eds M.W. Obal, N. Krey, and C. Bushardt (Cham:
01/last-ford-mustang-bullitt-has-rolled-off-the-assembly-line-at-flat-rock/ Springer), 361–373. doi: 10.1007/978-3-319-11815-4_104
(accessed February15, 2021). iCode. (2019). iCode—Power of Confidence. Available online at: https://icodert.
Friesen, E., and Ekman, P. (1978). Facial action coding system: a technique for the com/ (accessed February 3, 2021).
measurement of facial movement. Palo Alto 3, 5. Izard, C. E. (1990). The substrates and functions of emotion feelings: William
Fu, B., Noy, N. F., and Storey, M. A. (2017). Eye tracking the user experience–an James and current emotion theory. Pers. Soc. Psychol. Bull. 16, 626–635.
evaluation of ontology visualization techniques. Semantic Web 8, 23–41. doi: 10.1177%2F0146167290164004
Gacioppo, J. T., and Petty, R. E. (1985). Physiological responses and advertising Janakiraman, R., Sismeiro, C., and Dutta, S. (2009). Perception spillovers across
effects: is the cup half full or half empty? Psychol. Market. 2, 115–126. competing brands: a disaggregate model of how and when. J. Market. Res. 46,
Gattol, V., Sääksjärvi, M., and Carbon, C. C. (2011). Extending the 467–481. doi: 10.1509%2Fjmkr.46.4.467
implicit association test (IAT): assessing consumer attitudes based Keller, K. L. (1993). Conceptualizing, measuring, and managing customer-based
on multi-dimensional implicit associations. PLoS ONE 6, e15849. brand equity. J. Mark. 57, 1–22. doi: 10.1177%2F002224299305700101
doi: 10.1371/journal.pone.0015849 Keller, K. L. (2003). Brand synthesis: the multidimensionality of brand knowledge.
Gauri, D. K., Harmon-Kizer, T. R., and Talukdar, D. (2016). Antecedents J. Consum. Res. 29, 595–600. doi: 10.1086/346254
and outcomes of market mavenism: insights based on survey and Klinčeková, S. (2016). Neuromarketing-research and prediction
purchase data. J. Bus. Res. 69, 1053–1060. doi: 10.1016/j.jbusres.2015. of the future. Int. J. Manag. Sci. Bus. Adm. 2, 53–57.
08.018 doi: 10.18775/ijmsba.1849-5664-5419.2014.22.1006
Gerhardt-Powals, J. (1996). Cognitive engineering principles for enhancing Kulczynski, A., Ilicic, J., and Baxter, S. M. (2016). When your source
human-computer performance. Int. J. Hum. Comput. Interact. 8, 189–211. is smiling, consumers may automatically smile with you: investigating
doi: 10.1080/10447319609526147 the source expressive display hypothesis. Psychol. Market. 33, 5–19.
Gonzalez-Holland, E., Whitmer, D., Moralez, L., and Mouloua, M. (2017). doi: 10.1002/mar.20857
“Examination of the use of Nielsen’s 10 usability heuristics and outlooks Küster, D., Krumhuber, E. G., Steinert, L., Ahuja, A., Baker, M., and Schultz, T.
for the future,” in Proceedings of the Human Factors and Ergonomics Society (2020). Opportunities and challenges for using automatic human affect analysis
Annual Meeting, Vol. 61. (Los Angeles, CA: SAGE Publications), 1472–1475. in consumer research. Front. Neurosci. 14. doi: 10.3389%2Ffnins.2020.00400
doi: 10.1177/1541931213601853 Lee, W. J. (2020). Unravelling consumer responses to omni-channel
Grafsgaard, J. F., Boyer, K. E., and Lester, J. C. (2011). “Predicting facial indicators approach. J. Theoret. Appl. Electron. Commerce Res. 15, 37–49.
of confusion with hidden Markov models,” in International Conference doi: 10.4067/S0718-18762020000300104
on Affective Computing and Intelligent Interaction (Berlin; Heidelberg: Lemon, K. N., and Verhoef, P. C. (2016). Understanding customer experience
Springer), 97–106. throughout the customer journey. J. Mark. 80, 69–96. doi: 10.1509/jm.15.0420
Greenwald, A. G., McGhee, D. E., and Schwartz, J. L. (1998). Measuring individual Lewinski, P. (2015). Automated facial coding software outperforms people in
differences in implicit cognition: the implicit association test. J. Pers. Soc. recognizing neutral faces as neutral from standardized datasets. Front. Psychol.
Psychol. 74:1464. 6:1386. doi: 10.3389/fpsyg.2015.01386
Gregg, A. P., and Klymowsky, J. (2013). The implicit association test in Lewinski, P., den Uyl, T. M., and Butler, C. (2014a). Automated facial coding:
market research: potentials and pitfalls. Psychol. Market. 30, 588–601. validation of basic emotions and FACS AUs in FaceReader. J. Neurosci. Psychol.
doi: 10.1002/mar.20630 Econ. 7, 227–236. doi: 10.1037/npe0000028
Grewal, D., Motyka, S., and Levy, M. (2018). The evolution and future Lewinski, P., Fransen, M. L., and Tan, E. S. (2014b). Predicting advertising
of retailing and retailing education. J. Market. Educ. 40, 85–93. effectiveness by facial expressions in response to amusing persuasive stimuli.
doi: 10.1177/0273475318755838 J. Neurosci. Psychol. Econ. 7, 1–14. doi: 10.1037/npe0000012
Gross, J. J. (2014). “Emotion regulation: conceptual and empirical foundations,” in Liu, X., and Lee, K. (2018). “Optimized facial emotion recognition technique
Handbook of Emotion Regulation, ed J. J. Gross (New York, NY: The Guilford for assessing user experience,” in 2018 IEEE Games, Entertainment, Media
Press), 3–20. Conference (GEM) (Galway: IEEE), 1–9. doi: 10.1109/GEM.2018.8516518
Gross, J. J., and Thompson, R. A. (2007). “Emotion regulation: conceptual Loijens, L., and Krips, O. (2019). FaceReader Methodology Note. Leesburg, VA:
foundations,” in Handbook of Emotion Regulation, ed J. J. Gross (The Guilford Noldus Information Technology Inc.
Press), 3–24. Martini, U., Buffa, F., and Caporusso, L. (2016). Celebrity endorsement and
Hamelin, N., El Moujahid, O., and Thaichon, P. (2017). Emotion and advertising brand equity in outdoor-mountaineering. Micro Macro Market. 25, 357–378.
effectiveness: a novel facial expression analysis approach. J. Retail. Consum. doi: 10.1431/84720
Serv. 36, 103–111. doi: 10.1016/j.jretconser.2017.01.001 Matukin, M., and Ohme, R. (2017). “Implicit measures of attitudes in market
Hancock, P. A., Pepe, A. A., and Murphy, L. L. (2005). Hedonomics: the research,” in eds Ethics and Neuromarketing, A. Thomas, N. Pop, A. Iorga, and
power of positive and pleasurable ergonomics. Ergon. Des. 13, 8–14. C. Ducu (Cham: Springer), 113–122.
doi: 10.1177%2F106480460501300104 Matukin, M., Ohme, R., and Boshoff, C. (2016). Toward a better understanding
Harris, P., Pol, H., and van der Veen, G. (2020). “Customer journey,” The of advertising stimuli processing: exploring the link between consumers’
Routledge Companion to Strategic Marketing, eds B. B. Schlegelmilch and R. eye fixation and their subconscious responses. J. Advert. Res. 56,
S. Winer (New York, NY: Routledge), 67. 205–216.doi: 10.2501/JAR-2016-017
Hassenzahl, M. (2004). Emotions can be quite ephemeral; we cannot design them. Mele, M. L., and Federici, S. (2012). A psychotechnological review on eye-tracking
Interactions 11, 46–48. doi: 10.1145/1015530.1015551 systems: towards user experience. Disabil. Rehabil. Assist. Technol. 7, 261–281.
Hazlett, R. (2003). “Measurement of user frustration: a biologic approach,” in doi: 10.3109/17483107.2011.635326
CHI’03 Extended Abstracts on Human Factors in Computing Systems (Ft. Meng, Q., Hu, X., Kang, J., and Wu, Y. (2020). On the effectiveness
Lauderdale, FL),734–735. doi: 10.1145/765891.765958 of facial expression recognition for evaluation of urban sound
He, W., Boesveldt, S., De Graaf, C., and De Wijk, R. A. (2012). Behavioural perception. Sci. Total Environ. 710:135484. doi: 10.1016/j.scitotenv.2019.
and physiological responses to two food odours. Appetite 59, 628. 135484
doi: 10.1016/j.appet.2012.05.071 Missaglia, A. L., Oppo, A., Mauri, M., Ghiringhelli, B., Ciceri, A., and Russo, V.
Hirt, F., Werlen, E., Moser, I., and Bergamin, P. (2019). Measuring emotions (2017). The impact of emotions on recall: an empirical study on social ads. J.
during learning: lack of coherence between automated facial emotion Consum. Behav. 16, 424–433. doi: 10.1002/cb.1642
recognition and emotional experience. Open Comput. Sci. 9, 308–317. Munim, K. M., Islam, I., Khatun, M., Karim, M. M., and Islam, M. N. (2017).
doi: 10.1515/comp-2019-0020 “Towards developing a tool for UX evaluation using facial expression,” in 2017
Huber, F., Meyer, F., and Schmid, D. (2016). “Creating passionate love through 3rd International Conference on Electrical Information and Communication
rational value: a moderated mediation analysis considering identification Technology (EICT) (Khulna: IEEE), 1–6. doi: 10.1109/EICT.2017.8275227
Frontiers in Psychology | www.frontiersin.org 16 October 2021 | Volume 12 | Article 674159
Mauri et al. Measuring Emotions Is User Experience
Nielsen, J. (1992). “Finding usability problems through heuristic evaluation,” in Songa, G., Slabbinck, H., Vermeir, I., and Russo, V. (2019). How do
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems implicit/explicit attitudes and emotional reactions to sustainable logo
(Montréal, QC), 373–380. doi: 10.1145/142750.142834 relate? A neurophysiological study. Food Qual. Prefer. 71, 485–496.
Nielsen, J. (1994a). “Enhancing the explanatory power of usability heuristics,” in doi: 10.1016/j.foodqual.2018.04.008
Proc. ACM CHI’94 Conf. (Boston, MA), 152–158. doi: 10.1145/191666.191729 Stasi, A., Songa, G., Mauri, M., Ciceri, A., Diotallevi, F., Nardone, G.,
Nielsen, J. (1994b). “Heuristic evaluation,” in Usability Inspection Methods, eds et al. (2018). Neuromarketing empirical approaches and food choice: a
J. Nielsen and R. L. Mack (New York, NY: John Wiley and Sons), 173–202. systematic review. Food Res. Int. 108, 650–664. doi: 10.1016/j.foodres.2017.
doi: 10.1016/B978-0-08-051574-8.50020-0 11.049
Nielsen, J., and Molich, R. (1990). “Heuristic evaluation of user interfaces,” in Stewart, D. W., Hecker, S., and Graham, J. L. (1987). It’s more than what you
Proceedings of the SIGCHI Conference on Human Factors in Computing say: assessing the influence of nonverbal communication in marketing. Psychol.
Systems, (Gaithersburg, MD), 249–256. doi: 10.1145/97243.97281 Market. 4, 303–323.
Nielsen, J., and Norman, D. (2021). The Definition of User Experience (UX). Stöckli, S., Schulte-Mecklenbeck, M., Borer, S., and Samson, A. C. (2018). Facial
Nielsen Norman Group. Available online at: https://www.nngroup.com/ expression analysis with AFFDEX and FACET: a validation study. Behav. Res.
articles/definition-user-experience/ (accessed February 1, 2021). Methods 50, 1446–1460. doi: 10.3758/s13428-017-0996-1
Nielsen, J., and Pernice, K. (2010). Eyetracking Web Usability. San Francisco, CA: Teichert, T. A., and Schöntag, K. (2010). Exploring consumer knowledge
New Riders. structures using associative network analysis. Psychol. Market. 27, 369–398.
Noldus, L. P. J. J. (2014). FaceReader: Tool for Automated Analysis of Facial doi: 10.1002/mar.20332
Expression: Version 6.0. Wageningen: Noldus Information. Terzis, V., Moridis, C. N., and Economides, A. A. (2013). Measuring instant
Noordewier, M. K., and van Dijk, E. (2019). Surprise: unfolding of facial emotions based on facial expressions during computer-based assessment. Pers.
expressions. Cogn. Emot. 33, 915–930. doi: 10.1080/02699931.2018.1517730 Ubiquit. Comput. 17, 43–52. doi: 10.1007/s00779-011-0477-y
Norman, D. (2013). TheDesign of Everyday Things: Revised and Expanded Edition. Till, B. D., Baack, D., and Waterman, B. (2011). Strategic brand association
New York, NY: Basic Books. maps: developing brand insight. J. Prod. Brand Manage. 20, 92–100.
Norman, D. A. (2004). Emotional Design: WhyWe Love (or hate) Everyday Things. doi: 10.1108/10610421111121080
New York, NY: Basic Civitas Books. Travis, D. (2017). E-Commerce Usability: Tools and Techniques to Perfect the
Park, J., Spence, C., Ishii, H., and Togawa, T. (2021). Turning the other cheek: On-Line Experience. CRC Press.
facial orientation influences both model attractiveness and product evaluation. Trivedi, J. (2019). Examining the customer experience of using banking Chatbots
Psychol. Market. 38, 7–20. doi: 10.1002/mar.21398 and its impact on brand love: the moderating role of perceived risk. J. Intern.
Pettersson, I., Lachner, F., Frison, A. K., Riener, A., and Butz, A. (2018). “A Commer. 18, 91–111. doi: 10.1080/15332861.2019.1567188
bermuda triangle? A review of method application and triangulation in Trivedi, J., and Sama, R. (2020). Determinants of consumer loyalty towards
user experience evaluation,” in Proceedings of the 2018 CHI Conference celebrity-owned restaurants: the mediating role of brand love. J. Consum.
on Human Factors in Computing Systems (Montreal, QC), 1–16. Behav. 20, 748–761. doi: 10.1002/cb.1903
doi: 10.1145/3173574.3174035 Van Kuilenburg, H., Wiering, M., and Den Uyl, M. (2005). “A model based
Puccinelli, N. M., Motyka, S., and Grewal, D. (2010). Can you trust a customer’s method for automatic facial expression recognition,” in European Conference
expression? Insights into nonverbal communication in the retail context. on Machine Learning (Berlin; Heidelberg: Springer), 194–205.
Psychol. Market. 27, 964–988. doi: 10.1002/mar.20368 Van Osselaer, S. M., and Janiszewski, C. (2001). Two ways of learning brand
Quaglini, S. (2020). Applicazione di Tecniche Neuroscientifiche Alla Valutazione associations. J. Consum. Res. 28, 202–223. doi: 10.1086/322898
Dell’esperienza Utente: Il Caso Della Navigazione Dei Siti del Settore Verhulst, N., De Keyser, A., Gustafsson, A., Shams, P., and Van Vaerenbergh, Y.
Automobilistico. Doctoral Dissertation, Università’ Cattolica del Sacro Cuore. (2019). Neuroscience in service research: an overview and discussion of its
ProQuest Dissertation Publishing. possibilities. J. Serv. Manage. 30, 621–649. doi: 10.1108/JOSM-05-2019-0135
Riem, M. M., and Karreman, A. (2019). Experimental manipulation of emotion Vermeeren, A. P., Law, E. L. C., Roto, V., Obrist, M., Hoonhout, J., and Väänänen-
regulation changes mothers’ physiological and facial expressive responses to Vainio-Mattila, K. (2010). “User experience evaluation methods: current state
infant crying. Infant Behav. Dev. 55, 22–31. doi: 10.1016/j.infbeh.2019.02.003 and development needs,” in Proceedings of the 6th Nordic Conference on
Roy, S. K., Eshghi, A., and Sarkar, A. (2013). Antecedents and consequences of Human-Computer Interaction: Extending Boundaries (Reykjavik), 521–530.
brand love. J. Brand Manage. 20, 325–332. doi: 10.1057/bm.2012.24 doi: 10.1145/1868914.1868973
Rozin, P., and Cohen, A. B. (2003). High frequency of facial expressions Yi, Q., Xiong, S., Wang, B., and Yi, S. (2020). Identification of trusted interactive
corresponding to confusion, concentration, and worry in an analysis behavior based on mouse behavior considering web User’s emotions. Int. J. Ind.
of naturally occurring facial expressions of Americans. Emotion 3, 68. Ergon. 76, 102903. doi: 10.1016/j.ergon.2019.102903
doi: 10.1037/1528-3542.3.1.68 Zhi, R., Cao, L., and Cao, G. (2017). Asians’ facial responsiveness to basic tastes
Salgado-Montejo, A., Tapia Leon, I., Elliot, A. J., Salgado, C. J., and Spence, C. by automated facial expressions analysis system. J. Food Sci. 82, 794–806.
(2015). Smiles over frowns: when curved lines influence product preference. doi: 10.1111/1750-3841.13611
Psychol. Market. 32, 771–781. doi: 10.1002/mar.20817
Sanbonmatsu, D. M., and Fazio, R. H. (1990). The role of attitudes Conflict of Interest: The authors declare that the research was conducted in the
in memory-based decision making. J. Pers. Soc. Psychol. 59, 614. absence of any commercial or financial relationships that could be construed as a
doi: 10.1037/0022-3514.59.4.614 potential conflict of interest.
Schnittka, O., Sattler, H., and Zenker, S. (2012). Advanced brand concept maps: a
new approach for evaluating the favorability of brand association networks. Int. Publisher’s Note: All claims expressed in this article are solely those of the authors
J. Res. Market. 29, 265–274. doi: 10.1016/j.ijresmar.2012.04.002 and do not necessarily represent those of their affiliated organizations, or those of
Schreiner, M., Fischer, T., and Riedl, R. (2019). Impact of content
the publisher, the editors and the reviewers. Any product that may be evaluated in
characteristics and emotion on behavioral engagement in social media:
this article, or claim that may be made by its manufacturer, is not guaranteed or
literature review and research agenda. Electron. Commerce Res. 21, 1–17.
doi: 10.1007/s10660-019-09353-8 endorsed by the publisher.
Sheth, J. (2021). New areas of research in marketing strategy, consumer behavior,
and marketing analytics: the future is bright. J. Market. Theory Pract. 29, 3–12. Copyright © 2021 Mauri, Rancati, Gaggioli and Riva. This is an open-access article
doi: 10.1080/10696679.2020.1860679 distributed under the terms of the Creative Commons Attribution License (CC BY).
Small, D. A., and Verrochi, N. M. (2009). The face of need: facial The use, distribution or reproduction in other forums is permitted, provided the
emotion expression on charity advertisements. J. Market. Res. 46, 777–787. original author(s) and the copyright owner(s) are credited and that the original
doi: 10.1509%2Fjmkr.46.6.777_JMR6F publication in this journal is cited, in accordance with accepted academic practice.
Smith, S. L., and Mosier, J. N. (1986). Guidelines for Designing User Interface No use, distribution or reproduction is permitted which does not comply with these
Software. Bedford, MA: Mitre Corporation. terms.
Frontiers in Psychology | www.frontiersin.org 17 October 2021 | Volume 12 | Article 674159