ISO/IEC JTC 1 SC 42 Artificial Intelligence - Working Group 4
Use Cases & Applications   04/23/2024

Editor's comments and enhancements are shown in green. [✓ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 42 ID: Use Case Name: Emotion-sensitive AI Customer Service
Application
Domain		Retail
Deployment
Model		On-premise systems
StatusIn operation
ScopeExtracting sentiment and its intensity from customers’ input, and responding with appropriate attitude in order to improve the quality of customers’ inquiry.
Objective(s)To design an efficient solution for customers’ sentiment and intensity detection, especially in the situation of limited training dataset.
Short
Description
(up to
150 words) 		The emotion-sensitive AI customer service of JD.com Int., is supported by AI technology and deep learning method. It is developed for ameliorating accuracy of customer sentiment and intensity. In sentiment classification, it has achieved 74% accuracy and 90% recall score while in intensity detection, it has accomplished 85% accuracy and 85% recall. During the special sale of -618-, it has increased customer satisfaction by 57%.
Complete Description JD’s customer service representatives need to handle millions of requests on a daily basis. Regular AI customer service systems, 24/7 online, are capable of offering instant assistance, which alleviates the labor resources to a large extent. However, it is quite challenging, if not impossible, for those systems to interpret emotions from customer input and respond as friendly as human.
Under this background, based on huge data set of customer comments and rich experience of Natural Language Processing, our system can automatically detect sentiments like happy, angry, anxious, etc. Moreover, this system can also detect the intensity of customer sentiment. Furthermore, we adapt Convolutional Neural Networks, a widely used techniques in visual computing, to interpret the semantic meaning of customer’s expression. It can improve the system’s performance for sentiment classification and intensity detection. Moreover, with the adoption of transfer learning, the system can also be applied into various types of data. To overcome the difficulty of limited training data, we also use data augmentation method such as reverse translation and data noise to increase the variability of training data.
Up to now, the system has reached 90% recall and 74% accuracy rate for sentiment classification over 7 categories. The overall recall and accuracy for sentiment intensity are also around 85%?it has increased customer satisfaction by 57%.
StakeholdersCustomers targeted for the Customer Service system
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities		The low degree of humanization, and lack of semantic diversity for response. Reducing the number of human customer service.
Performance
Indicators (KPIs) 		Seq. No. Name Description Reference to mentioned
use case objectives
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
AI Features Task(s)Natural language processing
Method(s)Deep learning, transfer learning, data augmentation
Hardware
Topology
Terms &
Concepts Used		Deep learning: a class of machine learning algorithms use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation.
Transfer learning: we adopt multi-task learning method in this system. Jointly training different annotated data in same domain, this method improves the model performance for classification problems.
Data augmentation: we apply reverse translation to firstly translation Chinese into English and then translate it backward. We also use data noise to improve the data diversity.
Standardization
Opportunities
Requirements		The system can be promoted to as many customer cervices companies as possible once provide with enough training data for the specific Application scenario
Challenges
& Issues		Challenge: the system’s performance should be as good as the human customer server.
Issues: 1) limited training data; 2) sentiment classification among seven categories.
Societal Concerns Description Improving the corresponding efficiency of customer service, improving customer service experience: Reducing labor costs, and reducing operating costs.
SDGs to
be achieved 		Industry, Innovation, and Infrastructure
Data Characteristics
Description For sentiment classification: conversation data from after-sales customer services. It’s annotated by professional annotators into 7 categories of sentiments.
For sentiment intensity: Only including sentiment data with “anger” and “anxious”; it’s annotated into 3 degrees of intensity: “low, medium, high”.
Source Conversation data from JD.com real-time customer services
Type Text
Volume (size) Around 60,000 sentences for sentiment classification and 20,000 for sentiment intensity.
Velocity Batch Processing
Variety Real-time data from JD.com, including various categories of products.
Variability
(rate of change) 		Static
Quality High
Scenario Conditions
No. Scenario
Name 		Scenario
Description 		Triggering Event Pre-condition Post-Condition
1 Data Augmentation Using reverse translation and noise processing to increase the size and diversity of data. Increase the performance of model training.
2 Model Training Based on the large training data, with deep learning method, to develop model for sentiment classification (7 categories) or sentiment intensity (3 categories).
3 Evaluation Evaluate data performance on open dataset and specific data.
4 Execution Apply the trained model on real-time AI customer service. The trained model has been evaluated as deployable
Trainng Scenario Name:  
Step No. Event Name of
Process/Activity 		Primary
Actor 		Description of
Process/Activity 		Requirement
1 Complete data augmentation Design model for training AI algorithm engineers Using CNN for sentiment classification and intensity.
2 Complete model designing Transfer learning AI algorithm engineers Multi-task learning with different data in same domain.
Specification of training data
Scenario Name Evaluation
Step No. Event Name of
Process/Activity 		Primary
Actor 		Description of
Process/Activity 		Requirement
1 Complete model training Evaluation on open dataset AI algorithm engineers Evaluate different models’ performance on open dataset
2 Complete model training Evaluation on own dataset AI algorithm engineers Evaluate different models’ performance on own dataset
Input of Evaluation Independent testing data
Output of Evaluation
Scenario Name Execution
Step No. Event Name of
Process/Activity 		Primary
Actor 		Description of
Process/Activity 		Requirement
1 Finish model training Application AI engineers Application
2 Given customer’s input Data processing AI algorithm engineers Data processing
3 Finish data processing Model prediction AI algorithm engineers Model prediction
4 Completion of Step3 Making response AI algorithm engineers Making response
Input of Execution
Output of Execution
Scenario Name Retraining
Step No. Event Name of
Process/Activity 		Primary
Actor 		Description of
Process/Activity 		Requirement
1 Certain period of time has passed since the last training/retrainig Improve architecture of model AI algorithm engineers
2 Certain period of time has passed since the last training/retrainig Collecting new data AI algorithm engineers
3 Completing Step1&Step2 Model retraining AI algorithm engineers
Specification of retraining data
References
No. Type Reference Status Impact of
use case 		Originator
Organization 		Link
1 IT company XiaoIce In operation Microsoft Asia

  • Peer-reviewed scientific/technical publications on AI applications (e.g. [1]).
  • Patent documents describing AI solutions (e.g. [2], [3]).
  • Technical reports or presentations by renowned AI experts (e.g. [4])
  • High quality company whitepapers and presentations
  • Publicly accessible sources with sufficient detail

    This list is not exhaustive. Other credible sources may be acceptable as well.

    Examples of credible sources:

    [1] B. Du Boulay. "Artificial Intelligence as an Effective Classroom Assistant". IEEE Intelligent Systems, V 31, p.76-81. 2016.

    [2] S. Hong. "Artificial intelligence audio apparatus and operation method thereof". N US 9,948,764, Available at: https://patents.google.com/patent/US20150120618A1/en. 2018.

    [3] M.R. Sumner, B.J. Newendorp and R.M. Orr. "Structured dictation using intelligent automated assistants". N US 9,865,280, 2018.

    [4] J. Hendler, S. Ellis, K. McGuire, N. Negedley, A. Weinstock, M. Klawonn and D. Burns. "WATSON@RPI, Technical Project Review".
    URL: https://www.slideshare.net/jahendler/watson-summer-review82013final. 2013