site-logo

JAWS PANKRATION 2024

site-logo
HomeNewsTimetableCfPCommitteePromotionFollow UpPrivacy Policy

How to Streamline Help Desk Operations and Improve RAG Response Accuracy with Amazon Bedrock

Lv300

Lv300

8/25/2024 01:00 (UTC)

Session Info

This presentation explores how we revolutionized our cloud service help desk operations using Amazon Bedrock and RAG (Retrieval-Augmented Generation) technology.

 

We'll discuss the challenges faced by our help desk, including time-consuming document searches and unnecessary ticket escalations to managers.

Our solution: a serverless RAG Chatbot (Web Application) built on AWS, leveraging Amazon Bedrock and Knowledge Base feature.

This system integrates various internal documents, service specifications, past user interactions to provide context-aware responses.

 

We'll talk about the technical aspects of our implementation, including:

- Maintaining conversation history to enable contextual understanding

- Building a data pipeline for personal information masking

- Enhancing RAG response accuracy

 

We'll also address business-related challenges following implementation:

- Driving user adoption and change management within the help desk team

- Setting business goals , KPIs and metrics

 

Key takeaways include:

- The ease and speed of developing serverless RAG chatbots with AWS

- Best practices for applying RAG in help desk operations

- Techniques for improving RAG response accuracy

Kohei  Gamo

Kohei Gamo

- AWS Ambassadors(APN) -



Session Category
Machine learning


AWS Services
Bedrock
Glue
CDK

Session Materials


Session Summary (by Amazon Bedrock)
    The speaker discusses the optimization of helpdesk operations using AI chatbots and Bedrock. They introduce themselves as an AWS ambassador working on internal CCOE helpdesk. Key points: 1. Challenges in helpdesk operations: - New staff: Time-consuming to find answers and create responses - Experienced staff: Complex queries require gathering information and considering conditions 2. AI chatbot solution: - Web application using AWS services - Incorporates various data sources (manuals, past interactions, internal regulations) - Serverless architecture with Bedrock for LLM functionality 3. Design considerations: - DynamoDB for conversation history - Using different regions for backend and Bedrock - Data pipeline for PII removal 4. Implementation tips: - Create quality datasets for evaluation - Involve domain experts early - Design with KPIs in mind - Measure user adoption, AI response accuracy, and business efficiency 5. Data pipeline importance: - Utilizing past Q&A interactions as data source - Removing personal information from data 6. Improving chatbot accuracy: - Quantitative evaluation using frameworks like RAGAS - Identifying bottlenecks in the question-search-answer process - Applying targeted solutions based on identified issues The speaker emphasizes the importance of creating evaluation datasets, setting KPIs, incorporating past Q&A data, and continuously improving accuracy for successful implementation of AI chatbots in helpdesk operations.

©JAWS-UG (AWS User Group - Japan). All rights reserved.