🍃Prompt Optimization
The Garden Bank
We have a Garden Bank famous for exceptional fund management across the cosmos. With our stellar reputation for security, convenience, and efficiency, along with appealing investment opportunities, clients from every corner of the universe are eager to utilize our banking services.
Our bank is among the most efficient in the universe. However, due to a large number of customers and diverse services, we have numerous procedures to address all customer needs. To expedite this, we've developed a chatbot for our bank employees. Utilizing RAG, it offers quicker responses to customer inquiries.
About the Data
To protect bank customers' privacy, how can we show their real data? All the data came from Hugging Face's financial Q&A data: a small dataset and a bigger dataset. For the Bigger Financial Data, its samples selected from 100,629 training records and 6,421 testing records, guess why here didn't use "big data"? 😉 Find the answers in this trip!
In both datasets
question: is user query.
context: is bank's document that contains the correct answer.
The "correct answer"
ground_truth in Small Financial Data is the correct answer for the user query.
answer in Bigger Financial Data is the correct answer for the user query.
The "answer_before_prompt_opt"
It only exists in Small Financial Data. This column can be compared with answers generated after prompt optimization. But without this column, we can still evaluate prompt optimization's performance using a set of testing data.
So, both datasets have user query, context and the correct answer.
🌻 To get the Small Financial Data >>
🌻 To get the Bigger Financial Data >>
About RAG
RAG stands for Retrieval-Augmented Generation. When a customer request is received, bank staffs can enter a query into the chatbot. The chatbot utilizes RAG to retrieve relevant information from our system and generate a helpful response, guiding our staffs on how to proceed.
There are 2 steps in RAG:
Information Retrieval
Answer Generation
Why Prompt Optimization
With RAG, after information retrieval, a prompt is used for answer generation. To handle large numbers of queries effectively, prompt optimization allows us to improve this prompt in order to enhance the overall answer generation performance.
Prompt Optimization Tools, or Politics? 🙃
When considering prompt optimization, you may find various Python open source libraries available. However, a key risk associated with using these libraries is their stability. New libraries often undergo frequent changes, requiring users to constantly update their code. Conversely, libraries lacking active maintenance might not adapt to new Python updates, leading to compatibility issues.
After reviewing a bunch of promising open source libraries of prompt optimization, Lady H. opted for DSPy because of its mature framework, active upkeep, and robust support for structured AI pipelines.
At the time Lady H. decided to experiment with DSPy, the Cosmos Banking Union sent her a message claiming that AdalFlow was the better choice for cost and performance. The statement came from a group of people who are lack of hands-on experience but held enough power to influence every bank in the cosmos. Lady H. never liked them, she actually found them quite annoying. Still, if she ignored AdalFlow, the union might lower our Garden Bank’s score and damage its reputation. So, Lady H. had to experiment with both DSPy and AdalFlow 😐. How will this comparison turn out? Follow us to find out!
Last updated