QBRAG (QnA Boosted RAG) with Vectara

A company’s knowledge base often doesn’t answer the wide variety of questions a user could come up with. A customer support system ideally could answer specific (yet diverse) questions about the company’s systems and knowledge. Example: “How can I enter Cash Flow in ThruThink?” But sometimes the user asks generic questions, such as “What is Cash Flow?”, which could be sourced from the mind of a giant LLM and/or the internet.

My submission is essentially a synthetic data enrichment of the knowledge base by reusing generated questions and answers. The questions and answers were originally generated for fine-tuning a model. The motivation behind that was to apply fine-tuning on top of RAG so that the resulting agent would potentially be even more specialized and grounded.

However, once we have the QnA dataset (more than 10,000 question-and-answer pairs), we could also feed that into the RAG pipeline to provide more ammunition for augmentation. That could potentially support specific user queries with better matching than a generic, “non-focused” chunked knowledge base.

As part of the effort, I contributed to the question_extractor open-source project, which offers semantic markdown knowledge base chunking:

• Added the ability to use AnyScale’s system
• Added support for QnA JSON formats for Google models and certain Azure OpenAI models
• Added throttling to prevent vigorous hammering of embedding and LLM APIs to avoid bans

Technology used:

1. Vectara: embedding, vector database, ranking, etc.
2. AnyScale: LLM hosting, fine-tuning
3. React: front-end

The React front-end provided excellent agent UX with nicely interpolated and numbered references. Vectara implements advanced techniques such as hybrid keyword and vector search and re-ranking for Retrieval Augmented Generation. Even though some articles lament that large context window models will supersede vector databases, vector databases can still provide lower costs (prompts containing entire knowledge bases would incur proportionally large API costs and high latency), while still potentially providing more control over the pipeline.

2,041 participants across 165 teams competed at the hackathon. Thanks to Ofer Mendelevitch for helping with technical guidance and thoughts on RAG vs fine-tuning at the GitHub HQ event, and thanks to all the mentors and tech sessions related to the hackathon as well. I’ll keep marching forward and exploring more about customer support LLM agents for my startups.