Open-Source vs. Proprietary Chatbots: Vicuna-13B’s Performance Shocks with Impressive Quality
The Rise of Open-Source Chatbots
In recent years, chatbots have become more and more popular because they can talk to people in a natural way. They can do many things, like answer questions from customers, give information, and even help with mental health therapy. However, creating a chatbot that can understand and respond to human language is a challenging task. So, researchers and developers have been looking into different ways to make chatbots that can understand and respond to human language in a natural and effective way.
In this blog post, will discuss the latest open-source chatbot, Vicuna, developed by a team with members from UC Berkeley, CMU, Stanford, and UC San Diego. We will explore the features of Vicuna, its training, evaluation, and how it performs compared to other chatbot models.
Features of Vicuna
Vicuna is an open-source chatbot that was trained by fine-tuning LLaMA with conversations from ShareGPT that were shared by users. Vicuna-13B is the current version of the chatbot, which achieved more than 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases.
The cost of training Vicuna-13B is around $300. The code for training and serving, as well as an online demo, are open source and can be used by anyone for non-commercial purposes. Vicuna is made to give answers that are more detailed and well-organized than models like Alpaca.
The research team rated the quality of responses from different chat AIs. ChatGPT got a score of 100%, while LLaMA and Alpaca 7B got scores of 68% and 76%, respectively. Vicuna-13B's response quality, on the other hand, was found to be close to 92%, which shows that it is a very good chatbot model that can give well-structured and accurate answers.
Training and Evaluation of Vicuna
To train Vicuna, the team fine-tuned LLaMA on 70K user-shared ChatGPT conversations. This training was done using ShareGPT, which is a dataset of human-generated conversations. The fine-tuning process involved training the model on the ShareGPT dataset until it became capable of generating well-structured answers that are on par with ChatGPT.
To evaluate the performance of Vicuna, the team used GPT-4 as a judge. The preliminary evaluation showed that Vicuna-13B achieved more than 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases. This shows that Vicuna is a high-quality chatbot that can generate well-structured answers and perform well compared to other models.
Comparison with Other Models
The team compared the performance of Vicuna with other chatbot models like LLaMA and Stanford Alpaca. The results showed that Vicuna outperformed these models in more than 90% of cases. Vicuna was also found to generate more detailed and well-structured answers compared to Alpaca. This shows that Vicuna is a very competitive chatbot that can do better than other models.
Conclusion
Vicuna is an open-source chatbot that has been developed by a team with members from UC Berkeley, CMU, Stanford, and UC San Diego. It has been trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. The preliminary evaluation showed that Vicuna-13B achieved more than 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases.
Resources & Reefferencs
Visit https://vicuna.lmsys.org for readers to find further information and resources related to Vicuna.
Vicuna online demo https://chat.lmsys.org/
Visit this open platform for training, serving, and evaluating large language model based chatbots https://github.com/lm-sys/FastChat
lpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving https://arxiv.org/abs/2302.11665
