Text 2 SQL using Llama

A text to SQL conversion tool using Llama and SFT

About

Text-to-SQL is a challenging task that involves converting natural language questions into SQL queries. This project aims to build a text-to-SQL conversion tool using the Meta-llama-7B model with supervised fine-tuning using sql-create-context dataset b-mc2.The project involves training the Llama model using SFT and evaluating its performance on a test set of text-to-SQL examples.

Background

Text-to-SQL is a challenging task that involves converting natural language questions into SQL queries. Text-to-SQL is a challenging task that involves converting natural language questions into SQL queries.

Large Language Models

Large Language Models are advanced natural language processing models characterized by enormous size, typically containing tens or hundreds of billions of parameters. These models are trained on large-scale datasets using unsupervised learning techniques and have been shown to achieve state-of-the-art performance on a wide range of natural language processing tasks.

How LLM’s are Trained

write about pre-training , SFT and RLHF

Difference between SFT and RLHF

write about pre-training , SFT and RLHF

Different ways to fine-tune LLM’s

write about pre-training , SFT and RLHF

Methodology

Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks. In the full fine-tuning of LLMs, there is a risk of catastrophic forgetting, where previously acquired knowledge from pretraining is lost.

PEFT

PEFT is designed to fine-tune models while minimizing the need for extensive resources and cost. PEFT is a great choice when dealing with domain-specific tasks that necessitate model adaptation. Using PEFT we can balance retaining valuable knowledge from the pre-trained model and adapting it effectively to the target task with fewer parameters. 

LORA

PEFT is designed to fine-tune models while minimizing the need for extensive resources and cost. PEFT is a great choice when dealing with domain-specific tasks that necessitate model adaptation. Using PEFT we can balance retaining valuable knowledge from the pre-trained model and adapting it effectively to the target task with fewer parameters. 

Results

Here are some results tested on blind dataset

Ways to improve

  • We can try different modles with more parameters like Llama-70, GPT-4 etc

  • Better prompting can like explaining context of variables and tables can be used to improve the performance.

  • Use data centric approach with more SQL QA datasets and instruction based datasets.

  • RLHF can be incorporated by ranking the output and using human feedback to improve the model.

  • Customizing loss functions to penalize for retrieving the correct query, which will help in interpretability and explainability of the model.

  • Pre-checking the validity of questions can help in reducing the errors in the output.