{"meta": {"id": "zWGDn1AmRH", "review_idx": 0, "title": "Title: ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation\nAbstract: Text-to-SQL is the task that aims at translating natural language questions into SQL queries.\nExisting methods directly align the natural language with SQL Language and train one encoder-decoder-based model to fit all questions. \nHowever, they underestimate the inherent structural characteristics of SQL, as well as the gap between specific structure knowledge and general knowledge. This leads to structure errors in the generated SQL.\nTo address the above challenges, we propose a retrieval-argument framework, namely ReFSQL.\nIt contains two parts, structure-enhanced retriever and the generator.\nStructure-enhanced retriever is designed to identify samples with comparable specific knowledge in an unsupervised way. \nSubsequently, we incorporate the retrieved samples\u2019 SQL into the input, enabling the model to acquire prior knowledge of similar SQL grammar. \nTo further bridge the gap between specific and general knowledge, we present a mahalanobis contrastive learning method, which facilitates the transfer of the sample toward the specific knowledge distribution constructed by the retrieved samples.\nExperimental results on five datasets verify the effectiveness of our approach in improving the accuracy and robustness of Text-to-SQL generation.\nOur framework has achieved improved performance when combined with many other backbone models (including the 11B flan-T5) and also achieved state-of-the-art performance when compared to existing methods that employ the fine-tuning approach.", "claims": ["Claim1: has achieved improved performance when combined with many other backbone models (including the 11B flan-T5) and also achieved state-of-the-art performance when compared to existing methods that employ the fine-tuning approach.", "Claim2: In this paper, we propose a retrieval-augmented framework for Text-to-SQL generation.", "Claim3: Subsequently, we incorporate the retrieved samples\u2019 SQL into the input, enabling our model to acquire prior knowledge of similar SQL grammar.", "Claim4: Through contrastive learning, we guide the samples toward the specific knowledge distribution.", "Claim5: By utilizing the Mahalanobis distance, we employ a contrastive loss function to facilitate the transfer of the sample toward the specific knowledge distribution.", "Claim6: The obtained results serve as strong evidence to validate the effectiveness of our method.", "Claim7: Our ReFSQL framework, when applied to the RESDSQL backbone model, outperforms all baseline models on the two benchmark datasets.", "Claim8: The approach achieves state-of-the-art performance in methods that employ the fine-tuning approach.", "Claim9: Specifically, after adapting to our framework, the RESDSQL-based model improves the EM by 1.8 on Spider and 1.5 on WiKiSQL compared with the original model.", "Claim10: Our framework demonstrates flexibility and effectiveness, allowing for adaptation with numerous backbone models.", "Claim11: Remarkably, our framework yields notable improvements when applied to small-size models like T5-small.", "Claim12: This suggests that our framework can bridge the performance gap between small and large models, achieving comparable effects.", "Claim13: Furthermore, we conducted experiments using a larger scale model, Flan-T5, and observed a substantial improvement of nearly 2%.", "Claim14: These results indicate that our framework can deliver impressive performance even on large language models.", "Claim15: Additionally, we investigated the impact of model size.", "Claim16: As illustrated in Table 2, our framework consistently showcases larger performance gaps when compared to its T5-3B counterpart.", "Claim17: This observation aligns with previous findings in other fine-tuning tasks, highlighting the ability of larger pre-trained models to capture more knowledge effectively.", "Claim18: The results, presented in Table 4 reveal a surprising and significant performance advantage of RESDSQL(FlanT5)+NatSQL+ReFSQL overall strong competitors across all three datasets.", "Claim19: This finding suggests that our framework can also enhance the robustness of Text-to-SQL parsers.", "Claim20: The effectiveness of RefSQL is evaluated on five widely-used benchmark datasets, where it surpasses all baseline models, verifying its superior performance."], "review": "Review: Reasons to reject: 1.This paper is challenging to follow, and the proposed method is highly complex, making it difficult to reproduce.\n\n2.The proposed method comprises several complicated modules and has more parameters than the baselines. It remains unclear whether the main performance gain originates from a particular module or if the improvement is merely due to having more parameters. The current version of the ablation study does not provide definitive answers to these questions.\n\n3.The authors claim that one of their main contributions is the use of a Mahalanobis contrastive learning method to narrow the distribution gap between retrieved examples and current samples. However, there are no experiments to verify whether Mahalanobis yields better results than standard contrastive learning.\n\n4.The proposed method involves multiple modules, which could impact training and inference speed. There should be experiments conducted to study and analyze these effects.\nQuestions: None\n\n"}, "response": {"Weakness associated with claims": [{"Associated claims": [5], "Reasoning": "The review specifically mentions that there are no experiments to verify whether the Mahalanobis contrastive learning yields better results than standard contrastive learning. This directly questions the effectiveness claimed in Claim 5 about employing Mahalanobis distance for facilitating the transfer toward specific knowledge distribution.", "Label": "INSUFFICIENT_SUPPORT_EVIDENCE", "Weakness span": "The authors claim that one of their main contributions is the use of a Mahalanobis contrastive learning method to narrow the distribution gap between retrieved examples and current samples. However, there are no experiments to verify whether Mahalanobis yields better results than standard contrastive learning."}]}, "Clustered claims": {"DESCRIPTIVE": ["Claim3: Subsequently, we incorporate the retrieved samples\u2019 SQL into the input, enabling our model to acquire prior knowledge of similar SQL grammar.", "Claim4: Through contrastive learning, we guide the samples toward the specific knowledge distribution.", "Claim5: By utilizing the Mahalanobis distance, we employ a contrastive loss function to facilitate the transfer of the sample toward the specific knowledge distribution.", "Claim7: Our ReFSQL framework, when applied to the RESDSQL backbone model, outperforms all baseline models on the two benchmark datasets.", "Claim9: Specifically, after adapting to our framework, the RESDSQL-based model improves the EM by 1.8 on Spider and 1.5 on WiKiSQL compared with the original model.", "Claim11: Remarkably, our framework yields notable improvements when applied to small-size models like T5-small.", "Claim13: Furthermore, we conducted experiments using a larger scale model, Flan-T5, and observed a substantial improvement of nearly 2%.", "Claim16: As illustrated in Table 2, our framework consistently showcases larger performance gaps when compared to its T5-3B counterpart.", "Claim18: The results, presented in Table 4 reveal a surprising and significant performance advantage of RESDSQL(FlanT5)+NatSQL+ReFSQL overall strong competitors across all three datasets."], "INTERPRETIVE": ["Claim6: The obtained results serve as strong evidence to validate the effectiveness of our method.", "Claim12: This suggests that our framework can bridge the performance gap between small and large models, achieving comparable effects.", "Claim14: These results indicate that our framework can deliver impressive performance even on large language models.", "Claim19: This finding suggests that our framework can also enhance the robustness of Text-to-SQL parsers."], "OVERARCHING": ["Claim1: has achieved improved performance when combined with many other backbone models (including the 11B flan-T5) and also achieved state-of-the-art performance when compared to existing methods that employ the fine-tuning approach.", "Claim8: The approach achieves state-of-the-art performance in methods that employ the fine-tuning approach.", "Claim10: Our framework demonstrates flexibility and effectiveness, allowing for adaptation with numerous backbone models.", "Claim20: The effectiveness of RefSQL is evaluated on five widely-used benchmark datasets, where it surpasses all baseline models, verifying its superior performance."], "RELATED_WORK": ["Claim17: This observation aligns with previous findings in other fine-tuning tasks, highlighting the ability of larger pre-trained models to capture more knowledge effectively."], "OTHER": ["Claim2: In this paper, we propose a retrieval-augmented framework for Text-to-SQL generation.", "Claim15: Additionally, we investigated the impact of model size."]}, "id": "zWGDn1AmRH0", "pdf": "openreview.net/pdf?id=zWGDn1AmRH"}