INSTINCT

Abstract

Large language models (LLMs) have shown remarkable instruction-following capabilities and achieved impressive performances in various applications. However, the performances of LLMs depend heavily on the instructions given to them, which are typically manually tuned with substantial human efforts. Recent work has used the query-efficient Bayesian optimization (BO) algorithm to automatically optimize the instructions given to black-box LLMs. However, BO usually falls short when optimizing highly sophisticated (e.g., high-dimensional) objective functions, such as the functions mapping an instruction to the performance of an LLM. This is mainly due to the limited expressive power of the Gaussian process (GP) model which is used by BO as a surrogate to model the objective function. Meanwhile, it has been repeatedly shown that neural networks (NNs), especially pre-trained transformers, possess strong expressive power and can model highly complex functions. So, we adopt a neural bandit algorithm which replaces the GP in BO by an NN surrogate to optimize instructions for black-box LLMs. More importantly, the neural bandit algorithm allows us to naturally couple the NN surrogate with the hidden representation learned by a pre-trained transformer (i.e., an open-source LLM), which significantly boosts its performance. These motivate us to propose our INSTruction optimization usIng Neural bandits Coupled with Transformers (INSTINCT) algorithm. We perform instruction optimization for ChatGPT and use extensive experiments to show that our INSTINCT consistently outperforms the existing methods in different tasks, such as in various instruction induction tasks and the task of improving the zero-shot chain-of-thought instruction.

How to use your INSTINCT?

Step ①: Training the neural network for score prediction.

Step ②: Selecting the next soft prompt using the NeuralUCB algorithm.

Step ③: Generating the instruction using a white-box LLM.

Step ④: Predicting the label for a validation dataset using black-box LLM using the generated instruction.

Step ⑤: Evaluating the predicted results (i.e., the performance of the instruction).

Step ⑥: Extracting the hidden representation from the white-box LLM for the instruction. Adding the hidden representation and the evaluated score to the dataset which is used to train the neural network.

Results

How does INSTINCT generate higher-quality instructions across iterations?

Instruction induction

Our approach significantly outperforms APE and InstructZero in a variety of instruction induction tasks.

Figure 1: Improvement of our INSTINCT over baselines (in 30 tasks). II Image

Table 1: Average test accuracy achieved by (i) APE, (ii) InstructZero and (iii) INSTINCT. II Table

Improving instruction for summarization tasks

We demonstrate the capability of our INSTINCT in instruction optimization for the task of text summarization.

Table 2: Instruction optimization on SAMSum dataset (summarization task). Sum Table

Improving zero-shot chain-of-thought prompts

We improve zero-shot CoT prompts on multiple arithmetic reasoning tasks.

Table 3: The best zero-shot CoT instructions found by different algorithms and their scores. CoT Table

Further improvement with one-shot in-context learning

We see wider potential applications of our INSTINCT through its combination with in-context learning.

Table 4: Average test accuracy achieved by (i) INSTINCT, (ii) test-time-only one-shot INSTINCT, (iii) one-shot INSTINCT. The results including all tasks are given in the paper. One-shot Table

BibTeX

@inproceedings{lin2023use,
        title={Use Your {INSTINCT}: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers},
        author={Xiaoqiang Lin and Zhaoxuan Wu and Zhongxiang Dai and Wenyang Hu and Yao Shu and See-Kiong Ng and Patrick Jaillet and Bryan Kian Hsiang Low},
        year={2024},
        booktitle={Proc. ICML}
}

Use Your INSTINCT: INSTruction optimization usIng Neural bandits Coupled with Transformers