🍃How Does DSPy Work

The way DSPy work is quite sequential:

DSPy Workflow

1. Signature

DSPy uses signature to define the data schema of input and output. The task instruction is also included in it.

In the example signature below, it defines question and context as the input, and answer as the output.

Example DSPy Signature

2. Module

DSPy module defines the prompt strategy. “Prompt” refers to everything that goes into constructing the LLM’s input message. In the example module below,

• It outlines how an LLM processes the input data (e.g question and context), also known as reasoning style, such as the ChainOfThought in the example below.

• It has a forward function to define the actual computation flow: how inputs are transformed into outputs.

Example DSPy Module

3. Optimizer

The DSPy optimizer considers prompt components like task instructions, reasoning style, and few-shot examples (namely, example inputs & outputs) as parameters. It creates variations of these components, evaluates their performance using predefined metrics, and selects the prompt with the best optimization results.

To evaluate the performance, you need evaluation metrics. In the example below, the predefined metric is G-Eval, which measures answer correctness by comparing the answer with the ground truth.

Example DSPy Optimizer

The example optimizer here is MIPROv2. The way MIPROv2 works is:

1. It bootstraps few-shot examples from your training data, observing the data and summarizes the data.

2. It also observes the LM program formed by the Signature and the Module, then provide the program description.

3. A randomly selected "tip" is to give suggestions on how to generate prompts. For example, high_stakes focuses on correctness and it's often used with legal or financial data.

MIPROv2's Observations & Descriptions from the Compiling Log

1. After all the observations above, MIPROv2 will run multiple iterations till convergence by following the workflow below:

MIPROv2 Iteration Workflow

🌻 Click to see full code >>

The whole process can be logged in a text file. See an example here:

🌻 An example of compiling log >>

In the optimized prompt, you will notice that this example only had the task instruction got optimized:

Example Output: Instructions Before & After