Frequently Asked Questions (FAQ)
How do I compare across prompt templates?
Use template chaining. Then, you can use Table Layout in the Response Inspector to plot the templates on columns of a table. When comparing templates with manual inspection, we recommend sticking to a single LLM. If you want to compare across multiple, set up an evaluation pipeline.
I want to compare system messages, or template my system message. How can I do this?
Two ways:
-
You can simply add another copy of the model in the Prompt Node and change the settings manually. This latter approach won't work with template variables, however.
-
A more powerful approach is declaring a template variable
{=system_msg}in a Prompt Node or TextFields field:
This uses what we call a settings template variable, or a variable of the form {=setting_name}. The line will not be used directly in the final prompt, but will rather change the model parameters for that setting. (The value of that parameter in the models settings window will be overidden.)
For example, adding {=temperature} would let you compare across different temperatures, and your inputs would be a list of numbers (using either a TextFields or Items Node). For more information about settings variables, see Prompt Templating.
I'm running on dev mode with npm run start. Why do all these red error messages pop up?
That's normal. We haven't prioritized squashing all of these errors, as some of them are produced by third-party imports (e.g. Plotly). All of them (or mostly all!) are harmless, however. Just close them out.
Unless you are developing or extending ChainForge, you shouldn't use it with npm run start. If you have modified ChainForge source code,
and don't want to deal with the error pop-ups, do npm run build in react-server folder, then python -m chainforge.app serve
from the top-level directory to serve the built React app on localhost:8000.
I want to use my own model/chain. Can I do that?
Sure, see Custom Providers---you'll just need to write a Python completion function.