⛓️🛠️ ChainForge

ChainForge is an open-source visual programming environment for prompt engineering, LLM evaluation and experimentation. With ChainForge, you can evaluate the robustness of prompts and text generation models with little to no coding required. Features include:

• Query multiple LLMs at once to test prompt ideas and variations quickly and effectively. Or query the same LLM at different settings.
• Compare response quality across prompt permutations, across models, and across model settings to choose the best prompt and model for your use case.
• Setup evaluation metrics with code or LLM-based scorers and automatically plot results across prompts, prompt parameters, models, or model settings.
• Hold multiple conversations at once across template parameters and chat models. Template chat messages and inspect and evaluate outputs at each turn of a chat conversation.

ChainForge is in active development and is provided as an open beta test.

How to Install

Install ChainForge locally

pip install chainforge 
chainforge serve

Open localhost:8000 in a Chrome, Firefox, Edge or Brave browser. For more details on installation, such as setting API keys as environment variables, see Getting Started.

Or try out ChainForge online @ chainforge.ai/play

The web version is limited, yet includes a magical feature called the Share Button:

With Share, you can send your LLM experiments to others as links. Simply click Share to generate a unique weblink for your LLM experiment and copy it to your clipboard. For instance, here's an experiment that tries to get an LLM to reveal a secret key.

Note

ChainForge is compatible with Google Chrome and Mozilla Firefox. You will need an API key for the LLM(s) you wish to use. We do not store your API keys —not in a cookie, localStorage, or server. Because of this, you must set your API keys every time you load ChainForge. If you prefer not to worry about it, we recommend installing ChainForge locally and setting your API keys as environment variables.

About Us

ChainForge was created by Ian Arawjo during his time as Postdoctoral Fellow in the Variation Lab of the Harvard HCI group. He currently serves as the chief developer and maintainer. Collaborators include PhD students Priyan Vaithilingam and Chelse Swoopes, undergraduate Sean Yang, and faculty members Elena Glassman and Martin Wattenberg. In particular, Chelse developed the documentation and helped with our user study, while Sean Yang pioneered the new Generate, Replace & Extend generative AI features.

How to Cite

If you use ChainForge for research purposes, or build upon the source code, we ask that you cite our arXiv pre-print in any related publications. The BibTeX to use is:

@misc{arawjo2023chainforge,
      title={ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing}, 
      author={Ian Arawjo and Chelse Swoopes and Priyan Vaithilingam and Martin Wattenberg and Elena Glassman},
      year={2023},
      eprint={2309.09128},
      archivePrefix={arXiv},
      primaryClass={cs.HC}
}