The archive · Developer & Business Tools · Product decision · 2023
Hegel AI's PromptTools bet: code-first LLM evaluation drew 211 HN points, 3.1k stars
Hegel AI built open-source prompttools for evaluating LLMs and vector DBs in code; its 2023-08-01 Show HN drew 211 points and the repo later passed 3.1k stars.
Hegel AI
What the business is
Open-source, self-hostable Python tools (prompttools) for experimenting with, testing and evaluating LLMs, vector databases and prompts — the product wedge Hegel AI shipped into LLM evaluation.
How it started
Kevin and Steve said in their Show HN that they had started a month earlier, in July 2023. Steve had worked at Google with HuggingFace and Lightning on running the newest open-source models on TPUs; he saw the community contributing powerful models that were still hard to discover and evaluate — it was not clear when Llama or Falcon could replace GPT-4. PromptTools was their attempt to simplify and scale that evaluation: write a short Python script that runs checks across models, parameters and prompts, then pass the results into an evaluation function for scores, all executed locally.
What happened
The 2023-08-01 Show HN drew 211 points and 24 comments. Engineers called it exactly the tooling one realizes is needed when starting to build apps on LLMs; commenters compared it with promptfoo and ChainForge and pressed on whether LLM auto-evaluation can be trusted. The founders answered with their roadmap — structured-output validation, semantic-similarity checks, manual feedback, multiple-LLM judging, exports to CSV, JSON, MongoDB and Markdown — and invited PRs for function calling and Qdrant support. The repo kept moving: 3.1k stars, 256 forks and 638 commits by the 2026-09-05 crawl.
No ending yet — it is still running.
Background
Kevin and Steve started PromptTools in July 2023, a month before their Show HN, under the Hegel AI name: open-source, self-hostable Python tools for experimenting with, testing and evaluating LLMs, vector databases and prompts. The README's core idea is evaluation through familiar interfaces — code, notebooks and a local playground — with integrations for OpenAI, Anthropic, Google, HuggingFace and LlamaCpp plus vector stores such as Chroma, Weaviate and Qdrant.
Steve had worked at Google with HuggingFace and Lightning on running open-source models on TPUs. He saw the community shipping powerful models that were hard to discover and evaluate: it was not clear when Llama or Falcon could replace GPT-4. PromptTools was the answer — a short Python script runs checks across models, parameters and prompts, and results stay on the developer's machine, with no server forwarding and no collection of keys or outputs.
The 2023-08-01 Show HN drew 211 points and 24 comments. Engineers called it needed tooling, compared it with promptfoo and ChainForge, and questioned how far LLM-as-judge can be trusted; the founders responded with details on structured-output validation, semantic similarity, manual feedback and multiple-LLM scoring. By the 2026-09-05 crawl the repository showed 3.1k stars, 256 forks and 638 commits.
What has to be true
- Dated, specific traction: the 2023-08-01 Show HN (item 36958175) drew 211 points and 24 comments, and the hegelai/prompttools repo showed 3.1k stars, 256 forks and 638 commits on the 2026-09-05 crawl.
- A concrete founding bet stated by the builders: Steve saw powerful open models at Google that were hard to discover and evaluate, and PromptTools was built to simplify and scale that process.
- The wedge is a deliberate inversion: evaluation runs in local code and notebooks rather than a hosted platform, with no server forwarding and no collection of keys or inputs.
- The launch thread shows real pull: engineers called it needed tooling and debated LLM-as-judge circularity against promptfoo and ChainForge — the problem the founders said they were solving.
What can be applied
A founder's own unsolved pain is a credible wedge: Steve could not tell when an open model could replace GPT-4, so PromptTools made 'evaluate in code' a five-line habit that spread via open source.
Aftermath
As of 2026-09-05 the hegelai/prompttools repository is live with 3.1k stars, 256 forks and 638 commits; the README still positions it as open-source, self-hostable evaluation tooling from Hegel AI and links documentation, a hosted Streamlit playground and a Discord. The material records no funding, revenue or corporate outcome for Hegel AI, so nothing beyond the project's continued development is claimed here.
Sources
- Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
- hegelai/prompttools — GitHub repository (live as of 2026-09-05)
spotted an error? The archive wants to know.
Your turn
You just read one. Describe what you are building, and see who is betting on the same thing.
Free account · 3 free questions · no card