I Built an AI Chatbot That Learns From Every Mistake

If you manage a team that deals with patrons, members, donors, or the general public, you know this problem.

Your team has questions all day. Some of those questions have answers that exist somewhere, in a policy document, in an email from six months ago, in someone’s head. But finding the right answer quickly, consistently, without pulling a manager away from what they’re doing? That’s a different challenge.

That was the problem I was trying to solve for our patron support team at the BSO. The solution I built ended up being one of the most useful things I’ve done in the last few years.

Where It Started: Pete

This tool didn’t start with Claude. A while back, when ChatGPT was getting all the attention, I built an internal resource for the patron support team using custom GPTs. We named it Pete, Patron Experience and Ticketing Expert.

The idea: give the team something they could ask operational questions and get fast, accurate answers, rather than pulling someone else away from their work. We connected it to Notion documentation and added a set of relevant web resources. And honestly, Pete worked. The team adopted it.

But there were real limitations. It required team members to have ChatGPT accounts. There were usage limits on free accounts. The Notion connection wasn’t seamless. And I had no visibility into how it was performing or where it was getting things wrong.

I’d been moving away from ChatGPT in my own work toward Claude, the ecosystem matched better with how I build things and Claude is exceptional at coding. But I’d kept my ChatGPT subscription because the team was using Pete. It was time to rebuild.

The Rebuild: Same Methodology, More Moving Parts

Same methodology I used for the website build: Chat first, before I open any other tool.

I brought Claude Chat everything I had, Pete’s existing instructions, the sources we’d been using, all the documentation I’d accumulated, and had a conversation about what I wanted the new version to do differently. Hosted somewhere I actually control. No sign-in required. Moderable. And designed to improve with use over time based on real feedback from the people using it every day.

That conversation produced a plan and a full set of build instructions that I handed directly to Claude Code.

The original Pete took a week or two of trial and error to get right. The Claude rebuild took a morning, one to two hours to have it up and running.

What the Team Sees

A clean chat interface. No account required, no sign-in, no friction to stop them from using the tool. They type a question, the tool searches the knowledge base and relevant resources, and returns an answer.

Simple on the surface. What’s underneath is where it gets interesting.

The Feedback Loop

Every response has a thumbs up or thumbs down option. Agents rate the answers they receive in the moment, which gives me data on where the gaps are.

There’s also a Correct button. If an agent receives a wrong answer but knows the right answer, they can submit a correction directly.

When someone hits Correct or Thumbs Down, an automated process triggers:

  • A task is created in Asana and assigned to our Director of Patron Support, with the original question and flagged response attached so the manager has full context.
  • A Notion article is drafted, the AI attempts to write the correct policy and provides a recommended structure for the manager to review.
  • The manager updates the article and changes its status from Draft to Confirmed.
  • That confirmation automatically pushes the article into the live knowledge base.

The loop closes. No additional manual steps beyond the review. The next time that question is asked, the agent gets the right answer.

The Admin Panel

What I didn’t have with Pete: visibility. Now there’s an administration panel showing volume, common questions, flagged responses, and overall performance over time.

That’s going to give us insight into how the team is using the tool and into the customer service experience itself, where are people asking the most questions? Where is the team most uncertain? Where can we reduce friction in the patron experience?

What We Learned From the Rollout

The original Pete was well-received and genuinely used. But there’s something worth naming for anyone building a tool like this: there’s a real learning curve in helping a team understand what an AI tool can and can’t do.

Team members would ask Pete questions that no AI system could possibly answer, not because the tool was broken, but because the information didn’t exist anywhere it could access. Internal decisions that hadn’t been documented. An AI agent works with what it has access to. That’s a conversation worth having with your team before you roll anything out.

The tool is only as good as the knowledge base it reads from.

What It Costs

  • Hosting on Railway: $5/month (I have several applications on this plan)
  • Notion subscription (for the knowledge base): ~$20/month
  • Claude API usage: minimal for a small internal team, no additional cost beyond my existing plan

Compare that to the ChatGPT subscription I was maintaining with usage limits, no monitoring capability, and no ability to customize the environment. This costs less, does more, and I own it.

The Bigger Point

The problem this tool solves isn’t unique to the BSO. If you have a team that deals with the public, you have institutional knowledge that’s hard to access consistently. That’s the problem. The tool is just how I solved it.

Same methodology as the website: Chat first, get your thinking clear, build a brief, hand it to the right tool, direct it, test it, iterate. What you need is documentation, a clear problem you’re trying to solve, and willingness to sit with the process.


Leave a comment