AI systems at Capacities

During the past months we have released many new AI features. We've added support for more AI providers, chat connectors for external AI applications, media analysis to get information out of images, audio and websites, query creation via natural language and we've greatly enhanced our AI Assistant within the app.

Soon, you'll also be able to analyze PDFs and websites. While we have many other ideas about what we can bring into the app in the future, and how we plan to do it, I wanted to take a step back and reflect on how we've been able to build and deploy all these features to our users.

With all the infrastructure changes we made last year, we finally had the resources to deploy many more features. People have reached out to send very nice compliments for the pace of features released this year (thank you for that!), and we've actually had to update the What's next twice already.

To support all these updates, we needed a clear idea of what to build and how to build it. Even if it seems like we're now on a crazy streak (and hey, we kind of are) many of the features that are coming out now had already been in the works, or at least in a conceptual phase, for a significantly longer time. Media analysis for images as an example was released just this April, but its underlying system is something both Mike and Steffen used as part of my interviews when applying to Capacities by the end of 2024, and they had already conceptualized a large part of it at that point.

In this article I'll be talking about the backend that supports all these new AI features.

AI SDK vs. LangGraph

This is a question I've encountered many times, and that people have asked me every time I mention what I'm working on. Assuming you're working on a TypeScript codebase (which we are at Capacities), my response is usually about the abstraction level.

The AI SDK essentially abstracts the typical interface provided by the LLM providers' SDKs. This makes it extremely useful, since it supports a very wide range of LLM providers and brings them together into one single functional interface. Even if there's some custom parameters for a given provider, there's usually some openly typed customProviderOptions so that you can still adjust those settings without adding another dependency to your project, making the addition of this framework to a project somewhat of a lower stake commitment.

On the other hand, LangGraph was first built on Python and then extended to TypeScript. It stands out by providing an abstraction at the agentic orchestration level, meaning that different teams that build in distinct languages (say, data scientists working on Python and product engineers working on TypeScript) can still share the same concepts and mental model. However, their handling of states, memory and faults is determined by their own implementation and they're providing an entire suite of products based off of that abstraction (LangSmith, LangGraph Cloud, LangGraph Studio, ...). This is both a good thing and a bad thing. If you're completely building from scratch and can adopt their mental model, that'll make things easier. However, if you have existing abstractions, you must weigh the effort of refactoring them to fit LangGraph against the effort of replicating missing features using the AI SDK.

I don't believe there's any one feature that can only be built in one of the two frameworks, and both of them have their own limitations and hundreds of open requests in their GitHub repositories. One of the limitations we found for the AI SDK for instance was their lack of stateful file handling, which is now being added in the v7 update.

At Capacities, we already had a very specific data structure abstraction that we felt comfortable working with and was already built on top of the OpenAI SDK. While we could've adapted LangGraph abstractions into our interface, it felt safer, quicker and more natural to ship smaller extensions of our own AI implementation via the AI SDK.

The difficulty: generalizing across all users, models and use cases

When researching how to build AI Chatbots and applications, you'll very quickly come across the concept of "domain expert". If you're building a personal finance chatbot for instance, you'll probably include many concepts like "budgeting", "tax deductions" or "diversification" into the system prompt in order for the LLM to correctly characterize the kind of requests it's going to be working with.

However, looking at a note taking app like Capacities, that problem exponentially grows in complexity. A financial advisor will probably want their AI Assistant in Capacities to work as a financial domain expert, but the writer that wants to contrast different ideas for their next sci-fi novel won't care about that, and won't appreciate the extra input token consumption either.

One tricky assumption to make in our domain is that feeding it information about productivity systems and PKM concepts will work well, but the reality is that most users just want the AI Chat as a side screen to use for their own workflows. Focusing entirely on PKM knowledge will just trigger most LLMs to question the way users are shaping their information when it could be a perfectly valid system for their use cases.

We had to come up with a system that would understand Capacities, understand how users shape their information into object types unlike with other apps, understand the different property types of those object types, and then also understand the kind of data and links a particular user is leveraging within their notes. All that while utilizing an acceptable number of tokens so that a cheap model won't start hallucinating across different use cases and respecting the user's AI budget. Oh, and it also needs to work across multiple models from multiple providers so that people can rightly choose who's processing their data.

This has of course taken many iterations across many use cases. And our prompts will undoubtedly have to continue to adapt to future model releases. One internal rule I had here was that if a prompt doesn't work well on the cheapest model offered from a large LLM provider then it's simply not good enough. Models are becoming smarter and smarter, which makes it easier to get away with a bad prompt that won't generalize well for most users. I found this rule kept much of the phrasing around Capacities concepts somewhat evergreen throughout many model iterations.

Why did we go with an agentic architecture

Setting all the agentic orchestration hype aside, being able to break down instructions and context across multiple specialized units has been a game changer for us. As we kept enhancing our system prompt, and started adding tools to the AI Chat in Capacities to do things like searching across notes or linking content, it was fairly easy to land in context rot territory for cheaper models. Wrapping our system in a dedicated agentic harness to break down instructions across specialized units became essential to scaling capabilities without overloading context windows.

And this didn't just allow for more capabilities but also for better use of resources. If a user is just saying "hi" you don't need your LLM to suddenly know all about the 25 object types the user has configured for their notes. Having a system that can correctly identify what the right level of complexity is and determine how best to approach the response generation keeps the token usage low.

There is, however, a trade-off to consider: the more you break down your LLM calls into a specialized hierarchical system the more latency you'll get. Every time you add an intermediary to get to the right agent that will process the user's request, you're adding a processing step that needs to consume both input and output tokens.

From a high level, this is the agentic harness we are using:

The key detail here is that the orchestrator runs as a single continuous tool-loop where each capability surface is a tool the model can call. When it calls exploreContent with an operation like searchNotes, that operation is forwarded directly to the browser for client-side execution. When it calls webSearchAgent, a server-side sub-agent handles the web lookup and returns a compact summary reference, keeping the orchestrator's context window clean while the full payload stays in a server-side output store.

The orchestrator has full visibility of the turn at all times, can make continuation decisions after each tool step, and can stop or degrade gracefully when the budget is running low.

Request classification and routing

Before the orchestrator even starts, a lightweight request classifier examines the user's prompt. It uses a combination of heuristic checks (for truly trivial input like greetings or emoji) and a structured LLM call to determine three things: whether the request needs the full orchestrator at all, and which capability groups might be needed. If Auto mode is active in the chat, it also determines the tier of model complexity from your preferred configured provider.

Trivial requests skip the orchestrator entirely and get a direct budget-tier response. This saves a meaningful amount of tokens for the majority of casual interactions in a chat. For everything else, the classifier prunes the tool set to only the relevant capability surfaces, so the model doesn't see tool schemas it won't need.

Capability surfaces

Each capability surface wraps a group of related client-side tools behind a single tool schema:

1. exploreContent: Groups all read operations: searchNotes, getNoteContent, getMediaContent, exploreConnections, generateQuery, getQueryResults. Also integrates web search as an optional server-side operation when external context is needed.
2. createContent: Groups all write operations: createObject, updateObjectProperties, appendObjectContent, deleteObject, createTask, createEntityLink, getObjectTypeShape. These operations are forwarded to the browser where the user sees a live approval prompt before any modification happens.
3. planAgentLoop: The bulk task planning surface. When the orchestrator receives a multi-item request (like "create book objects for all the titles mentioned in my reading list"), it uses this capability to generate an ordered execution plan with phases, constraints, and acceptance criteria. The orchestrator then executes those phases itself using the other capabilities, calling the task evaluator between write phases to verify completion.
4. answerSupportQuestion: Routes to either the support agent (for Capacities features, settings, troubleshooting) or the knowledge management consultant (for PKM methodology, organization strategies, object type modeling advice).
5. writingAssistantAgent: The final formatting pass. Once the orchestrator has gathered all the information and completed any necessary operations, it delegates the final user-facing response to the writing assistant. This agent receives the accumulated context via server-side output references rather than raw text dumps, keeping its input focused and predictable.

By collapsing 13+ individual client tools into 4 capability entry points, the orchestrator's tool schema stays compact and its routing decisions stay accurate even on cheaper models.

Tool call forwarding and approval

Write operations (create, update, append, delete) flow through a tool-call forwarding layer that bridges the server-side orchestrator with the browser. When the orchestrator decides to create an object, it doesn't execute that directly, it emits a tool-call event that the client renders as an approval prompt. The user sees exactly what will be modified and can approve or deny.

This is the same pattern that makes the AI Chat feel transparent rather than autonomous. The model's reasoning is streamed live, the tool calls are visible, and modifications require explicit consent. On the server side, the orchestrator's tool-loop pauses when a client write tool is requested and resumes only after the approval result arrives.

Budget-aware continuation

The orchestrator runs within a real-time budget envelope. A shared budget accumulator tracks cost across the main stream and all nested sub-agent calls. After each tool step, a continuation decision engine evaluates whether to continue, pause, or degrade:

• If the budget is running low, the system degrades gracefully, it'll still deliver a partial response rather than stopping abruptly.
• If too many loops have been done, execution pauses.
• If a write tool is awaiting browser approval, the loop yields and the stream completes, saving state so the system can check for timeout on the next interaction.

This replaced a simpler "cut off at budget limit" approach we had before, and the improvement was immediately noticeable for users running complex multi-step tasks on limited budgets.

Context compaction

As conversations grow longer, the message history can exceed the model's effective context window. Rather than truncating abruptly, our context compactor estimates the token count of the full history (with script-aware counting that handles CJK characters correctly) and, when the estimate approaches the window threshold, removes older turn pairs while preserving system messages and inserting a placeholder summarizing what was removed. The compaction targets are based on the model's context window, and always leave room for both the system prompt and tool definitions overhead.

Resumable streaming transport

A significant infrastructure addition was the resumable streaming protocol built on top of the AI SDK's UIMessageStream. Each stream is backed by a pub/sub pattern implemented on Redis, the server publishes stream events to a per-stream channel, and the client subscribes. If a client disconnects mid-stream (network interruption, tab switch, etc.), it can reconnect and resume from where it left off by subscribing to the same channel. The stream's active state is tracked via Redis keys with compare-and-swap semantics so that concurrent requests (multiple tabs) don't step on each other.

Model Context Protocol and learnings

While developing the tools for all the new agents, the release of the MCP server was quite helpful. Not only could we monitor how tool execution behaved on our infrastructure, but we could also evaluate the real life behavior of the context passed for each of the tools. Since the MCP tools were built on top of the new public API implementation that was released later, we were able to validate both AI system assumptions as well as indirectly the new implementation of the public API in a production environment.

We saw that the user reception was great, and the reported feedback was largely focused on the setup and authentication part rather than the tool execution itself. This meant that different models working in external apps with largely different contexts were doing a good job at deciding which tool from our server they were meant to execute. Even though we already saw this in our internal tests, checking that behavior consistency in production gave us more confidence in deploying the next iterations of our AI chat.

This also fed into the decision of keeping a single chat agent. If models without all the context our internal Capacities chat has were behaving quite well for all MCP operations, breaking the chat agent down into more agents (say one for reading and searching content, and another for updating objects and appending content) would just add latency without necessarily improving the user experience. Ultimately the MCP tools are an abstraction that simplify the public API to an interface we thought our internal AI agents could make a more efficient use of.

Embeddings and media processing

A crucial part of the AI systems are both the embedding and the media processing pipelines. These do involve a more complex infrastructure setup:

1. We extended our Postgres database to be able to handle embeddings storage and trace them back to the corresponding object they relate to.
2. We added a message queue to asynchronously mark content as “pending-embed”, so that a periodic worker on the server side can eventually accumulate all changes and overwrite embeddings.
3. For media handling, a similar system was needed. We once again extended the Postgres database to store media analyses that arrived as a result of a worker doing a LLM call after a media analysis message arrived on the processing queue. These media analysis responses had to then also be embedded into the previous system to take full advantage of semantic search.
4. The media handling system had to then consequently be extended multiple times to support all the other media types, each with their own release timeline and tests.

These two parts are vital to provide an AI system that can utilize typical media files that users want to write notes about, but also to better interpret and understand relations and similarities across all the content of a user space.

The roadmap and iterative approach to both AI features and infrastructure

The engineering approach in Capacities has largely impacted the delivery calendar of all these features, and I think it's something we have to look back on and be proud of, but also keep as a model of reference for future projects.

When it comes to AI systems, executing our roadmap felt like a straight arrow:

• First we switched from using just the OpenAI SDK to the Vercel AI SDK, and allowed more provider options when bringing your own key. This allowed us to refactor much of the server code to support the upcoming projects.
• Then we worked on the embeddings pipeline and planned the right infrastructure deployment that would later be reused for the media pipeline.
• In the meantime, we worked on extending the Capacities AI budget to also support multiple providers since we now utilized the AI SDK, and refactored all the logic further to increase maintainability as the system grew in complexity.
• Then we started working on the public API as well as the media processing handling in parallel. Once the setup for the image analysis was deployed, we decided to use the MCP server release as a test ground for the agentic system conception and the public API new implementation.
• We have then continued working in parallel on the AI Chat agentic system and the rest of the media analysis file types.

Even if we had breaks in between to patch bugs, the fact that we could trace each implementation step in relation to each other and in relation to the larger features users had been requesting, and how smoothly the execution went, is a great win, especially for a small team with limited resources. By not imposing timelines for any specific feature, we've been able to validate systems and extend them in a way that felt natural and maintainable to then determine what the next release could include at that time. Even if some features could've been hacked together sooner and attract more subscribers in the short term, we would've had a great technical debt and maintenance overload in the longer term. Speaking plainly, we prioritized product quality over deadlines.

Note on privacy and security

When approaching the implementation and the infrastructure for all these systems we've been very mindful about the privacy and concerns of our users. We have a group of users in our community that are vocally against AI usage, and that's why we keep it as an opt-in setting. For people who do want to use AI in Capacities, we want them to know that they're in control of their data. And that goal has guided many of our design choices:

• All the tools that the AI Chat has access to request approval before a modification happens, you can see the live thinking process of the model and determine if you want to or not want to approve those changes.
• We keep the documentation up to date on which features access what kind of data, so that users have full transparency on where their data goes behind the curtain even if they don't have very strong technical knowledge.
• Capacities AI uses models that are hosted via cloud providers with whom we have very strict data processing agreements, ensuring that our users' data can't be legally used for training or stored in their systems.
• In the case a model option is fully hosted under strict EU regulations on European servers, we show that directly within the UI so that users can select them if that's their preference.

Personal take

Late last year I started to notice a burnout among people I know who also work in product development. With all the crazy headlines about AGI, job security, global finances and their derived impact on companies' strategy shifts many people are losing the passion for building products and growing an inner grudge against these new tools. I remember when saying that I was working on some new AI features at my company everyone got super excited and asked me a ton of questions, and now that has shifted to an instant mood killer.

These are people that understand that technology is not magic. I've heard of too many people at larger companies who are focused on using, or forcing others to use, these technologies to continue performing the same tasks in a much tighter deadline to achieve a never-heard-of productivity metric. LLMs not only write code but they can also help you question your architectural decisions, detect edge cases and identify readability and maintainability improvements. That is, there is another way of using this tool that still allows to put care into the product and continue growing as an engineer. There's also a lot of excitement out there because of how cool these features we can build are, and rightfully so. I don't think there's ever been a point where technology felt more like magic, even if we know it really isn't.

Every now and then I like opening LM Studio to check what my own laptop is capable of running. This last week, loading the Gemma 4 E4B model and playing around a bit really surprised me. I don't think that local models are going to match what the bigger companies are putting out there in the short term, but I do find myself opting out of the more expensive models without taking a giant hit to my developer workflow, and wonder if there'll come a time when instead of choosing a lower tier model from a cloud provider I'll just decide to spin up a model on my laptop for free and achieve similar results. There's also some bigger moves towards running LLMs for phone applications, and I imagine that'll probably continue feeding the development of these models that could potentially run on every device locally.

I do want to acknowledge that without the company culture at Capacities my views about AI in general could be vastly different. The point I made earlier about "letting go of performance metrics" is not something many engineers can do within their organizations, and I personally believe that'll come at a detriment for both them and the products they're building. I do think it's important to find time to research, try out new things, reflect on what you've learned so far and expand your interests in whatever area it is that you are curious about. Not out of fear of falling behind or not meeting expectations, but out of passion for what brought you into the field in the first place. Of course working at a place that encourages this feels like a luxury, but I can't see how other companies expect to hire (and keep) dedicated engineers with the current industry trends at the large companies.

Hope you enjoyed reading the article!