[{"data":1,"prerenderedAt":424},["ShallowReactive",2],{"blog-migration-to-postgres":3,"related-building-capacities":371},{"id":4,"title":5,"authorKey":6,"body":7,"category":351,"date":352,"dateFormatted":353,"description":354,"extension":355,"icon":356,"iconColor":356,"imagePath":357,"keywords":358,"meta":362,"navigation":363,"path":364,"readingTime":365,"relatedArticles":366,"seo":368,"stem":369,"__hash__":370},"blog/blog/migration-to-postgres.md","Scaling Capacities: Why we swapped Dgraph for PostgreSQL","luis",{"type":8,"value":9,"toc":333},"minimark",[10,14,40,49,53,60,63,81,88,92,95,100,103,106,126,129,149,153,164,189,193,200,204,207,237,241,264,276,282,285,288,292,295,306,310],[11,12,5],"h2",{"id":13},"scaling-capacities-why-we-swapped-dgraph-for-postgresql",[15,16,17,18,22,23,30,31,39],"p",{},"When building a ",[19,20,21],"em",{},"tool for thought"," like ",[24,25,29],"a",{"href":26,"rel":27},"https://capacities.io/",[28],"nofollow","Capacities",", the data model is the product. Our users literally create webs of connected notes, so choosing a graph database initially felt like the most authentic architectural decision to the founders. ",[24,32,35],{"href":33,"rel":34},"https://docs.dgraph.io/",[28],[36,37,38],"strong",{},"Dgraph"," was chosen because it promised native graph traversal and a schema that matched the user's mental model, it made it easy to move fast and express queries in a way that closely matched how we think about data.",[15,41,42,43,48],{},"However, as time went on, they realized that the \"perfect\" theoretical fit was becoming an operational nightmare and with the release of ",[24,44,47],{"href":45,"rel":46},"https://capacities.io/whats-new/release-45",[28],"Full Offline Mode",", Capacities was much less reliant on having a graph architecture on the backend. After joining the team, making a switch to a more efficient database became my first project as the new backend engineer.",[11,50,52],{"id":51},"the-problem-the-graph-tax","The Problem: The Graph Tax",[15,54,55,56,59],{},"Our primary challenge wasn't functionality, it was ",[36,57,58],{},"CPU consumption",". Dgraph’s resource usage was unpredictably high, even with modest datasets. We found ourselves at a crossroads: horizontally scale a complex cluster (without the traffic to justify it) or vertically scale our servers to much more expensive machines.",[15,61,62],{},"We weren't alone. Community reports confirmed our suspicions:",[64,65,66,74],"ul",{},[67,68,69],"li",{},[24,70,73],{"href":71,"rel":72},"https://github.com/dgraph-io/dgraph/issues/1938",[28],"Dgraph Issue #1938: High CPU usage",[67,75,76],{},[24,77,80],{"href":78,"rel":79},"https://discuss.dgraph.io/t/one-node-in-dgraph-cluster-showing-unusual-resource-usage/19832",[28],"Unusual resource usage discussions",[15,82,83,84,87],{},"After doing some research and having many discussions, we finally decided it was time to move to something \"boring\" and reliable: ",[36,85,86],{},"PostgreSQL on a cloud provider",".",[11,89,91],{"id":90},"redesigning-the-graph-in-a-relational-postgresql-database","Redesigning the Graph in a Relational PostgreSQL Database",[15,93,94],{},"Migrating from a loosely enforced graph schema to a strictly typed relational database is a massive normalization challenge. I worked closely with Steffen, our founder, to audit every assumption I had about the data. The migration wasn't just a data transfer; it was a complete structural redesign.\nMy first task was to deconstruct our Dgraph schema. We needed to translate \"nodes and edges\" into a performant relational model.",[96,97,99],"h3",{"id":98},"the-links-and-notes-schema","The \"Links\" and \"Notes\" Schema",[15,101,102],{},"The main challenge was normalization. Dgraph’s data model is inherently denormalized, but to take full advantage of PostgreSQL we had to redesign our schema around normalization principles. This forced us to rethink how we modeled data and to carefully review every backend query, ensuring we weren’t merely trading CPU bottlenecks in a graph database for greater response times in a relational system due to slow joins.",[15,104,105],{},"We eventually landed on a clean, highly indexed structure where tables could be categorized as:",[64,107,108,118],{},[67,109,110,117],{},[36,111,112,116],{},[113,114,115],"code",{},"objects"," tables",": Contains the core nodes and their properties.",[67,119,120,125],{},[36,121,122,116],{},[113,123,124],{},"links",": A dedicated table representing the edges (from node A to node B).",[15,127,128],{},"To safely replace the engine while the car was moving, we followed a three-step technical strategy:",[64,130,131,137,143],{},[67,132,133,136],{},[36,134,135],{},"The DatabaseService Interface",": We wrote an abstracted interface in our backend. This allowed us to rewrite Dgraph interactions in one place and provided a roadmap of every query we needed to port.",[67,138,139,142],{},[36,140,141],{},"Choosing Kysely to interact with Postgres",": It gave us the best of both worlds, strong TypeScript typing and the freedom to write raw-like SQL for complex traversals.",[67,144,145,148],{},[36,146,147],{},"Recursive Power",": Traversing connected notes in SQL is no small feat. We utilized WITH RECURSIVE Common Table Expressions (CTEs), heavily testing them with EXPLAIN ANALYZE to optimize our indexing strategy.",[11,150,152],{"id":151},"tooling-why-kysely","Tooling: Why Kysely?",[15,154,155,156,163],{},"We spent significant time evaluating ORMs and different alternatives. We ultimately chose ",[24,157,160],{"href":158,"rel":159},"https://kysely.dev/",[28],[36,161,162],{},"Kysely"," for three reasons:",[165,166,167,173,179],"ol",{},[67,168,169,172],{},[36,170,171],{},"Type safety",": It provides end-to-end type safety without a heavy runtime overhead.",[67,174,175,178],{},[36,176,177],{},"Raw SQL control",": Unlike traditional ORMs that hide the SQL, Kysely feels like writing raw SQL without losing strong typing.",[67,180,181,184,185,188],{},[36,182,183],{},"Compatibility with complex queries",": Our most critical queries, which traversed the graph, required the use of ",[113,186,187],{},"WITH RECURSIVE"," and good column indexing.",[96,190,192],{"id":191},"optimizing-with-explain-analyze","Optimizing with EXPLAIN ANALYZE",[15,194,195,196,199],{},"The recursive logic was the core of the new system. We spent days running ",[113,197,198],{},"EXPLAIN ANALYZE"," on our queries to determine the perfect indexing strategy. Kysely’s migration tool made it incredibly simple to iterate on these indexes as we discovered bottlenecks in our local Docker environments.",[11,201,203],{"id":202},"the-zero-downtime-rollout","The Zero-Downtime Rollout",[15,205,206],{},"We couldn't just turn off the lights. We followed a rigorous rollout plan:",[165,208,209,215,221,227],{},[67,210,211,214],{},[36,212,213],{},"Double Writing",": We modified our backend to write every change to both Dgraph and RDS.",[67,216,217,220],{},[36,218,219],{},"Consistency testing:"," During local and staging testing, we compared read results from both databases to ensure consistency in our queries.",[67,222,223,226],{},[36,224,225],{},"The Two-Week Migration",": We ran a background job that slowly moved historical data from Dgraph to Postgres. This took two weeks to ensure we didn't overwhelm the production instance.",[67,228,229,232,233,236],{},[36,230,231],{},"Feature Flags",": Using a flag system, we toggled individual ",[113,234,235],{},"DatabaseService"," modules from Dgraph to Postgres one by one. This allowed us to monitor real-world performance for specific queries and \"roll back\" instantly if we saw a spike.",[11,238,240],{"id":239},"the-dirty-data-challenge","The \"Dirty\" Data Challenge",[15,242,243,244,247,248,251,252,255,256,259,260,263],{},"When we moved to staging on RDS, we hit a wall: ",[36,245,246],{},"Legacy Data.","\nA few years ago, Capacities used string literals like ",[113,249,250],{},"\"default\"",", ",[113,253,254],{},"\"root\"",", or ",[113,257,258],{},"\"basic\""," where we now expected UUIDs. We wanted to keep our Postgres columns as strictly typed ",[113,261,262],{},"UUID"," types for performance, but we couldn't just delete old user data.",[15,265,266,269,270,275],{},[36,267,268],{},"The Solution:"," We built a custom ",[36,271,272],{},[113,273,274],{},"QueryBuilder"," module on top of Kysely. It included a literal mapper that swapped these legacy strings for \"fake\" but consistent UUID values. To ensure this was safe, we ran a background job on production to extract all unique legacy values and verify the set was small enough to map manually.",[15,277,278,279],{},"After this, we thought we were in the clear until we started the background migration job. Suddenly, the logs were filled with errors: ",[113,280,281],{},"invalid byte sequence for encoding \"UTF8\".",[15,283,284],{},"Dgraph, written in Go, was surprisingly permissive with what it stored. It had allowed \"broken\" Unicode values, null bytes, and certain malformed emoji sequences to sit in the database for years. PostgreSQL, however, is a strict guardian of data integrity. It rejected these values outright.",[15,286,287],{},"We had to build a sanitization layer into our migration script to strip out these \"ghost\" characters and fix malformed UTF-8 sequences, including broken emoji encodings on the fly. It was a stark reminder that moving data isn't just about moving bits; it's about translating between two different philosophies of data validation.",[11,289,291],{"id":290},"the-results-110th-of-the-cost","The Results: 1/10th of the Cost",[15,293,294],{},"The impact was immediate and dramatic. As we toggled each module, we watched the server CPU metrics drop.",[15,296,297,298,301,302,305],{},"By moving to Postgres, we were able to reduce our database costs to ",[36,299,300],{},"1/10th"," of their original price. This meant ",[36,303,304],{},"70% annual savings on overall infrastructure"," costs.",[96,307,309],{"id":308},"key-learnings","Key Learnings",[64,311,312,318,327],{},[67,313,314,317],{},[36,315,316],{},"Boring is Beautiful",": Postgres has decades of community support, documentation, and tooling. When things go wrong, the answer is usually one Google search away. Postgres can also be highly optimized for a wider range of use cases than other database technologies, so it made us feel safer because we could prototype different implementations and compare metrics without completely switching to another technology.",[67,319,320,323,324,326],{},[36,321,322],{},"Interfaces are your friend",": The ",[113,325,235],{}," abstraction was incredibly helpful for us to gather all calls of Dgraph throughout the codebase and understand exactly what functionality needed to be provided by a new PostgresService module. At Capacities we’re big fans of dependency injections, and this was just one more example where it proved tremendously useful.",[67,328,329,332],{},[36,330,331],{},"Managed Services > Self-Hosting",": For a small team, the \"premium\" of a managed database instance is a bargain compared to the engineering hours spent managing self-hosted Dgraph clusters. Capacities is not a database company, the focus is better spent innovating in knowledge work instead of in managing infrastructure.",{"title":334,"searchDepth":335,"depth":335,"links":336},"",2,[337,338,339,343,346,347,348],{"id":13,"depth":335,"text":5},{"id":51,"depth":335,"text":52},{"id":90,"depth":335,"text":91,"children":340},[341],{"id":98,"depth":342,"text":99},3,{"id":151,"depth":335,"text":152,"children":344},[345],{"id":191,"depth":342,"text":192},{"id":202,"depth":335,"text":203},{"id":239,"depth":335,"text":240},{"id":290,"depth":335,"text":291,"children":349},[350],{"id":308,"depth":342,"text":309},"dev","2026-01-12","Jan 12, 2026","How we achieved a 70% infrastructure cost reduction by migrating from a self-hosted graph database to a managed relational model.","md",null,"/blog/migration-to-postgres.jpg",[359,360,361],"databases","infrastructure","migration",{},true,"/blog/migration-to-postgres","7 min",[367],"building-capacities",{"title":5,"description":354},"blog/migration-to-postgres","kgphIaNQLkaCWAm1Hvs_Ex_4W0CrKP3k5bh_C1JeBNE",[372],{"id":373,"title":374,"authorKey":375,"body":376,"category":407,"date":408,"dateFormatted":409,"description":410,"extension":355,"icon":356,"iconColor":356,"imagePath":411,"keywords":412,"meta":415,"navigation":363,"path":416,"readingTime":417,"relatedArticles":418,"seo":421,"stem":422,"__hash__":423},"blog/blog/building-capacities.md","How we decide which new features to add","steffen",{"type":8,"value":377,"toc":405},[378,381,384,387,390,393,396,399,402],[15,379,380],{},"We often get asked how we decide which features we add to @CapacitiesHQ. 🗺️",[15,382,383],{},"Here’s the answer. 👇",[15,385,386],{},"Very important: Every feature starts with a need or problem. There should never be a feature if it’s not solving a real problem. This cannot be stressed enough. Otherwise, you’re building a product with no real value and a lot of clutter.",[15,388,389],{},"We draw most feature inspiration from our community's feedback and by using the product ourselves. This helps us identify problems and friction points that stop users from getting their work done. We sometimes joke that our community is our first employee: a rigorous tester, opinion consolidator, and key source of inspiration.",[15,391,392],{},"Feature requests and explicit product improvements can be added to our feedback board, voted and commented on by other users. We review and cluster all requests and analyze their importance. This helps us to get a better understanding of the different needs and problems of our users.",[15,394,395],{},"In long and intense discussions, we combine all requests, comments, and ideas with our intuition and vision for what Capacities should become. We always radically aim for usefulness and simplicity – Capacities should just work and help, nothing else.",[15,397,398],{},"When features are big and fundamental, we share proposals or plans in our community for feedback. This ensures that we constantly have our users at the table with us. We fine-tune and iterate until we find a draft that fits into the greater picture.",[15,400,401],{},"Then we start creating. After a first draft, we introduce an early version to a small, selected group of alpha testers. In intense and incredibly helpful discussions, we optimize the last 20% in a combined effort. We refine the feature by addressing issues, polishing details, and enhancing interactions.",[15,403,404],{},"After a feature is released to all users, it’s not done. We constantly improve, iterate, and measure how it fits into users' workflows. Every feature is part of a continuous evolution of the overall product. We constantly strive to build a unified and consistent product – a beautiful environment to think and work in: a Studio for your Mind.",{"title":334,"searchDepth":335,"depth":335,"links":406},[],"team-product","2024-03-04","Mar 6, 2024","What's our decision making process? Here's the answer.","/blog/building-capacities.jpg",[413,414],"Feature development","Build in public",{},"/blog/building-capacities","2 min",[419,420],"why-the-name-capacities","how-it-started",{"title":374,"description":410},"blog/building-capacities","yUEsybivkdEpilRjL0YjnGhgRSjIYeDYSFuYsPOVPcQ",1775642834219]