I'll admit it: databases scare me. I've always avoided databases in my projects because the idea of sorting and scheming all of my necessary data is an intimidating task. Up until now, if I needed to store data then I would save it to a local JSON file or put it in localStorage. However, you can only do that for so long until you start hitting limits in your implementation. So this month I've been dedicating myself to learning databases. Ideally, I just want to think in terms of JSON, so it's no real surprise that my first major database exploration has been with CouchDB, which is a NoSQL database maintained by the Apache Software Foundation. And I have to say, it's been a relaxing experience so far.
Before I discuss CouchDB and why it's so great, it's important to define the bigger picture. Proposed in 1970, SQL (Structured Query Language) is a programming language used to manage data in relational databases. Relational databases use relations (typically called tables) to store data, and they match that data by using common characteristics between them. SQL is used to create both data (in objects called tables) and the schema for that data, which describes fields as columns and a single record as a row.
Younger in comparison, NoSQL (Not Only SQL, or Not SQL) was coined in 1998 and defines a database that is self-describing, which means it does not require a schema or enforce relations between tables in all cases. All of its entries (typically called documents) are JSON, which are complete data object entities that applications can read and understand. NoSQL refers to high-performance, non-relational databases that utilize not just document-based schemas but a wide variety of data models for its entries. These databases are prized for their ease-of-use, scaling performance, resilience under pressure, and ample availability.
A CouchDB server instance has databases, which in turn store documents. Each document has a unique identifier within the database, and CouchDB provides a RESTful HTTP API for manipulating the server and its contents. Documents are the bread and butter of CouchDB, consisting of any number of fields and attachments that can contain any amount of various value types. Documents also include metadata that’s maintained by the database system (like revision history). Documents form the heart of the database and can change dynamically beyond their original definition. CouchDB applies document updates without locking the entry, to improve performance for all connections. So if another client editing the same document saves their changes first, the remaining client gets an edit conflict error when saving because the document metadata (revision history) has changed. To resolve the update conflict, the latest document version should be opened, the edits reapplied, and the update reattempted. Manipulating entire documents (as opposed to their properties) either succeeds entirely or fails completely. CouchDB never contains partially saved or edited documents because the document commitment system is ACID (Atomic, Consistent, Isolated, and Durable).
CouchDB isn't the only NoSQL database out there. In fact, there are a lot! One of the most popular options is MongoDB, which admittedly has more resources (i.e., a larger community with more tooling and marketing) than CouchDB. And MongoDB supports ad hoc querying, which makes the transition from SQL even easier if you're coming from that background. However, there is no one single perfect database for all scenarios. Each database type offers certain features and prioritizes certain tenants of database design. This is best explained by the CAP theorem, which says it's impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees: Consistency (each client always has the same view of the data), Availability (all clients can always read and write), and Partition tolerance (the system works well across physical network partitions).
CouchDB favors availability. For example, CouchDB supports full replication to mobile devices and desktop, so if you need a local database on your mobile application (or if your desktop users need to work offline and sync their work back to a server) then you should use CouchDB. CouchDB uses a replication model called Eventual Consistency. In this system, clients can write data to one node of the database without waiting for other nodes to come into agreement. The system incrementally copies document changes between nodes, meaning everything will eventually be in sync. Another important distinction is that CouchDB relies on B-tree indexes. This means that whether you have 1 entry or 15 billion, the querying time will always remain below 10 milliseconds. This is huge for performance, making CouchDB low latency and very appealing.
MongoDB, on the other hand, favors consistency. It uses a single authoritative main server that accepts all writes and updates its replicas accordingly. Replicas can be read, but they may possibly not be completely up-to-date at that specific point in time. Consequently, there is no versioning like in CouchDB. In fact, updates can happen in-place, using an algorithm to pad frequently growing documents.
There are drivers and tooling available for CouchDB in several languages, including JavaScript. CouchDB itself is manipulated via a RESTful API, and it's surprisingly easy to get started that way:
const URL = "http://127.0.0.1:5984"
function createDB(dbName) {
var req = new XMLHttpRequest();
req.open("PUT", URL + "/" + dbName, true);
req.setRequestHeader("Content-type", "application/json");
req.send();
}
function updateDB(dbName, docName, data) {
var req = new XMLHttpRequest();
req.open("PUT", URL + '/' + dbName + '/' + docName, true);
req.setRequestHeader("Content-type", "application/json");
req.send(JSON.stringify(data));
}
createDB("baseball");
updateDB("baseball", "Rangers", {"pitcher":"Nolan Ryan"});
But building an interface like this from scratch is a huge hassle and I don't recommend it! Unless absolutely necessary, you should use a community-supported driver so your interface is well-developed and well-tested. When using CouchDB in JavaScript, I recommend PouchDB, which works both in the browser and in Node. By default, PouchDB ships with a IndexedDB adapter for the browser and a LevelDB adapter in Node. By using the included adapters in both the browser and in Node, you can use PouchDB as an actual local database that's capable of syncing to a remote store, or you can simply use PouchDB as the interface for manipulating a remote store directly. PouchDB's API works the same in every environment, so you can spend less time worrying about browser/server differences and more time writing clean, consistent code. PouchDB can also be easily used with frameworks like React and Vue. You can either listen for events on a synced database and handle those events, or you can use a package like pouch-vue to handle the database directly in your components.
When I initially started digging into databases this month, I wasn't too sure how I wanted to practice my new skills. Coming from several website projects, I was more interested in working on a game or at least something more fun than functional, per se. The last game-related project that I worked on was my Pinwheel MUD Engine, and I still have much to do in terms of that project. However, the worldbuilding resources behind Pinwheel have been a personal project of mine for a long time, and many aspects are practically begging to be plucked out and formed into their own independent projects. My Pinwheel MUD is envisioned as a multiplayer hybrid of both an artificial life simulation and a role-playing game. Early in development, one of the creatures I built using the engine was a small human-like frog that exhibited a wide array of characteristics. The concept immediately gripped me and was both intuitive and fun to develop, so I'm expanding on them in a new project called Froggins.
In this game concept, you're charged with caring for a nomadic tribe of frog-like creatures who migrate across a marsh and exhibit a wide array of social traits. Part of the game is simulation-based and involves nurturing your little pets as you grow the tribe. Another part of the game is RPG-based and focuses on their adventures in the marshlands. Both gameplay modes meld together to create a story unique to the player. You can trade them with other players to cultivate desired traits within your tribe, and once your creatures mature then you can imbue them with heroic abilities and send them out on quests. The game concept is still in early development and I hope to provide more details in a future blog post.
The concept is inspired by early artificial life simulations like Creatures and browser-based virtual pet games like Neopets. Since you can train and battle your creatures, it's also a lot like Pokémon. Virtual pets have always been fascinating to me because it's interesting how much we can grow to love a digital representation of a living thing. They're ultimately just 1s and 0s on a virtual machine somewhere but they can evoke emotions all the same.
CouchDB underpins the entire project, which persists on a server handling all game logic. Clients connect via the browser and relay user interactions back to the server, which then responds with the outcome. Building this server and it's corresponding client has been an eye-opening undertaking. One of the early obstacles involved creating a communication protocol that's shared between the WebSocket server and the Vue client. I standardized the format for sending and receiving these messages. And I call them ribbits, in celebration of the project's subject matter.
Simply speaking, CouchDB itself is an HTTP server that's capable of serving HTML directly to the browser. You can also attach binary files directly to a database, as well. As a result, you can use actually serve any web application and its resources directly from a CouchDB instance, removing the intermediary server component from any traditional client–server model. The loaded page can even talk directly to the database using the JavaScript served in the page itself, which completely eliminates the server layer in some implementations! CouchDB’s features are a foundation for building standalone web applications backed by a powerful database. Even CouchDB’s own built-in administrative interface, Fauxton, is a fully functional database management application built using HTML, CSS, and JavaScript. CouchDB and web applications go hand in hand.
This type of application was once called a CouchApp before the functionality was integrated into CouchDB's design documents, which store JavaScript code in the database. When that happened, the use of CouchDB as a combined standalone database and application server became no longer recommended by CouchDB's maintainers. There are significant limitations to a pure CouchDB web server application stack, including security, templating, and tooling. The developers of CouchDB recommend that we use CouchDB only as the database layer, in conjunction with a web application framework like Vue, Angular, or React.
Having dipped my toes into the waters of NoSQL and CouchDB, my fears and anxieties about databases have washed away! I've set up my own CouchDB instance on a server and configured it for use on my local machine. Now that I've learned the basics, I can't wait to integrate it into my pre-existing projects and any new ones!