DokuWiki

Edit the page and hit Save. See syntax for Wiki syntax. Please edit the page only if you can improve it. If you want to test some things, learn to make your first steps on the playground.

03/ Month - DEV Neighborhood

[[https://www.reference.com/world-view/static-balance-dcdd696e1db32ba6?ad=dirN&qo=paaIndex&o=740005&origq=balanced|(Image: [[https://dribbble.com/search/shots/popular/20src=|https://dribbble.com/search/shots/popular/20src=]])]]Here's a riddle. My web app keeps all of its information in a SQL database. I can spontaneously tear it down, release the code to a various hosting platform, and the app will still serve all the exact same information. Running my app in production expenses $0.03 monthly.

How is this possible?
That's simple. You have a different database server running someplace that shops all of your app's state.

No, my app never ever speaks to a remote database server.
Oh, then you're utilizing a proprietary, handled datastore like Amazon DynamoDB or Google Cloud Firestore.

Nope, my whole stack is open-source and platform-agnostic.
Then what?

I integrated SQLite, Litestream, and Docker.
My tool is called LogPaste. It enables users to create shareable URLs for text files. I utilize it in my open-source KVM over IP gadget so that users can quickly share diagnostic logs with me.

Sharing text files isn't precisely innovative, however serverless information duplication may be. Here's a demonstration of me moving my LogPaste app server in between 2 different hosting platforms: Heroku and fly.io. There's no database server or information migration action, however all of my information continues in between platforms:

The finest part is that I didn't require to customize my app's code at all. It simply composes to a regional SQLite database, and Litestream amazingly deals with information duplication in the background.

In this post, I'll discuss how I incorporated Litestream into my app and how you can do the very same to change your costly, complex database servers.

Information determination for individuals who dislike database servers
My disgraceful developer trick is that I can't preserve a database server.

I have actually been constructing my own software application items and services for the previous 8 years, and I have actually never ever utilized a database server in production. I do not wish to be accountable for backups or software application upgrades, so anything that needs MySQL, Postgres, or Redis is a dealbreaker for me.

Rather, I have actually constantly utilized Google-managed datastores like Cloud Datastore, Firebase, and Firestore. However every couple of years, Google constructs an absolutely brand-new datastore service, deprecates its old one, and disposes all the migration work onto its clients. I didn't wish to produce another service on top of a tech stack that Google would most likely exterminate quickly.

Litestream: the serverless database server
A couple of months earlier, I saw that Ben Johnson, author of the popular Bolt database, had actually handled a brand-new job: Litestream. It's a basic, open-source tool that reproduces a SQLite database to Amazon's S3 cloud storage.

It appeared cool, however I wasn't especially delighted about it. I never ever utilize SQLite, so what did I care?
I didn't have anything versus SQLite, however the style appeared not practical. Unlike other databases that send out information to an external server over the network, SQLite composes whatever to a regional file. I constantly stressed, "What occurs if I lose that file?"

Thinking of it more, I understood I 'd dismissed Litestream since I didn't utilize SQLite. However Litestream resolved the really barrier keeping me from embracing SQLite ... Perhaps this deserved a shot.

Even much better, Litestream might be my ticket out of Google Cloud Platform. SQLite runs anywhere, so I 'd have liberty in picking server hosting platforms. Litestream supplies supplier versatility on the storage side, as it supports any S3-compatible service, consisting of BackBlaze B2, Wasabi, and Minio.

Litestream sounded rosy in theory, however you can't evaluate an innovation till you check it in production. I required a log upload service, and it appeared like the ideal job to check Litestream.

Developing the fundamental performance
LogPaste required to accept HTTP PUT demands from the command-line, so I composed this easy HTTP handler in Go:

The InsertEntry application looks how you 'd anticipate. It's a standard SQLite row insertion:
This permits LogPaste to accept HTTP demands from the command  [[https://www.accuit.ca/managed-it-services/|accuit]] line like this:

That works, however it's composing the SQLite database to the regional filesystem. I required to incorporate Litestream to allow cloud storage.

Layering in Litestream for cloud information syncing
Among Litestream's greatest strengths is that it's entirely independent of the application it serves. My LogPaste code never ever calls into a Litestream API, nor does it need any unique setup to enable syncing. Litestream silently does its task in the background.

I produced a custom-made Docker image to integrate Litestream and LogPaste. Normally, Docker images need to hold Simply One Service, however I in some cases flex this guideline to help with release. It's orders of magnitude simpler to release a single, independent Docker container than 2 containers that require to collaborate with each other.

LogPaste's Dockerfile begins by constructing the LogPaste binary from source, and after that it takes down the Linux executable for Litestream.

Next, Docker copies a customized litestream.yml file into the image. This is Litestream's setup file.

The replicas.url field includes the cloud storage area for my database. access-key-id and secret-access-key are the IAM-style qualifications Litestream requires to access the cloud storage container.

You can hardcode these worths into the setup file, however Litestream supports environment variables and inserts them at runtime. That's a hassle-free function, as it enables you to keep your litestream.yml file under source control without keeping any delicate qualifications. It likewise makes the Docker image portable - anybody can produce their own LogPaste server by recycling my image. Setting environment variables for their cloud storage container.

The next little bit of Litestream reasoning remains in LogPaste's docker_entrypoint script, which runs when the Docker container launches. It begins by taking down the app's most current database photo from cloud storage:

The -if-replica-exists flag informs Litestream that it's alright if no pictures exist on cloud storage yet. Otherwise, you 'd have a chicken-and-egg issue. Your app might never ever introduce since there's no cloud database to bring back, however Litestream can't duplicate the database to cloud storage since the app has actually never ever run.

Next, the entrypoint script generates a Litestream procedure, which views LogPaste's SQLite database and continually streams any modifications to cloud storage:

The small hack remains in that routing &. It informs the script to run the Litestream procedure in the background, which is how I can carry out 2 long-running procedures in the very same Docker container. Ben Johnson has actually released a cleaner option, however I'm utilizing the hacky variation for ease of presentation.

The entrypoint script ends by introducing the Logpaste app, which is a basic HTTP server:
To run the Docker container with all the environment variables occupied, I utilize this command:

Here's how everything fits together in production:
Utilizing LogPaste in production

I'm utilizing LogPaste in production for TinyPilot, my open-source KVM over IP gadget. Due to the fact that users run my software application on gadgets they own, I can't see any diagnostic info when they report concerns. LogPaste supplies a hassle-free method for users to share their logs with me.

LogPaste has actually managed all of TinyPilot's debug logs for the previous couple of months, and it's worked well. The expense for information duplication genuinely is simply $0.03 each month:

My usage case is,  [[https://www.accuit.ca/exchange-email-hosting-canada/|hosted exchange canada]] undoubtedly, relatively mild. Just a handful of users submit their logs every day, so there might be discomfort points with this setup under much heavier work.

It's likewise crucial to keep in mind that Litestream can't fix disputes in between numerous database composes, so each database can have just one application server with compose gain access to.

Still, I have actually been exceptionally impressed with Litestream, and I'm excited to utilize it in more situations.
Self-hosting LogPaste

It's easy to deploy if you desire to host your own circumstances of my LogPaste app. You can even tailor the text on the homepage so that it states your item's name rather of "LogPaste."

For instance, here's TinyPilot's variation:
I have actually composed implementation directions for a couple of various platforms:

Additional reading
- Litestream: Litestream's main documents. MIT-licensed source code. Paperwork for LogPaste. - litestream-s6-example: A more innovative and robust technique for running Litestream together with your app in a Docker container. It utilizes s6-overlay to reboot the Litestream circumstances on failure.

Architecture diagram by Loraine Yow. Thanks to Ben Johnson for his deal with Litestream. His early evaluation of this short article. Thanks to the members of the Blogging for Devs Neighborhood for offering feedback on this post.

Edit summary

Note: By editing this page you agree to license your content under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International