Make sure you read up on latency requirements for k8s too. It’s definitely doable but etcd for example needs really low latency between nodes to not cause issues by default.

If you only have one master like you described, it’d probably be fine.

I already replied to a different thread, but figured I’d comment on some of the other options too. My vote is for Silverbullet, but I’ve tried way too many note taking tools.

• Joplin: I ran into multiple syncing issues that caused data loss and large numbers of conflicted files. I’m pretty sure these were all fixed a long time ago, but it was annoying. The dev was always good about fixing issues when they came up. It takes forever to sync on my devices and only syncs while the app is open with the screen on. The format it exports markdown files in isn’t standard, so I had to write my own scripts to export from joplin to markdown and preserve metadata.
• Standard Notes: I was willing to pay for this, but it’s extremely slow. Their support said it’s because it loads everything into memory, which I’d expect to be terrible on mobile with large databases. It’s also pretty limited in what you can do on the free self-hosted version.
• Obsidian: I really like obsidian’s ui/ux, and my only complaint is that it’s not OSS. I’d even be happy if they offered a self-hosted sync solution. There are some third party solutions for syncing, but they aren’t as smooth as the paid sync.
• Trillium: I love Trillium. I would vote for it, but it recently entered into maintenance mode. The community is working to start a new fork and I’m sure it will be great, but it’s too new to know where things will go yet. Trilium lets you encrypt specific notes and also has a cool plugin system where the plugin scripts are just notes in the database. It does have a mobile interface, but it’s a bit limited compared to the desktop interface and also doesn’t have an option to sync notes to use offline.
• Silverbullet: My current choice. I use it between windows, macos, and an android phone. I leave all three clients on sync mode all the time. The interface is minimalistic, but offers everything I need for notes and documentation so far. One of the rare “markdown” tools that actually save your content to markdown files and not to a database with the ability to export to markdown. It also has a cool feature built in where it indexes all of your notes/tasks/paragraphs and lets you build queries around them sort of like the dataview plugin for obsidian.
• Emacs: I haven’t seen emacs mentioned yet, but emacs+org-mode is still great. The mobile apps just don’t live up to the desktop experience, and you’d still have to figure out how to sync your notes yourself. Logseq’s outliner format is a similar feel afaict

Another +1 for silverbullet from me too. I was skeptical about only having a PWA for mobile access, but it actually works really well.

It’s open source, so somewhat more customizable than obsidian since you can see the inner workings of it. There aren’t too many active community plugins yet, but there is a relatively new concept of ‘Space Scripts’ where you can write simple functions/commands directly in a note (markdown file) to extend silverbullet without even needing to write a real plugin. That’s been an amazing addition for me.

Ollama and localai can both be run on a server with no gpu. You’d need to point a different web ui to them if you want though

Storage is hard to do right :(

If you can get away with it, use a separate NAS that exposes NFS to your other machines. Iscsi with a csi might be an option too.

For databases, it’s usually better to not put their data on shared storage and instead use the databases built in replication (and take backups!).

But if you want to go down the rabbit hole, check out ceph, glusterfs, moosefs, seaweedfs, juicefs, and garagehq.

Most shared file systems aren’t fully posix compliant so things like file locking may not work. This affects databases and sqlite a lot. Glusterfs and moosefs seen to behave the best imo with sqlite db files. Seaweedfs should as well, but I’m still working on testing it.

+1 for Nomad. Ive used k8s a lot and still use it, but i prefer Nomad for home purposes. You dont even need a consul cluster to run it anymore so it’s pretty simple to start.

The second isn’t a bad idea if it’s in combination with the first. Then you have an image you can restore with most of your config and you can just restore the rest from the normal backups.

What I do right now is I have a rclone sidecar container that uploads files in a directory every few seconds, and I also have another init sidecar that runs before the main application and downloads those files (incl sqlite dbs) to the normal disk. This works okay but feels pretty clunky and can still result in stuff getting corrupted because I’m just backing up the db files and not using any sqlite commands to actually back up the db to another file that isn’t in-use first.

How do you handle a job going from one nomad node to another? Or do you pin jobs like grafana to specific hosts?

Thanks! I’ll do some testing over the weekend and see how it goes.

While I’d love to be able to use it for postgres, I figured that wouldn’t work out well so probably won’t try it any time soon. I do have several apps that use sqlite databases though, do you think those would have any issues? e.g. trilium, ntfy, ghost

The main downside to most of the distributed/clustered storage that I’ve tried is they always seem to corrupt sqlite db files due to not supporting locking or some other posix feature. Reading through some older github issues, it looks like that is something the dev of seaweedfs fixed hopefully.

Hey, your stack is pretty similar to mine. One thing I recently started testing is Seaweedfs. I saw it listed in your repo too, how are you liking it so far? And do you use it on all of your nodes?

UptimeKuma is great, I use it for the simple “are my services up?” and is what I pay most attention to.

I still use zabbix for finer grained monitors though like checking raid status, smartctl, disk space, temperatures, etc.

I’ve been trying out librenms with more custom snmp checks too and am considering going that route instead of zabbix in the future

deleted by creator

Bookstack is really nice and user friendly. It’s probably one of my favorites.

Dokuwiki is simple and stores files in plaintext.

I haven’t used wiki.js much but I’ve heard good things about it too.

Another option if you don’t need to share the wiki with anyone would be a note tool like Trilium. It has built in support for stuff like mermaid or excalidraw diagrams.

Don’t forget to setup backups for whatever wiki you do go with, and make sure you can restore them when your wiki is broken ;)

That makes sense, it does sound better to keep it within nixos! I’ve mostly been using nixos to bootstrap servers that run nomad+docker, so beyond the system-level config, I haven’t done a lot with additional software yet.

Make sure your backups are solid and can’t be deleted or altered.

In addition to normal backups, something like zfs snapshots also help and make it easier to restore if needed.

I think I remember seeing a nextcloud plugin that detects mass changes to a lot of files (like ransomware would cause). Maybe something like that would help?

Also enforce good passwords.

Do you have anything exposed to the internet that also has access to either nextcloud or the server it’s running on? If so, lock that down as much as possible too.

Fail2ban or similar would help against brute force attacks.

The VM you’re running nextcloud on should be as isolated as you can comfortably make it. E.g. if you have a camera/iot vlan, don’t let the VM talk to it. Don’t let it initiate outbound connections to any of your devices, etc

You can’t entirely protect against zero day vulnerabilities, but you can do a lot to limit the risk and blast radius.

Iirc crowdsec is like fail2ban but blocks ips reported by other servers, not just ones attacking your server. Kinda like a distributed fail2ban I guess?

I only recently started using nix and NixOS. How’s the update process for nextcloud? Can you use the self updater?

Were you downloading master or the latest release? If you’re interested in using it, post the issue you have on their GitHub. The main dev is super helpful

I’m not 100% sure, but wasn’t ssdnodes one of the companies that offers really cheap deals without actually giving you the specs they say?

E.g. they say 64gb ram, but you actually get a VM with memory ballooning enabled and then your account gets suspended if you consistently use that much ram

For backups, consider using rsync.net. for a server, have you looked at dedicated servers before? OVH has some cheap servers every once in a while that should be better in theory than most VPS.