We’ve spent the last 5 years building umbrelOS to make self-hosting accessible. Yesterday, we launched our dream hardware: Umbrel Pro.
Specs:
- 4x NVMe SSD slots for storage (tool-less operation)
- Intel N300 CPU (8 cores, 3.8GHz)
- 16GB LPDDR5 RAM
- 64GB onboard eMMC with umbrelOS
The chassis is milled from a single block of aluminum and framed with real American Walnut wood.
Here is a video of the manufacturing process if you want to nerd out on the machining details: https://youtu.be/4IAXfgBnRe8
Also, we built a "FailSafe" mode in umbrelOS, powered by ZFS raidz1. The coolest part is the flexibility: you can start with a single SSD and enable RAID later when you add the second drive (without wiping data), or enable it from day one if you start with multiple drives.
We also really obsessed over the thermal design. The magnetic lid on the bottom has a thermal pad that makes direct contact with all 4 NVMe SSDs, transferring heat into the aluminum. Air is pulled through the side vents on the lid, flows over the SSDs, then the motherboard/CPU, and exits the back. It runs whisper quiet.
Lots more details on our website, but we’ll be hanging out in the comments to answer any questions :)
FWIW, this is what Gemini thinks you are likely doing. Is this correct, or close?
The Trick: The "Sparse File" Loopback
Since ZFS doesn't allow you to convert a single disk vdev to RAID-Z1, Umbrel's "FailSafe" mode almost certainly uses a sparse file to lie to the system.
Phase 1 (Single Drive): When you set up Umbrel with one 2TB SSD, they don't just create a simple ZFS pool. They likely create a RAID-Z1 pool consisting of your physical SSD and two "fake" virtual disks (large files on the same SSD).
The "Degraded" State: They immediately "offline" or "remove" the fake disks. The pool stays in a DEGRADED state but remains functional. To you, the UI just shows "1 Drive."
Phase 2 (Adding the 2nd Drive): When you plug in the second drive, umbrelOS likely runs a zpool replace command, replacing one of those "fake" virtual disks with your new physical SSD.
Resilvering: ZFS then copies the parity data onto the second disk.
Great question! Close, but not exactly. We do use a sparse file but only very briefly during the transition.
We start with 1 SSD as a single top level vdev. When you add the second SSD you choose if you want to enable FailSafe or not. If you don't enable FailSafe you can just keep adding disks and they will be added as top level vdevs. Giving you maximum read and write performance due to striping data across them. Very simple, no tricks.
However if you choose FailSafe when you add your second SSD, we then do a bit of ZFS topology surgery, but only very briefly. So you start with a ZFS pool with a single top level vdev running on your current SSD. And you just added a new unused SSD and chose to transition to FailSafe mode. First we create a sparse file sized to the exact same size as your current active SSD. Then we create an entirely new pool with a single top level raidz1 vdev backed by two disks, the new SSD, and the sparse file. The sparse file acts as a placeholder for your current active SSD in the new pool. We then immediately remove the sparse file so this new pool and dataset is degraded. We then take a snapshot of the first dataset, and sync the entire snapshot over to the new pool. The system is live and running off the old pool for this whole process.
Once the snapshot has completed we then very briefly reboot to switch to the new pool. (We have the entire OS running on a writable overlay on the ZFS dataset). This is an atomic process. Early on in the boot process, before the ZFS dataset is mounted, we take an additional snapshot of the old dataset, and do an incremental sync over to the new dataset. This is very quick and copies over any small changes since the first snapshot was created.
Once this sync has completed, the two separate pools now contain identical data. We then mount the new pool and boot up with it. Then we can destroy the old pool, and attach the old SSD to the new pool, bringing it out of degraded state. And the old SSD will be resilvered in the new pool. The user is now booted up on a two wide raidz1 dataset on the new pool with bit-for-bit identical data that they shutdown on with the single ssd dataset on the old pool.
Despite sounding a bit wacky, the transition process is actually extremely safe. Apart from the switch over to the new dataset, the entire process happens in the background with the system online and fully functional. The transition can fail at almost any point and it will gracefully roll back to the single SSD. We only nuke the old single SSD at the very last step, so either we can roll back, or they have a working raidz1 array.
It sounds bad that the raidz1 goes through a period of degradation, but there is no additional risk here over not doing the transition. They are coming from a single disk vdev that already cannot survive a single disk failure. We briefly put them through a degraded raidz1 array that can also not survive a single disk loss, (no less risky than how they were already operating), to then end up at a healthy raidz1 array that can survive a single disk loss, significantly increasing the safety in a simple and frictionless way for the user.
Using two wide raidz1 arrays also get's a bit of a kneejerk reaction but it turns out for our use case the downsides are practically negligible and the upsides are huge. Mirrors basically give you 2x read speed over two disk raidz1. And less read intensive rebuilds. Everything else is pretty much the same or the differences are negligible. It turns out those benefits don't make a meaningful difference to us. A single SSD can already far exceed the bandwidth required to fully saturate our 2.5GbE connection. The additional speed of a mirror is nice but not really that noticeable. However the absolute killer feature of raidz is raidz expansion. Once we've moved to a two disk wide raidz1 array, which is not the fastest possible 2 disk configuration, but more than fast enough for what we need, we can add extra SSDs and do online expansions to a 3 disk raidz1 array and then 4 disk raidz1 array etc. As you add more disks to the raidz1 array, you also stripe reads and writes across n-1 disks, so with 4 disks you exceed the mirror perf benefits anyway.
In theory we could start with one SSD, then migrate to a mirror with the second SSD, and then again migrate to a 3 disk raidz1 array using the sparse file trick. However it's extra complexity for negligible improvements. And when moving from the mirror to the raidz1, you then degrade the user AFTER you've told them they're running FailSafe. Which changes the transition process from a practically zero additional risk operation, to an extremely high risk operation.
Ultimately what we think this design gives us is the simplest consumer RAID implementation with the highest safety guarantees that exist today. We provide ZFS level data assurance, with Synology SHR style one-by-one disk expansion, in an extremely simple and easy to use UI.
Thanks for the thorough answer. It is a little wacky and complicated but I agree it should be safe. I'm not really in the target market for your software but the hardware does look very nice. Good luck with it.
If you start with one SSD, how can you later make that into a raidz1 of two? Also, a raidz1 of two block devices does not seem like a really great idea.
It's powered by Nous Hermes Llama2 7b. From their docs: "This model stands out for its long responses, lower hallucination rate, and absence of OpenAI censorship mechanisms. [...] The model was trained almost entirely on synthetic GPT-4 outputs. Curating high quality GPT-4 datasets enables incredibly high quality in knowledge, task completion, and style."
The main difference is setting everything up yourself manually, downloading the modal, optimizing the parameters for best performance, running an API server and a UI front-end - which is out of reach for most non-technical people. With LlamaGPT, it's just one command: `docker compose up -d` or one click install for umbrelOS home server users.
Gpt4all[1] offers a similar 'simple setup' but with application exe downloads, but is arguably more like open core because the gpt4all makers (nomic?) want to sell you the vector database addon stuff on top.
I like this one because it feels more private / is not being pushed by a company that can do a rug pull. This can still do a rug pull, but it would be harder to do.
Maybe I've been at this for too long and can't see the pitfalls of a normal user, but how is that easier than using an oobabooga one-click installer (an option that's been around "forever")?
I guess ooba one-click doesn't come with a model included, but is that really enough of a hurdle to stop someone from getting it going?
Maybe I'm not seeing the value proposition of this. Glad to be enlightened!
It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.
It's an entire app (with a chatbot UI) that takes away the technical legwork to run the model locally. It's a simple one line `docker compose up -d` on any machine, or one click install on umbrelOS home servers.
Good luck! The token/sec will be under your expectations or it will overheat. You really shouldn't play games with your data-storage. You could try it with an old laptop to see how bad it performs. Ruining your NAS for this is a bit over the top to show, that "it worked somehow".
But i don't know, maybe your NAS has a powerful processor and is tuned to the max and you have redundancy and don't care to loose a NAS?
Or this was just a joke and i fell for it! ;)
Curious what aspects of Umbrel do you find not satisfactory? I know you said you'd like to see something less crypto focused. Is it the selection of crypto-oriented apps in the app store that you don't like to see? (And I totally get if that's the case, just trying to understand.)
Ah unfortunately that's not supported. When running umbrelOS on a Raspberry Pi, it stores everything on the HDD/SSD you connect to the Pi on install to make sure your data isn't susceptible to microSD card's wear and tear. So if you change the HDD/SSD, it will format the drive and use it as the main drive. And connecting two or more drives to the Pi isn't supported because the Pi lacks power to support more than 1 drive at once, causing abnormal behavior.
Thanks for the taking the time to try Umbrel out, great observations!
1. Re crypto apps, I figured some additional context may help. Before our today's release, Umbrel was a self-hosting OS primarily geared towards Bitcoin node users. Today, we migrated the Bitcoin node to the Umbrel App Store and took the last step in our transition to becoming an app-agnostic general purpose OS. So expect to see a lot more non-Bitcoin apps hereon!
2. Yes, agree. We'll have Plex and Jellyfin live in the app store soon.
3. The main issue we found with using a single domain on the local network is that many Android phones and PCS have flaky mDNS support, in which case name resolution for "*.local" would simply fail. This is why we decided to use ports. Perhaps we can look into using ports on the local network and domain on a VPS.
4. Good suggestion! Feel free to share your recommendations.
6. Until now, a common use case of our users has been remote connection between Umbrel and their Bitcoin wallets over Tor. This is why remote access was baked directly into Umbrel and turned-on by default.
However, as we've now evolved from the Bitcoin space, we'll prioritize offering the ability to disable remote Tor access functionality in the next update, and make it opt-in instead of opt-out.
Caddy has state-of-the-art certificate automation and TLS support, and with that module, it automatically updates DNS records if users have non-static IPs. It'll also serve certs for localhost domains (use *.localhost IMO).
Re 3, that's why you need to run a DNS server in your LAN, like pihole or adguard or coredns. And don't use .local, use .home.arpa instead, or use a DDNS domain like DuckDNS and make it resolve to your LAN IP with your DNS server. And use Caddy (shameless plug)
1. Makes sense, looking forward to progress there.
2. Excellent. I’d consider one of the Wireguard VPN servers be prioritized as well.
3. I wouldn’t use mDNS for it, I would either require and integrate the PiHole configuration or come with a DNS server as well (leaning towards PiHole here). I’d suggest long-term planning on integrating DNS/DDNS and LetsEncrypt. I use a combo of a DDNS container for CloudFlare and a wildcard DNS generated by nginx proxy manager.
4. I’d go for one “simple” CMS, like Ghost, and one fully featured, like WordPress.
5. Will check it out.
6. Appreciate it being an option, I’ve signed up for the mailing list to get a notification when it is available so I can make another run at it.
I’m pleased to see the support for deploying directly to your own Umbrel without going through the App Store / pull request process. This is one of my biggest frustrations with Unraid.
It would be nice to have first class support for deploying stuff this way - not just for testing. I would like to deploy custom containers / compositions on my Umbrel and see them alongside stuff installed from the official repository. Ok to require an external guy repo as upstream for this, but better to work entirely local.
> Today, we migrated the Bitcoin node to the Umbrel App Store and took the last step in our transition to becoming an app-agnostic general purpose OS
Hello, do you have plans to interop with an established selfhosting distro and package scheme? Yunohost, Freedombox and Libreserver come to mind. If you'd rather go the containerized/virtualized way, there's a dozen or so distros based on Docker/LXC/K8S to make selfhosting easier.
I'm always happy that people are building stuff for selfhosting (though like others i'm skeptical of anything cryptocurrency-related), so please don't take it as a dismissal of your work, but i don't understand the appeal of building yet another solution and package format that's not interoperable with the others who have been out there for 5/10 years and provide good services to plenty of users already.
To be fair, apart from Dockerfiles there's not exactly any decent specification for declarative sysadmin (network ports, filesystem access..). The selfhosting field could certainly use a specification for selfhosted packages across distros, because the current situation places a strong burden on volunteer maintainers to keep up with updates.
> Which ones do you have in mind? Would you count ChromeOS as one of those, too?
A few i had in mind (from my bookmarks): Cloudron, Sandstorm, HomelabOS, libre.sh, UBOS, Unraid, Helm, CasaOS, servers.coop's Capsul. In my opinion, in those virtualized solutions Sandstorm is the only one that's not a simple GUI for docker/LXC and had some actually interesting research going on (especially in terms of security). That's for generic selfhosting solutions, and i personally have no strong opinion about these as i'm more interested about bare-metal solutions that work on low-end hardware (Freedombox/Yunohost/LibreServer).
To this list you can add the free ansible/docker recipes used by friendly hosting coops such as webarch.coop or disroot.org. I'm guessing many other CHATONS.org/Libreho.st federation members also publish their recipes, but i wouldn't know for sure.
I don't count ChromeOS as anything as my understanding is it's just a web browser with a custom kernel? I may be missing something as i've never used it, and if i don't have the source code and/or have to pay Google a single cent to use it i most probably will never try it out.
Thanks for the information! To be honest, i'm still not interesting to fall into anything maintained by Google, but i see the value you're proposing.
Personally, when it comes to desktop virtualization, i'm very happy with QubesOS. It's not designed for graphics performance, but it's to my knowledge the only distro providing decent security for multi-VM graphical workloads, and their research keeps going!
I agree, that's my least favorite part of the site as well. We'll update it with a browsable directory of all the available apps in our app store soon.
Licensing is a nuanced subject so we've written a detailed post on it. But in short, our license grants you the same rights that come with an open source license, except the right to sell or commercialize Umbrel.
No, it's not Open Source. It's a perfectly justifiable decision, but it's not Open Source, nor is it equivalent.
Yes, your license means that individual users can make little patches to customize the product to their needs, and even share these customizations with other users. That's great!
But the license effectively prohibits borrowing code from your codebase for use in other projects, meaning your code does not become part of the aether of Open Source code that anyone can build upon. That's a very important part of what it means to be "Open Source".
It also effectively prevents any large-scale modification or forking effort, since maintaining patches as the underlying codebase evolves is a hard job, and the license prohibits people from funding such effort. If users want timely security updates, they will need to stick close to your version of the codebase. So the lock-in is there.
Again, this is all a perfectly justified direction for you to take. I don't blame you at all, and I definitely understand that it's Amazon's fault that we cannot have nice things. But it's not Open Source.
On a tangentially-related note, a little tip: You have defined all noncommercial organizations -- including education, public research, and government -- as being permitted users. That may be dangerous. I was the founder of Sandstorm, and these organizations were exactly the ones most interested in paying for our product -- literally the only big sales we ever made were a couple universities, a big research org, and a government. Despite being non-profit, these orgs have lots of money and a need for self-hosting.
Thanks for the insightful reply. Everything you said makes total sense.
Re noncommercial organizations being permitted users, this was a conscious decision on our part. We're purposefully building Umbrel purely for consumers, and don't plan to serve any commercial or noncommercial organizations. We want to align our incentives directly with consumers instead of enterprises, and this is purely to help us focus on building for the user-base that excites us the most.
I'm a big fan of Sandstorm btw. It was way ahead of its time.
I see. Well, I hope you are able to succeed at that. For what it's worth, we initially focused on consumer use but weren't able to find a path to revenue there.
Despite a lot of noise on HN, we had only a few hundred signups for our paid hosting service. We built super-scalable hosting tech but it turned out we could have hosted them on a single big VM all along... oops.
I think the problem is that the apps, while functional, weren't competitive with their SaaS competitors, and so the only reason to use the hosting service was if you really cared about the Open Source aspect. Maybe if we had a killer app that was actually better than any SaaS alternative, we could have gotten somewhere? But we never found that.
Meanwhile, we got a lot of feedback from people working at big orgs that were forced to self-host for regulatory reasons. Such orgs are terribly served by the current software market, since they can essentially only buy software from companies that specialize in building regulated software, and those companies generally build software that is expensive and terrible.
Real-time collaboration essentially didn't exist in this market, making our apps actually better than what these organizations had! But we had absolutely no expertise in selling to orgs like this, and we never really figured it out. We should have hired for it much earlier, or maybe even brought on another co-founder with enterprise sales experience.
So, we were unable to get anywhere before investors pulled the plug.
With that said, I always say you should not trust anyone's advice. Your story is different and you need to do what makes intuitive sense to you. If your intuition is right, you succeed. But you certainly can't succeed by going against your own intuition, so if someone says something that doesn't make intuitive sense to you, ignore it.
Thank you for your work on Sandstorm! I live in the highly-regulated, not-for-profit world. Sandstorm's architecture was very compelling for us. Self-hosting was a good place to start and the Capability-based Security solved a lot of problems.
I agree that the overall functionality for the apps wasn't quite up to snuff. The problem is, self-hosted apps are structurally under-resourced relative to their hosted peers. This is because SaaS providers can amortize development costs over a much bigger user base while simultaneously capturing operations efficiencies.
In my world, we want all the latest stuff, but refuse to let anyone host our data. We wind up with a small vendor pool that specializes in meeting regulatory requirements instead of making good software. I am very interested in finding technical solutions to this problem. Or at least, technical solutions that create new options to solve the operational and cultural problems!
To be fair, I think Umbrel has a very different consumer story: Since it has a much stronger crypto focus, it is targeting consumers much more likely to spend (crypto)money, I think, than the average consumer.
And a key difference is that Cloudron and Umbrel may monetize selfhosters, which I believe Sandstorm did not endeavor to do at all.
Yeah that doesn't surprise me, you pretty much have to pay them regularly if using Cloudron, even as an individual, whereas nobody hosting their own Sandstorm server ever had to pay anything. Even when Cloudron was open source, updates weren't automatic unless you subscribed.
I think personal servers is pretty key, so I'm glad there are a few endeavors working on it.
Especially when one of the VC firms that funds your project is also the one behind formulation of the PolyForm set of licenses, I'd imagine. At least, PolyForm is better, in some sense, than fully closed source projects built atop other MIT/BSD/Apache licensed projects (say, the V8 JavaScript engine ;), and never shared.
The only reason I dislike non-OSI approved licenses are, the "users" of such licenses want to have their cake and eat it too: As in, they want to project open source ethos while also denying the advantages/rights otherwise afforded by Open Source, as defined by the OSI.
Imo, source-available licenses are justified only when companies using it are honest about their intentions and forthcoming about the license's limitations. Nothing specific on Umbrel, but generally, misdirection by firms insistent on source-available licenses as being some convenient 'middle-ground' is off-putting, to say the least!
I've followed Umbrel since I first stumbled upon it in August 2020, and of course, I'd have liked them to be open-source (since I don't believe software is their core advantage, rather their brand is; but then again, what do I know): https://github.com/getumbrel/umbrel/issues/291#issuecomment-...
That said, Umbrel already brings a lot to the table... its licensing is a predictable HN distraction from discussion on its true potential.
Except it denies me the right to purchase support services from the vendor of my choosing. It also denies me the ability to hire a contractor of my choosing to create a derivative of the product or even contribute changes to the product. It further denies me the ability to receive sponsorship from another individual to create a derivative work (This falls under commercializing, but the other two would not be me doing the commercializing).
These are all rights I have with Open Source software that is denied by your non-commercial license. Reading your blog post, you seem to not understand Open Source nor do you seem to understand the implications of your own license very well.
We’ve spent the last 5 years building umbrelOS to make self-hosting accessible. Yesterday, we launched our dream hardware: Umbrel Pro.
Specs: - 4x NVMe SSD slots for storage (tool-less operation) - Intel N300 CPU (8 cores, 3.8GHz) - 16GB LPDDR5 RAM - 64GB onboard eMMC with umbrelOS
The chassis is milled from a single block of aluminum and framed with real American Walnut wood.
Here is a video of the manufacturing process if you want to nerd out on the machining details: https://youtu.be/4IAXfgBnRe8
Also, we built a "FailSafe" mode in umbrelOS, powered by ZFS raidz1. The coolest part is the flexibility: you can start with a single SSD and enable RAID later when you add the second drive (without wiping data), or enable it from day one if you start with multiple drives.
We also really obsessed over the thermal design. The magnetic lid on the bottom has a thermal pad that makes direct contact with all 4 NVMe SSDs, transferring heat into the aluminum. Air is pulled through the side vents on the lid, flows over the SSDs, then the motherboard/CPU, and exits the back. It runs whisper quiet.
Lots more details on our website, but we’ll be hanging out in the comments to answer any questions :)