Part of the challenge is that the RAM overhead is still high, so it takes pretty expensive devices to run a strong local setup. That said, I see the path. Individuals and companies will want local compute not only to save model costs, but also for privacy, offline work, and custom tools and harnesses.
My guess is hybrid will be the default for a while.
Part of the challenge is that the RAM overhead is still high, so it takes pretty expensive devices to run a strong local setup. That said, I see the path. Individuals and companies will want local compute not only to save model costs, but also for privacy, offline work, and custom tools and harnesses.
My guess is hybrid will be the default for a while.
Curious what kind of PC you are running these on?