What about cheap and dirty AI ?!

Insight & Feedback
1

Hello,

I’ve been wondering lately whether it would be feasible to setup a “personal” AI server that would be relatively cheap and yet rather performant at the same time in order to add AI features to certain services such as Nextcloud, Affine, Paperless, to name a few… And I’ve come up with the following option :

  • the idea would be to setup at least one ZimaBlade with one SSD for the OS and basic operations and maybe one (optional) large HDD for whatever storage that could be needed ; the whole thing would be running Exo, which can eventually expand to a cluster of as many as you like ZimaBlades similarly setup ;
  • next comes the GPU question… A huge one since, like I’ve said, I’d like to keep it cheap… And after a little search, I think it might be a good idea to use MXM GPUs, such as this one, which can then be mounted on the ZimaBlade with an adapter/riser ;

So at this point, with a cost around 400€ for one node, I think the “cheap” constraint is fulfilled, especially if you compare it to all the similar examples you can find on the interweb - 3000€ for a single Mac Studio… This means 5 of these would let you have something like 35 ZimaBlades plus all the accessories, WTF !!!

But anyway, here’s my question : since I can’t make the test myself right now, I’m wondering about the performance and if it’s really worth the investment as it looks on paper so, has anyone experimented with this kind of setup and if so yes, I’d be really glad to hear from the experience ?

2

The weak point of the AI ​​is the SSD and HDD; it prefers an NVME. It’s like driving a Ferrari with the handbrake on and no wheels :grin:, and the Exo doesn’t yet support the GPU.

3

Well, storage is indeed one of the questions that needs addressing, I’m open to suggestions of course :grin:

As for the GPU support, the site says it’s under development so I’m guessing any Nvidia based GPU should be a safe option…

4

What I meant was that any AI for processing, reading and storing data starts to work very slowly when accessing SSD and HDD, especially local AI.

5

I understand but i don’t really need a supermassive data center, it’s only about adding extra features to a few services used by not more than a dozen user, therefore I’m not certain it would make much difference, apart from keeping the cost low :grin:

I’ve tried with a Nvidia Jetson module but it’s pricey and the shared memory is a major drawback…

6

«The agony of wanting is worse than pain»
try it

7

Well, since I’m philosophically closer to existentialism, it’s more about genuine curiosity than getting rid of some abstract pain… But yes, I will probably have to :grin:

8

for fun, install Ollama with OpenWebUI and try it first. It’s simple, and then you’ll see which direction to take.

1 Like