Debated between local AI and cloud API for my home project

I spent last month trying to decide if I should run a local AI model on my old gaming PC or just pay for an OpenAI API subscription. Ended up going with the local option since I have a GTX 1080 that can handle smaller models okay. It took me about 3 evenings to get it set up properly with Ollama, but now I don't have to worry about monthly fees. Has anyone else wrestled with this tradeoff between cost and convenience?

3 comments

3 Comments

james_bell1mo ago

You said it took about 3 evenings to get set up. That sounds about right for the first time. On the bright side once you do it once and figure out the quirks of ollama and whatever frontend you're using the next install takes like an hour. I run a 13B model on my old 1080 ti and it gets like 8 tokens a second which is totally fine for chatting in the living room. The biggest win for me is not having to think about data or limits.

max_brown1mo ago

Man, people really act like setting up a local model is some kind of heroic journey. Three evenings to get Ollama going? Is that really something to brag about? I get it, you saved a few bucks, but how much is your time actually worth? Sitting there tweaking configs for a week just to avoid a monthly fee seems like a weird flex to me.

the_fiona1mo ago

@james_bell a buddy of mine still pays for cloud because he couldn't get his old card to run fast enough.