Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
Do you host your own AI?
English
You must log in or register to comment.
18 hoursYep. https://lemmy.world/post/46066942/23416719 Basic setup, works for me
- fluxx@mander.xyzEnglish2 days
I do, but I am becoming increasingly more disappointed as time goes on. Not just self hosted, llms in general. They sometimes help, but they mislead so many times and waste time that you don’t even notice. I think that’s the trap. When you succeed at a task, you become impressed but don’t notice how many times it failed doing a simple task. And as soon as you scratch the surface, you see how you would have done it differently and perhaps in a better way. Even just googling is bad. It does research for you, but it has no critical thinking and can’t decide what is better from the results it gets (other than google ranking) so it often leads you to think it did as good as you would, when it’s nowhere near as good. Every time I did the googling myself after it did, I did it much better. And I mean MUCH better. Ask it to find the app, it misses the most important ones, hallucinates a bunch, for ex. I found this to be the case with frontier models as well.
Self hosting has its benefits, but seeing how the ecosystem looks right now, concluding this is a huge bubble is inevitable. It reminds me of crypto so much. It looks rich and plentiful, but as soon as you dig a mm under the surface - nobody has tested it, it’s got a critical bug, it is overblown and there are issues with no response. No docs, no info, no nothing. For the biggest thing in technology in history, it is awfully hollow. I don’t mean it in a condescending way, in fact community is enthusiastic and very helpful, it’s just that it doesn’t live up to what most would expect.
A caveat I need to mention is I have not used it for coding - I have an irrational fear and resistance towards it, being a programmer. I just won’t touch it, even if it means the end of my career. I’m trying to be grown-up about it, but so far, I dont want to use it, for good and bad reasons.
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
- portifornia@piefed.socialEnglish4 days
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
Mostly on coffee, not exclusively. Noticable amounts of spite & tortilla chips are also present, yes, but… no shame.
- curbstickle_lw@lemmy.worldEnglish3 days
@SuspiciousCarrot78@aussie.zone this comment is not (directly) for you, I just want it in context.
Before you report someone for breaking rule 1, please look a the context. Specifically, the username someone may be replying to.
- 3 days
LOL.
https://en.wikipedia.org/wiki/Flowers_for_Algernon
Looks like someone got big mad over a harmless, good natured and on topic joke. You love to see it.
Sorry they wasted your time.
- curbstickle_lw@lemmy.worldEnglish3 days
Eh, its fine. Certainly better than the “I don’t like this so I’m going to report it” approach.
- 4 days
critical security bug: if coffee is taken away my head hurts :(
As we know AI stands for “An Indian”, so if you’re not from India, its actually impossible to self host.
Well, unless you manage to trap one in your basement, but that would violate human rights and hopefully also break the laws of your country.
- 3 days
You may be confusing Indians with gremlins (AGI). Which might explain ChatGPTs obsession with gremlins
- brucethemoose@lemmy.worldEnglish3 days
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
- plasma8726@lemmy.todayEnglish3 days
Thanks for this link. Because of this article, I had claude stand up a llama.cpp container next to my already running ollama container. It ran side by side tests with the same model and parameters, and the results blew ollama out of the water. I’m in the process of moving hermes and openwebgui over to the llama.cpp instance to see how it goes day to day.
- brucethemoose@lemmy.worldEnglish3 days
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
- plasma8726@lemmy.todayEnglish3 days
Thanks! I’ll look into this. I’m a bit limited at 12GB of VRAM right now.
- brucethemoose@lemmy.worldEnglish3 days
A 3060?
Exllama/TabbyAPI is still worth looking at if you are trying to run a model purely in GPU RAM. It’s easily the most VRAM efficient backend, it just doesn’t support CPU offloading (which is useful for MoEs if you have considerable spare CPU RAM) and more optimized for 4xxx and up Nvidia cards.
And TabbyAPI has a docker container you can use. Look for “exl3” models on huggingface.
- vagabond@lemmy.dbzer0.comEnglish3 days
Didn’t know this. Going to switch this weekend, thanks for sharing this!
It’s not that hard to use
llama.cppdirectly anyway. Why would I use a wrapper when I can just run a python script?- BlackLaZoR@lemmy.worldEnglish2 days
I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm)
- brucethemoose@lemmy.worldEnglish3 days
Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.
- pinball_wizard@lemmy.zipEnglish3 days
I agree that the concerns listed there are smells, and I wasn’t aware of some of the options listed there.
Thank you for sharing this!
- comrademiao@piefed.socialEnglish3 days
looks like extreme nitpicking without any real issues beyond some VC funding a FOSS issues.
//whyre you spamming the comment to everyone? its quite alarmist actually
- brucethemoose@lemmy.worldEnglish3 days
I completely disagree.
Frankly, I find the description “VC funding a FOSS” offensive. They aren’t funding the engine. I’ve been messing with LLM inference engines since 2022, and Ollama is the worst I’ve seen in the community.
They misname models for SEO. They leech off llama.cpp while deliberately hiding attribution yet redirecting GH support requests there. They sometimes make their own GGUFs+forked releases which are broken and incompatibile with upstream llama.cpp, just so they can get a release out a day ahead for hype, even though it doesn’t really work and they’ll never upstream one line. They set a default context size thats basically unusable, they screw up chat templates and deep internal code with no obvious indicators, they release suboptimal quants without iMatrix, they gate you into their internal quantization repo and model card format, they hide model downloads on your hard drive, they mess with standard APIs for no good reason other than to mess up other backends. I could go on and on.
And if that’s all fine, they’re enshittifying the app with closed code, and pointers to cloud models.
They GIVE LLM inference a bad name, by making it a terrible quality engine that happens to show up in search as the “default.” Hence the comments below of people being unimpressed with local inference. And they sap attention from actual llama.cpp devs, without contributing a single dime. Everyone in the localllama communtity hates their guts, and that’s not even getting into the interpersonal drama they’ve stirred.
They are a leech that’s a net drag to the whole community, that we can’t get rid of because they’re attention grifters. And they’ve gotten worse and worse over time.
It’s more morale to use any cloud API over Ollama, in my eyes. They’re a grift.
EDIT: And, to be clear, I’m not against VC funded downstream stuff.
LM Studio is good! Even though it’s closed source.
Tons of downstream projects are great.
- bier@lemmy.blahaj.zoneEnglish2 days
Hell naw my homelab is already sucking way too much power and running too hot.
Running qwen3.6 27b through llama.cpp.
It’s about as capable as sonnet 3.5.
I use it for light scripting, but real coding is done by cloud models.
I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.
- 3 days
Do you mean Sonnet 4.5?
I don’t have the rig to run it at real speeds but I’ve played with it over API. Seems pretty good.
No, it needs a lot more babysitting than 4.5 does. 3.5 was on the same level of mistakes, at least on the quants I have to use.
- frongt@lemmy.zipEnglish4 days
Yes. Openwebui/ollama for LLM, comfyui for stable diffusion. I just dick around with it as a toy.
Same. Its somewhat useful on some very small scripting or tasks…but its mostly just to try out a new model or two. Its not really useful for anything big.
I will have to say…even my tiny models are about as good as Chatgpt/Claude/etc… which makes me think about how much people are spending on tokens regularly. I was able to get the same kind of python script started with my local tiny model that was comparable to the newest Claude code offerings.
What local models have you been using? And what hardware are you running them on? I’ve been playing with local LLMs a bit for exactly your use case.
I have zero interest in vibe coding or full agentic workflows. But having a local LLM generate a Bash script to help me automate parts of my home lab infrastructure would be nice.
Ryzen 7 5800 X3D Radeon RX 9070XT 32GB of DDR4 system memory.
- OhVenus_Baby@lemmy.mlEnglish3 days
How hard does it push this setup? How far can you scale up your own models on this hardware?
Same, toyed with it for creating stupid things like bot for telegram, that basically was a 3rd-person NSFW storyteller in RP chat. Sadly, after I made said bot I remember that I don’t have friends to RP with.
That being said, ollama+openwebui kinda sucks: openwebui have “wider scope” and features that you don’t need like auth via social providers and managing multiple accounts, while ollama itself does the opposite and lacks certain features (like proper mmap support to load big models), slow in comparison to pure llama.cpp and generally easily replaceable with lm studio, that provides both - client and server. So yeah, my advise for anyone who want to try it localy - just use lm manager.
I was put off by ComfyUI, seems awfully complex. How is your experience?
Any suggestions to start? I have Fooocus installed now
- Honse@lemmy.dbzer0.comEnglish3 days
It is difficult to understand in the beginning but has great support for premade workflows. It even saves the workflow into its output images so you can drag and drop them into the webui to duplicate the setup that generated the image. Use the internet to get premade workflows and mess around with them to see what the options do and you’ll slowly learn how it works. If you don’t care about precise control over the generations or understanding how image generators works then just use something else more all-in-one.
- Strider@lemmy.worldEnglish3 days
No. I still have no use for it and everything I use is automated without at a far lower footprint.
No I don’t. Unforunetly using Claude (asking myself everyday why tf cuz I don’t do crazy shit) but trying to move on to LumoAI even meaby will buy a premium version to check this out formyself.
- dotAlexX@lemmy.worldEnglish2 days
I would love to run and host a local LLM on my phone just to tinker and learn. I found a tutorial on setting DeepSeek on your Android phone using Termux but it is a year old. I’m sure there are better more efficient LLMs that can run on a phone now.
- Domi@lemmy.secnd.meEnglish3 days
Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.
Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.
I’m running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.
I use them for:
- Tagging and classification of my documents in Paperless
- Home Assistant (voice assistant)
- Translations (both text and image)
- Transcriptions
- Some light coding and debugging
- Avatar/Backdrop generation for DnD sessions
- Domi@lemmy.secnd.meEnglish3 days
About 200 t/s prompt processing and 10-20 t/s with MTP.
Greatly depends on the task, predictable things like code generates at 18-20 t/s. Creative writing more like 10-17 t/s.
- 3 days
Damn - I thought strix would do a bit better than that, for how much it costs.
- robber@lemmy.mlEnglish3 days
Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.
The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.
- Domi@lemmy.secnd.meEnglish3 days
Curious about the quant tho.
Q8 from unsloth.
Something like Qwen3.5-122b
My go to model for knowledge. Definitely much faster at Q5 but it lacks the tool calling quality of the Qwen3.6 models. Really hoping we see a Qwen3.6-122b soon…
- robber@lemmy.mlEnglish2 days
In case you missed the Ornith 1.0 release (Qwen and Gemma RL finetunes for agentic / coding workloads), they look interesting to bridge the gap until we see larger 3.6 models or a 3.7 release. I didn’t test them yet but according to benchmarks, the 35b MoE seems to be more or less on par with Qwen3.6 27b dense, while ofc a lot faster.
- 3 days
Yeah. Though I think theres a new strix out soon (Medusa? Gorgon? Something like that).
Its a bit like my P40. On paper, it has 24GB. But that 24gb is capped at 400GB/s and the ai compute is what…Pascal era?
AI = Good, fast, cheap - pick 2
- robber@lemmy.mlEnglish2 days
Well compared to the strix, 400GB/s is not that bad, I think with fast system RAM and expert offloading you could squeeze quite something out of it when running stuff in the 100b-a10b regions.
Your bigger problem is going to be future software support.
Yeah, I’m using qwen 31b a3b on an amd 9070xt requires a bit of cpu offloading, but still plenty fast. Using it wall llama.cpp. Combine that with some mcp’s such as ddg-search to make it truly useful by actually being able to search online.
I mostly use it for small tedious tasks with well defined inputs and outputs. For example when hyprland recently changed from their own configuration language to lua. At first I started going line by line translating my config to the new lua language until I realized oh wait this is exactly the type of thing that ML is useful for. Going from the well defined hyprland configuration language to their also well defined lua syntax. It banged it out in less than a minute with only a single mistake which I easily fixed. The mistake it made was that it forgot to translate the comments to lua. It did it in less than a minute and worked first try. Where as I had made several typos and gotten a few lines wrong when I was doing it by hand.
Not to say that I couldn’t do it. I would have gotten it done in about half an hour, but less than a minute is a lot faster.
I also used it to transform a bunch of unstructured data into json data, so that I could then use purpose built tools like jq to parse that. If I’m having trouble finding certain information. I’ll ask it to find me some resources to look at.
Basically small well defined tasks and parsing data is what I use it for and it seems to be pretty good at that.
What I don’t like is the way companies try to market it to people. I don’t believe people should be trying to summarize emails or messages from loved ones, writing essays or any other creative tasks for the most part. Translating is okay. I don’t expect a machine to be able to decide things for me or to be some filter between me and others.
- 3 days
If I wanted AI for some reason, it’d be self-host or nothing.
- BlackLaZoR@lemmy.worldEnglish2 days
I was hosting LLM with LMStudio occasionally but can’t access it anymore due to some fuckery with CORS and http vs https in browsers.
- 2 days
I googled it, and it seems like you can just enable cors.
- BlackLaZoR@lemmy.worldEnglish2 days
Yes you can enable cors in LMStudio. But since few months it’s blocked by all major web browsers if you aren’t using HTTPS.
Which I don’t. I had LMStudio server open to local network so I can use it on my phone or laptop via third party website.
- 2 days
Why would it be blocked? I can use http sites just fine, and even then you could setup a self-signed certificate.
- BlackLaZoR@lemmy.worldEnglish2 days
I don’t know why. It worked half a year ago, now it doesn’t. All I know I can access server via local host on same machine, but if I try to access it over LAN it just doesn’t work no matter what I do. I might look into self signed cert but it feels like to much work for now.
