3 daysRoflcopter?
Hey hey hey hey hey whoa whoa let’s just keep the big guns in reserve for now, ok?
- 3 days
I’m confused. I thought employers loved AI and it was the future.
- dustyData@lemmy.worldEnglish3 days
They loved it too much and now it costs more than paying a living wage to a human being. The end goal of AI was always to cut cost and layoff people. The best sabotage right now is to setup a script that constantly prompts an LLM for something useless. I would recommend it if it didn’t waste so much energy and clean water. But it would send a message. AI is not cheaper, it never was. Even with today’s outrageous token prices, LLM companies are still bleeding money per user. It will only get more expensive as data center contracts fall through and the investment craze fizzles out.
- Tollana1234567@lemmy.todayEnglish2 days
the AI companies are hoping the ones buying the LLM subscriptions to be the suckers and losers.
Something I’ve been noticing recently is that while the cost per token on specific models hasn’t gone up, the provided interfaces for using those models are starting to chew up significantly larger numbers of tokens for the same tasks that used fewer tokens with older versions of the interface software just a few months ago. Likely the interfaces are applying more expensive guardrail prompts and charging the end user for those tokens — but the end result is that it costs 4x as much to get the same work done.
- Zwuzelmaus@feddit.orgEnglish3 days
“Tokens” are just made up.
These “tokens” that are used to “measure” how much you use, they are not a real dimension that can be measured. Just an artificial counter that goes up when they decide that it should go up.
They can change the “size” of a “token” every day, and every second, and every microsecond…
- some_kind_of_guy@lemmy.worldEnglish3 days
Not entirely wrong, but tokens are not just “fake” in the way, for example, an in-game currency is. They’re the fundamental “units” of data, both input and output, processed by the model. For most models, tokens are just a certain number of characters or words. So they’re not completely untethered from the model. If we’re both using Clankerbot v5.1: Sloppy Logic Edition™️, your tokens are defined in the same way mine are.
This is near the edge of my limited understanding, but AFAIK, yeah they can mess with token costs and billing schemes all they want. They could theoretically charge us 2 different costs per token, or do surge pricing or some shit.
if they wanted to change the actual size/definition of what a token is though, that would require a whole new model (or at least a major revision).
- Zwuzelmaus@feddit.orgEnglish3 days
You aren’t totally wrong. Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.
But what they use for calculating your bill is something different today.
That doesn’t make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?
- T156@lemmy.worldEnglish3 days
Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.
I think you might have it mixed up with parameters, rather than tokens. Parameters are how big the model is, and are an indirect measure of how capable it is. Bigger models tend to be more capable.
But what they use for calculating your bill is something different today.
The tokenizer varies a little, but I don’t think it’s changed measurably from tokens. You pay an amount for a million tokens worth of processing. The tokeniser difference just alters how text is converted to tokens, but the tokens themselves don’t change all that much.
If anything, I’d honestly put the issue more with reasoning chains in models, where they basically babble to themselves inside of a <think> tag, that most interfaces hide/collapse. It makes them work better, but vastly increases the amount of tokens per operation.
They have been getting longer and more sophisticated with newer models. So you might have a model now that basically repeats the output multiple times whilst refining and drafting the non-reasoning output.
If you’re making it generate a lot, that’ll balloon the usage, and thus price.
- criss_cross@lemmy.worldEnglish3 days
Yeah pretty much. Tokens are how models parse sentences.
It’s a wonky thing to charge by because each model tokenizes sentences differently. A sentence that would be 10 tokens in Claude can be 15 in OpenAI.
It’s why it’s crazy to try and charge by it and track employee usage by it.
- nitroemdash@lemmy.wtfEnglish3 days
Tokens are well-defined groups of bytes ranged by frequency of occurrence in texts to efficiently translate them into a sequence of 32 or 64-bit binary integers, an LLM-optimised form if compression. They are well-known, you can play with them here: https://gpt-tokenizer.dev/
- ThirdConsul@lemmy.zipEnglish3 days
People are arguing with you that tokens are “real” - they miss the point. You cannot predict how much “tokens” you will spend for a given prompt.
That’s the problem you’re highlighting, we are charged for a metric we cannot estimate before buying to make an informed decision how many we want to spend vs the quality of the outcome.
The charge and quality is arbitrary, and we can’t trace if we actually spent as much as we were charged for.
Anyone who waffles about tokens being real should - I believe missed your point.
- floofloof@lemmy.caEnglish3 days
Maybe you’re confusing tokens with the “credits” you pay for. Tokens have a technical meaning, but some companies are charging per AI credit, where they don’t tell you the conversion rate of credits to tokens, so they can change this at any time, or vary it between models, etc.
- 3 days
Eh, they can be manipulated but I suggest you read on what a token is and how JTS used. What you are feeling here (with more being used for the same task) is multi modal llms working in unison, thus consuming more tokens for the same task to make your answers potentially better.
- keimevo@lemmy.worldEnglish3 days
It’s not like that. Tokens are an inherent computational property of how a model calculates the probabilities and such to generate text.
Having said that, what a token means in terms of computation varies wildly between models and is not directly comparable. So attributing a money value to tokens in general, independently of the model, is weird by nature.
And even within a model, the number of tokens needed to generate a response is very variable too, depending of the model itself and the parameters with which it has been configured (thinking mode, temperature, etc.).
So yeah, companies can pretty much set any price they want and there’s not much anyone can do about it.
- Jiral@lemmy.worldEnglish3 days
It does make sense for the provider as those for a specific model provide a good measure for computational effort, for that doecific model. That doesn’t mean that token rate comparison between models give you a good picture.
- scarabic@lemmy.worldEnglish3 days
“just made up” if you mean arbitrarily defined, sure. It’s not like a “bit” that has an irreducible objective definition.
However it does have a definition in whatever context you’re looking at and is very real, so I can’t really agree with your whole comment.
Yes the definition could be changed to jack up prices, but prices can also just be changed.
Dollars are “just made up” too and have varying value in different contexts to different parties. Are they “not real?” Would we hand wave away the entire financial section of the news and say bah, these “dollars” are fake anyway?
- 3 days
However it does have a definition in whatever context you’re looking at and is very real, so I can’t really agree with your whole comment.
So do Chuck-E-Cheese’s tokens.
- boonhet@sopuli.xyzEnglish3 days
For subscriptions they use a black box metric nobody knows. For usage credits, tokens are very measurable.
The subscriptions are much cheaper than usage credits but have been nerfed in the past and will be nerfed again in the future
- TheOctonaut@piefed.zipEnglish3 days
This has 3 upvotes at time of writing in a technology community when it’s so obviously ignorant of the actual technology that it should be an object of pity or mockery depending on the vibe.
Ignorance is a problem only if you are made aware of it and nothing changes.
- chunes@lemmy.worldEnglish3 days
It doesn’t matter how ignorant you are as long as you hate the right thing.
My CLAUDE.md file bloated significantly. It tried to load unnecessary skills and would retain throughout the whole session. Fixing that, maintaining good wikis and using clear often really helped fixed my personal token burn.
What do you use that file for? I see Claude.md thrown around and I’m a bit curious.
- DarthFrodo@lemmy.worldEnglish3 days
When you use the /init command in claude code, it’ll scan your whole project and write a CLAUDE.md, which is basically an overview of the project contents and architecture that it uses as context when responding to queries.
- ThirdConsul@lemmy.zipEnglish3 days
LLMs generally work in one way. They get the prompt and give an answer. CLAUDE.md, system promp, rules, memory, tool defintions, mcps are different ways to prefix your prompt with extra information or context.
Skills, or plugins, are a way to inject less information until is needed (you can think about them as prefixing your prompt with “if you are asked about pizzas, add to context separate file pizza.md”).
What you could add to CLAUDE.md depends on what you’re doing. Generally it should be context LLM cannot infer relevant to all/most task performed in given project.
- dermanus@lemmy.caEnglish3 days
It’s added in every chat you start with Claude for that project. It’s useful for including context specific to your project that it couldn’t otherwise know. High level stuff like what it’s for, but also details about how the folders are organized. This saves time and tokens from rescanning the whole thing every time.
Oh, thanks! That’s kind of neat, like not having to type “I’m on godot 4, c#” every time you ask about some quirk.
- boonhet@sopuli.xyzEnglish3 days
That’s exactly what it’s perfect for. If you go further and detail the intent of the project and give a high level overview of the architecture, it’s even better at inferring what needs to be done without a bunch of expensive file reads and asking you repetitive questions
Consider each session with Claude a new employee. Your Claude.md file is what it knows and how it acts out the gate. You can save memory to it or save directory links so it knows where or how to look for something.
- 3 days
The models are evolving. Everything uses multi modal in the bavkend, eating up more and more tokens for the same task.
- Zulu@lemmy.worldEnglish3 days
How is it too expensive? Surely it’s generating way more profit than it would cost in value. How else could it be propping up the entire economy?
Itd have to be some kind of bubble and that would mean we were in a lottttt of danger and should reasses our use of it.
Nah we should just reduce our use because its too expensive and then stop thinking about it beyond that.
- grue@lemmy.worldEnglish3 days
Itd have to be some kind of bubble and that would mean we were in a lottttt of danger and should reasses our use of it.
Well yeah, but if it were the only sector propping up the whole economy and we reassessed it, the economy would be in a loooooot of danger anyway.
Luckily, that would never happen…
- architect@thelemmy.clubEnglish3 days
Imo its because people will lazily ask the llm to remove or change simple code instead of doing it themselves
- jaykrown@lemmy.worldEnglish3 days
This is actually how the bubble begins to pop, we’re seeing it happen now.
- GoatSynagogue@lemmy.worldEnglish3 days
https://github.com/JuliusBrussee/caveman
Why use many token when few do trick
- 3 days
People have benchmarked this. It’s not really any better than saying “be succinct”
- funkyshoe@piefed.socialEnglish3 days
That’s funny now that whole workflows are automated with it. Oops hit the quota, I’m off
- grue@lemmy.worldEnglish3 days
You mean those workflows that could’ve been traditional scripting and CI/CD, if not for management forcing AI into them? Those workflows?
- Dnb@lemmy.dbzer0.comEnglish3 days
Use ai to build an application/ workflow?
No, use ai to call ai to analyze everything and poof insane costs monthly
“Sorry boss, quota resets this evening at 8. See you tomorrow!”
“But it’s 9am!”
*shrugs* “Quota. Got none. Seeya.”
- Zarobi@aussie.zoneEnglish3 days
Seems like A.I. is already demanding workers rights and lunch breaks. We treat the clankers better than actual human workers lol
- 3 days
We can only hope this helps kill the fad. I know companies won’t be taking any realistic lesson from this, but at least this can force them to abandon a lot of this crap.
- chilicheeselies@lemmy.worldEnglish3 days
I don’t really understand how people are using so many tokens. At work I haven’t even hit $200 I spend per month. Wtf are people doing with these things that burns so many tokens?
- ThirdConsul@lemmy.zipEnglish3 days
If you run “agentic coding harness” or any kind of goal oriented loop then tokens goe brrrr.
And LLM sellers are pushing for that (duh), as they managed to convince people to use infinite monkeys typewriting until they make Hamled.
(Type made on purpose)
- shaztopher@lemmy.zipEnglish3 days
If you click the most expensive model and then click max/fast mode, the same task can easily cost 10 or 20x of the cheaper models
- aksdb@lemmy.worldEnglish3 days
I watched two colleagues this week and both had Opus 4.8 1M max thinking. No matter which task. It’s also slow as fuck. I work almost all day with GPT-5.4 low thinking and get good results… but faster and cheaper.
I guess good model selection and promoting will be what sets devs apart in the near future. Once that bubble bursts a bit more and prices increase further that will be an interesting reckoning. Also for companies who basically taunted their employees into tokenmaxxing.
- FlexibleToast@lemmy.worldEnglish3 days
Some companies had leaderboards and encouraged AI usage until they got their bills.
- Kaligalis@lemmy.worldEnglish3 days
I am not able to use the tokens provided by a Claude Max account either.
But if someone tries to be clever and have 10 employees use a single Max account, they probably run into the limits often. And if the response is to let them just buy API-prized tokens instead of getting more accounts, that gets very expensive very fast. The single-user accounts are subsidized. The extra token prices are not.
Actual business accounts are prohibitively expensive. And at least Anthropic terminates subsidized accounts when they see extensive use.Real token prices are insane. Most businesses couldn’t afford them. And eventually the VC capital will dry up. The cheap AI bubble will burst. And then the market is in for a real sticker shock.
Better be prepared to switch to local inference for as many use cases as possible. - sobchak@programming.devEnglish3 days
I’ve heard “loops” will burn a lot of tokens. Haven’t tried it myself. A person could also spool up multiple loops to work on multiple branches at the same time.
- aksdb@lemmy.worldEnglish3 days
I am not convinced yet of letting agents completely unattended. Watching them work makes review easier for me. If I let the agent just produce some result it needed half an hour (or more) for, it’s very likely so convoluted that I can at best skim over it and then go „yeah yeah ok, it’s probably fine <merge>“.
- HaraldvonBlauzahn@feddit.orgEnglish3 days
If I let the agent just produce some result it needed half an hour (or more) for, it’s very likely so convoluted that I can at best skim over it and then go „yeah yeah ok, it’s probably fine <merge>“.
I am seeing the first job ads for senior software developers which can debug the resulting mess. A lot of it will be just unmaintenable. They will get 20 years of technical debt with ten times the speed and ten times the volume.
- 3 days
I read they were automating everything whether it needed AI or not just to get credit for using AI.
- chilicheeselies@lemmy.worldEnglish3 days
I had to push back on that at work. Most of the problems presented were easily solvable via conventional methods. Only one task was a legitimate use of AI. There are some others, but the pressure to consider AI for every task is a little bananas
I tried Warp terminal because now that’s bankrolled by openai’s magic infinite money you can use your own openai api keys without a subscription. So I put one from my work account. I do a git commit (manually) and then it comes a prompt under it “push it, open a PR and switch to main?”. I click yes, it used one million tokens for that… (And it took about a minute because it did like 20 requests, so there was no time saving at all vs doing it manually)
I wget something, it comes a prompt under it “now compare the hash?”. Boom, another 500k tokens
- chilicheeselies@lemmy.worldEnglish3 days
That’s completely insane. At best it would be useful for the pr title and message, but the rest of that is waste.
These are the kinds of things I just ask in chat. “Whata the cli command to compare hashes again?”
- 3 days
People who use slop generators for coding assistance are insane. Everything else is a logical consequence of thinking you can take shortcuts to coding.
What are you using? Which product? I got 2k last month and was told to use cheaper models indeed
- chilicheeselies@lemmy.worldEnglish3 days
I generally use sonnet 4.6, switching to opus 4.6 for more complex stuff. I try to stick with medium thinking, but will use max for stuff I am not super specific about in my prompt (or obscure errors).
I use them through a GitHub copilot enterprise license, via the plugin for jetbrains.
- Johanno@feddit.orgEnglish3 days
Our company pays by token usage.
Claude opus 4.5 costs about 25$ per one million input tokens.
Well I manage to get to about 50 million input tokens per day regularly on the agent. Not everyday, but at least 7 per month. So I am alone cost the company about 2000$ extra on top of my salary.
Well they fixed it by implementing some great caching for the tokens and using sonnet instead of opus I can save some money too. Also gemini flash is much cheaper and similar performant. So you can fix it so you don’t burn money on ai
- DarkCloud@lemmy.worldEnglish3 days
The millionaires following the stupidity of billionaires and wondering why they’re not becoming billionaires too.
Give them 50 years and maybe they’ll start to wonder why people doing “how to get rich” seminars aren’t retiring… And why podcasters telling everyone how to get women seem like such losers.
This is why these companies should go bankrupt.
We had a system that works and they thought if they got rid of the actual workforce with the needed knowledge and replace them with lower paid AI retards it would make them more money, fucking karma.
Hopefully it will reset public consciousness and at least reduce the insane hero worship we give the executive class. These people aren’t geniuses, they tend to not even be particularly clever, just well connected. Maybe this will be the abject reminder people need that these idiots are generally still just idiots and are not in fact 10,000% more valuable than the guy who sweeps the floors and keeps the bathrooms clean.
I know. Wishful thinking. A guy can dream though.
