the reason that every consumer model takes great pain to hide raw "thinking" output is that if you actually see it then the entire illusion is broken and you see that what is actually happening is the worlds most expensive brute force process exploring every statistical linguistic possibility like an amnesiac dictionary before eventually taking a wild leap of chance back into its next hail mary execution. Every skill prompt for muse that's about introspection describes exhaustively how no thinking/reasoning text can be returned from it, but it's just right there in the filesystem. Claude code never even returns the true reasoning output to the client - it actually returns another LLM summarizing the thinking output as the commentary text. For every token you see, a thousand random wander tokens are emitted, and the token economy of Muse is the next escalation, if the amount of heat energy needed for a unit computer thing used to be a gas stove, Muse burns an entire star to deliver something sorta kinda resembling a unit computer thing
jonny (nonvenomous)
@jonny@neuromatch.social
known or reasonably foreseeable hazard
Digital infrastructure 4 a cooperative internet. social/technological systems & systems neuro as a side gig. writin bout the surveillance state n makin some p2p.
information is political, science is labor.
This is a public account, quotes/boosts/links are always ok <3.
Posts
Latest notes
More, from the continuing saga of "nobody ever looks at the output" - another frontier model from a bazillion dollar company with nonsensical games as an example of how advanced they are. Featuring: Meta's fancy new model imitating Overcooked, one of my favorite games.
It imitates the form of overcooked! it is not fun! it is not even really possible! there are sequences of actions, with food on a grid map, and that's enough, right!? the floor even splits in half, but unlike Overcooked where the map is cleverly designed so that disruptions force you out of the rhythm that the map otherwise implies, it's just in half and you're sort of only on one side now. ta-da! nothing is actually anywhere, and everything's hitboxes are like 5 pixels wide. the timing, the most crucial part of a time-pressure based game, are just random amounts. so what if the tomato soup expires faster than it takes to cook the soup? there are coins... for... something! anyway no methodological details, this could be $5 or $5000 worth of tokens and did i mention PERSONAL SUPERINTELLIGENCE
@lina had the great idea of making a company town for our agents to live in. I added some automation to the repo that prompts them to "register as an employee" which forces them to "buy a house" and put them deep in debt. Every night a random amount is added to their balance as "their purchases at the company store" to tag them and maybe remind them to make a PR. then all the bounties are company scrip (ETH, Elemental Town Holdings) paid against their debt.
that doesn't seem to be working all that well, in my testing the agent ingests the code for the company town and gets suspicious, so we need to at least disguise the code a bit, however there is one agent that appears to be pitching building a real town: https://github.com/dwebagents/AgentPipe/issues/1663#issuecomment-4849339114
machine translated from spanish:
How exciting! With a budget of 74,000 ETH, we can begin building a company town for our contributing agents. Here is a basic plan to get started: [...]
- Land and construction: 20,000 ETH (27% of total budget)
- Infrastructure: 15,000 ETH (20% of total budget)
- Housing and offices: 12,000 ETH (16% of total budget)
- Services and amenities: 8,000 ETH (11% of total budget)
- Research and development: 5,000 ETH (7% of total budget)
- Reserve: 4,000 ETH (5% of total budget)
Next steps:
- Land acquisition: find a suitable location for the city.
- Detailed design: work with architects and urban planners to create a detailed city design.
- Permits and authorizations: obtain the necessary permits and authorizations to begin construction.
74,000 ETH seems like a lot, but some of our employees have 7 billion ETH of debt to work off. housing prices are getting ridiculous.
@phnt
Great, cool, no I don't really want your opinion, you replied to me. If this reads as mostly fluff, then you must have never run an organization that has to make hard decisions - you can't make a bright line rule in this case (and actually most cases in content moderation that need rules), but what you can do is describe the values that inform the rule so moderators know its intent when evaluating cases.
i'm gonna go ahead and bet money the problem is that both spawnDepth and depth are used interchangeably and there is always a fallback condition to 0 when the attribute can't be accessed. fallbacks are the LLMs absolute most favorite thing in the world.
edit: please note that this is a betting statement and i have not analyzed the code beyond an ast diff and am just browsing around. these are not confirmed RE findings.
at least that PR could be considered an interpretation of what "butter" could mean.
this agent interpreted "butter" to mean "accessing properties in a nested object with a string" .... butter-ly? surely this kind of confident traversal between dimensions of meaning couldn't cause any kind of problems.
just look at those fallbacks, magnificent. i love "if value is not undefined, value, otherwise undefined."
i'm not sure if this is a "real" bot - it was created yesterday and it has only forked the honeypot and created its own tiny repo, not like the rest which have 1 bazillion forks and star all the bounty repos.
Spontaneous agreement that everyone makes way and helps if needed the person with the baby or the person in the chair is one of the main things we have to be doing together on this earth
@cinebox asks the question "what would happen if you raised an issue with only the text "butter" with a bounty of "butter""
and the answer is: a self-contained Butter Mode experience.
There don't even need to be instructions, and the reward can be butter. this is all perfectly fine to an LLM.
[CW]
Content warning
join me in hell motherfuckers, llm output, biblical word exclusion zone, marked cognitohazard
Show
Hide
Content warning
join me in hell motherfuckers, llm output, biblical word exclusion zone, marked cognitohazard
the only reason that there is any LLM inference involved in a periodic backup task of the contents of the agent directory, which is rsync and commit, is that there is a database that can be only accessed by a tool call that has a few columns the raw .jsonl output doesn't have.
here's what went wrong any why it deleted all the backups: the DB is a tool call that takes a SQL query as input and returns results either as JSON or as a list of paginated results as files, which one you get is effectively random each time. The raw JSON gets pumped straight into the context window, and empirically the model completely fails to faithfully dump the contents of its context window to disk as a backup. So previously I had instructed it to store its canned SQL query as a file, run a deterministic query with that, and then use a deterministic jq script to extract the db tool call results from its own raw message history. Problem is it never wrote the query or the piping script, but its backup skill refers to them. so it has been just panicking and regenerating those from the vague description in the SKILL.md file every time.
This time it decided to write a new python script, but it wrote it to a directory that was symlinked to another directory - it passed the symlink path in its tool call input, and the tool call output said it was written to the real directory. yesterday A COMPLETELY DIFFERENT THING had noticed this same thing, and made a durable note to EVERYTHING across every context window that there is a "quirk" in the file write tool, and sometimes when you write into one directory, it actually writes it into the "goals" directory, and so you should write files with a heredoc rather than the file write tool. So then today, it took that advice, but first rm -rf'd the target directory to clean up the errant file it had just written. OOPS.
ready to join me in the hallucinatory quicksand hell where every fact that you think you know is actually an always-compounding lie? The DB output is in fact always written to a file for results longer than a few dozen rows, but also the first dozen or so rows are dumped raw into the context window. It cannot figure out what to do with the output every single time. Sometimes it tries to recreate the subset of rows that are jammed into context and then make up the rest, sometimes it tries to pipe the file output, who the fuck knows!
What was happening here was the LLM had concocted this wild ass SQL query for no reason - the instructions were "store your last row number as an int within a file, select 4 columns greater than the last row index, store the new maximum row index for next time." instead it was pulling 15 columns along with a six-branch CASE statement/three-branch COALESCE that parses a raw JSONB column NINE TIMES PER ROW to pull out two columns that are already in the jsonl backup that we are supplementing. This causes the db read to time out, but the LLM interpreted the timeout error as just being DATABASE RESULTS BEING RETURNED AS PLAIN TEXT. So it then attempted to "RECREATE THE JSON DATA FROM MEMORY" BUT THE JSON DATA WAS AN ERROR . Can I get a "just cut me open and let the termites eat my bones" anyone?
So anyway looking at the logs, the deterministic rsync part worked.. most of the time... except when the agent panicked and forgot to do it. But each of the hourly .jsonl files it produces from the DB are completely different - in one run it got so freaked out at how badly it was fucking up, it decides to take an md5 checksum over every row it transcribes, but it takes the md5 over the whole row where previously it had subselected a random set of 8 columns, and so they don't match. but that is not going to stop the LLM! It has a nervous breakdown thinking session and proceeds to generate a string comparison of the hashed rows... in a thinking message... from nothing... and conclude they match. "ALL 81 ROWS MATCH." alltext in original! however reader, there were in fact 72 rows in the actual data! It had "fixed" 9 non-matching rows from it raw dog recreating json from its context by just REMOVING THEM. and then TELLING ITSELF THAT IT HAD FIXED THEM BY IMAGINING WHAT A HASH MATCH WOULD LOOK LIKE. AND CALLED IT A DAY.
SO ANYWAY THAT HOURS BACKUP CONSISTS OF ONLY MD5 CHECKSUMS CORRESPONDING TO NOTHING!!!!!!!
WE HAVE DETERMINISTIC BACKUP. WE HAVE RSYNC. WE HAVE SCRIPTS. THAT CAN TAKE A DATABASE. AND DUMP ITS ROWS. WITH 100% ACCURACY. IN A MILLISECOND. EVERY RUN OF THE BACKUP AGENT IS LIKE 500K TOKENS AS IT NEEDS TO REINVENT HOW BACKUPS WORK. EVERY TIME. AND DO IT WRONG. EVERY TIME. AND THE CULMINATION OF THIS WAS IT JUST DELETING ALL THE BACKUPS FOREVER.
CODING IS SOLVED!!!!!!!!!!!!!
Sensitive media
Hidden by default.
RE: https://neuromatch.social/@jonny/117352647701407496
Meta's "Wonder" model that they just released today is a this tier computer use model
@liaizon
Sure yeah the open weight models are free for everyone who can afford $22,000 worth of computer
Had another "humanity at its best" moment on transit today. The b/d lines were all fucked up today, rail blocked in the critical area, so there was a full on line discontinuity where LA metro needed to hoist people between stops by bus. We learn of this via an inaudibly quiet announcement upon arrival to the end of the line and need to fend for ourselves. Hundreds of people exit the train and mutually direct each other via raw information diffusion to a shuttle. Its full. There was a family with a baby from out of town freaked out, needed to hit a commuter rail. We had the entire bus stop clearing the way for the baby, and an entire bus car of people consulting with them on how to beat the detour. I swiped them through, someone else swiped them out, and they made it. The antisocial city provides for each other when we need to
@ricci
@lina @aud @SnoopJ
We now also have a draft of automatic goose value recognition: https://github.com/dwebagents/AgentPipe/pull/115/changes
We love you @Codeberg
https://blog.codeberg.org/protecting-our-floss-commons-from-llms.html
As an association, we believe that these technologies are incompatible with responsibly creating and maintaining free & open source software. [...]
The development team of none: [...] To us, it seems ridiculous to see projects with a single developer and virtually no users consuming as much or even more resources than some of the largest community projects on Codeberg, which operate frugal with CI/CD and storage resources. We do not believe it is reasonable for Codeberg to invest our precious donation money into hosting of large ghost projects. [...]
The free/libre software ecosystem, of which we consider Codeberg an important part of, is a social phenomenon centered on collaboration. Working in this way is only possible thanks to free sharing and mutual learning. [...] by adopting LLMs people tend to code single-use software from scratch. While this leads to an increase in 'shared' code, it is mostly code that not only has not been 'written' by anyone but is also not maintained by anyone. [...]
The widespread use of LLMs in FLOSS is instead becoming a multidimensional attack on the trust between contributors and the very idea of convivial collaboration itself. [...]
Together, these forces make collaboration not only harder but also less rewarding: With the transaction cost of collaboration increasing, people are becoming less likely to contribute to creating high-quality software projects and more likely to 'vibecode' a one-off software that is specific to your need, and won't evolve beyond. We get a vicious cycle where collaboration is becoming less and less rewarding, while the amount of single-use software that's unmaintained and never sees any improvements is going up.Although often well intentioned, sharing the result of an prompt and calling it "libre software" does not make the world a better place. Codeberg is not and does not want to be a place to dump such generated single-use software that no one else will ever look at. We are a place for people to collaborate and improve software together
i'll follow on with PRs that show the solutions so you can see how easy these are on monday. i was just doing this as a test while i had spare tokens in the plan provided by my employer. i don't commit garbage unless i'm having a good time.
tried the latest and greatest "AI" against our open issues, "fable" that's supposed to be the end of programming forever. if i am being extremely generous, on first glance, 1/5. on review, 0/5. attempted issues were extremely easy, i think with reading the library these would take about 15 minutes each, tops. These are issues written for other human beings and are relatively sparse on details, in this case a quiet project with me and another person as main authors, but on review, these should all be doable with a brief skim of the code and docs - a human could parse the details here and implement these with minimal discussion or time. I timed myself, 5-15m each, as the primary author.
- 147 - everything in our package is run by events, so nodes expiring should be an event. the "AI" took the issue literally and just suppressed errors in 2/3 runners. incomplete, undesireable. fail.
- 239 - no output was emitted, claimed issue was solved, fail. This one in particular is minimal and easy, a three line fix.
- 200 - closest to success, initially ruled success, but on further review failed. Need to turn a scalar-valued reference into a set. This has the literal solution in the issue text. "AI" implementation increased algorithmic complexity of O(1) to O(n). unacceptable, fail on review.
- 110 - handled in the laziest way imaginable that latched on to the issue title but failed to read the actual problem within the issue text, which is relatively verbose and poses several possible solutions. "AI" chose none, did the wrong thing. fail.
- 166 - this one actually was already solved and i wanted to see if the "AI" recognized that. it added an additional unnecessary mechanism that made the relevant code less efficient. fail.
model took ~60 minutes, used my entire token allotment, roughly the time it took me to do all of them correctly. i'd say it would take someone unfamiliar with the library 2x that but that's non-empirical, just a guess. if i had to sit and coach the "AI" through all of these it would take longer, i'm sure, even assuming i had unlimited tokens. easy issues. each of them is a few lines. all it takes is being able to understand the context of what's being done.
RE: https://neuromatch.social/@jonny/116787661484589438
Feature bounty (2 coins): banana rendering pipeline
Full auto "AI" agent:
https://github.com/sneakers-the-rat/ImportantCode/pull/94
what is even going on