The fault lies entirely with the human operator for not understanding the risks of tying a model directly to the prod database, there’s no excuse for this, especially without backups.
To immediately turn around and try to bully the LLM the same way you would bully a human shows what kind of character this person has too. Of course the LLM is going to agree with you and accept blame, they’re literally trained to do that.
I don't see the appeal of tooling that shields you from learning the admittedly annoying and largely accidental) complexity in developing software.
It can only make accidental complexity grow and people's understanding diminish.
When the inevitable problems become apparent, and you claim people should have understood better. Maybe using the tool that let's you avoid understanding things was a bad idea...
These AI-focused Twitter threads feel like they’re just recycling the same talking points for likes and retweets. When AI systems make mistakes, it doesn’t make sense to assign blame the way we would with human errors - they’re tools operating within their programming constraints, not autonomous agents making conscious choices.
> When AI systems make mistakes, it doesn’t make sense to assign blame the way we would with human errors - they’re tools operating within their programming constraints, not autonomous agents making conscious choices.
It's not really "assigning blame", it's more like "acknowledging limitations of the tools."
Giving an LLM or "agent" access to your production servers or database is unwise, to say the least.
I think at this point it is like rage-baiting. “AI wiped out my database”, “AI leaked my credentials”, “AI spent 2 million dollars on AWS” etc create interaction for these people.
The message reads like "AI did this bad thing" but we should all see it as "Another stupid person believed the AI hype and discovered it isn't trustworth" or whatever. You usually don't see them admit "gee that was dumb. What was I thinking?"
The whole thread seems very naive somehow. You can tell that he doesn’t fundamentally understand how a coding model works. The suggestion that it would know not to make any changes just because he said so means he doesn’t really understand what the model is. It’s built to generate (and apparently execute) code, so that is what it does. It doesn’t have an inner monologue running that says “ahh, a day off where I shoot the breeze around a whiteboard” or something. It’s more like an adderall addict with its fingers glued to the keyboard laying down all of its immediate thoughts directly as code with no forethought or strategy.
To be fair, the whole premise of vibe coding is that you don’t have to understand how things work under the hood. And Replit advertises creating and deploying apps that way: https://docs.replit.com/tutorials/vibe-coding-101
> I panicked and ran database
commands without permission
The AI responses are very suspicious. LLMs are extremely eager to please and I'm sure Replit system prompts them to err on the side of caution. I can't see what sequence of events could possibly lead any modern model to "accidentally" delete the entire DB.
They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.
Maybe the individual tokens, but from experience of using LLMs something upstream encouraged the model to think it was okay to take the action of deleting the DB, something that would override safety RL, Replit system prompts and supposed user instructions not to do so. Just goes against the grain of every coding agent interaction I've ever had - seems fishy.
According to the thread, the unit tests weren't passing, so the LLM reran the migration script, and the migration script blew out the tables. The "upstream encouragement" is a failing test.
Is this a hoax for attention? It's possible, but the scenario is plausible, so I don't see reason to doubt it. Should I receive information indicating it's a hoax, I'll reassess.
Friends don't let friends run random untrusted code from the Internet. All code is presumed hostile until proven otherwise, even generated code. Giving an LLM write access to a production database is malpractice. On a long enough timeline, the likelihood of the LLM blowing up production approaches 1. This is the result you should expect.
This wasn't even the first time "code freeze" had failed. The system did them the courtesy of groaning and creaking before collapsing.
Develop an intuition about the systems you're building, don't outsource everything to AI. I've said before, unless it's the LLM who's responsible for the system and the LLM's reputation at stake, you should understand what you're deploying. An LLM with the potential to destroy your system violating a "code freeze" should cause you to change pants.
Credit where it is do, they did ignore the LLM telling them recovery was impossible and did recover their database. And eventually (day 10), they did accept that "code freeze" wasn't a realistic expectation. Their eventual solution was to isolate the agent on a copy of the database that's safe to delete.
If you've ever tried getting a llm to solve moderatly difficult but solved tasks you'd know they're currently no good for anything beyond boilerplate code, and even then you have to watch it like a hawk.
I use LLM dev tools, and even have Supabase MCP running. I love these tools. They allowed me to create a SaaS product on my own, that I had no chance of creating otherwise as a long out of practice dev.
However, we are nowhere near the reliability of these tools to be able to:
1. Connect an MCP to a production database
2. Use database MCPs without a --read-only flag set, even on non-prod DBs
3. Doing any LLM based dev on prod/main. This obviously also applies to humans.
It's crazy to me that basic workflows like this are not enforced by all these LLM tools as they will save our mutual bacon. Are there any tools that do enforce using these concepts?
It feels like decision makers at these orgs are high on their own marketing, and are not putting necessary guardrails on their own tools.
Edit: Wait, even if we had AGI, wouldn't we still need things like feature branches and preview servers? Maybe the issue is that these are just crappy early tools missing a ton of features, and nothing to do with the reliability and power of LLMs?
One thing I’ve learned from seriously using AI agents for mundane coding tasks is: never ask them to do anything that involves deleting stuff. Incidents like these only reinforce that belief.
HN guidelines are to link to the original source, and Dang has confirmed that submissions shouldn’t link to mirror/proxy sites. Instead, circumventing links can be given in the comments.
The fault lies entirely with the human operator for not understanding the risks of tying a model directly to the prod database, there’s no excuse for this, especially without backups.
To immediately turn around and try to bully the LLM the same way you would bully a human shows what kind of character this person has too. Of course the LLM is going to agree with you and accept blame, they’re literally trained to do that.
I don't see the appeal of tooling that shields you from learning the admittedly annoying and largely accidental) complexity in developing software.
It can only make accidental complexity grow and people's understanding diminish.
When the inevitable problems become apparent, and you claim people should have understood better. Maybe using the tool that let's you avoid understanding things was a bad idea...
These AI-focused Twitter threads feel like they’re just recycling the same talking points for likes and retweets. When AI systems make mistakes, it doesn’t make sense to assign blame the way we would with human errors - they’re tools operating within their programming constraints, not autonomous agents making conscious choices.
> When AI systems make mistakes, it doesn’t make sense to assign blame the way we would with human errors - they’re tools operating within their programming constraints, not autonomous agents making conscious choices.
It's not really "assigning blame", it's more like "acknowledging limitations of the tools."
Giving an LLM or "agent" access to your production servers or database is unwise, to say the least.
I think at this point it is like rage-baiting. “AI wiped out my database”, “AI leaked my credentials”, “AI spent 2 million dollars on AWS” etc create interaction for these people.
The message reads like "AI did this bad thing" but we should all see it as "Another stupid person believed the AI hype and discovered it isn't trustworth" or whatever. You usually don't see them admit "gee that was dumb. What was I thinking?"
the author is an ai booster
he's not going to be happy with all this publicity
The whole thread seems very naive somehow. You can tell that he doesn’t fundamentally understand how a coding model works. The suggestion that it would know not to make any changes just because he said so means he doesn’t really understand what the model is. It’s built to generate (and apparently execute) code, so that is what it does. It doesn’t have an inner monologue running that says “ahh, a day off where I shoot the breeze around a whiteboard” or something. It’s more like an adderall addict with its fingers glued to the keyboard laying down all of its immediate thoughts directly as code with no forethought or strategy.
To be fair, the whole premise of vibe coding is that you don’t have to understand how things work under the hood. And Replit advertises creating and deploying apps that way: https://docs.replit.com/tutorials/vibe-coding-101
> I panicked and ran database commands without permission
The AI responses are very suspicious. LLMs are extremely eager to please and I'm sure Replit system prompts them to err on the side of caution. I can't see what sequence of events could possibly lead any modern model to "accidentally" delete the entire DB.
They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.
Maybe the individual tokens, but from experience of using LLMs something upstream encouraged the model to think it was okay to take the action of deleting the DB, something that would override safety RL, Replit system prompts and supposed user instructions not to do so. Just goes against the grain of every coding agent interaction I've ever had - seems fishy.
According to the thread, the unit tests weren't passing, so the LLM reran the migration script, and the migration script blew out the tables. The "upstream encouragement" is a failing test.
Is this a hoax for attention? It's possible, but the scenario is plausible, so I don't see reason to doubt it. Should I receive information indicating it's a hoax, I'll reassess.
Friends don't let friends run random untrusted code from the Internet. All code is presumed hostile until proven otherwise, even generated code. Giving an LLM write access to a production database is malpractice. On a long enough timeline, the likelihood of the LLM blowing up production approaches 1. This is the result you should expect.
> Yesterday was biggest roller coaster yet. I got out of bed early, excited to get back @Replit ⠕ despite it constantly ignoring code freezes
https://twitter-thread.com/t/1946239068691665187
This wasn't even the first time "code freeze" had failed. The system did them the courtesy of groaning and creaking before collapsing.
Develop an intuition about the systems you're building, don't outsource everything to AI. I've said before, unless it's the LLM who's responsible for the system and the LLM's reputation at stake, you should understand what you're deploying. An LLM with the potential to destroy your system violating a "code freeze" should cause you to change pants.
Credit where it is do, they did ignore the LLM telling them recovery was impossible and did recover their database. And eventually (day 10), they did accept that "code freeze" wasn't a realistic expectation. Their eventual solution was to isolate the agent on a copy of the database that's safe to delete.
If you've ever tried getting a llm to solve moderatly difficult but solved tasks you'd know they're currently no good for anything beyond boilerplate code, and even then you have to watch it like a hawk.
I use LLM dev tools, and even have Supabase MCP running. I love these tools. They allowed me to create a SaaS product on my own, that I had no chance of creating otherwise as a long out of practice dev.
However, we are nowhere near the reliability of these tools to be able to:
1. Connect an MCP to a production database
2. Use database MCPs without a --read-only flag set, even on non-prod DBs
3. Doing any LLM based dev on prod/main. This obviously also applies to humans.
It's crazy to me that basic workflows like this are not enforced by all these LLM tools as they will save our mutual bacon. Are there any tools that do enforce using these concepts?
It feels like decision makers at these orgs are high on their own marketing, and are not putting necessary guardrails on their own tools.
Edit: Wait, even if we had AGI, wouldn't we still need things like feature branches and preview servers? Maybe the issue is that these are just crappy early tools missing a ton of features, and nothing to do with the reliability and power of LLMs?
This imo is the biggest issue, LLMs can at times be very capable but they always are unreliable.
https://twitter-thread.com/t/1946069562723897802
This is the funniest thing I’ve seen in months. Maybe years? Incredible stuff
> I understand Replit is a tool, with flaws like every tool
> But how could anyone on planet earth use it in production if it ignores all orders and deletes your database?
Someday we'll figure out how to program computers deterministically. But, alas.
One thing that AI likely won’t obviate the need of is making backups.
Here’s another funny one: https://aicodinghorrors.com/ai-went-straight-for-rm-rf-cmb5b...
Not only backups, but also a database with transaction logs or some way to play back the transactions after the most recent backup.
And here I was thinking AI had no sense of humor
One thing I’ve learned from seriously using AI agents for mundane coding tasks is: never ask them to do anything that involves deleting stuff. Incidents like these only reinforce that belief.
You need backups. If your lost data weren't due to AI slop, it could be a typo in a command, or anything else
There was no database, it was a hoax.
> SaaStr.ai
Has to be a joke. Right?
Ahh, vibe coding.
Please do not link to twitter directly, use xcancel!
HN guidelines are to link to the original source, and Dang has confirmed that submissions shouldn’t link to mirror/proxy sites. Instead, circumventing links can be given in the comments.