Nicholas Carlini - Black-hat LLMs | [un]prompted 2026

What Hacker News said

Strong rec knowaveragejoe · 2026-04 · on “Project Glasswing: Securing critical software for the AI era”

I'm sure they do, yet the models really are getting scarily good at this. This talk changed my view on where we're actually at: [this video]

Recommends panstromek · 2026-04 · on “Claude Wrote a Full FreeBSD Remote Kernel RCE with Root Shell (CVE-2026-4747)”

The talk "Black-Hat LLMs" just came out a few days ago: [this video] Looks like LLMs are getting good at finding and exploiting these.

Recommends ogig · 2026-04 · on “Claude Wrote a Full FreeBSD Remote Kernel RCE with Root Shell (CVE-2026-4747)”

This very question was asked to Nicholas Carlini from Anthropic at this talk: [this video] The answer is complex, worth watching the video. But mainly, they don't know where to place the line. Defenders need tools, as good as attackers. Attackers will jailbreak models, defender might not, it's the safeguard positive in that case? Carlini actively asks the audience and community for "help" in determining how to proceed basically.

Recommends eichin · 2026-04 · on “Claude Code Found a Linux Vulnerability Hidden for 23 Years”

[this video] is the presentation itself. The prompts are trivial; the bug (and others) looks real and well-explained - I'm still skeptical but this looks a lot more real/useful than anything a year ago even suggested was possible...

Recommends woeirua · 2026-04 · on “Project Glasswing: Securing critical software for the AI era”

You should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: [this video]

Recommends conradkay · 2026-04 · on “Project Glasswing: Securing critical software for the AI era”

I would've basically agreed with you until I'd seen this talk: [this video] Maybe a bad example since Nicholas works at Anthropic, but they're very accomplished and I doubt they're being misleading or even overly grandiose here See the slide 13 minutes in, which makes it look to be quite a sudden change

Recommends underdeserver · 2026-04 · on “Project Glasswing: Securing critical software for the AI era”

I would suggest watching Nicholas Carlini's talk and Heather Adkins and Four Flynn's talks from unprompted: [this video] [LINK] My takeaway is that fuzzing is not just complementary, it also gives a stronger AI a starting point. But AI is generally faster and better.

Recommends vmg12 · 2026-04 · on “Small models also found the vulnerabilities that Mythos found”

The technique Anthropic uses was demonstrated by Nicholas Carlini in a talk he gave 2 weeks ago and it's very simple, when asking LLMs to review code, ask them to review one file in a single session. Here is the video with the timestamp (watch through to ~5:30, they show two different ways of prompting claude). [this video] [this video] IMO the big "innovation" being shown by Mythos is the effectiveness with prompting LLMs to look for security vulnerabilities in specific files one at a time and automating this prompting with a simple script. Prompting Mythos with a single file per session is why I suspect it cost Anthropic $20k to find some of the bugs in these codebases. I know this same technique is