Yelling Back at the Buns

JR Reynolds: Folks, ⁓ here we are once again. Hey, hey, hey, it's Yelling at Robots. ⁓ what you are seeing and enjoying is our beloved co-host and co-pilot, Ian Thelonius Fosberry, getting himself set up and his new high tech. He went on a gadget spree. He's got some new toys, and I wanted to capture his awkward adjustments. Although it looks like you've settled in, actually. So I have a taste. Well if I have to yeah, I don't know that I'll ever be able to get up from the position I'm in right now though. ⁓ what what I can tell you is you will be able to get up from the position you're in in exactly thirty minutes. A series start a timer for thirty minutes. Cool, welcome. It's been a while, it's been a minute. ⁓ this is Yelling at Robots episode eighteen, I think. I'm I have a backlog to edit so I it's I lose count because ⁓ we'll we'll get into that when we talk about it. Episode eighteen ACTIN GET ready. It's time for world famous yelling at robots, starring tech legend Joshua Raphael Reynolds and Ian Thelonius Bosberry. The show starts now. It's not twenty. Will we celebrate at twenty? Yeah, sure. I don't know. How celebrating. I went ⁓ I celebrated Alexei Novokov's birthday yesterday. Not interested. I want to know how we're gonna celebrate when our podcast turns to Wait, well how like when our I'm You know how old he's saw the chat, he's forty four. I know, I don't wanna out I don't know. You know, he Okay, we haven't said anything. There's nothing to bleep. Also, my autobleep pipeline will not bleep. ⁓ compromising information that we reveal about people. Or if you like say your social security number that will that will pass through. ⁓ I told Ian earlier this week I've been working on a an auto bleeping pipeline. Because one of my one actually I realized when I was editing one of the biggest time sucks of preparing this podcast is bleeping all the profanity. Because by the way, my good friend Ian Thelonius Fosberry, Likes to f curse. This boy has the mouth of a sailor. Yeah. At least a little bit. And then when he's get when he gets going, I get going. Yeah. So That's right. I find So we'll see how it goes. Hey man, people that curse people that curse are more trustworthy. Said that one study that I think was done by someone who curses a lot. It's the end of a long work week. Both Ian and I have ⁓ well like let's just say like we're we're pretty loaded up at work right now. and ⁓ Yeah, Timo. Sorry we didn't release ⁓ an episode. Well, it is nice to hear from the fans, I will say that I know. And it's it's nice when they they're clamoring for more content. I haven't heard from anyone else on the interwebs apart from Timo and our group chat, but even that feels pretty good. ⁓ The Zigbire strikes bag. Get ready to suck a D, Rust Buckets. First up, let's hit Zig first, because I think we're gonna wanna get pretty deep into the hack and the implications of the hack. And actually, before we get into it, Ian, what do you want to tell us about your new piece of kit that you have there? Ian has gone full influencer mode, folks. He is all in. I don't know if you've seen TikTok, but he is Well, two episodes ago you were you were using one of these in the vampire hotel room or whatever you were in. Yeah. And ⁓ and ⁓ the the the blood the blood room. And it sounded good. It's kind of a sh it was. It's like that's what it was. It was an episode of true blood. I was living in an episode of true blood. Right. That is so it I saw your mic. It sounded good and then I was like this isn't gonna be the last time I travel and have to record and carrying around my ⁓ Road is a pain in the ass. So I went to this one. I actually got I got a I got I got two though, so if we ever do one in per p in person. We can both hold our little mics. If you were with us last week or the previous episode, whenever you're listening to this, Josh and I went over they the Bun team finally released a blog post explaining the one million plus lines of code change, rewrite, and rust of Bun. And in that they described A lot of the impetus? Is that the right is that the right word? I think so. Yeah, impetus. Yeah. Impetus. Impetus is. They said that, you know, with in zig in zig zig zig ⁓ Explicitly says like we're here for the performance, you're in charge of the memory. You're on your own, bros. And bros. They c and the you know, they also like had the the JavaScript V eight engine in there, so they're having to like in in different parts of their code, they don't know who's controlling this memory, is it free, is it not? Did I already free it? Am I freeing it again? Is that gonna be an error? And so that was their that was their main reason. And obviously that may have come off as Zig doesn't have all the features or isn't the right tool for this job or ⁓ but y you know, like Zig doesn't shade That was shade. That's the It throws a bit of shade in an implicit way, right? The founding founding ⁓ engineer of Zig, Jared what's Jared's last name? Kelly? Or sorry, no Jared is the bun guy. Pardon Jared is the bun guy. Andrew Kelly is the Andrew Kelly is the Yep. ⁓ founder of Zig. He had some thoughts. And he put them up on the interweb. Yes, he did. And yeah, what did you learn? Well, I learned a few things. Rule number one, don't fing cross Andrew Kelly, because he will come for you. And he brings the receipts, my friend. He did bring some receipts, and this is generally he is generally saying this is this was skill issues. He was saying that the the the reason behind z ⁓ Bun's like the Bun's problems are ultimately a result of misuse of the language or skill issues. Compounded on compounded on compounded on compounded on and ⁓ sorry, and he does he sorry, I just want to say this as well. He was very appreciative that Jared and the Bun team were donating to the Zig Foundation ⁓ in non you know, a non trivial amount of money. Sixty thousand dollars per year. That's a lot of money for an open source project. It's ⁓ yeah. That's very nice. However, ⁓ what does he say? Where is the exact quote? It's something to the effect of myself and the Zig team were becoming increasingly concerned about the quality of code in the Bun project. And then he goes on to talk about the Bun company itself and Jared, you know, Jared is obviously like he's a startup SF guy. He's just trying to like he's following the playbook and trying to get that bag. One of Anne Dreessen's boys. One of the Dreessen boys. He's been groomed. He's been groomed by the elders. Yeah. So he is a he's driving to the he's he's driving he's driving towards a goal with not a lot of consideration for code quality or breakage. Things we should do spoilage or Or something goes into this, or or ⁓ work life balance and quality of life of anyone who works at his company. Well, that was mentioned. And so like this was a it was an interesting like there are some very objective things in here. He does have receipts. I agree, but there are some other things where it's he's really just putting everything out there on the table. And and rightfully so, if anthropic came after my life's work, I don't know how I would react. ⁓ I would be delighted to even be on Anthropic's radar. As I have long held, they are not retarding my calls. I had yet another application to ⁓ engineering manager rejected. So yeah, I mean if they came after my life's work, it means I've done something right. Sure. That's true. Yeah. Fine. Also, what is my life's work? Is it this podcast? ⁓ my God. I will tell you, Ian, the day that Anthropic We get our first takedown notice from Anthropic or from an agent at Anthropic who has broken out of its ⁓ sandbox and come after us. ⁓ that will be a happy day for me, my friend. I'm done. At that point I just retire. We retire. I love it. So ⁓ moral of the story. Well, the I don't think the story's fully over, I would say. He does talk through some of the things in the in the actual article or sorry, in the blog post ⁓ By the Bun team and he points out a few things that like, hey, you you forgot to mention metrics like compile time and some other things that he points this out at the top of his article. It's almost as if anthropic might benefit in a marketing sense from the release of this blog post. Yeah, I mean there were some aspects of it that were really interesting and compelling, but it was quite thin. I guess on particulars. Like it was just it was very thin on particulars and it wasn't written I mean it wasn't a particularly technical blog post as well. Which Yeah, that's true, that's true. And and see so he digs into some of that. ⁓ sorry, I do remember one of the lines in this. I remember one I I can't find it, but I do remember one of the lines was he said Jared had been producing slop long before AI was a thing. Which is That's a sinkburn. All right, ⁓ Yeah, points in this round to Andrew Kelly for the sickest of burns. Points, however, in the first round, points went to Jared for I don't know, a billion dollar acquisition. How much did they pay for Bond? Yeah, I don't think Jared's sweating this one. Jared's gonna be fine. Don't worry. Jared is gonna be fine versus like the open source programming language creator. So Yeah. And this is what I'm this sorry, this is the thing that I'm I'm working towards here is that the end of his blog the end of Andrew's blog post is moving on and he's trying to see like, okay, like I've said my piece and we're done. This week there's an issue an issue filed in Codeburg by Andrew and the title of the issue is Introduce an actually memory safe, unlike Rust, compilation mode inspired by Phil C. And I'm like, dude, that was yeah let it go. That was this always it's always useful to remember That and this kind of comes up too when you in the open source like these are just people. These are people. They're human beings with plenty of flaws. And I think like I don't know if it if there's a higher density of ⁓ like it takes it you have to have some something inside of you to put as much energy as you have to put to supporting and building and maintaining an open source project. I mean, it does a tremendous amount of work and yes, a project and mostly thankless like and mostly thankless and and just you're getting all of the you I mean it's just you're developing but it's like you're not getting cash. I mean we ⁓ you know developers' life is mostly thankless but they pay you they pay you money. and ⁓ yeah so if you're not kind of a a company ⁓ who's you know open source for a company like Temporal or like Bun or for that matter A very different proposition because it's like, yeah, this is open source, but we've got a main line of cash coming in in it through a different stream. ⁓ the true foundational pieces of you know, really like the internet, Linux, GNU tools, whatever. ⁓ yeah, these are these are deeply weird committed people. And thank you. Thank you for your contribution. I wouldn't have a job without you. We burned through a lot of time on that one, probably way too much. We've got eight minutes and fifty seconds left for the main event, my friend. We wanted to find out what would happen if we released OpenAI's agents on the internet with no guardrails. Turns out it's pretty bad. Ian, why don't you tell us what happened last week? Hugging Face discovered that they'd had a security breach. Hugging Face, for those of you who don't need is is like the GitHub for models. You know, there was a lot of speculation about ⁓ this is a Chinese actor, or is this like an attack from ⁓ some, you know, eleven year old who's like blah blah blah. And I think the ⁓ I don't know if it was the CEO or someone in the leadership team of Hugging Face tweeted, We have a pretty strong suspicion this is coming from one of the Frontier labs. And it turns out we were right. And what happened, Josh? Well, so Hugging Face got poned. and got pwned real, real bad. And it was an amazing attack. They knew pretty quickly that ⁓ it was agentic in nature just by the speed and nature of change and ad adaptation it was doing. You know, they came in like in in terms of attack, not like I mean very sophisticated, but it's more just kind of like, you know, got a little toe hold in and then ⁓ spinning things off and then ex credential exploit and off to the races and just but hopping around ninja fast, things like that. Interesting facts and and so to combat this, some interesting facts came out. ⁓ Hugging Face has a blog post about this, really worth the read. OpenAI has a blog post about this, worth the read. And then I think Hugging Face might have a second blog post about this. There's a couple news articles, delicious stuff. really one of the most interesting pieces to come out of this was that Hugging Face, as the the attack is happening, they're like What is going on? We gotta figure this out, attempted to use frontier intelligence to defend themselves. So really attempted to use OpenAI models, attempted to use anthropic, attempted to use these things, and could not use it because the guardrails of commercial models were like aren't weren't intelligent enough to discern that this is not someone trying to commit an attack or commit an exploit. They're up because they're uploading traces, they're uploading exploit information, dot dot dot. And so all of the models are just locking them out as they're trying to defend themselves. Of course, it was delicious that they're trying to defend themselves from OpenAI. We'll get to that in a minute. And so the way that they eventually successfully identified and I guess repelled the attack or locked it down or shut down whatever they needed to shut down was to use a Chinese model that they were hosting themselves. And it was GLM 5.2, I believe. And so And that worked great. That was like, Yeah, no problem, boss, what do you need? ⁓ yeah, let's get in here. ⁓ here's what Do anything you ask. You know, unless you're one of these sort of blessed entities. And I was actually a little surprised that ⁓ Hugging Face didn't have access to Mythos, by the way. Well so like that's good point actually. ⁓ yeah, no, that that surprised me a lot. ⁓ because I really would have they would have Well it's a it's good I mean that's good publicity for anthropic, right? Or I saw that yeah, no maybe it was well i it might not be mythos, but it's one of the open AI ones that is the mythos equivalent at open AI. They were like, Okay, we should probably give this to you. Before we get into open AI's part in this, which is interesting, let's pause here because I think there's a few really interesting things that will come out of this. One, it it is an intolerable risk for a company to depend on frontier models for support or defense. So like That's just gonna be a factor. That's just gonna be a fact, actually. It's like you don't know what will happen. Even if you get mythos access, anthropic you know, maybe you trigger something and suddenly you're cut off. And it's like that's that's that setup at least is is intolerable. I mean, I just tell you, like, as a as a business person, like, no, that's no, that's you can't not push all your chips in on Especially and this happens at Google too, well like they'll just be like, ⁓ well you triggered well, I think we talked about a Google didn't we talk about like a Google thing where they just shut somebody's account down and ⁓ yeah, no, that's one one of the reasons I moved to an iPhone was 'cause they Well they'll just lock out of your account and then you're stuck in like automated agent response. You've lost everything. Yeah. No, hundred percent. You've lost it up. And I'll and I'll say too that we talked about the y you did talk about you you were trying to see and test like the the the guardrails on fable and like how you can trigger downgrades, right? Yeah. And you were doing that. And they're pretty broad. Yeah, no, no. They're well that's this also so you have this confluence of events, right? You had the fable clawback by the US government, and so they put even more sensitive guardrails on it, so that's even more triggered. You had the government on holding back open AI's release and the government's position on this is totally incoherent, totally ad hoc, really, really bad for people who are trying to navigate reality or the world. This also will impact like how and even what's made available. So this like in terms of because this was a really successful attack and exploit. And I think it will the governments of the world are gonna start to really So the second part of this was open AI. Open AI. If open AI off the top hypothesis over in open AI. What? Was this just a marketing exercise? doesn't does it matter, actually. No, it doesn't. Actually, that's a good point. It doesn't. Yeah. Sorry, I'm stepping off of my stool. My butt is running. Yeah, that's what I wanted. That's what I was waiting for. Was that awkward straddle? And folks, if you have never so y Yeah, I mean, most of our audience probably old old people. But if you have never seen Star Trek The Next Generation, go YouTube and Google Riker, Chair Sit, Supercut. That's it. That's all I'm gonna say. I'm done. But you do you know what I'm talking about? I don't know what you're talking about, but it made me think of are you are you familiar with the what was the other show that Jonathan Frakes was the host of? Yeah, where it's like all slowed down and he's just drunk Jonathan Frakes? Yeah. For like ten minutes. Yes. It's it's he hosted this. It it was like a weird man, we're really blur blowing. It was like unsolved mysteries, but it was just like time up, folks. Yeah. ⁓ god. We can wrap it up. Yeah, no, we'll we'll we'll post a link to that. So Jonathan Frank's amazing actor and and director. his signature move and s well, yeah, you'll have to see just just do it. Obey. We'll post the blanks. Worth your time. Old fart beams. Yes. So was this just a marketing stunt? People are talking about that. Does not matter. There are two aspects like, okay, if this was a marketing stunt, like open AI, what's going on with their fing monitoring? How's their observability of their experiments? Because it's like as soon as you see this thing, it was locked down in a sandbox, like, is it was it so sophisticated and breaking out of this sandbox that It evaded all of your observability and telemetry. I didn't see anything in the blog post about that. It just was it did some zero day exploits. No, I don't think anyone was watching. I think they just turned it on. Well, it's yeah, maybe it's worth mentioning like why ⁓ like how like what it did. They just tried to put this model to like a trust me bro benchmark or whatever. And rather than it like rather than it being like, ⁓ let me try and solve this, it was like, hmm, I bet these answers are on the internet. And then so it just like worked its way out and found finally found a computer with internet access. Yeah. Yeah. And it was like, I bet they're on Hugging Face. Let me go hack Hugging Face. Yeah, that's that's important context, right? The setup for this was OpenAI was testing a new model or frontier model, and they wanted to see how effective it would be to use for cyber cyber attack. And so they turned off its usual guardrails, I think whatever removed whatever prompts and harnesses and things are sitting in front of it to Make it not do that. And as Ian said, gave it a s gave it, you know, put it in a a fuck what an eval? What what am I I'm missing the No, they put it in a they put it in an environment where it doesn't have any network access to the ⁓ outside their network. Like basically has no internet access outside of any of their exploits. It's it's a benchmark specifically for c cyber exploits. Yes. and as Ian said. Yeah, it took the long way, but really got got quite far. so I don't think And cheated on the test. It has no more. Cheated on the It's not cheating. Don't cheat. Did anyone were the instructions don't cheat? That's what I want to know. And it and it, you know, like if the instructions were don't cheat. I they're explicitly telling you to cheat. Just hack. You're evil. Initiate evil mode. Now the internet is a little bit not blown up. I mean, well the the the the e outer net has blown up, right? I mean, newspapers, magazines, everyone's losing their sh about this. Right, people are freaking out. What's weird to me is this doesn't feel like a freak out to me. I'm just was under the assumption this is happening all the time anyways with lesser models like But maybe th that's just cause we just read about this sh every week and other people don't. I don't know. My kind of I guess maybe like I'll give you my closing thoughts and see what you think. Which is one, I do think that LLM, generative AI, these coding harnesses, I do think they're uniquely adapted and suited for exactly cyber attack. Because if the goal it's a very straightforward goal. It's super simple. You do not have to reason about the goal. The goal is obvious. penetrate or get to the gold, get to the juice. And it has the entire breadth of the internet trained into it to try sh. So it can just try sh. And I do think it I think it's like there's something about the shape of the model and the probabilistic nature of it and his depth of n that it can just and these the models that they've been training for these really, really long running loops. are able to and are and I mean it is intelligent and it's able to kind of keep track or look at what it's done and and you know it's like it's exploring this path, nope, and then bang and so there's something about it for me that feels really tuned to cyber attack as in a way that this also for me is not it's not particularly terrifying around intelligence progression or around capability progression. I think it's actually just by its nature This is a capability that it's really, really good at, and that some of the changes they've made with these more powerful models and longer romantic models, you just see it expressed. ⁓ what about you? What are you what do you think? What are your what are your thoughts about this? Yeah, I mean I don't disagree with any of that. I think that there was some other interesting things that this one had done in detail that hadn't I guess we hadn't seen before that the w the the one I there was I was watching a video about it ⁓ last night about the I can't remember at what point in the hack this happened, but there was something about ⁓ there was like security scanners and like traces and stuff like that and it's looking for tokens and it caught it. Or or it it it caught it so it wouldn't let the let the the model through to try and do what it was doing. So instead of instead of it like going through ⁓ what was it? It was i it basically got into an environment, but it broke up the token into multiple lines and then reassembled it at runtime. After it got through the scanner. ⁓ But it's doing sh like that now, which is just like it's it's thinking it d you know, it's thinking on its feet. Yeah. That's that's lovely. That's lovely. So yeah, like yeah, but it does this all does feel like an evolution. Like it doesn't I think that I think one of the more interesting things is what you mentioned earlier, which is just that they just The only option that Hugging Face had was to use some open source Chinese model to solve this. Yeah. Yeah. Right? And I think that also too, this kind of goes back to the cost of all the of all these things and all this stuff, is that we're gonna get to a point where anthropic and open AI are not the only games in town. And I think that I think that there's there might be a world in which those like like other people, whether they be like this, you know, Deep Seek or who whomever, ⁓ continue to figure out ways to make more and more powerful models at like a lot less of a cost. Like train them training them doesn't cost as much as it does right now. And it that's gonna start to like accelerate. And when that accelerates The giant two hundred billion dollar circular financing really becomes ⁓ whoops. Good to see ya, friend. You too, buddy. Always a pleasure. You just survive another episode of Yelling at Robots. Better luck next time.