Bots Dumping Sand in the Internet's Lube

JR Reynolds: Hey, hey, hey, it's yelling at robots, episode ten. Folks, we're in the doubles. We doubled up. Double dege, here we go. In five, four, three, two, one. ⁓ you know one thing? Sorry. Sorry, buddy. One thing I realized actually is ⁓ you're not supposed to say action until after you do the clapper. So that's I I always say action before. Huh? Leave all this in. Five, four, three, two, one. Action! Get ready. It's time for world-famous yelling at robots, starring tech legend Joshua Raphael Reynolds and Ian Thelonius Fosbery. The show starts now. ⁓ that was that was beautiful. That was some nice camera work, man. It was good. I knew it would I knew it would work well because you were like you couldn't see what was happening. By the way, I don't know if you notice, you now have ⁓ cameraman credit here. Amazing. Yeah. Before we get started Ian, I have a serious question for you. We're now in the double digits. This is essentially we are young podcast adults, young creator adults. Is it time, do you think? And I this is an honest question, is it time to have a guest? Yes, a hundred percent. I feel like it's ⁓ we gotta we gotta mix it up. There's we gotta do that. I also got some other feedback this week. Ooh You have like a one in three chance of guessing from who. But was it Alexi? Yeah, ⁓ no, you gotta let this guy out of the gates. Folks, ⁓ yelling at robots today, 0.02% more better, thanks to feedback. Feedback, this episode is sponsored by feedback, the gift of giving your perspective to someone else so that maybe they can improve. Okay, up first, the ⁓ slopification of the internet. We'll talk about three stories. One is on the leading edge of The slopification, which is Stripe and Cloudflare are partnering to unlock and allow agents to basically provision infrastructure and build the internet as they see fit. This is an article from InfoWorld. We'll talk about multiple articles and the collapse of GitHub. GitHub is completely falling down, and every response to the failure seems to make things worse. Ian has some personal experience on that. We'll go into detail there. And we'll come back because we weren't able to get to it in our last few episodes, to ⁓ Vercell's the exploit that happened at Vercell ⁓ through a chain of AI-related failures. And this was a very bad security exploit. and sort of these three, these three stories are kind of interconnected and touching on. What's going on with AI and the internet in different ways, and we'll see where that leads us. Let us begin. The red pills are kicking in. Cloudflare and Stripe, let's get busy. Up first, ⁓ yeah, the InfoWorld article, Are We Ready to Give AI agents the keys to the cloud? Cloudflare thinks so. Sure. Why not? Let's f do it. Before before we get into it, and let's take a moment, actually. Ian, are you ready? To give agents the keys to the cloud. Just no question. No, you're not. Absolutely f not. All right. If it was in a self-contained box that wouldn't affect the internet and everyone's livelihood and like whatever, sure. Cool experiment. But like this is the internet that we all use. And like Cloudflare is a very large part of that. Like the Cloudflare, any Cloudflare outage in the past two years is like Everybody takes the day off. It's really a linchpin and a fundamental aspect of the internet. Cloudflare going down is is the equivalent of I don't know, all of AWS going down. So for those of you who don't know who are listening, if you if you're not familiar with Cloudflare, they started as a ⁓ CDN, which is a content delivery network. So to explain this quickly, what you what if you have if you have a website or a application or something that is accessed through the internet and you start to get some scale to it. And you start to get a lot of requests, you don't want those all going to the same two or three or four computers you have deployed because you will get crushed. A lot of the requests are requesting the same sh over and over and over. They're not actually like updating data or doing anything like that. They're just asking for the same data over and over and over. So what you do is you It's taking too long for my pornography to download. Why is this part of the provision? That's actually so long. That's just a picture. It's just a picture. So what you do is you you put a CDN in front of your actual system, and what a CDN does is it pushes and caches that data that gets requested over and over and over into all these other edge nodes on the internet that is just like so when I make a request to Josh's website for the same thing I requested before it doesn't go all the way for f for for porn. It doesn't go all the way to Josh's website anymore. I'm just putting in color commentary, don't worry. You can just keep going. Keep going. It just comes it comes back directly from the node that's closest to where I am j like geographically. And it it speeds up websites at scale very, very quickly. And they've since added a lot of other features and other things that you can do with Cloudflare, but that's what they are and how they started. And as you can imagine, that they're a very successful CDN, one of the most successful CDNs, if not the most successful. And so they serve v a lot of the internet. They do. And they've they've really they I mean they sc they kind of went from there into a lot of services that I don't know. I guess I would say like lubricate the internet. It's kind of the internet's lube. To keep going with our fork. Yeah, keeping it keeping it on theme here. To keep it on theme. You can think of Cloudflare like the lube for the internet, right? It's making everything move smoothly. It's well, maybe maybe you can think of it like a lube and a prophylactic for the internet, right? It's protection. It actually does provide a lot of protection. It does. ⁓ no, I mean it it w like probably one of Cloudflare's most critical services it provides is denial of service, attack, re prevention and reduction. And they're really good at it. And actually I don't I I I'm actually I n I'm not sure that I've ever worked on a Like a production system that didn't use Cloudflare for DDoS production. Like it's it's so easy, you just throw it in front, set it and forget it. But the denial of service is actually more important than what Josh is talking about. Sorry, I did forget I forget to mention that. That's really important. The story itself, I mean, like and you know, is basically providing the connections, the authorization, and the mechanisms for agents to do it all. Like basically, do do every piece of putting something on the internet, essentially, which which which consists of provisioning infrastructure, buying domain names, like setting it setting it up, mm, deploying code, all paid for and during the setup process through Stripe integrations. And so I g I think I did I actually read this article or was this another one that I just was like, Ian, check this out. Check it out. Yeah, and there like, you know, in terms of kind of like guardrails, I think was was really just kind of like, well, each agent gets a budget. You know, it's like, ⁓ don't go. Yeah, like $100 a month max. Yeah. What do you think? What do you think, buddy? Like, what's gonna happen? What do you think will happen? What like I like if we've learned anything from like The stuff we've talked about, like of of like sure again, agents and all the all the harnessing, it's it's improving and it's getting better, but like it's there's still a lot of slop out there. And like instead of instead of like focusing on fixing the code slop, we're gonna add some provisioning infrastructure slop too. Cloud slop. And and also too, like I don't I I'm not fully clear, like I I understand that Stripe projects or whatever the hell it's called is integrating with a lot of different things and it's like this d it it's basically like claude code but stripe code, stripe CLI. It's a similar it it's looking like a similar tool. But are they gonna also like are are you also allowed to do like payment integrations? Like this is literally what we talked about with Bartine. Like that's one small business in the Netherlands like maybe doing a POS. This is fing stripe. Like they do all the payments on the internet. good luck, dip s That's my that's my closing. If you're comfortable digging digging out a big pile of sh slop, buddy, you're gonna have a job for years. You know? You're gonna be just fine. That web of trust is actually a venomous spider web. Join the lads as they learn how Vercell just got bit. ⁓ so let's talk about this Vercell thing, because I've ⁓ slowly loading it back into my brain. Sure. ⁓ Do you ⁓ d do you have it in your brain? Are you are you fully remembering what happened? Well, no, the the security breach was somebody gained access to Vercel's internal systems and was able to extract a massive quantity of secrets. We're talking NPM tokens, we're talking API tokens, tokens, tokens, tokens, and not the not the tokens that you spend, the tokens that ⁓ cost you money when they are re when they are sold on the dark web. ⁓ And and I don't remember if there were kind of subsequent secondary attacks through like compromised NPM tokens that then they're publishing compromised packages. We should have a whole episode where we talk about supply chain, like what's happening with the supply chain right now, because ⁓ it's it's really, really bad. Yeah. So the interesting details about this exploit were it was a a single individual employee at a company, not for sale, at a company. That was integrated into Vercel. Versel had an integration with this company. This company was called Context AI. But they've got their security l like tight, bro. SOC2 certified, bro. SOC2 certified by dev, bro. And what is ⁓ what is wait, what it real quick for the people who don't know, what is SOC2 and why would you want to be SOC2 compliance? Security standard. Baseline security controls are in place. You know, there's some you you it's safe to have some assumption of competent security. Now, I've also been in the security business long enough to know like some assumption, like, hey, we all got in a room and we said the word security together. And we said, do we have a disaster recovery plan? Yeah. Is it written down? Yeah. Okay. Certified. Punk. Like this is right. But but but but but that's actually better. That is better. Than a huge majority of companies out there. And actually, it does lead to better outcomes. So beside the point. But the important point to here was that they were SOC2 certified by a company called Delve. Delve, which is a we have talked about before. We've talked about Delve. Delve is a disgraced startup that has been kicked out of Y Combinator, that is an AI SOC2 certification company that lets AI get into your systems and SOC2 certify you and lets AI check all your and certify it, and was essentially a total fraud. So they weren't there was no AI. They were just pulling in the documents and then saying that the AI had certified it and issuing certifications. And they will you I had like investigation ongoing, but the founders will probably serve jail time. This is this is really very fraudy frauddy McFrauderson. And I could it's so easy for me to see like in Y Combinator startup land just like Again, th the cycle of just like, well we're would hey, you it's like move fast and break things. It's like fake it till you make it. Like not with security certification, guys. That's when fake it till you make it stops. Okay? Is when you're signing legal documents. Yeah. Fake it don't do not fake it till you make it. Because the only you're only gonna make it like the only thing you'll be making is toilet moonshine in prison. Yeah. Exactly. ⁓ but so full circle, context AI, SOC2 certified, yeah, our security is good from Delve, a fraud. They were given, and I don't know what the integration was, but they were given they had an OAuth. ⁓ How even does the chain work? Somebody Yeah, it was a context AI employee was infected with a Luma Luma info stealer. Long story short, they got fished fuck by some application that sounded cool. To authenticate with it, you know, this is and by the way, think about when you use your Google ⁓ account to sign into something, right? And Google says, Hey, like this application is asking for permissions to your calendar and your email and your name. Is that cool? And here's where the trust comes in. And it's like, yeah, sure. And by the way, we're all completely conditioned to never question the trust and just be like, Give me the thing, give me the goods, ca, cac, cac, cut. Their Google workspace was fu absolutely horribly configured to even allow an untrusted rando application to ask for permissions to everything. And this like this guy, I don't know who this person was who's just like, I mean, what was the application? Does it say what they were trying to install? Doesn't quite say. It says that the attacker did gain they got access to context AI's AWS environment. We'll see. Eventually. And then yeah. I mean, so what this person did was just Off full full access into the entire context AI system. And then they were off to the well, then they were off to the races, and I'm sure just like, you know, crawling and try to figure out where the edges are, you know, doing good good hacking stuff. Yeah. And because context AI, I think because context AI is a service provided or embedded into Vercell, they were able to use that as an entry point into Vercell systems. And to Verse's Google Workspace and Vercel's built ⁓ bulletin describes the same event from its perspective, stating that the attacker used this access to take over the employees Google Workspace account. ⁓ man, wow. I mean, good job, hackers. Ian, what's your take on it? It's all breaking. And it doesn't I some of this I think some of this is driven by like letting AI do the wrong things and like not making good AI decisions and agents and yada yada yada. But some of it is just also driven by like bad leadership decisions and the same we gotta go faster than the other guy. Yeah. ⁓ at any cost. I do think that there is a a thematic bet in the technology industry right now. ⁓ and maybe it's always been there, but we've never been able to to go so fast. that speed at all costs. Speed at any cost. Yeah. Speed to one speed running to oblivion. Speed running to speed run speed running. Speed run everything. Speed run to the edge of the cliff. And then hope And pray that you can wily coyote your way. Wait, no, you can roadrunner your way. But I got bad news. I got bad news, Jack. You're the coyote. Coming up next, GitHub still fed. So tell us about what's going on with GitHub. I mean it's been it's been a it's been for a while. You've probably seen the there's an image floating around the internet of ⁓ GitHub's uptime. It is just gre a big giant green line back in the day. And then there's an arrow that says, This is where Microsoft bought it, and it just starts to go like ⁓ and it over the years it just gets worse and worse and worse. And I don't know how accurate that line actually is, but I will say this. Anecdotally. Anecdotally it feels right. It feels correct. And it also like I I I can't remember at this point I cannot remember I can't remember a month going by without a GitHub issue. For twenty twenty six, I can't remember a week going by. And in the last two weeks, it has been holy sh Man. And I don't know how much of this is driven by like AI stuff, but this is just the degradation of a company that is that used to be like the shining. They were the sh ⁓ my god. What happened in the last two weeks was one, they're they have they've had this new feature. I don't know how long it's been on, but they've they somebody turned it on at work and we've started using it in a few repos. It's called a merge queue. And so that's basically like if you have a if you have a repo where there's a lot of people trying to like merge to merge to trunk. Like the main branch. There's a queue that you can just say like merge when ready to make sure that okay, all the checks in the CI have passed, like all your tests have passed, all your lintings passed, and it's ready to go. It's rebased correctly, yada yada yada. So it's basically like it's not ready to merge yet because someone else is merging and all these other people are merging, there's a conflicts. So you say like merge when ready, and it just puts it in this queue that will automatically merge it once It once GitHub system sees a moment to like, okay, it's your turn, now you merge. Mm-hmm. So it's it's it's handling automatically or it's attempting to automatically handle also the sort of whatever conflict resolution or rebasing, whatever your policies are. If it can, if it thinks it can. Like main is good, I need to rebase or whatever other thing is, and then Then I'll trigger all of the safety checks and if everything's good, in you go. We're probably using it when we don't need to because we don't have that scale. Like we're not like Uber or like Twitter or something like that. Yeah, but still, I mean nothing wrong with like fire and forget, right? Sure. The issue here is that ⁓ GitHub is made on top of the Linux tool git. And it literally broke Git. So as a developer, you have trust in Git and what Git promises you of like it's an it's it's basically an an event system of changes and version control over time. And what this merge thing started doing was merging off of the wrong place in the Git ⁓ git tree. And so you were losing commits on main and you had no idea that was happening. I mean it was basically like rewriting head and pointers so that those those prior commits pointing to the wrong place. Completely lost. Basically reverting previously merged commits and it and it was random. It wasn't there was no in a non determined there was no ⁓ yeah there was no there was no way to determine like how it was doing that. It was just completely random, which means that you had randomized erasures of code That had been previously deployed. And so unless you have like regression tests on your system, you might not even know about this for days. Just one thing to throw on there. So basically, like the way Git works, if you if you're using it in a very simplistic way, it's very difficult to shoot yourself in the foot. Yes. But it also has the power that you know it will give you like there's plenty of plenty of foot guns. That was the first like. mind-blowing incident. ⁓ there's been a couple since. The one that has been ongoing this week, which I don't know if they've fixed yet, is the pull request I guess their pull request system is backed by Elasticsearch and they had some kind of it there's a big warning at the top of all your pull request page that say some pull requests might be missing because of a search incident, blah blah blah. Don't worry, no data has been lost. We are just trying to fix the API, blah, blah, blah, blah, blah, blah, blah. Okay. So but as a consequence, the none of the you can't merge any of the f pull requests right now. So I had to merge like three different PRs the old fashioned way this week, which is like on my CLI, like I will I will merge the branch into main and I will push main and GitHub will reconcile it, like ⁓ this branch has been merged. Which is how I used to do stuff on smaller teams. How many people are are manipulating and ⁓ managing your repos? Like how many people are you having to coordinate this? Well some of our so some of our repos yeah, so the repo that I did that on is a repo that's been a small team and doesn't necessarily work the same way as the majority of our s our repos are standardized and have like main locks, so you can't push to main. So you can't merge. You can't you can turn that off if you need to. Yeah. Yeah, you can. But it's So the the one that I was working on. You you it it works the way it works that way so that you can run npm run release and it will properly release things and and version the software and push a Docker image and yada yada yada. ⁓ so, anyways, the interesting thing about this, I think, is that it's gotten to the point now where you're seeing some pretty influential people. The the most the the the the number one that I that I saw was Mitchell Hashimo Hashimoto, the you know, CEO and founder of HashiCorp, and one of the biggest open source guys we've ever seen, one of the biggest GitHub livers lovers we've ever seen, has said that his new project, Ghosty, which is a w wonderful terminal that I use, is leaving GitHub. And that's crazy. And when you start to see that sort of stuff happen, I don't know. I don't know if this is a stoppable train at this point. So how's GitLab, buddy? It works. It works fine. If you go to a lot of the status pages for What you know, the it's a typical thing. I'm gonna screen share and just show show the good people what who don't who have never seen one of these what a status page looks like. This makes my yeah, we're gonna share buddy. It's gonna be quick, don't worry. no, it's not the quickness. I guess I could have fine. We can vibe it. We can vibe edit it. But this is what a status page look like. This is like a standard template that most of these things use. ⁓ and all systems are operational now. But actually there's there's a better one. There's ⁓ the but I'll tell you right now, like hold hold on, hold on, go back to the This is the real one that actually shows so there's th there GitHub has like someone has made a different GitHub status page that is more up to date than their actual GitHub. Okay. Like their official one. And this is the one where you can see like the last ninety days of uptime. Had you gone to this like six years ago, this would be just solid green. As a platform as opposed to as individual. And it's not anymore. Yeah. And I will I'll also say like if we go to cloud status page. Cloud status. I mean wow. Not green. Wow. Jesus. And this is what we're working with, man. This is the internet. Yeah. Which by the way, every one of those red lines you see, you will not get your money refunded. FYI. Exactly. Those tokens are gone. Dip sh God, they're the way I'm still I'm still happy to take that interview, by the way. I will if if you're getting tired of being called an Claude, Claude anthropic company, there's an easy way out. This is an easy way out of this. Just hire Josh. Buy me off. He'll become your biggest evangelist. ⁓ folks, let me tell you, it's not as easy as you think in there. You know, that's ⁓ I've seen. I've seen under the covers. I've spoken with mythos. Well, takeaways. But yeah, this is the takeaways you like the I I don't know, man. The industry at large, do you like you remember a time when And this is very difficult. And this has to has this goes back to what you were saying earlier about the, you know, 80% and the 20%. There's a a well-known engineering ethos, if you will, ⁓ that achieving nine nines, which means like you are your uptime is 99.9, can I do this? 99.9999, which Google, I think some some system at Google was the first to do it, is a monumental feat. But It was very sought after by a lot of these Silicon Valley companies and teams for a very long time. And a lot of these systems like GitHub used to have nine nines. And now it's just like, well, we'll settle for nine zero and some numbers after that. It's like they've given we I feel like people have just given up 'cause be in the name of speed and we need to move fast. Hmm. And ⁓ I don't know, man. It sucks. No 'cause I like I don't like when things are broken. So nine nines is too many, by the way. Like that's too many nice you guys. Like for my pacemaker, I want nine nines, you know? But like for you know, to like for my pornography transaction to complete I'm fine with like four nines. Four nines is great. Like that that means I'm only like four hours without pornography a year. So that's still pretty good. ⁓ I really I definitely in my next team meeting, I'm gonna definitely frame all reliability and SRE conversations around pornography impact. For sure. It's a great way to make it accessible to the young people. So ⁓ yeah, nine nines. It's not directly controllable. This is an outcome of the system and the systems and the system of systems and the system of systems. Many, many systems. And for something but and and where it's like I don't know, and it's interesting and you know because like one of the things that people have been pointing to is, you know, GitHub doesn't have a CEO. It actually doesn't have someone directly responsible for this. It has a CPO, it has a COO, it has people who are responsible for part of parts of it. There's not actually one person who is like, this is that's that's on the headless monster. Yeah. It's a headless monster. That was and and there's a lot of questioning about the decision behind that. And I don't know like So that's one aspect of it, but it's like, and yeah, you know, Microsoft, this, you know, when when GitHub was acquired, it stopped being a I mean, my understanding is it stopped being kind of a product in and of itself and it started becoming ⁓ an acquisition strategy for Microsoft to engage with the developer community in a different way. And yeah. It may have lost its purpose then, or maybe that was driving factor. But what I'm getting at is like it's it's hard to know why. Like we can point to a lot of things. And and we can point to a lot of contributing factors of ⁓ you know, yeah, I'm sure the load has gotten nuts. Definitely. But what the number of users and AI agents like number of repos and number of like Yeah, it's it's it ⁓ it's it it is spiked beyond what they ever thought they would have to deal with, I bet. Sure. That's a yeah, that's a that's a charitable. Would the numbers of crazy have brought this down if there had been Someone in like someone making hard decisions at the top. Some adults in the room. We're gonna we're gonna Okay, you know what? Some adults in the room. Yeah, like we're like we're not taking new signups. Sorry. Like until we kind of understand or have a strategy around how to deal with all these repos being created and agents just doing things. Like, we're not taking new signups, guys. You can make or you can make ten repos. That's it. Whatever. Or I don't know. I don't know. I mean, that's just that's that's one example of one giant company where it's very visible right now. Yeah. The same decisions are being made at many other companies that support the internet and the same crazy Crazy decisions about like moving fast and giving the agents the keys to the world are happening at other companies, and this is all moving pretty quickly. And I don't know, I don't know. I I I it it's it feels it it like the the chaos feels like it's increasing week a week. It is increasing. And it's gonna get worse before it gets better. And on that note, folks, that's another yeah folks, Foz, you want to give us our signature our signature tagline? We're f it will get worse before it gets better. Definitely. A hundred percent. And it will probably get much, much worse. And there will probably be a major subsystem of the internet that will completely collapse due to a loss of comprehension ⁓ as a system and as an organization, due to ⁓ too much automation, too much trust in systems, blah blah blah. So if I have one piece of advice for everyone who's kind of engaging and entering into the system, it is download the pornography on your computer now. That's it. Because if you lose access to that. Well, because if you lose access to that, like what are you gonna do all day? Alright, folks, that's Yelling at Robots episode 10. We're double-digiting. ⁓ That's a ramp! You just survived another episode of Yelling at Robots. Better luck next time.