Keith Casey: Webhooks & webhooks.fyi

• https://webhooks.fyi
• ngrok.com
• Webhooks to Revolution the Web from @progrium
• What are Webhooks 
• The Self Provisioning Runtime 
• CaseySoftware

Colin: Welcome to Build and Learn. My name is Colin.

CJ: And I'm CJ and today we're joined by Keith Casey. Keith is a product expert at ngrok and has also worked at Okta and Twilio. He's all about getting great tech into the hands of awesome people, and he lives in Austin and is big participant in the Austin Tech community. He also shares thoughts on his blog @ caseysoftware.com. He's also co-authored a book about API design. And so we're super excited to have Keith here. And, we're gonna talk about webhooks today and webhooks.fyi. If you haven't seen it, head over and take a look at this knowledge base that, ngrok has put together. Before we get too deep into it, just wanted to welcome you, Keith, and, in case I said anything wrong there, please, feel free to correct the record.

Keith: No, that was great. That, that Keith Casey sounds really impressive. I'm not sure how it got to be about me, but, no. thanks for having me here, guys. looking forward to this chat.

Colin: As we've been getting into, like looking at who we want to have on the show, it seems like the theme so far has been finding people from our past and putting them in front of a microphone and. Keith is no exception. There, sounds like inroads from everything from coworking to Techstars to startups that we've used, I think you were dev at Twilio when we like met in person for the first time. so yeah, getting into some hackathons and things.

Keith: And I think that's really cool because the people I was meeting 12 years ago with Twilio, that's when I was there way back in the ancient days. The people that were there that were passionate even that they're, even if they're the most junior first day on the job, developers after that, Applying that passion for 6, 8, 10, 12 years in this case. They've moved on to some pretty cool, impressive roles. I look around and I'm like, I remember when this guy was writing his first lines of code and now he's C T O over here. Or I remember when this person just moved to town and now they're a significant like impactful community leader. And it's awesome to see like friends and allies grow up like that and move into their own and see how that passion has affected the world around them.

CJ: Totally. Like just in those 12 years, someone going from writing the first line of code to being a cto O as a microcosm tech in itself lets you see a co a full career progression over the course of 12 years versus other industries where it might take you 40 years or something to kinda climb the corporate ladder.

Keith: Oh yeah.

CJ: fun to.

Keith: when I was first in the field or Twilio way back in the day, we had to tell people what an API was. We had to teach people like building on APIs was okay. AWS had been around for a couple years, but not many people were using it at scale, When we think about how fast people's careers have progressed, we have to think, that's because part of that is the technology under us has moved so quickly and in such radically different directions. If you told us 20 years ago, no, most people won't have their own servers. They'll be living entirely in this thing called the cloud. And the cloud is really just other people's servers. But there'll be APIs connecting in all these third parties, like really defining and making your app work. There's people that would've said, wow, that's cool. And there's other people that would've been terrifi. The correct answer. Somewhere in between those, but that's how see why our careers have progressed the way they have, is that everything under us has moved so quickly in interesting ways. It's really powerful to see.

CJ: Yeah. I think that's also A fun lens to look at all of these AI changes through that. okay, this is an inflection point, and if you are feeling stuck in your ways and you're not open to considering new ways of doing things or being agile and flexible about what this next year. Is going to look like things are going to get wild. I don't know. I think Colin, you were showing me some stuff with the GitHub co-pilot brushes or whatever the other day and I started playing around with it and I was really jealous having spent spending most of my time in Vim man, these co-pilot brushes in VS. Coat are just blowing my mind right now. so yeah, it'll be an interesting next decade to see all of the progress that happens within tech and our careers and those folks that are getting into AI for.

Colin: Yeah. And that's a great segue to talk about what we're talking about today, which is webhooks. I was actually digging into kind of what made webhooks popular, and you might be able to shed some light on this too. the idea that when you're at Twilio, you weren't just trying to get people to use Twilio's api, you were trying to teach them how. why they even wanted to use an API in the first place, or what an API even is. and just to set the stage, we found like the earliest mentions of webhooks was from Jeff Lindsay, all the way back in 2006. And the idea that thinking of a world today where webhooks are available on almost every app, but there was a point at which there, there were no webhooks. And there were no APIs. today we're gonna be talking about webhooks and specifically this webhooks.fyi site that you guys have built at ngrok. what have you guys seen as far as like the rise of webhooks? Popularization of webhooks?

Keith: Yeah, absolutely. So I mean you call it, Jeff Lindsay was one of the first people to put the word together a webhooks. And at that point it was actually two words. just a bit of trivia for you there. but in 2006 and seven, he was out there talking about this. And fundamentally he went back to the, some of the earliest principles in object-oriented design and the sort of the encapsulation model of object oriented design. Cause remember, when you're building object, To rewind back to, CS 1 0 1 and you've got this object, you really don't want to expose the internals of how that works because now you get tied into the specifics, the, you get into all the idiosyncrasies. You now have to worry about that state of the object. There's a concept called message passing. And message passing is just the state of this thing has changed. Here's how it's. that's it. That's, that goes all the way back to, small talk in the sixties and seventies, and it's a exceptionally powerful concept. Now when you're talking about individual components within a system, that works out really well. When you then say, okay, we're gonna take those components instead of operating the same memory space where we could just pass things over a bus or a port or something like that. Let's put them thousands of miles. In different architectures, in different data centers, with all kinds of things in between them, we now have to figure out how in the world do they do message passing? And it turns out webhooks was the model to do that. And simply it's, it's JSON over HTTP or XML over HTTP. It conceptually, when you realize, oh wait a minute, it's this concept that's now close to 60 years. just updated for the tech stack we're working on. It's cool to think about, like we've never gotten away from those core principles. We've just applied them to the medium that we're working in, and He, Jeff Lindsay was started talking about webhooks2006 and seven. he joined a little company called Twilio not too long after that. And basically Jeff Lawson, the CEO of Twilio, went to Jeff Lindsay and said, I like what you're doing. We have some problems that I think that's a solution. Let's blend this. Let's pull this in. And so I attribute a lot of the early webhooks, one to Jeff and the other to Jeff. So Jeff Lindsay brought to Jeff Lawson and they made it work in Twilio and. Frankly we are blessed at Twilio. We were one of the first people into that space to do it at scale and we were admired for a lot of different things. And one of the things was the model behind those webhooks, cuz fundamentally it's this thing, this text message, this phone call, this recording has changed state. Being able to pass that message across a wire, regardless of your tech stack ended up being exceptionally powerful. Then when you take the macro trend of, uh, AWS, of GCP, of all these things coming in the market, and you go, wait a minute, my tech stack is no longer a stack. It's a bunch of stacks sitting in a bunch of different places, and oh, by the way, it has dependencies on apps that I'm building. Has dependencies on apps that you are building, has dependencies on third parties. WebBook is the only way to do that reasonably. There are other approaches, but when you say, Hey, look, I don't care about the tech stack, I don't care about your architecture, I don't care about anything else other than are you speaking JSON over HTTP. That makes life really easy.

CJ: Totally. And I think also like when you think about a normal web framework and you're building out these typical features, whether it's, the classic, you're just taking the post body from a form and submitting that to. Some endpoint, or if you're taking the post body of that form and using JavaScript with, fetch or Ajax or whatever to submit that as j s o to a server endpoint, you can build your web application and your web hook handler using all of the same tools under the hood, which I think is also pretty killer because yes, there are these other solutions that, you might be using. some message queuing system or some other sidecar thing that happens to reach into your database, but it becomes much, much easier to reason about, in my opinion, your web hook handlers if they're following the same patterns that your, the rest of your web application is following. And yeah, I, it's, I don't know, it's it's interesting to see how. the trend in industry has landed on or decided that JSON over HTTP is the end all be all. And I think it's a lot of it is because like we don't really have another pattern that has emerged for just typical web development that is significantly better and, something that we might all want to migrate to. and so yeah, at Stripe we. Looked very closely at our web hook solution and thought about oh, what if we changed it to all of these other different potential things? Then, would people still adopt it? It's okay, 99.9% of people building on the web or building mobile applications or whatever, are going to still use webhooks because of the way that they're set up and the way that they work.

Keith: Yeah, all of us know, and we have a favorite HTTP tool. All of us know we have a favorite JSON tool. When that's the baseline you're starting from, it's hard to get people to move to anything else. and I appreciate that. I appreciate that. It's hard right now because we can work with tools we're familiar with. We don't have to introduce new patterns, new technologies or god forbid, a vendor specific protocol that's we're done with those hopefully.

Colin: Speaking of tools, can you give us kind of the high level of what ngrok is trying to solve as a company, and as the kinds of tools that you guys are working on, and what inspired the creation of webhooks.fyi, what were you guys seeing as far as these simple tools being used in really powerful ways?

Keith: Yeah, absolutely. Ngrok, founder and CEO is Alan Shreeve. He was employee number six or seven at Twilio. If you send an SMS using Twilio, you probably used his code at some point. But while he is working with those webhooks, way back in the day, he ran into this very simple thing of when you pull out his phone and send a text. he'd wanna route that to the app he was building locally. And there are a couple other alternatives at that point. And he said, you know what? I wanna learn, go and I wanna figure this out. So he wrote the first version of ngrok basically to do that, to scratch his own itch. And then he started adding things like, request capture and replay to be able to say, I've got this web hook. It came in successfully instead of having to pick up my phone again. Let me just replay that. . Awesome. That was really cool. And most people find ngrok through some of those webhooks because if you look at any web hook guide out there, odds are step one of integrating with stripes webhooksis probably go install inro, which is awesome. fundamentally what ngrok is, it's a platform for providing secure connectivity into any. So today we're talking about webhooksand local development. It can also be IOT devices on a customer site. It can be a Kubernetes cluster sitting inside your data center. it could be the deep in the bowels of that ancient D data center, getting access to the database behind the scenes.

CJ: we're assuming that you, dear your listener, know what a web hook is

Keith: Oh, yeah.

CJ: we'll just gloss over. I think it's fine if we gloss over it because we want people to go read webhooks.fyi, which is gonna teach you all about webhooksand what they are and and how they work. But I think it's worth at least explaining the fundamental issue with like why that request was not delivered to the local running application and

Keith: Yeah. Yeah. So when you get, when you're working with Stripe, Twilio, slack, whatever, when the change in state happens, a payment being processed as text message being received, a new slack message coming across, that web hook, that payload has to go somewhere. So the web hook is just telling you the change in state has happened. Here's some information. And so when you're running in your local environment, there's DNS issues, there are SSL issues, there's all these things to get from your local environment. getting from your local environment out is super easy. that's what our browsers do every day. Getting in is a horrifying complexity of dns, of ssl, of na traversal, of all these different things. And so what Ingra does is it opens a. over 4 43 outbound to our cloud. And then the cloud handles a, generates a URL that then that web hook can call into that then gets routed down into your machine,

CJ: Yeah. So it's if you are running your app locally, if you're building a web application locally and you're running it and it pops up on local host 3000, like your friend that's in a different city cannot go to local host 3000 and see your program running because it doesn't have access to the internet. And I think it's also probably worth mentioning that It's hard to get into someone's computer on purpose. , like the operating systems are protecting us. Yeah, it should be really hard for any outside service to get in and this is intentionally difficult and yeah, luckily anytime we're working with third parties, we can. Depend on ngrok to poke a hole through that magical firewall that we've created and give access. I, is there any like security considerations, I guess this is of like tangential, but that people should be aware of when running ngrok because you are poking a hole in that, yeah, in that inbound

Keith: Yes. Poking a hole is a phrase that comes up pretty often because at first glance, it's scary. it's scary cuz you're like, wait a minute, I'm doing what? so what we did a few, about a year or so ago now, we launched, or we had them for a while, but we doubled down all of them. We've got some capabilities you can put on top of that tunnel so you can do IP restrictions. So if you've got IOT devices, you can say only our command and control server has access to it. You've got, mutual t. So you can have shared certificates on each side and a handshake to secure that connection. you can also put an identity provider in front if you do dash OAuth equals Google. Now you can force someone to log in through Google before they have access to your connection. And in fact, you can add restrictions on top of that. In fact, let's see, by the time this airs, we added some updates to our free plan to be able to add OAuth. So you've got very simple OAuth protections built into the free plan itself, along with web hook verification. So before you let any traffic into that connection. We ngrok in the cloud can verify that payload is from stripe, from Twilio, from Slack, whatever. And then anything that it can't verify gets. and it never makes it to your server. So yes, it is poking a hole, but yes, we can layer on a variety of security, policies on top.

Colin: Nice. We're poking a safe.

CJ: Yeah.

Colin: There's a, these are your taglines that you can have these for free, Keith, so

Keith: a safe hole. Thank you. Thank you. I appreciate the the spirit it was offered in. How's that?

Colin: Absolutely. Yeah. And I what's exciting to me, webhookshave always fascinated me because I, when I teach APIs, I like to think of it as like turning a light bulb on across the world and like a web hook is a. A post request to a thing and hue lights and other things. you can have a web hook that, when a payment goes through Stripe, make this light flash green. Or play, your cash register sound in, in Slack, which is fun. And there's a lot of ways, like Zapier today is one of the bigger, Low-code integrate X to Y type of apps. And most of that today is run by webhooks. So you may, if you're new to webhooks, you may be using them already. You're just not, catching and sending them yourself. things like lambda functions in the API gateway and aws, like a lot of that is really powering webhooks, which is really great. and I think that there's two sides of this, right? If you're a company or an app and you wanna provide webhooks to an end user, or if you're more commonly your a developer who wants to consume webhooks like stripe, webhooks, slack, webhooks, things like that. what did some of that inspire, what webhooks.fyi is built for, or how did that come about for you guys?

Keith: Y Yeah, it completely inspired it. When we were working on the web hook verification within ngrok, one of the first things I was tasked with when I joined up was, find a list of all the webhooks out there and let's figure out which ones we're gonna support. it's a very reasonable request. So I googled, web hook directory and web hook, like best practices and everything. And it turns out there wasn't much out there. Everyone had their list of their favorites, but no one actually dug into how are these verifications actually handled. So I was sitting there making a Google doc of, 50, end up with like a hundred and some different webhooks of. API company you can think of was probably on that list. I think that list has grown to 150 or so at this point. and we are looking at how do you verify the payload, which ones support ssl, which ones don't, what are the payload options for each of those? And while we are building our own implementation of that verification, we realize, like realized like there was no directory, there wasn't a place for capturing this information. And I'm sitting there with this big Google Doc, I'm like, why don't we turn this into a. and just share it with people. And then we started digging into what are some of the best practices around this. So we first analyzed, we, we took the top a hundred or so and put them on the site and we said, okay, how are the different web hook providers, what verification methods do they offer? some are HMAC 256, some are this, some are that, we dug into, which ones are over HTTP versus HTTPS. And we dug through probably 10 or 12 different aspects of all of. And we released it. And we released, here's what we see in the wild in the field right now. Here's what we all know are good things. Let's make a list of the ideal and call out and celebrate the people who are doing it well. And so on the site, webhooks.fyi, there's a web hook directory and there's a bunch of list of web hook providers and some that have red X's and some that have green check marks. And we're trying to encourage people to do things.

CJ: this is a fun list. and also, there's people that are not on this list that I'm like, oh, I wonder if that would encourage them to migrate from pub sub hubub to webhooks or some, something similar. But, I think also it's worth talking and breaking down a little bit. Some of those characteristics that you went through. So do they support, TLS or how are they verifying signatures or payloads or just, what are of the differences there and maybe, yeah. Do you have any that are like the gold standard that like, oh hey, everyone should go implement their webhooks, like this company.

Keith: Yeah. that was the fun thing is we had this list of a hundred and some different webhook providers and we categorized all of them and we said, okay, which of the ones we should implement first? We narrowed, did the list down to about 50. Then we said, okay, now let's actually start building the verification code to verify these 50. So you guys wanna take a guess of how many different verification processes there were for 50 web.

Colin: Probably 50

Keith: 50.

CJ: yeah, I think, I'm going to cheat here a little bit and guess like 10 based on the

Keith: there were 49, 49 different approaches. There was one, I forget who it was offhand, but there was one that duplicated somebody else and that was it. it was horrifying.

Colin: It's what happens when you roll your own auth,

CJ: Holy smokes.

Keith: Yeah. Yeah. and this is just the authentication. This is just a verification. This isn't the additional capabilities. but when we look at some of those capabilities, we have things like, Shared secrets. So this is the model that, let's see. I think Okta uses it, DocuSign. Oh, Twilio implements this too. In the dashboard itself, there'll be some sort of secret that's generated and then that gets used in the signature creation. And then on the receiver side, you've got that secret and then you generate the same hash and you use that secret to make sure, yes, this was definitely came. Twilio, this definitively came from Okta. What we've seen a lot of in the wild is people not verifying payloads at all. In fact, I do a demo where I open a very simple Twilio app and I have people texting in. I go, Hey, by the way, if you have Postman, there's my u URL endpoint on the board right now. Why don't you go ahead and do a post and see what happens? and people start posting, data to my endpoint right there on the screen. And sometimes things go off the rails. It will keep it pg, pg rated here, but sometimes it goes off the rails. and I point out to people, I say, look, this is a silly Twilio app in front of an audience. I said, what if you are waiting for a Stripe web hook to confirm, yes, Colin paid for this shopping. if I know your endpoint and you haven't verified that payment did come, or that payment notification came from Stripe, somebody else could make a post and Mark Collins' order is paid. And how would you know that? How would you ever find that out? And that's. It's scary. So anyway, we go through these different processes and we wanna encourage people to verify whether that is a shared secret, like a Twilio or Okta. There's some that use, like asymmetric keys. I think SendGrid and PayPal use those. So you've got a, a key on each side, and then it signs with the private. And you can verify it with the public key and say, yes, definitively this came from PayPal. some of 'em are using like is a whole other challenge, but it's a, it's still, it comes down to, it's a way of being able to say, yes, this definitively came from the source. And some of the providers actually support multiple ones. I think SendGrid supports two or three different approaches, so you can kinda gauge your verification method to, according to the skills and the abilities that you.

CJ: That's cool. one thing too that has been interesting as we're like reviewing Stripes web hook situation is if you're building a web hook provider that's sending these web hook notifications, one of the challenges is versioning your event payloads. So that if you have, you're adding a new property, but you don't wanna break or you're removing a property even worse. You don't wanna break any of those web hook consumers, and so you have to version it. So now like each web hook endpoint has a different version and the payloads have different versions and stuff. And so when looking@webhooks.fyi, under Web Hook security, if you look at this like data list notifications option, that is really attractive to me because like then you don't actually have to include. any like too much stuff that's gonna get versioned, right? Like you can build a payload that will be very consistent across decades worth of versions, ideally. But then the consumer, when you receive the notification, depending on your S D K, that is then pinned to some specific A p I version can fetch the latest copy of that object that you're being notified about. You get the latest state, but you don't actually necessarily have to get that in the payload of the web hook.

Keith: Yeah, so a little bit of background. a dataless notification, we said earlier, like the message passing of saying, here, the state of this thing has changed. Here's all the information about it. In a dataless one, instead of having all that information, all it has is a reference to the object that changed and the fact that it did change. And then you go back and you use that your eye to then pull back and say, okay, the thing has changed. This shopping cart status has changed. Tell me, that's actually what I recommend, not just for versioning, it mitigates some of that, but also if you have really sensitive information. So think of a healthcare use case where you don't necessarily wanna broadcast that information over the wire.

CJ: It's also if I think about it a little bit, it's also, really great if you have, like a globally distributed network. Of like sharded databases where like some of your customers are in, the US and some are in the Australia. If you're just sending this like super lightweight dataless payload that says, here's the object ID that was updated, then like the API request from Australia will be routed to some AWS data center in Australia to fetch the object faster than, yeah. If it's located somewhere else. I don't know, feel like there's. Some,

Keith: a really good approach. I hadn't thought about that. I need to add that to the site.

CJ: it's, the, yeah, I think, yeah, if you're looking at those web hook security options, I think dataless is, pretty attractive, like going forward.

Keith: we also have a section on there for like operational experience because while we started with a security mindset, we said, okay, as people are building and designing and working with WebBook, how do they operationalize 'em? you had mentioned like versioning of the payloads and things. There's also the, some capabilities around forward compatibility of when a web hook comes in. How do you support multiple keys? Because there's always that moment of like with a shared secret of my secret has been. I need to revoke the old one and generate a new one. Depending on the traffic and your web hook patterns and everything, you could actually miss requests, so the ability to revoke the old key and generate a new one. If you can have both of them running simultaneously, you don't necessarily miss any requests.

CJ: Yeah. And I've seen sometimes like where each end point that you create in the third party, so. If you imagine the way that this usually works is you log into the third party, whether that's Slack or Zoom or Box or whatever, and there's some input box in your developer application settings where you're gonna enter a url. In many of these, the ones that will give you like a signing secret will generally give you a different well, Might give you a different signing secret per web hook endpoint that you put in. So sometimes if people wanna roll the key, they'll just put the same u URL twice , and then they'll get back like different keys and then like they'll have two running for a little while and then you can delete the old one. but yeah, I think it's, operationally it sounds like a bit of a nightmare, cuz you have to oh, let me run through the entire process of checking and verifying the. and then let me like catch an exception and do the same exact thing again to re-verify with like potentially, N Keys that you're handling. So

Keith: and we all know that complicated processes scale really well, right?

CJ: Yeah, it is just like any, anything that's complicated is going to break. And I see that PagerDuty is one of your web hook providers and so that's something you're gonna need when you build one of these complicated processes and someone tries to roll the key. .

Keith: Yeah, PagerDuty is actually doing a number of things. they've got, like the zero downtime rotation is what we call that one, so that you're not going to miss webhooks in between and you're not gonna break downstream systems. But, I'm trying to think. I think HubSpot also implements some stuff around forward compatibility. So not just changing the key, but changing the signature method. because frankly we know that at some point we're gonna have to change our signing methods. I will also say there's, I think at least one or two companies that use MD5 for signatures. Don't,

CJ: Yeah.

Keith: don't we do name them on the site. I'm not gonna name them here cuz that's impolite, but at some point we're gonna have to change our signature methods. So having that forward compatibility, that multiple URL pattern, like having those things ends up being super powerful.

Colin: Yeah. If you think that might be you can go to webhooks.fyi to verify. but you can also submit your company. So if you do follow these, there's a way to submit and contribute, like to add to that list, right?

Keith: Oh, yeah. I should say we set this up entirely as a community site. so there's very light, any inro marketing on it, I think it says in the header and the footer, and that's about it. but this is all a public GitHub repo. go to, github.com/ngrok, webhooks.fyi. See where we're really tricky with the naming there. but we accept poll requests from every anyone. So we have, there's a web hook company. I think it's sw. That submitted, themselves and a bunch of their customers, we want to make this the definitive place that people can come to learn about webhooks and still have it totally neutral. So we don't sell webhooks. We're not interested in that. We just want the internet to be a little more secure, and we think this is one of the ways to do it.

Colin: I think it's a great tool for if you're doing your first, you have a guide for web hook consumers as well as web hook providers. A lot of these are on the provider side, the security, things like that. But then on the consumer side, there's a whole bunch of kind of conventions that you have to learn. a lot of times, let's just say you're, you don't have your end groc tunnel on right now. If you have that turned off and Stripe tries to send a message to you, they're gonna do things like retrying until there's a certain point when they just say, I'm gonna give up and not keep sending it. what does that look like on your end? How do you verify as a consumer that you're not playing the same event more than once? there's some reasons why you would wanna replay a same event more than once, but, even that, that idea of dataless notifications becomes pretty interesting in that instance where it's have we seen this event before? Okay, if we have, we're not gonna do anything with it. Or let's go check and see what's changed each time versus some people I do know try to use webhooks as a pure integration framework, and there's some challenges there if you miss events. , right? Because it's not polling, you're not. but being able to go reach into the system with your API key to go get that latest data can help with that. I think for us, we do use a lot of webhooks to tell us something changed, but we usually use that to go pull the service to see. All the changes and maybe all the ones that we don't know about. so it's more of, instead of polling every hour or every day, we'll pull when we get a message from the system, which is also a good way of thinking about that.

Keith: Yeah, absolutely. And There, there's the, provider section. There's a consumer section, best practices for both. Even the people that mostly consume. We found that internally, they're still building systems that are generating webhooks. Even if they don't go outside your firewall, outside your infrastructure, that's fine. Remember, originally, we start talking about the value of webhooks is the fact that we all know HT d p and we all know J S O N. Don't generate your own custom protocol for these things. Even if your only consumers are inside your company, still follow best practices because all it takes from, for that web hook to then be shared publicly is some senior exec at your company going, wow, we have partners, we have customers who would love that information too. And now suddenly you're entirely internal tool set just got published to the world.

CJ: Yeah. I had a couple of questions. If we could circle back to the, like the building and hosting of the site on GitHub. it, I really like this approach in building and public Also, I noticed that it's using Mark Doc, which is pretty cool. That's fun to see in the wild. , but also, yeah, have you found this to be like a pretty successful way to engage in open source? I, it's also seems like a great lead gen for ngrok and just generally how's it going? Would you recommend this to other companies that are trying to participate in open source

Keith: we decided to, to launch it on GitHub because it was quick. It was easy. we, had already collected all the information behind it. So really the marginal effort was getting a reasonable design in front of it and then generating some, Graphics to show like what the different web hook flows were and things like that. So I think it's, I've been very happy with the results so far. Yeah, we have Google Analytics on it, but we don't have a ton of like marketing analysis tools or anything like that. but realistically, like seeing people use this, hearing how people Been informed by this process, I think is great. We had, I think two, maybe three weeks ago, zoom announced that they're doing a major update to their webhooks and they announced the platform, they announced all the changes, and in their documentation itself it said, we use webhooks.fyi to make these decisions.

CJ: Whoa, that,

Keith: Yeah. That was so cool. Like our initial goal with Web outside FOI was to take the information we had and organize and make it available to others. The aspirational goal was that we start nudging the industry towards better practices, was amazing to have somebody like Zoom come forward and say, your work had inspired us to do this better. it was a mic drop kind of moment.

Colin: Yeah, there've been these webhook communities for a while. Like I've been in the webhooks Google group for a long time, but like people who are gonna go consume or create their own web hook system, I don't know that they're going to join a Google group or go. I think like webhooks.com used to be a directory, but like that site doesn't exist anymore. There's a. Webhooks, r slash webhooks. Not a very lively place. not really where people are going. I think, having the powered by ngrok and just like being a little bit of an organizer and giving, some very good opinions and also like making it open so that people can contest some of those, oh, you know, so. Something new has happened that we now believe should be, the new standard for security or a new option, right? People can do a poll request and contribute to it. it's very similar to like, whenever someone decides to go write their own authentication from scratch, you want as many eyeballs on that as possible. And that's why so many people use these open source authentication tools because so many people have seen them. And I think, you guys are giving some credibility. But then everyone's gonna benefit from it. Zoom's adding credibility to it. We're at orbit. We're gonna take a look at our webhooks and see how we stack up against these, right? And figure out like, what are we missing or what are we doing right? and it's a good way to just close any of those windows that might, we might have left open accidentally.

Keith: Yeah, and frankly, Sometimes you don't know what questions to ask. Like when we first started digging into this, like I wasn't even thinking about forward compatibility of changing the hashing algorithm. I was thinking about changing keys cuz that's something we just have to do occasionally anyway. But changing the ha hashing algorithm, I said, oh yeah, Allen Shreve, our ceo. He's yeah, we need to think about this. And I was like, oh. Yeah, actually that makes a lot of sense. And despite being, in Web Oak for 10, 12 years at this point, it wasn't something I'd really thought of. And so we're trying to let the next people know, Hey, look, here are a consideration you have. Even if you don't choose to think about forward compatibility, that's fine, but at least you knew that it's something that you might want to consider.

CJ: Totally. Another thing that comes to mind with for forward compatibility too, and like a lot of the comments that I'm making come back to adding stuff into SDKs. So like when we talk about Dataless or Yeah, it's, you're gonna lean heavily on the sdk, when looking at H Mac. verification, which is what we use at Stripe. There are helper methods in the Stripe SDKs in all of the official SDKs that will do all of the fancy, crypto creating the H Mac, you just have to give it, here's the signature and here was the request, and then let the SDK figure it out. And what's nice about that is that, again, you're, you can depend on the SDK for forward compatibility. So as maybe as the, as the payload is using different signing. Algorithms, you can upgrade those inside of the SDKs. And so then it's just a matter of oh, let me bump my SDK version and maybe put in a new key or drop the. The public key somewhere in order to handle the verification so that, like I also, I have implemented lots of web hook endpoints that use H Mac and they don't have helpers in the SDKs. And every single time it's ah, how does this work again? And like it's so finicky cuz you have to get it perfect or it won't work at all. And so it's great that you have the code on, WebBook FYI,

Keith: that's one of the kind of horrifying things we found. Like that moment of, how do I do this again? We found, I don't know, 10, 15% of the docs didn't actually have SDKs or code samples. It was like a bulleted list of get this field, then do this, and I'm like, wait a minute. You are trying to appeal to developers in general. You don't know their skill level, you don't know they're understanding these topics. You're expecting them to do it right the first time. I'm like, come on, this is, you're making life miserable for them. And I always defer to use the SDK if it's available. but every company has to prioritize and they're going to not be able to tackle every language, every framework out there. And I, I don't criticize anyone for that, but there's gonna be times where you are working in an environment that they do not have the tooling for. So you have to figure out, okay, can I do this myself?

CJ: right? Is that, so it sounds like inside of Enro now you can use it in production to handle message verification. Is that. is exposing SDKs from the ngrok side that handle that? Is that something that y'all have thought about at all? providing those open source as oh, hey, this is just like the standard H Mac thing. You pass in Slack or whatever. I don't know. And then it just will verify things for you.

Keith: Yeah. the way it works on our side is that you choose your web hook provider. So if you're on the dashboard, you choose Slack and you put in, I think it's the shared secret for Slack, and then we handle it in the cloud. We've considered, how do we provide general tooling for that? That's really hard. We've tried to say, can we do a inject your own logic, like a Lambda style kind of thing for web book verification that scares the crap outta me executing other people's code in our environment as eh, Yeah, those are some interesting questions that I'm not sure we're willing or ready to answer at this point, but as long as you're in the flow, you can do a lot of things.

CJ: totally. The fact that there were 49 different implementations across 50 providers also, is so frustrating because the way that I like to implement a web hook endpoint is just, I want like slash webhooks and I want every third party that I integrate with for that application to all go to slash webhooks. And now you have to like fork on the source, like where did this come from? And now you

Colin: big case statement,

CJ: Exactly big case statement. And some people think that, synchronous webhooks are cool and asynchronous webhooks are cool. Like all this stuff. I'm like, Hey, that is another huge, I think selling point of letting N Groc just do it for you is that, hey, if you figure out all of the verification piece and I don't have to worry about that, then my web hook endpoint is literally just, take the payload, stick it in the database, and then kick off a background job to process this later and then respond with 200. And I'm like gonna walk away. and yeah.

Keith: there was a great article going around recently called, the Self-Provisioning Runtime. And it was the whole mindset of that is as you're building these stacks and these operating environments, if we can build tooling and support infrastructure so that the developer building their app only has to worry about their business logic. they get more efficient, they get more effective, and they can scale. They can operationalize things that much easier. It's really the, that's the mindset behind most of the DevOps philosophy is how do we make sure that developers stay focused on building business logic? And so we actually had the same concept on the ngrok side of things. So how do we just take the security stuff off? just make it so that it works by default? And you don't have to do this because implementing OAuth, implementing webhooks, like all these things suck, .

CJ: Yep.

Keith: it's the exact same patterns over and over again, but it's not entirely the same pattern. It's really close to the same pattern. It's 49 out of 50 times. It's different, but it's just different in little ways, and it's irritating.

Colin: Absolutely. We found that link. We'll put it in the show notes cuz I think, that idea is I think with the stripe, listen. framework too, like being able to listen to those webhooks. I see this, we'll probably talk to Alan about this when we have him on the show, but ngrok go looks really exciting too, just being able to like, have your app start listening. For you rather than have like I, I like running ngrok locally and just like seeing, those things. But as a developer or even as someone who's focusing on developer experience, like that time to first hello world or time to that first light bulb going off and it's all these steps that we have to do to be secure are only gonna slow that down if you don't provide ways of, Self provisioning, runtime or helpers in your SDKs. and so it's a good thing to think of if you're developing developer tools or if you're in developer marketing or developer advocacy and you're trying to figure out, yeah, everyone seems to fall off at this point where we ask them to go do a ed list of 10 things. what can we take off their plate? What can we push to the edge there? And yeah, it's, I think the thing that I'm most mind blown why right now is just how many things. I'll go back to Twilio right now. and, between Jeff and Alan and yourself. yeah, it was a, around that time was a, was what a time to be on the internet and all the things that we get to benefit from today.

Keith: I, I think say 2009 through 2012, there was a lot of foundation laid for kind of an internet renaissance. and some of that has led to all the, these API companies, a lot of it's led to some of this machine learning stuff. Machine learning like ChatGPT would be a lot harder to use and implement right now if we didn't have the APIs behind the. And the APIs rewind 12 years ago and people were just getting their hands and their heads around these things. So I think, yeah, it was a cool time to get started in the space and I look forward to what's the cool thing to get started with now?

CJ: I was just checking to see if OpenAI had webhooks, but I'm like, what would that use case even be? instead of webhooks, they just have like really long requests. you make an API call and it takes, I'm not bagging on it, but yeah, like two seconds later you'll get back your check completion. but. earlier we talked about the Stripe listen command. So at Stripe we built a custom CLI for integrating against Stripe, and it has a bunch of different features. One of them happens to be Stripe, listen, which is a tunneling, a tunneling. Mechanism so you can receive your stripe webhooks without having to set anything up. So it will set up the WebBook endpoint and also deliver it. and as part of that we have a VS code extension. So like from inside of VS code you can start the listener. And I found that there's also a ngrok for VS code extension. And I assume this works very similar, where you can just like fire up ngrok directly inside of your editor.

Keith: Yeah, just like that early thing of a wanting to set his phone down, keep his hands on keyboard. We have that exact same mindset for this. Phil Nash, another Twilio alum, built that vs. Code extension years ago at this point. but it's all about how do we keep you in the environment that you're working in? How do we not interrupt your flow? you wanna be effective? Let's make it so that you are and just get outta your.

CJ: totally love it.

Colin: Awesome. I think that's a great place to leave people wanting more. They we're gonna put a bunch of links. I found a really good article from 2007 from Jeff Lindsay talking about how webhooks are going to revolutionize the web. It's a good read. gets into a little bit of the message handling and the small talk type message passing that, that we talked about here today. but where can people find more about you and all of your work on the internet? Keith?

Keith: Yeah, absolutely. my day job is inro, so ngrok.com, that's the place for everything. If you need to reach me specifically, you can email me danger@ngrok.com. I respond to that pretty quickly. if you wanna find me online, everywhere else, I'm Casey Software. That's c a s e y software. and definitely check out webhooks.fyi. It is the source. We really wanna make it available to everyone and, inform the internet. Make it a little more.

CJ: Thanks a ton for all your hard work on that resource. I think it's a wealth of knowledge and information and yeah, it's making the whole community better. So really appreciate that, Keith. yeah, so as always, you can head over to build and learn.dev to check out the links to the resources that we've just mentioned and the show notes. That is all for this. Thanks folks. We'll see you next time.