Oxide and Friends

Bryan and Adam reflect on Oxide and Friends in 2025--favorite moments, episodes, and images. Happy new year and see you in 2026!Your hosts are Bryan Cantrill and Adam Leventhal.Some of the topics we hit on, in the order that we hit them:RFD 576: Using LLMs at Oxide (hacker news comments)OxF: Oxide and Friends 6/2/2025 -- AI Discourse with Steve Klabnik (around 1:08:00)Shell Game podcastOxF s05e12 – Hell is other networks — April 4, 2025"No Egress" was a ChatGPT joke!OxF s05e33 – A Grown-up ZFS Data Corruption Bug — November 26, 2025Simpsons scene deleted in syndicationOxF s05e29 – AI in Higher Education with Michael Littman — October 17, 2025OxF s05e28 – Systems Software in the LargeOxF s05e18 – AI, Materials, and Fraud with Ben ShindelOxF s04e21 – Adventures in Data Corruption"Duck season, Fire!"MLG Airhorn aka "the jj airhorn"OxF s05e31 – FuturelockLaura's blog post: A disappearing Service ProcessorOxF s05e34 – Death by Uptime"Painfully concrete" - ChatGPTOxF s05e03 – Holistic Engineering with Robert MustacchiOxF s05e30 – RIP USENIX ATCTeam DTrace meets Dennis Richie, redux"Fart Boy"OxF s05e?? – Books in the Box V — The latest annual book recommendation episode.Oxide Bingo by John HollowayOxF s05e16 – Scaling ManufacturingOxF s05e22 – Founder vs. InvestorOxF s05e27 – Character LimitStretch goal for 2026: finally a C&DOxF s05e24 – Diving InBryan's blog: College Baseball, Venture Capital, and the Long MaybeOxF s03e31 – Hiring Processes with Gergely OroszOxF s04e06 – Crucible: The Oxide Storage Service"Don't worry, Alan, no one will listen" -> one of our most popular episodesOxF s04e23 – RFDs: The Backbone of OxideOffice Space: Michael Bolton as AI em-dash"This isn't nostalgia, it's epistemology" - ChatGPT"Weaponized weariness" - ChatGPT"Dry Fatalism" - ChatGPT"Cougar turned in his wings..."OxF s05e16 – Solutions Software Engineering with Matthew SanabriaAlexander Hamilton: amazing. Also the world's pre-eminent subtweeter and blogger?If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

We hit a new (and disturbing!) failure mode recently when a production rack that had been up for several months saw every (!) compute sled's service processor become simultaneously unresponsive. Bryan and Adam were joined by the members of the Oxide team who debugged the vexing issue -- and reached its surprising root cause.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Cliff Biffle, Matt Keeter, and Will Chandler.Previously, on Oxide and Friends:OxF s05e03 – Holistic Engineering with Robert MustacchiOxF s04e14 – Rebooting a datacenter: A decade laterOxF s01e26 – The Pragmatism of HubrisOxF s05e20 – Debugger-Driven Development (omdb)OxF s05e07 – Transparency in Hardware/Software InterfacesOxF s05e31 – FuturelockOxF s05e33 – A Grown-up ZFS Data Corruption BugSome of the topics we hit on, in the order that we hit them:hubris #2304: STM32H7 Ethernet driver stops yielding CPU after many packetsgist — Summarizing the Hubris side of investigationsMatt's blog: Hunting a spooky ethernet driver bugIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Hey hey! We recently tripped over a ZFS data corruption bug–introduced over 18 years ago! Bryan and Adam discuss with members of the Oxide team as well as Matt Ahrens, the co-inventor of ZFS.In addition to Bryan Cantrill and Adam Leventhal, speakers included Alan Hanson, Matt Keeter, Andy Fiddaman, James MacMahon, and special guest, Matt Ahrens.Previously, on Oxide and Friends:OxF s4e6 - Crucible: the Oxide Storage ServiceOxF s5e28 - Systems Software in the LargeSome of the topics we hit on, in the order that we hit them:ZFS fsync can trigger ZIL transaction reordering and data corruptionRFD 177: Implementation of Data Storagethe "fix" that introduced data corruptionPRs needed!If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Oxide founders, Bryan and Steve, as well as Oxide investor, Seth Winterroth, were joined by Liz Zalman and Jerry Neumann, authors of the book Founder vs. Investor, discussing the collaboration and conflict in company formation. Adam was also present.In addition to Bryan Cantrill and Adam Leventhal, our guests included Liz Zalman, Jerry Neumann, Seth Winterroth (Oxide investor), and Steve Tuck (Oxide founder / CEO).Some of the topics we hit on, in the order that we hit them:Founder vs. InvestorTopic[@M:SS](link into recording) Leventhal's ConundrumPRs needed!If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

We're big users of async Rust at Oxide, and recently we found (another) very odd and hard to debug pathology related to async Rust that we dubbed "Futurelock". Oxide engineers who diagnosed the problem join Bryan and Adam to describe Futurelock and discuss methods to identify and avoid it.In addition to Bryan Cantrill and Adam Leventhal, speakers included our Oxide colleagues Dave Pacheco, John Gallagher, Rain Paharia. Sean Klein, and Eliza Weisman.Previously, on Oxide and Friends:OxF s05e22 - When Async Attacks!OxF s05e26 - Technological Revolutions with Jerry NeumannSome of the topics we hit on, in the order that we hit them:Oxide RFD 609: FuturelockPRs needed!If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!