Hey hey! We recently tripped over a ZFS data corruption bug–introduced over 18 years ago! Bryan and Adam discuss with members of the Oxide team as well as Matt Ahrens, the co-inventor of ZFS.In addition to Bryan Cantrill and Adam Leventhal, speakers included Alan Hanson, Matt Keeter, Andy Fiddaman, James MacMahon, and special guest, Matt Ahrens.Previously, on Oxide and Friends:OxF s4e6 - Crucible: the Oxide Storage ServiceOxF s5e28 - Systems Software in the LargeSome of the topics we hit on, in the order that we hit them:ZFS fsync can trigger ZIL transaction reordering and data corruptionRFD 177: Implementation of Data Storagethe "fix" that introduced data corruptionPRs needed!If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!