It all started as a simple error message from a piece of software that had run seamlessly thousands of times over the past two years. And yet, once a week in late 2016, this error message would rear its ugly head. At first, it was thought to be a user error. But after some testing, it was clear that we were dealing with a software bug. Thus began a month-long hunt for this elusive bug, becoming as much a personal vendetta as a solution to an intractable problem. Software bugs are a fact of life; born out of the ever-increasing complexity of our world. Humans work hard at reducing their frequency and effect, but they can never be eliminated. In our case, what complicated matters was our inability to replicate the problem, which would have allowed us to reliably test and fix the issue. No matter how many times we tried, we simply could not reproduce this bug, and yet, it continued to haunt us week after week. The problematic code was only 8 lines long. After several weeks of chasing, these eight lines were re-written and simplified. This should have been the end of the hunt, but instead of using the re-written code, it was decided, perhaps foolishly, that the hunt must go on. Eventually, the whole company was pulled in to aid in the hunt. The source of the bug was eventually discovered, but at what cost? Listen to your bug-hunter host Hany Fahim explain how it all went down. Connect with Hany at his company <a href='https://stack.io'>stack.io</a> and <a href='https://www.linkedin.com/in/hanyfahim/'>LinkedIn</a>.   Don&apos;t forget to leave us a review and subscribe to our channel to keep up with the latest episodes!

<description>&lt;p&gt;It all started as a simple error message from a piece of software that had run seamlessly thousands of times over the past two years. &lt;br/&gt;&lt;br/&gt;And yet, once a week in late 2016, this error message would rear its ugly head. At first, it was thought to be a user error. But after some testing, it was clear that we were dealing with a software bug. Thus began a month-long hunt for this elusive bug, becoming as much a personal vendetta as a solution to an intractable problem.&lt;br/&gt;&lt;br/&gt;Software bugs are a fact of life; born out of the ever-increasing complexity of our world. Humans work hard at reducing their frequency and effect, but they can never be eliminated.&lt;br/&gt;&lt;br/&gt;In our case, what complicated matters was our inability to replicate the problem, which would have allowed us to reliably test and fix the issue. No matter how many times we tried, we simply could not reproduce this bug, and yet, it continued to haunt us week after week.&lt;br/&gt;&lt;br/&gt;The problematic code was only 8 lines long. After several weeks of chasing, these eight lines were re-written and simplified. This should have been the end of the hunt, but instead of using the re-written code, it was decided, perhaps foolishly, that the hunt must go on.&lt;br/&gt;&lt;br/&gt;Eventually, the whole company was pulled in to aid in the hunt. The source of the bug was eventually discovered, but at what cost?&lt;br/&gt;&lt;br/&gt;Listen to your bug-hunter host Hany Fahim explain how it all went down.&lt;br/&gt;&lt;br/&gt;Connect with Hany at his company &lt;a href='https://stack.io'&gt;stack.io&lt;/a&gt; and &lt;a href='https://www.linkedin.com/in/hanyfahim/'&gt;LinkedIn&lt;/a&gt;.&lt;br/&gt; &lt;br/&gt;Don&amp;apos;t forget to leave us a review and subscribe to our channel to keep up with the latest episodes!&lt;/p&gt;</description>

It all started as a simple error message from a piece of software that had run seamlessly thousands of times over the past two years. And yet, once a week in late 2016, this error message would rear its ugly head. At first, it was thought to be a user error. But after some testing, it was clear that we were dealing with a software bug. Thus began a month-long hunt for this elusive bug, becoming as much a personal vendetta as a solution to an intractable problem.Software bugs are a fact of lif...

E06 - Unpacking an Anomaly

Most companies don’t discover they have been hacked until months later. But imagine catching a hacker red-handed? Your course of action would be very different.   Instead of dealing with the aftermath, your focus would be on urgently identifying the source and then blocking the hack.   This is exactly what <a href='https://stack.io'>stack.io</a> CEO and Founder Hany Fahim did when dealing with his first hack, back when he was a young Systems Administrator. Yes, this episode tells a tale from some years ago. So long ago that our host Hany was using a Blackberry and struggled to remember the course of events. This slight memory haze introduces a humorous interlude in the episode, with current-day Hany and his younger self in a quibble about the true course of events. The hack started with an internal search tool used company-wide to retrieve important information. Its speed had slowed to a crawl. Hany investigated and discovered a huge backlog of searches. The culprit? The task at the head of the queue was trying to back-up the database. Hany terminated the task only to have it pop-up again minutes later. He traced the request to a web server, then sent an email to the hardware team and to the entire company about the issue. The situation was resolved, or so he thought. Suddenly, monitoring alarms started sounding from a highly secured back-up system deep inside the data centre. The hack was coming from inside the office! Hany had the hardware team track it - directly to a vice president’s office. Knowing the VP was on vacation, Hany went into his office and discovered the hackers were remotely logging into the VP’s terminal and using it to back-up the company database. Hany unplugged the computer. It turned out that this was only the beginning. We won’t give the entire story away here . . . otherwise, that would spoil the episode for you! Hany notes that humans, not machines, are generally the weak-link when it comes to security. Similar to this tale, the recent SolarWinds hack in December 2020, which breached many Fortune 500 companies and US government agencies (including Homeland Security and the National Nuclear Security Administration), were also connected to human error, rather than the machines. Connect with Hany at his company <a href='https://stack.io'>stack.io</a> and <a href='https://www.linkedin.com/in/hanyfahim/'>LinkedIn</a>.   Don&apos;t forget to leave us a review and subscribe to our channel to keep up with the latest episodes!

<description>&lt;p&gt;Most companies don’t discover they have been hacked until months later. But imagine catching a hacker red-handed? Your course of action would be very different.&lt;br/&gt; &lt;br/&gt;Instead of dealing with the aftermath, your focus would be on urgently identifying the source and then blocking the hack. &lt;br/&gt; &lt;br/&gt;This is exactly what &lt;a href='https://stack.io'&gt;stack.io&lt;/a&gt; CEO and Founder Hany Fahim did when dealing with his first hack, back when he was a young Systems Administrator.&lt;br/&gt; &lt;br/&gt;Yes, this episode tells a tale from some years ago. So long ago that our host Hany was using a Blackberry and struggled to remember the course of events. This slight memory haze introduces a humorous interlude in the episode, with current-day Hany and his younger self in a quibble about the true course of events.&lt;br/&gt; &lt;br/&gt;The hack started with an internal search tool used company-wide to retrieve important information. Its speed had slowed to a crawl. Hany investigated and discovered a huge backlog of searches. The culprit? The task at the head of the queue was trying to back-up the database. &lt;br/&gt; &lt;br/&gt;Hany terminated the task only to have it pop-up again minutes later. He traced the request to a web server, then sent an email to the hardware team and to the entire company about the issue. The situation was resolved, or so he thought.&lt;br/&gt; &lt;br/&gt;Suddenly, monitoring alarms started sounding from a highly secured back-up system deep inside the data centre. The hack was coming from inside the office! Hany had the hardware team track it - directly to a vice president’s office. Knowing the VP was on vacation, Hany went into his office and discovered the hackers were remotely logging into the VP’s terminal and using it to back-up the company database. Hany unplugged the computer. It turned out that this was only the beginning.&lt;br/&gt; &lt;br/&gt;We won’t give the entire story away here . . . otherwise, that would spoil the episode for you!&lt;br/&gt; &lt;br/&gt;Hany notes that humans, not machines, are generally the weak-link when it comes to security. Similar to this tale, the recent SolarWinds hack in December 2020, which breached many Fortune 500 companies and US government agencies (including Homeland Security and the National Nuclear Security Administration), were also connected to human error, rather than the machines.&lt;br/&gt; &lt;br/&gt;Connect with Hany at his company &lt;a href='https://stack.io'&gt;stack.io&lt;/a&gt; and &lt;a href='https://www.linkedin.com/in/hanyfahim/'&gt;LinkedIn&lt;/a&gt;.&lt;br/&gt; &lt;br/&gt;Don&amp;apos;t forget to leave us a review and subscribe to our channel to keep up with the latest episodes!&lt;/p&gt;</description>

Most companies don’t discover they have been hacked until months later. But imagine catching a hacker red-handed? Your course of action would be very different. Instead of dealing with the aftermath, your focus would be on urgently identifying the source and then blocking the hack.  This is exactly what stack.io CEO and Founder Hany Fahim did when dealing with his first hack, back when he was a young Systems Administrator. Yes, this episode tells a tale from some years ago. So long ...

E05 - A Windy Day

<a href='https://www.stack.io/'>stack.io</a> CEO and show host Hany Fahim puts on his Sherlock Holmes hat in this episode of Tales from the Ops Side. He&apos;s investigating a missing file of sensitive data, while his client and their partner point fingers at each other over fault. This client had an agreement to provide sensitive data on a nightly basis to a partner. This was automated, the process called a cronjob. Hany&apos;s lunch was interrupted in late 2017 when an urgent request came in from this client. The problem: One night the data was never received by the partner. Hany shifted into troubleshooting mode, working with his client and examining the guts of their system. It appeared the job did run as scheduled, even though the partner did not receive the file. What happened? Digging deeper into alert files, database logs and network graphs didn’t shed any more light on the problem. After examining all the evidence and chasing down logical leads, Hany was no further ahead. That night at home, he was distracted by the problem. After a late-night of research, he was no further ahead. Over a cup of strong coffee the next morning, he spotted an obscure forum post he had disregarded the night before. The post gave his investigation a new avenue to explore - Linux Process Accounting.  With more sleuthing, hiking through the historical bowels of the internet, and combing through over 1000 lines of code, Hany was rewarded with an answer and knew what the client&apos;s problem was and who was at fault. For the whole story, all the technical details, and Hany&apos;s insider view, listen to the episode. Connect with Hany at his company <a href='https://stack.io'>stack.io</a> and <a href='https://www.linkedin.com/in/hanyfahim/'>LinkedIn</a>.   If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!

<description>&lt;p&gt;&lt;a href='https://www.stack.io/'&gt;stack.io&lt;/a&gt; CEO and show host Hany Fahim puts on his Sherlock Holmes hat in this episode of Tales from the Ops Side.&lt;br/&gt; &lt;br/&gt;He&amp;apos;s investigating a missing file of sensitive data, while his client and their partner point fingers at each other over fault. &lt;br/&gt; &lt;br/&gt;This client had an agreement to provide sensitive data on a nightly basis to a partner. This was automated, the process called a cronjob. Hany&amp;apos;s lunch was interrupted in late 2017 when an urgent request came in from this client.&lt;br/&gt;&lt;br/&gt;The problem: One night the data was never received by the partner. &lt;br/&gt; &lt;br/&gt;Hany shifted into troubleshooting mode, working with his client and examining the guts of their system. It appeared the job did run as scheduled, even though the partner did not receive the file. &lt;br/&gt; &lt;br/&gt;What happened?&lt;br/&gt; &lt;br/&gt;Digging deeper into alert files, database logs and network graphs didn’t shed any more light on the problem. After examining all the evidence and chasing down logical leads, Hany was no further ahead. &lt;br/&gt; &lt;br/&gt;That night at home, he was distracted by the problem. After a late-night of research, he was no further ahead. Over a cup of strong coffee the next morning, he spotted an obscure forum post he had disregarded the night before. The post gave his investigation a new avenue to explore - Linux Process Accounting.  &lt;br/&gt; &lt;br/&gt;With more sleuthing, hiking through the historical bowels of the internet, and combing through over 1000 lines of code, Hany was rewarded with an answer and knew what the client&amp;apos;s problem was and who was at fault. &lt;br/&gt; &lt;br/&gt;For the whole story, all the technical details, and Hany&amp;apos;s insider view, listen to the episode. &lt;br/&gt; &lt;br/&gt;Connect with Hany at his company &lt;a href='https://stack.io'&gt;stack.io&lt;/a&gt; and &lt;a href='https://www.linkedin.com/in/hanyfahim/'&gt;LinkedIn&lt;/a&gt;.&lt;br/&gt; &lt;br/&gt;If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!&lt;/p&gt;</description>

stack.io CEO and show host Hany Fahim puts on his Sherlock Holmes hat in this episode of Tales from the Ops Side. He's investigating a missing file of sensitive data, while his client and their partner point fingers at each other over fault.  This client had an agreement to provide sensitive data on a nightly basis to a partner. This was automated, the process called a cronjob. Hany's lunch was interrupted in late 2017 when an urgent request came in from this client.The problem: One night the...

E04 - Dr Strangecron or: How I Learned to Stop Worrying and Love the lastcomm

Host Hany Fahim tells an intriguing tale of the global ripple effects of a political event and the phenomenal power of social media. Hany was - somewhat successfully – at home assembling a chair one evening in 2015. Suddenly, he received news of a huge spike in traffic. Normally, this would point to a number of things, such as a marketing event or even an attack—but nothing seemed to add up. Hunting for answers, Hany traced the users’ IP addresses. Interestingly, they all seemed to lead back to Brazil. That seemed like a red flag; was this a calculated attack? Rather than trying to puzzle it out on his own, Hany consulted the social media grapevine for answers: Twitter. When Hany learned of the Brazilian government’s 48-hour ban on WhatsApp, the puzzle pieces began to fall into place. Brazil’s citizens were circumventing the popular app’s ban by using VPNs. Luckily, Hany was able to use this information to find ways to accommodate the spike in traffic, rather than defending against it. How Brazil Kept Us Up At Night is a thought-provoking look at how business, politics, and technology are deeply intertwined in our social lives. Hany reflects on human ingenuity in the face of government censorship and the positive impacts of social media when it comes to problem solving for ops. Connect with Hany at his company <a href='https://stack.io'>stack.io</a> and <a href='https://www.linkedin.com/in/hanyfahim/'>LinkedIn</a>. If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!

<description>&lt;p&gt;Host Hany Fahim tells an intriguing tale of the global ripple effects of a political event and the phenomenal power of social media. &lt;br/&gt;&lt;br/&gt;Hany was - somewhat successfully – at home assembling a chair one evening in 2015. Suddenly, he received news of a huge spike in traffic. Normally, this would point to a number of things, such as a marketing event or even an attack—but nothing seemed to add up. &lt;br/&gt;&lt;br/&gt;Hunting for answers, Hany traced the users’ IP addresses. Interestingly, they all seemed to lead back to Brazil. That seemed like a red flag; was this a calculated attack? Rather than trying to puzzle it out on his own, Hany consulted the social media grapevine for answers: Twitter. &lt;br/&gt;&lt;br/&gt;When Hany learned of the Brazilian government’s 48-hour ban on WhatsApp, the puzzle pieces began to fall into place. Brazil’s citizens were circumventing the popular app’s ban by using VPNs. Luckily, Hany was able to use this information to find ways to accommodate the spike in traffic, rather than defending against it. &lt;br/&gt;&lt;br/&gt;How Brazil Kept Us Up At Night is a thought-provoking look at how business, politics, and technology are deeply intertwined in our social lives. Hany reflects on human ingenuity in the face of government censorship and the positive impacts of social media when it comes to problem solving for ops.&lt;br/&gt;&lt;br/&gt;Connect with Hany at his company &lt;a href='https://stack.io'&gt;stack.io&lt;/a&gt; and &lt;a href='https://www.linkedin.com/in/hanyfahim/'&gt;LinkedIn&lt;/a&gt;.&lt;br/&gt;&lt;br/&gt;If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!&lt;/p&gt;</description>

Host Hany Fahim tells an intriguing tale of the global ripple effects of a political event and the phenomenal power of social media. Hany was - somewhat successfully – at home assembling a chair one evening in 2015. Suddenly, he received news of a huge spike in traffic. Normally, this would point to a number of things, such as a marketing event or even an attack—but nothing seemed to add up. Hunting for answers, Hany traced the users’ IP addresses. Interestingly, they all seemed to lead back ...

E03 - How Brazil Kept Us Up At Night

Host Hany Fahim tells a roller-coaster tale of an emergency situation that occurred one beautiful summer evening in 2012. When multiple client websites his company supported mysteriously went down without warning, Hany raced against the clock—quite literally, it turned out—to get everything back up and running. Although his alerts indicated that high load was at the heart of the issue, when Hany checked things out, he was in for a surprise. Nothing seemed to indicate anything should be wrong. So, what could have possibly all these different systems to go down all at once? After finding an oddly simple solution, Hany traced the root of the problem back to an unlikely place: time keeping. This discovery took Hany down a research rabbit hole—one that changed his understanding of space and time, and the meaning of interconnectedness. Hany reflects on the unpredictability of the universe itself in this episode, and his tenacious spirit that summer night shows how important it is to rise to the occasion in Ops when things go awry. Earthquakes and The Moon is an intriguing deep-dive into science, time, and the physics of the universe. Hany’s story twists and turns to show how a single second, the Moon, and an earthquake half a world away can change the course of history. Hany is founder and CEO of <a href='https://stack.io'>stack.io</a> and you can connect with him there, or on <a href='https://www.linkedin.com/in/hanyfahim/'>LinkedIn</a>. If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!

<description>&lt;p&gt;Host Hany Fahim tells a roller-coaster tale of an emergency situation that occurred one beautiful summer evening in 2012. &lt;br/&gt;&lt;br/&gt;When multiple client websites his company supported mysteriously went down without warning, Hany raced against the clock—quite literally, it turned out—to get everything back up and running.&lt;br/&gt;&lt;br/&gt;Although his alerts indicated that high load was at the heart of the issue, when Hany checked things out, he was in for a surprise. Nothing seemed to indicate anything should be wrong. So, what could have possibly all these different systems to go down all at once?&lt;br/&gt;&lt;br/&gt;After finding an oddly simple solution, Hany traced the root of the problem back to an unlikely place: time keeping. This discovery took Hany down a research rabbit hole—one that changed his understanding of space and time, and the meaning of interconnectedness. Hany reflects on the unpredictability of the universe itself in this episode, and his tenacious spirit that summer night shows how important it is to rise to the occasion in Ops when things go awry. &lt;br/&gt;&lt;br/&gt;Earthquakes and The Moon is an intriguing deep-dive into science, time, and the physics of the universe. Hany’s story twists and turns to show how a single second, the Moon, and an earthquake half a world away can change the course of history.&lt;br/&gt;&lt;br/&gt;Hany is founder and CEO of &lt;a href='https://stack.io'&gt;stack.io&lt;/a&gt; and you can connect with him there, or on &lt;a href='https://www.linkedin.com/in/hanyfahim/'&gt;LinkedIn&lt;/a&gt;.&lt;br/&gt;&lt;br/&gt;If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!&lt;/p&gt;</description>

Host Hany Fahim tells a roller-coaster tale of an emergency situation that occurred one beautiful summer evening in 2012. When multiple client websites his company supported mysteriously went down without warning, Hany raced against the clock—quite literally, it turned out—to get everything back up and running.Although his alerts indicated that high load was at the heart of the issue, when Hany checked things out, he was in for a surprise. Nothing seemed to indicate anything should be wrong. ...

E02 - Earthquakes and The Moon

Tales From the Ops Side is a fascinating, behind-the-scenes look at the running of an Internet Ops company. Hosted by Hany Fahim – owner of Toronto-based stack.io – Tales From The Ops Side is a must-listen for anyone working in or interested in Internet technology. Even if you’re not part of the Internet tech world, you’ll find this show informative and entertaining! Hany explores such things as the interconnectedness between the moon, tides, and time, handling pressured deadlines, assisting nations dealing with oppressive government policies, and what it takes to launch and run a tech startup. As a storyteller, Hany is a natural, weaving in little details that add depth and colour to his narratives. He uses analogies to make his tech stories truly accessible to all. And Hany’s curiosity takes listeners on some wild and eclectic journeys, with extensive exploration of both causes and solutions! You’ll be surprised at what you discover and learn!

Tales from the Ops Side

Details

Recent Episodes