That got me thinking about the other flavors of this problem I’ve hit over the years. Desync bugs are interesting precisely because they’re not one pattern — they’re a category. Protocol ambiguity, timeout handling, connection reuse: the mechanism varies, but the result is the same: you asked for A and got B, and everything looked fine.
I first ran into this back in 2008 with MogileFS and a race condition. A slow database caused an fread to time out waiting for a response. The client treated it as “not found” and moved on — but the TCP buffer still held the original response. The next request read that stale data instead of its own response. We were asking for item A and getting item B. The fix was simple once diagnosed: close the connection on any timeout rather than reuse it.
At Reverbnation I ran into it again — twice. The MogileFS Ruby client had a similar connection safety issue. We ended up streaming (and caching in CDN) the wrong song. And separately, Dalli (the Memcache client) had its own desync problem (can’t find a link to this one), but there’s a recent one with a similar thread — issue #956 — where response data could bleed across requests with timeouts.
What makes these bugs so insidious:
The HTTP desync bugs found in Discord’s infrastructure show the same pattern at the HTTP layer — responses routed to the wrong client because something upstream got out of sync. Different protocol, similar critical section.
I don’t expect these to stop appearing. Any time you have pooled connections and concurrent access, the conditions are there. The next one is probably in a library I’m using right now.
The other interesting part about these types of bugs, is that the code one writes to send a request and receive a response seems trivial, and this type of issue can hide in a library for a long time before it’s unearthed.
The framing problem shows up outside of HTTP too. This post on phantom patches describes GNU patch being unable to distinguish actual diff content from diff-shaped text embedded in a commit message — same underlying idea, completely different domain. When the format doesn’t have unambiguous boundaries, parsers disagree, and something gets applied that shouldn’t.
Problem over Person ` In a team meeting, one of my leaders was struggling with their scrum master. Their engineers keep telling them about boring standups and no improvement.
I asked one of my annoying questions,
Me: “what would you like to do about it?”
EM (Engineering Manager): “Discuss with the group about how I can give the scrum master feedback.”
Me: “Why are they coming to you?”
EM: “They talked about it with the team, but nothing has changed.”
Me: “What does that mean?”
EM: “The engineers all feel this way.”
Me: “Has this come up at a retro?”
EM: “well,they don’t feel comfortable talking about it with the scrum master in the room.”
Me: “why not?”
EM: “You need a manager to give feedback to the scrum master”
Through the questions, we discovered a string of assumptions and fell in to a trap.
Some potential beliefs that got us here:
Any of these MAY be true, but outside of misconduct we can often step up to the problem and work it just like we might a bug.
We can build the communication and influence skill to help this engineer share the problem with the team at a retro, almost like it’s a bug report.
How can the team flex their continuous improvement muscle and own this issue and its solution? How do we, as a leader, engage build the skills so our engineer can describe the problem in a way their team will understand and rally around it?
Practice makes perfect.
Some example framings of this problem:
You might hate these and that’s okay. Communication is contextual and you know your voice and how to frame the real issue. (NB: AI can help brainstorm here, but the engineer needs to own the tone and keep it short and sweet; I’ve seen lot of disconnects/hard feelings because folks tried to use AI to help them express an issue only to find out some nuance and values got lost).
You’ll notice some of the examples above are harsher than others, but none are ABOUT another person’s shortcoming. They’re about the symptoms and the problem.
Leaders help engineers prepare to engage in their own discussion or retro WITHOUT a manager. This would be a great time for leader and engineer to use role play and help them practice both expressing the issue and responding to inevitable follow-up questions and challenges.
Once the problem is on the table, the whole team can help own the issue. They have created the situation they’re in together depending on how long they’ve been quiet or even gone through the motions like zombies every day for months without saying anything. The objective is to break the seal and get the issue on the table for discussion.
As leaders, we need to be on the lookout for what behavior and core skills are needed before we jump in to the rescue. Stepping back and letting the team engage CAN lead to more conflict, but over time they’ll build their muscle at working through their own issues rather than calling on outsiders to rescue them. Leaders hope for an interesting organizational problem or performance issue – ESPECIALLY on another team because we can BLAME someone.
As a leader, when did you swoop in to solve an issue when you needed to find a dfferent stance? Other resources for further reading:
]]>I was thinking of using a home-grown system to do this, but upon searching, and some pings from some experts, it became clear there’s a lot of prior art here. Some of the options are pretty expensive, but others aren’t.
Some of the needs of our platform in general. Some of which could be addressed by these platforms:
Another distro we could use other than traditional Debian based Raspberry Pi OS http://tinycorelinux.net/faq.html
]]>Goals:
One could just run VMs using virtualization on the respective BSD distro above. Using: https://github.com/churchers/vm-bhyve. Tried this and was able to start a VM pretty easily.
Other options include more virutalization web management frameworks on top of KVM and/or Xen nee Citrix
]]>Some links to other folks working on this Exobrain, or cataloging everything they know:
Some of the containers pointed me to tools that I find fascinating:
The Bluez library included in 18.04 has some issues reconnecting to Bluetooth devices (independent of the Dell XPS). Fix is to includ a PPA as described in this article.
Update: apparently Dell/Canonical has implemented their own workaround for this in package oem-workaround-bt-auto-resume-lp1774582
Update 2020-01-06: This may have been fixed in mainline Ubuntu by an update to the linux-firmware package. https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1849779
tee /lib/systemd/system-sleep/rfkill-bluetooth-workaround <<EOF
#!/bin/sh
set -e
if [ "\${1}" = "pre" ]; then
rfkill block bluetooth
elif [ "\${1}" = "post" ]; then
rfkill unblock bluetooth
fi
EOF
chmod 755 /lib/systemd/system-sleep/rfkill-bluetooth-workaround
for more information see this bug
This may be irrellevant with kernel update for 18.04 Bionic and 19.04 Disco https://bugzilla.kernel.org/show_bug.cgi?id=199689
]]>