Nasty bug only found in production
-
[Grinding Gear Games](https://www.pathofexile.com/forum/view-thread/2253250/page/1#p15963161):
At 6:53pm on November 26 (New Zealand time), we deployed this 3.4.5c patch and it contained a significant bug. Due to a typo in a database key comparison function, if two users returned to the character selection screen at exactly the same time, there was a chance that one person was logged into the other's account. This was not apparent to us during testing because it requires a lot of players to be online before it occurs. Upon it being reported, we took the realm down at 8:26pm and restored the old version that did not have this problem. 417 players had their accounts accessed by another user. Of those, only 150 actually tried to log into a character. Most of these either logged out within a few seconds or were kicked off as the owner logged in again immediately.
I'll admit I'm really curious what their code looked like to make that possible in the first place. Their backend is primarily Linux/C++. but I'm not sure if that is just the main game server code, or also webapi's too. I know the website is php, but based on the history of serious exploitable bugs (almost all of which required crashing a game server before state was saved to the database for local rollbacks of failed crafting attempts, off hand I can only think of one other that didn't and it was a [devops failure](http://www.pathofexile.com/forum/view-thread/861418/)) I'm skeptical that they're using php for any critical infrastructure.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, weighing all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
-
[Grinding Gear Games](https://www.pathofexile.com/forum/view-thread/2253250/page/1#p15963161):
At 6:53pm on November 26 (New Zealand time), we deployed this 3.4.5c patch and it contained a significant bug. Due to a typo in a database key comparison function, if two users returned to the character selection screen at exactly the same time, there was a chance that one person was logged into the other's account. This was not apparent to us during testing because it requires a lot of players to be online before it occurs. Upon it being reported, we took the realm down at 8:26pm and restored the old version that did not have this problem. 417 players had their accounts accessed by another user. Of those, only 150 actually tried to log into a character. Most of these either logged out within a few seconds or were kicked off as the owner logged in again immediately.
I'll admit I'm really curious what their code looked like to make that possible in the first place. Their backend is primarily Linux/C++. but I'm not sure if that is just the main game server code, or also webapi's too. I know the website is php, but based on the history of serious exploitable bugs (almost all of which required crashing a game server before state was saved to the database for local rollbacks of failed crafting attempts, off hand I can only think of one other that didn't and it was a [devops failure](http://www.pathofexile.com/forum/view-thread/861418/)) I'm skeptical that they're using php for any critical infrastructure.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, weighing all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
Dan Neely wrote:
Due to a typo in a database key comparison function
Something like:
if (logonToken != currentLogonToken)
perhaps?
cheers Chris Maunder
-
Dan Neely wrote:
Due to a typo in a database key comparison function
Something like:
if (logonToken != currentLogonToken)
perhaps?
cheers Chris Maunder
An `==` where a `!=` was expected or vice versa is almost a given. What I'm not getting is what the slightly bigger picture looked like. They said it only happened when 2 people logged in simultaneously. But since each login attempt should be independent of any others that suggests that somewhere they're at least temporarily storing a value used in the login process as a global/static/etc variable, without that person A and person B's login threads/etc should have zero shared state and be unable to affect each other. I can't see any legitimate reason to do that, which means that if true the bit of shared state would be a much bigger issue than a bad comparison.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, weighing all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
-
An `==` where a `!=` was expected or vice versa is almost a given. What I'm not getting is what the slightly bigger picture looked like. They said it only happened when 2 people logged in simultaneously. But since each login attempt should be independent of any others that suggests that somewhere they're at least temporarily storing a value used in the login process as a global/static/etc variable, without that person A and person B's login threads/etc should have zero shared state and be unable to affect each other. I can't see any legitimate reason to do that, which means that if true the bit of shared state would be a much bigger issue than a bad comparison.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, weighing all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
Another option is they are loading state (eg session state) based on current user values and getting back the wrong state.
cheers Chris Maunder
-
[Grinding Gear Games](https://www.pathofexile.com/forum/view-thread/2253250/page/1#p15963161):
At 6:53pm on November 26 (New Zealand time), we deployed this 3.4.5c patch and it contained a significant bug. Due to a typo in a database key comparison function, if two users returned to the character selection screen at exactly the same time, there was a chance that one person was logged into the other's account. This was not apparent to us during testing because it requires a lot of players to be online before it occurs. Upon it being reported, we took the realm down at 8:26pm and restored the old version that did not have this problem. 417 players had their accounts accessed by another user. Of those, only 150 actually tried to log into a character. Most of these either logged out within a few seconds or were kicked off as the owner logged in again immediately.
I'll admit I'm really curious what their code looked like to make that possible in the first place. Their backend is primarily Linux/C++. but I'm not sure if that is just the main game server code, or also webapi's too. I know the website is php, but based on the history of serious exploitable bugs (almost all of which required crashing a game server before state was saved to the database for local rollbacks of failed crafting attempts, off hand I can only think of one other that didn't and it was a [devops failure](http://www.pathofexile.com/forum/view-thread/861418/)) I'm skeptical that they're using php for any critical infrastructure.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, weighing all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
Dan Neely wrote:
I'm really curious what their code looked like to make that possible in the first place.
I would guess that Time is relative my dear Watson.
int seed = time(NULL);
srand(seed);
key = generate_session_key();
// With this crappy code each server-side session key is identical for each login for 1 second.
// On the NT kernel the average time slice/quanta per context switch is 15ms so there will always be at least a 15ms window even if seeding using a clock with millisecond accuracy.//Never ever seed your generator with only time.
Just a guess. Although the phrase 'at the same time' implies the bug was time dependent. Best Wishes, -David Delaune
-
Dan Neely wrote:
I'm really curious what their code looked like to make that possible in the first place.
I would guess that Time is relative my dear Watson.
int seed = time(NULL);
srand(seed);
key = generate_session_key();
// With this crappy code each server-side session key is identical for each login for 1 second.
// On the NT kernel the average time slice/quanta per context switch is 15ms so there will always be at least a 15ms window even if seeding using a clock with millisecond accuracy.//Never ever seed your generator with only time.
Just a guess. Although the phrase 'at the same time' implies the bug was time dependent. Best Wishes, -David Delaune
I think that is a really good guess at what might've happened. But, a production-ready web server wouldn't do that, right? You're saying that devs are generating some session and have written buggy code, right? It's interesting that devs do often write "genius" code for things they don't understand completely and for which there are already fully-tested solutions.
-
I think that is a really good guess at what might've happened. But, a production-ready web server wouldn't do that, right? You're saying that devs are generating some session and have written buggy code, right? It's interesting that devs do often write "genius" code for things they don't understand completely and for which there are already fully-tested solutions.
you mean like javascript "rooms" ;P ?
#SupportHeForShe Government can give you nothing but what it takes from somebody else. A government big enough to give you everything you want is big enough to take everything you've got, including your freedom.-Ezra Taft Benson You must accept 1 of 2 basic premises: Either we are alone in the universe or we are not alone. Either way, the implications are staggering!-Wernher von Braun
-
you mean like javascript "rooms" ;P ?
#SupportHeForShe Government can give you nothing but what it takes from somebody else. A government big enough to give you everything you want is big enough to take everything you've got, including your freedom.-Ezra Taft Benson You must accept 1 of 2 basic premises: Either we are alone in the universe or we are not alone. Either way, the implications are staggering!-Wernher von Braun
