Forums

Help › Forums

A smidge about servers, the internet and whatnot

Thu, Aug 13 2015 9:06 PM (40 replies)
  • PhiEaglesFan
    617 Posts
    Fri, Jul 31 2015 3:38 AM

    anonymous:
    The inability of the servers that WGT has chosen has never been able to carry the load .

    So, I found this gem recently and before I knew it I was about to hijack another thread in an attempt to help make some sense (because it's frustrating to me when people say very wrong things like this) when I caught myself.  So here I am instead with a new thread.

    Welcome aboard.  The exits are here, here and here.

    I warn you now.  This is long.  If reading comprehension is difficult for you, turn back now or abandon all hope.

    This is my attempt to describe, in layman's terms, what's actually going on with this whole internet and server thing in a clear and easy-to-understand manner.  You're welcome to sit back, enjoy the read and participate in this discussion.  You're also welcome to move along quietly if this doesn't interest you.  What you are not welcome to be is an idiot or an antagonist.  So, with all of that said...

    Generally speaking, of course...

    This game (like most websites and web apps) is/are spread across multiple application (or app) servers.  What's an application server, you ask?  Well, these are the servers on which the game and the website run.  App servers physically execute the code.  They are typically rented, in quantities, from big server farms like Akamai, but there are many other companies.

    There are different sets of application servers for the website and the gameclient.  This is why the website can be down and the game can still work.  That doesn't happen often, but it does happen.  I've witnessed it first hand, albeit a long time ago.

    Now, we're not even counting any potential additional servers they have setup for any backend uses such as databases, CMS tools, etc. There are always additional backend app servers.  What's the back-end, exactly?  That's all the stuff that goes on that you don't see.  For example, you don't see when WGT updates the database(s) with your scores.  You also don't see them process their nightly feeds.  You know how they have short downtimes occasionally, but you come back online and literally nothing is different?  They were updating their feed code or perhaps adding a new feature to the backend tool that helps them design and set-up tournaments.  Maybe they were laying a foundation for something coming in the next release. But continuing on....

    There are also any number of supporting content servers to backup the app servers.  These are, surprisingly enough, from where you get your game content.  The game photos, the assets, the clubs, the balls, the avatars, everything you download to your browser, everything that you see that makes this game (and website) all lies on these various content servers.  That picture of BP #18 green that you need for the next shot?  Content server say whaaat.

    There's always a content server within a few hundred miles of you, probably even closer  (unless your country does not internet, in which case, how is it that you are reading this?).  That is basic internet efficiency.  Don't go long when you can go short. Makes sense, right?

    So in reality, you're very rarely communicating with WGT's own content servers, except in two circumstances:

    • NO content server has what you need (for whatever reason, usually a rarity). 
    • You live in WGT's backyard and thus are actually closer to them than the closest content servers.

    There is also a possibility that WGT keeps their own content server's IP private and prevents any direct consumer access to it.   It is possible only the network of content servers is permitted to communicate with it. That would be much more safe and secure.

    So, instead of communicating with WGT directly you're more than likely communicating with any number of these mirror content servers that are strategically spread across the country and throughout the internet capable world.

    Here's a very simplistic, but realistic take on how a single "roundtrip" might work:

    Your computer makes a request to WGT.  Again, let's say you need that BP #18 green photo because it's not in your local cache.

    Out goes your computer's request into the big bad land of the internets.  Your request gets sent to your ISP who then more or less redirects the request to your local content server.  Content Server looks at your computer's request and says,

    "I see you need this green photo for BP #18.  Well you're in luck.... I've got that asset / file you need right here.  No need to talk to WGT's content servers all the way over in San Francisco for that.  Let me get it for you."

    And the friendly content server gives you what you need, because it can do it faster locally than WGT can from San Francisco.

    On their own, content servers will periodically update themselves from WGT's own content server or when WGT forcibly says, "hey all Content Servers, update this right now".  This would happen when they release a new course and they need to make the new course files available to us.

    Now that you are a content and application server guru, let's address our poor quoted fellow from above.

    Every time you load the game or website in your browser there's this thing called the load balancer (not a person, it's software).  The load balancer's primary job is to chaperone you to the least congested app server.  Timeout for a second on this point...

    ... do you long time guys remember back in the day, the discussions about psj-05, psj-02 and people bouncing around servers thinking one was better than the other.  Those were the application servers.  FYI, psj stands for San Jose. Alright, time-in.

    WGT likely didn't write their own load balancer.  There's already software for that, so we can assume it works as expected.

    To give some perspective, my company's servers run about about 15%-25% load at any given time and we're pretty large.  We're closer to 50%-60% load during the holiday season but that is still way under red line.  We service a much, much larger volume of traffic per day than WGT could ever hope.  As per Alexa.com, about 1.3 million more unique views per day.  We have *8* application servers for the entire front-end website.  That's it.  8 servers handling 1.3 million people a day with hundreds, sometimes thousands of requests coming in to our app servers each minute.  All handled with relative ease. 

    Server rental isn't cheap, but it's chump change in terms of overall software development expense.  Let me remind you that WGT has a freakin' helicopter. Servers are no expense here. Undercutting your app server budget would be suicide and contrary to every company's goal.

    I'm just guessing on this, but I would imagine, speaking front-end only, that a single application server could probably handle WGT and it's 18,000 (wow that's smaller than I thought) unique visitors per day in its entirely.  Maybe, maybe not, either way it's moot because having only one app server would be quite foolish.

    We know that they have more than one application server.  The browser's developer console will tell us that (unless you're using IE7/IE8 in which case I pray for your soul).

    It's not their servers, it's not their game.  None of it.   No matter how many unqualified people attempt to tell you that. 

    There may be congestion between the many OTHER non-WGT servers your data packet or request has to traverse before it even gets to one of WGT's servers.  Someone may be dDos'ing the hell out of one of the major hubs that your data just happens to need to pass through.  That'll choke you pretty good.  However, it'll look like it's WGT's fault.  In reality it's actually AT&T's fault or some hacker in another part of the world who has a vendetta against the internet and the people who use it.

    The point is it could be anything.  It's the internet.  It's wild.  Millions of non-moving parts, except for some old hard drives maybe.  Data flying everywhere at light speed.  Numerous mitigating factors to consider. The trucks are bursting through the tubes at light speed people!  

    Now sure, one of WGT's application servers might hiccup and need a reboot now and then, but that's few and far between and is no cause for alarm.  You always put yourself in a position where, in an emergency, you can shutdown an application server or two (or half in some cases) and force the entire load onto the rest of your app servers.  And you're typically still handling this with ease.

    Companies that demand 24/7 up-time will shut down a couple of application servers at a time, deploy code, restart those servers, then shutdown two more and continue that process until all servers are updated.  This is the way you have to do it if you demand 100% up-time.  After that you bust the application server caches and out goes the new code.

    None of this impacts your computer's ability (or inability) to display local flash graphics in a consistently smooth manner.  You'll need to ask your GPU, your CPU, Flash itself and your other installed software about that.  If you don't have a GPU, then we've probably answered your question about why your meter sucks. Forget all the rest of it.

    If you do have a GPU (an actual card that you installed and can uninstall, that isn't 15 years old... and isn't Intel Accelerated Graphics, which is usually a joke) then you better have the flash hardware acceleration setting turned on, otherwise you're not even using your GPU to render WGT.  And I think we can all agree that's just a sad situation.  Please remember to RESTART your browser if you change your hardware acceleration setting in order for it to take effect.  This will not solve everyone's problem.

    If you do all that, and try it in full screen mode, and low flash quality and if it still sucks, it's probably flashes fault.  Perhaps your GPU is really not up to the task or, more likely some combination of both.

    I play this game on lowest flash settings using the full screen mode with hardware acceleration on. I pretty much can't play in non full-screen view.  The meter problems become more problematic then (which I've yet to really figure out why, perhaps not a native resolution.)

    I'm using a rig I built myself 6(?) years ago.  It has an AMD Phenom 9950 Quad-Core 2.6 GHz CPU in a fairly average mobo. The GPU is a Hawk Radeon 5750.  I run 32-bit on Win7.  I have multiple IDE hard drives.  No SATA hard drives.  My windows experience index is a paltry 5.3

    This isn't exactly top of the line stuff here, but my WGT experience is mostly decent overall and my meter is usually quite smooth.   I don't have to run external programs or manage a bunch of windows processes and background services to improve the meter.  However, if I do notice a problem, my processes and services are the first thing I'm checking. 

    Rule #1: Always assume it's you until you can definitively prove it's not.

    I use regular old Firefox (latest) when I play.  I frequently stream music from Sirius or YouTube and chat in IRC all while playing with no performance drop.  The meter consistency doesn't change when I turn the music stream on and off, because the meter is not affected by the internet connection.

    However, I'm not immune to the occasional meter blip either, but almost never does it actually cost me strokes (it's really not that bad or that often).  I also don't play more than 2 or 3 single-player games or more than 1 multiplayer game in a row.  The longer Flash is open, the slower and more bloated it becomes. 

    I know, I know,  your video card runs the newest COD at 100+ frames in elite settings (don't know if that's real, don't call me on it).  This and that are not same.  COD is Direct3D/OPENGL.  This is... wanh wanh wanh ...Adobe Flash; pushed to the max.

    Now I'm not saying this is exactly how WGT runs, because I don't know exactly how WGT runs, but I've been around the block enough to know the gist of how this process works for most software development teams and where the similarities lie.  I might be off on the exact numbers, but I am definitely not off on the concept.  Trust me folks.  It all just sounds a little too true to be false, doesn't it?

    ---

    Once again.  Thanks for listening and reading. I tried to write this in a semi-entertaining way, since it is basically just an information dump.  Maybe you learned a thing or two.  If you did, that's fantastic.  If you didn't, then you probably already knew this stuff anyway.  If you want to learn more, ask.  I don't portend to know it all, and will immediately tell you when I don't know.

    So, let's blame WGT for what's actually their fault, like our camera situation right now (angry fist).  Let's not send them up the creek for things for which they aren't responsible.  Let's be better people, better educated, better forum.

    I told you I was good at long posts.  Insomnia be damned.

    Take care all.  Hit 'em well.

  • YankeeJim
    25,827 Posts
    Fri, Jul 31 2015 6:19 AM

    PhiEaglesFan:
    I also don't play more than 2 or 3 single-player games or more than 1 multiplayer game in a row.  The longer Flash is open, the slower it becomes. 

    What gets that Flash speed back, refreshing the browser or closing out and reopening it? I've seen this degradation but have always written it off to me playing too long and getting lazy.

  • PhiEaglesFan
    617 Posts
    Fri, Jul 31 2015 6:36 AM

    Closing out and reopening.  You're releasing what's being held in memory when you do that.

  • PaulTon
    10,731 Posts
    Fri, Jul 31 2015 6:38 AM

                         A very well written, interesting, entertaining and informative post...

                                                     TOUCH DOWN EAGLES !!

                                                                 

  • PhiEaglesFan
    617 Posts
    Fri, Jul 31 2015 6:39 AM

    PaulTon:

     

                                   

    Very nice.

     

  • mkg335
    5,491 Posts
    Fri, Jul 31 2015 6:53 AM

  • mkg335
    5,491 Posts
    Fri, Jul 31 2015 6:56 AM

    Should be required reading...ty PEF.

  • mmillings
    600 Posts
    Fri, Jul 31 2015 7:36 AM

    I agree with Mark. Great post EaglesFan. Go Steelers!

  • Boomerboy44
    1,514 Posts
    Fri, Jul 31 2015 8:23 AM

    A better understanding of how things work is always appreciated.  Go Boy's

  • andyson
    6,415 Posts
    Fri, Jul 31 2015 10:33 AM

    PhiEaglesFan:
    Let me remind you that WGT has a freakin' helicopter.

    Are you absolutely sure of that?  Yeah, we've all seen the video of Chad in the helicopter while they take aerial photos of a course, but that happens only for a few days maybe a couple of times a year.  And your not going to fly the company copter cross country to Pinehurst or Merion.  With a range of around 400 miles and a cruising speed around 100mph a copter is not practical for those trips.  What other uses for a WGT company helicopter could there be?

    But I searched the FAA aircraft registry anyway.  Nothing showed up for WGT or World Golf Tour.

    PhiEaglesFan:
    We service a much, much larger volume of traffic per day than WGT could ever hope.  As per Alexa.com, about 1.3 million more unique views per day.  We have *8* application servers for the entire front-end website.  That's it.  8 servers handling 1.3 million people a day with hundreds, sometimes thousands of requests coming in to our app servers each minute.  All handled with relative ease. 

    # of views/day = # of people/day ? 1 view per person?  That can't be right PEF.

    All that aside, I do agree the servers have zero to do with a jerky meter.  Remember when you could unplug or disable your internet connection and still take a swing?  Then the game would sit and wait for a response from the game engine in the app server.  Since then they've changed the disconnect handling and the game now quickly detects when your connection goes down.

    The meter animation and click detection is all done locally on our computer.  Its WGT's code running on Flash within our chosen browser competing for CPU time with everything else running on our computer.  So I understand why people are told to shut other apps down.

    But my 6 month old 4th gen i7, 16G RAM, nvidea GPU with h/w accel on, and Skype, Excel, Notepad, and a graphics program all running idles in the game at 8 to 9 % CPU usage and I still get an occasional bad meter!

    So I disagree when you say its all in our system.  WGT still ignores Adobe's recommendations for Optimizing Flash Performance by running the meter animation at 60 frames per second when Adobe says "In general, use the lowest possible frame rate for better performance" and " Anything more than 30 frames per second is often unnecessary." 

    Why is 60fps bad?  I recommend reading the section named Runtime Code Execution Fundamentals in the above document but I will try and explain.

    At 60fps Flash must get all its work done and render the meter in 16.6 msec. And it must render the meter every 16.6 msec or it won't be smooth.

    Unfortunately Flash prioritizes its work so that rendering the screen is done last.  All event processing is done first. Events are mouse clicks, mouse movements, timers, Incoming messages from the internet, etc. 

    So if any work comes in while the meter is running, and that work takes more than 16.6msec to complete, the meter will stutter.

    If WGT reduced the frame rate to 30fps then Flash only needs to update the screen every 33 msec. thus giving more time to complete other work before skipping a beat on the meter.

    Its like you agreed this morning to have that report on your boss' desk at 3pm.  But some emergency things cropped up during the day you had to handle so the report doesn't get there til 5pm.  Is he happy?  Does he say "you need to clean off your desk and optimize your time management!"

    The other thing WGT could do is stop sending friend status updates to the game client while it's your turn to shoot.  That's work the game client has to do.  We're told to shut down all other apps for a smooth meter yet WGT finds it OK to send un-needed (IMO) work directly to the game client while we're shooting! SMH  But that's a whole 'nother thread and this post is already too long!

RSS