Hey, that’s a week. Neat. Thoughts and questions for people who’ve found their way here:
Anything in particular you want to see? I have pending requests for another post about datacenters, something on customer service, and a piece on planning for usage spikes. If there’s anything in particular you want me to talk about, let me know.
For that matter, if there’s a general category of stuff which is more interesting, let me know that, too.
I fiddled around with the look of the blog a bit over the course of the week. Comment links are now at the bottom of each post instead of the top. I don’t imagine anyone really cares, but if you want those links at the top as well as the bottom, I could do that.
There is a Livejournal feed, which I should put in the sidebar. There is also now a Livejournal feed containing just excerpts, since the fairly large posts do chew up a bunch of room: imgnry_cgs_shrt. Not the most memorable name in the world but there’s a length limit on Livejournal syndication names.
I’m away on business Monday, so see you again probably on Tuesday. Thanks for coming by.
I read this almost exclusively on my SK; I don’t care where comments go as long as there’s a border between the posts so I can see what it attaches to. (Well, and as long as that border doesn’t break the SK’s ability to handle the CSS, as seems to be the case far too often.)
As for topics, I’m really interested to know what patch deployment looks like from your perspective.
Coincidentally, I just followed a del.icio.us link about startups and smacked into a great discussion of MMO patch deployment. I find the idea of continuous deployment to be horrendous, for reasons which I will post about soon, but everything else in that post and the followup is on target.
actually I was hoping you could talk a little bit about getting into the gaming industry. You could start with how you started and then how you progressed.
I’m actually trying to get into the business side of game operations right now, though obviously its difficult becuase everyone is cutting back. But tips from your experience would help. But so far I like this blog a lot 🙂 keep up the good work!
Have any games you’ve worked on (if you can say) offered an SNMP or similar interface so pull data from in-game? I thought about this when reading your post about graphs; provide numbers for Cacti et al. Things like number of players online, blocking DB transactions, client lag times, chat rooms, who knows…data from inside the game, not from the hardware/OS.
Similar, yes. Whether or not you’re using SNMP, I think it’s essential to be able to get that sort of data in a programmatic fashion for use with Cacti and whatever other monitoring system’s one’s using.
Perfmon is also a popular method for exposing this data, which is not surprising since it’s a core Windows technology.
“Anything in particular you want to see?”
Well, I’d love to see something about best-practice in cabling-up servers in racked in cabinets.
For instance, should switches be in the same cabinet as the servers? If not, how best to lay the cables from the cabinet with servers to the cabinet with switches?
I work for a charity, and as tidy as the network guys tried to be, the cables out of the back of the four cabinets we have is a sight to behold. I tried helping them to tidy the clutter, but between the kvm, power, and network cables, it just doesn’t seem possible.
Josh: the last project I worked on – Dungeon Runners – had a simple telnet interface that spewed XML statistics. I can’t remember now exactly what Ops used to display the graphs, but at least at one point it was Cacti. Other games did similar things, but I don’t know the exact details because I didn’t use their interfaces or work on them. 🙂
At the time, I was talking with folks in Operations about switching over to HTTP because the telnet interface required a custom ten-line Perl script to grab the data, but grabbing XML full of name/value pairs over HTTP was built-in to some part of their system. For my current project, the plan is to have a full HTTP server implementation embedded, and to expose quite a bit more data and functionality through it: statistics for Ops, core dumps for devs, server administration controls, etc.
I would love to hear about “the Anatomy of a Shard”. I would guess that smaller games could segment shards with processes, but games such as WoW or EVE must span multiple physical servers per shard, I really find the network infrastructure to be a point of interest and how network protocols can be of help of hindrance to server performance.
Now, I just started reading the blog from the bottom up and I hope you haven’t gotten to any of the above points yet.