Forum Outages

[Renojustin]

I've just noticed a heck of a lot of server downtime lately, for the last several days in a row. How long is this going to continue and can we expect uber awesomeness on the forums after whatever maintenence or upgrading is finished?

[NTJedi]

I hate the outage message being displayed... reporting a "SHORT" downtime which is a downright lie when it's down for almost all of July4th.

The message should provide an estimated timeline such as:
down for about 1 hour (shortest)
down for 4 hours (moderate)
down for 8 hours (long)
down for 15 hours (longest)

[Gandalf Parker]

LOL we want that crystal ball on some of the Dom3 units so we can predict good or bad events and get an appropriate message the very next turn. http://forum.shrapnelgames.com/images/smilies/happy.gif

I think that "short" in this case was the technician version.

B: "When will it be back?"
T: "In a short time."

B: "Its been an hour. When will it be back?"
T: "In a short time."
B: "Thats what you said an hour ago."
T: "And an hour ago I meant it, but it didnt go the way we planned."
B: "Do you know what went wrong?"
T: "Not yet."
B: "When will you know?"
T: "In a short time."
B: "And when will it be back up?"
T: "A short time after I figure out what went wrong, unless of course it doesnt."

Its not that they are lying to us. They really dont know. Any upgrade is going to only take a short time, unless of course it doesnt. And in that case, it will always be a short time longer. Asking for a more reasonable time estimate would just be wasting more of the techs time. http://forum.shrapnelgames.com/images/smilies/happy.gif

Gandalf Parker
-- System Administrator of Internet Systems for as long as Internet has been Internet.

[Gandalf Parker]

Actually if what is going on, is what I think is going on, then I feel vindicated. http://forum.shrapnelgames.com/images/smilies/laugh.gif

[Ironhawk]

Your colorful story aside, Gandalf, it remains unchanged that they were in fact being misleading in thier message to us the users. Of course I very much appreciate this service which Shrapnel offers us for free (moronic ads aside for the moment), but they could simply have said "We are taking the forums down for much needed maintainence. Please bear with us as we work through this upgrade process."

[Gandalf Parker]

It is usually better not to give any time estimate.
Im not sure if the message we saw was Shrapnels or part of the UBB software. I should know, but I dont remember the exact message in the software.

Knock on virtual wood, but personally Im pleased that it comes back after each incident with the entire forum database and user database intact. Ive been asked to update forums like this, or migrate them, and Ive usually chickened out.

[MaxWilson]

I agree, no time estimate is probably the best. On the other hand, a day is a short time to non-forum-junkies. I suppose they could have gone the other way and said, "The forums will be down for a non-zero but finite duration."

-Max

[Lingchih]

This is normal tech-speak for "we don't know when we will be done with upgrade". I commend them for doing it over the fourth of july holiday, when most of us were doing the vacation thing.

[NTJedi]

I'm just saying the forums out of service message needs to be more accurate in providing a ballpark time estimate. In my opinion it's rude and unprofessional to tell your customers to "check back soon" and remain unavailable for most of the day!

ShrapnelGames has been in business for awhile and thus shouldn't behave as if this is their first year running the website.

[Ballbarian]

I commend them for doing it over the fourth of july holiday, when most of us were doing the vacation thing.




I second that.
My hat is off to the Shrapnel gang for spending what should have been vacation and family time tuning the forums (instead of bumming around like I was). http://forum.shrapnelgames.com/images/smilies/happy.gif

[Omnirizon]

"dammit Jim, I'm a systems administrator, not a time-efficiency/public relations expert!"

[Coldshard]

I demand cake!

http://forum.shrapnelgames.com/images/smilies/wink.gif

[Gandalf Parker]

Omnirizon said:
"dammit Jim, I'm a systems administrator, not a time-efficiency/public relations expert!"


Heehee. That sounds way too familiar. A board room with the CEO of the company who just wants an answer, the Customer Support people who just want something to tell the customers, and the System Techs saying "your guess is as good as ours at this point". Im trying to mediate the whole thing. Once I pulled out some DnD dice and rolled them across the table then declared it would take 10 hours.

[HoneyBadger]

Gandalf is 110% correct, and that so called "colorful story" is precisely what I go through every day in my line of work, except that everything should be repeated atleast 3 times until the customer wakes up, plugs in their brain, turns up their phone and/or hearing aide, and rediscovers their knowledge of basic English.

And it's not only better not to give them a time estimate, it's actually much better to conceal all guesses, thoughts, and suppositions, because if you give the customer anything on which to base a thought-process, except cold hard facts (and those hold their own danger), they will inevitably come to the most deceitful, vile conclusion possible.

It's war, basically. An American war in which the enemy usually ends up better off than they started, just as soon as they stop being stubborn and start losing, getting out of our way, and letting us do our job.

[NTJedi]

I know the amount of time to fix the forums can be totally unknown, but then this should be expressed on the message to its customers. Saying "SHORT TIME" and "CHECK BACK SOON" is downright lying.

IF the amount of time to fix the forums is unknown then tell the customers "UKNOWN AMOUNT OF TIME".

How would you like dropping your vehicle at the mechanic and he told you "Short amount of Time" and "Check back soon"... yet it wasn't fixed until the next day!
Naturally a vehicle is more important, but understand my point how ShrapnelGames should be HONEST.

[MaxWilson]

The thing is, NTJedi, I don't think it was dishonest. Perhaps ShrapnelGames planned for the outage to be even shorter than it was, but it wasn't very long.

-Max

[HoneyBadger]

NT Jedi, a day is an extremely short time, geologically speaking.

[Endoperez]

I was away from these forums for a short time. Five days, to be exact (3rd through 7th). I didn't notice any problems, so I'd have to say their estimate was correct.

[Gandalf Parker]

Being hopeful is not dishonest. You make it sound as if they knew that it would take all day. They were probably told that it would take very little time, and they certainly didnt foresee any possible problems or they wouldnt have tried it at all.

However the event has given us all a fine measuring stick for how addicted we are. http://forum.shrapnelgames.com/images/smilies/happy.gif

[NTJedi]

MaxWilson said:
The thing is, NTJedi, I don't think it was dishonest. Perhaps ShrapnelGames planned for the outage to be even shorter than it was, but it wasn't very long.

-Max



and


Gandalf Parker said:
Being hopeful is not dishonest. You make it sound as if they knew that it would take all day. They were probably told that it would take very little time, and they certainly didnt foresee any possible problems or they wouldnt have tried it at all.




Well I actually work with enormous enterprise customers and have experienced small tasks suddenly taking much much longer. From my experiences when a small task develops into a serious problem which requires rebooting, verifying permissions and running repairs it reaches a point where the time for fixing the issue is "Unknown".

At this point the message should be updated, the same as you'd expect a mechanic to call you if he reached a point where the time to fix your vehicle became "Unknown".

[MaxWilson]

I hope that means you will apologize for calling Shrapnel "dishonest." What you appear to have meant is that you thought the message was "unprofessional." I still may not agree (it WAS a pretty short time) but I can at least see where such a sentiment would be coming from.

-Max

[NTJedi]

MaxWilson said:
I hope that means you will apologize for calling Shrapnel "dishonest." What you appear to have meant is that you thought the message was "unprofessional." I still may not agree (it WAS a pretty short time) but I can at least see where such a sentiment would be coming from.

-Max



NO

If my mechanic kept telling me to "Check back soon" and he wasn't finished until 14 hours later I would also call him dishonest the same as ShrapnelGames.

[HoneyBadger]

What if your car was totaled though? And the mechanic was offering to rebuild it from the ground up, for free? Then how long would "soon" be? It's my understanding that the forum is either going to be soon, or is in the process of, being completely remade under new programming/code, from the ground up. That may be what's happening now. I got this from Mindi, by the way, so the information itself should be nice and accurate, even if my own understanding and ability to communicate said, is less than complete.

I've been online for longer than I'm willing to admit (my first internet experience happened at 2400 baud, so you do the math), and I don't stress about anything that takes less than 1 to 3 months to get done. That's "soon" to me. And as much as I post on here, I have lots and lots of other things in my life to occupy me.

[JimMorrison]

The "check back soon" part did bother me a bit (as a severe forumoholic), but geez, this is a bit of a severe reaction, you might want to put some lotion on it. http://forum.shrapnelgames.com/images/smilies/redface.gif

The worst part was, the second day, when it was the same relentless "check back soon" message, hope started to dwindle. And on sunday, when they did it AGAIN, I didn't even bother checking back soon at all. To me, "soon" in computer time has a sort of Attention Deficit imminence. Like, really, if you refresh the page right now it might work. No? How about now? I like unknown a lot. I can deal with unknown, it tells me not to concern myself with it, because I don't have enough information to understand, and no one who DOES have that info is going to take the time to explain it, rather than working on the issue.


BUT, on the third hand (yes I have three hands, AND??), I am extremely disappointed that after this 3 day goose chase of promised upgrades - there seems to be absolutely nothing changed whatsoever in the forums. However as Gandalf has pointed out, I am quite pleased and relieved to note that we seem to have all our data here still, so someone is doing a good job regardless of their erroneous and probably canned downtime message. <3

[Gandalf Parker]

Your mechanic would be unlikely to be spouting some default message.
For all you know (I wish I could remember from working with UBB) that message was not put there by Shrapnel. It might be like the default error messages that a browser gives you when it cant find something on a website, which is configurable but few ever bother to change it.

But it does bring a common complaint to mind from the system techs. "If I think Im about to fix it, why would I go to the user page and update the time frame? What would I change it to? Something like 'I know its been a couple of hours but I still think it will be fixed soon'?". I always hated leaving a job to update a wag time. Thats Customer Supports job to placate the whiners... umm... I mean users. The only time I ever updated a wag was to put "OK the easy quick update patch blew up, so now it will be 6 hours for us to recover the system back to what it was before we tried to upgrade it"

[MaxWilson]

JimMorrison said:
BUT, on the third hand (yes I have three hands, AND??), I am extremely disappointed that after this 3 day goose chase of promised upgrades - there seems to be absolutely nothing changed whatsoever in the forums.



"On the gripping hand," you mean. http://forum.shrapnelgames.com/images/smilies/happy.gif

I dunno, it seems like the forums might be a bit faster to refresh when I post. It's hard to tell. Kind of like it's hard to tell whether shields work against missiles the way the manual describes or at half-strength. http://forum.shrapnelgames.com/images/smilies/happy.gif

-Max

[Bwaha]

Come on guys, give them a break, if we were paying for a service then I would expect notification. Here's a sugestion though, make a batch file that sends a email to forum members telling them when the forum is going to be up or down. Thank you for this game and the forum that is simply wonderful. http://forum.shrapnelgames.com/images/smilies/biggrin.gif Bwaha

[Tifone]

I agree with everything said by Bwaha (what a nick) ^^

And I add that even if I like this forum, I'm not so addicted I can't live 3 or 4 days without it ^^

[Gandalf Parker]

Everything has its pros and cons. Mass mailing like that has many drawbacks. It pretty much eats a server, and it can get Shrapnels server on a blacklist.

[Ballbarian]

How would you like dropping your vehicle at the mechanic and he told you "Short amount of Time" and "Check back soon"... yet it wasn't fixed until the next day!



Most mechanics expect us to pay them for fixing our cars. If on the other hand my mechanic is a friend who happens to be letting me drive his car and it breaks down, I don't think I would feel right complaining about how long he said it would take to fix it before letting me drive it again.

And I must say that equating that message to "lying" and being "dishonest" is a real stretch. "Short amount of Time" and "Check back soon" involve generic measurements of time and mean different things to different folks.

[Bwaha]

Easy now folks, No reason to get hot under the collar. So far its a done deal and over with. Just let it go, take a deep breath, and relax... now that feels better now, heh. http://forum.shrapnelgames.com/images/smilies/biggrin.gif Bwaha

[Sombre]

I'm glad they've arsed about taking it up and down the last couple of days.

It gives us fair if slightly rude warning that we should start downloading and backing up attachments.

[NTJedi]

HoneyBadger said:
What if your car was totaled though? And the mechanic was offering to rebuild it from the ground up, for free? Then how long would "soon" be?



Find me one mechanic quoting to rebuild a car from ground up and advising it will take a "Short Time" and to "Check Back Soon". LOL LOL

That's like trying to find a doctor who would say quadruple bypass surgery is a minor operation. LOL LOL

I'll have what HoneyBadger is drinking.

[Gandalf Parker]

Sombre said:
I'm glad they've arsed about taking it up and down the last couple of days.

It gives us fair if slightly rude warning that we should start downloading and backing up attachments.


Excellent point.

[NTJedi]

Ballbarian said:
"Short amount of Time" and "Check back soon" involve generic measurements of time and mean different things to different folks.



Yes, they are generic measurements... and any company should make the greatest attempt to make its customers happy. This website has customers who visit daily and since ShrapnelGames knows they have daily customers a "Short amount of Time" is not 12 hours for these daily customers.

My point is a more accurate and honest message should be displayed during downtime.

[Gandalf Parker]

How can you possibly provide a more accurate assessment than "we think it will be soon" on any project they havent done before?

[sector24]

The ridiculosity of this thread makes me smile. Please continue.

[Itchykobu]

Ridiculous? Or is it......THOUGHT PROVOKING!?!?! ;-)

[Mindi]

Look, we're preparing for a major upgrade and system migration to new forum software that will also include a complete redesign of just about all of our sites (with the exclusion of the store). While we do not like outages anymore than you do, sometimes they are necessary and in this case we are having a little pain from time to time now to prevent a major outage later.

We determined that during the US holiday this past weekend would be a good time to do some of this work as a lot of people here would have other plans and not be online as much. While I wish we could give exact times, it's simply not possible when you are doing the kind of work we are talking about. For the most part, the outages just had a message that said "The forums are closed" and at one point we put up a generic message about maintenance.

What we have learned from the maintenance and testing we did this past weekend has helped us in reducing the amount of time the forum will have to be down when we do the site conversion. Some companies have taken days to weeks to do the type of conversion we are talking about doing and we are trying to get it down to a one or two day outage over a future weekend at most.

Please be patient as I think the results will be well worth it with many new features for the users with the new software. Everyone will be hearing more about this conversion soon and when it comes time to do the site migration we will be giving people notice as to what weekend it will be and an approximate timeframe. Please keep in mind though that things can change and no process is foolproof, so every timeframe will be an estimate and is not written in stone.

[Mindi]

Oh and thank you to whomever applauded that we took up time during our long holiday weekend to reduce impact on users. Richard and I worked on this pretty much for three days straight and well into the wee hours of the morning during those days. We're not looking for any pat on the back, but it's nice to know some people understand the complexities of what we are trying to do and the effort involved.

[Lingchih]

Mindi said:
Oh and thank you to whomever applauded that we took up time during our long holiday weekend to reduce impact on users. Richard and I worked on this pretty much for three days straight and well into the wee hours of the morning during those days. We're not looking for any pat on the back, but it's nice to know some people understand the complexities of what we are trying to do and the effort involved.



You're welcome http://forum.shrapnelgames.com/images/smilies/happy.gif

Btw, the forums came back up every night, pretty late, granted, but every night. I just did my postings in the wee hours of the morning.

[JimMorrison]

NTJedi said:...and any company should make the greatest attempt to make its customers happy. This website has customers who visit daily and ...



See, this is where you stop, and take that deep breath that Bwaha urged you towards. You see, this website does NOT have customers. This website, does NOT provide a paid service to anyone. The literature provided with your software, and the purchasing agreement does NOT entitle you to 24 hour services of a professionally run and operated website. While they do a great job of actually presenting one to you, it is in fact a courtesy that this website is provided - ultimately no matter how well meaning your griping is, the tone comes off only as ungrateful and demanding.



-- This thread really is more of a gross overreaction than anything Mindi, surely nothing that should have warranted you needing to provide such lengthy reassurances. http://forum.shrapnelgames.com/images/smilies/wink.gif

[Bwaha]

I remember a game that charged five dollars a turn. and more for special missions. If I recall right it costs about fifty bucks to do a turn. I don't want to return to this type of gaming. So in closing, stop complaining. Just have fun. http://forum.shrapnelgames.com/images/smilies/biggrin.gif Bwaha

[HoneyBadger]

Thank you for making my point for me, NTJedi, in your own confused and convolutey way. It's nice when I don't have to work to win an argument, even if the defeated party doesn't always grasp the implications of their own suicide.

And thank you Mindi, Richard, various moderators, and other behind-the-scenes personalities, for your gynormous amounts of hard work and dedication!

[Edi]

Thanks for taking the time to explain things, Mindi, and to Bwaha and JimMorrison and others for their contributions to it. It's a good thing I did not see this before or I'd have been liable to blow my stack bigtime.

A few outages of a few hours each over a holiday weekend are nothing, especially ahead of a major upgrade and considering the service is normally up 24/7/365 and provided free of charge.

At work I get customers every now and then who get out of line and start yelling and screaming and accusing me and my coworkers of dishonesty and other things and generally being abusive. What we do is we tell them we have absolutely no obligation to put up with behavior like that and to call back later when they can behave civilly and then we hang up on them. Some of the complaints in this thread fall strictly into this latter category.

[Lingchih]

Hold on Edi. Don't blow your stack. This has been a fairly civil discussion so far. I agree with you totally, but let's keep it civil.

[Edi]

Lingchih said:
Hold on Edi. Don't blow your stack. This has been a fairly civil discussion so far. I agree with you totally, but let's keep it civil.


Not going to do so now, since everything ahas been settled. As I said, if I'd seen it earlier, I probably would have. Now I've no problem with it. http://forum.shrapnelgames.com/images/smilies/happy.gif

[NTJedi]

Gandalf Parker said:
How can you possibly provide a more accurate assessment than "we think it will be soon" on any project they havent done before?



Please read my previous post #622020 - 07/08/08 12:05 PM

No need for me to repeat myself.


JimMorrison said:
You see, this website does NOT have customers. This website, does NOT provide a paid service to anyone. The literature provided with your software, and the purchasing agreement does NOT entitle you to 24 hour services of a professionally run and operated website. While they do a great job of actually presenting one to you, it is in fact a courtesy that this website is provided - ultimately no matter how well meaning your griping is, the tone comes off only as ungrateful and demanding.



Actually this website does have customers because we provide traffic to their website. The more we visit the website the more we are exposed to their advertising of products and the more likely we are to recommend them to friends. There are many categories of customers, not just those who purchase service A. Everyone visiting the forums are potential future customers.

I'm not demanding anything I'm providing advice to make the website more customer friendly during outages. Providing a more accurate timeline is an improvement compared to a generic measurement which can be misleading since some will intrepret "short time" as being 1 hour.


Again I'm providing advice so messages are not misleading and intrepreted as dishonest by some during forum outages.

[Edi]

NTJedi:
There is no way to say this nicely, so I won't even try to sugarcoat it. You are practically the only one who has his panties in a bunch over this and right now your posts are sounding like nothing more than a continuous whine. They come across as sanctimonious lecturing and touting your personal opinions as fact. Here's a hint: They are not.

As has been said previously, "short" is a relative term and the outages we have experienced here have been short given all the factors that are relevant. This website is not a time-critical application where entire businesses are hanging in the balance and if it's down a few hours a few times ahead of a major upgrade, that's short, end of story.

I get to deal with customers complaining about subscription services not working or having problems every day and there's a certain difference in the way paying customers and non-paying customers are handled. If a customer has purchased a service that has a 2-hour response time come Armageddon or Apocalypse, then he by damn gets that response, but he's also paying through the nose for it. If it's a normal DSL customer, time to fix the problem is 2-5 days with best effort delivered. If they don't like it, they can take their complaints and shove them, though I'm not allowed to say that to their face.

This forum here falls into the goddamn services-provided-for-free category, so the users really do not have any right to expect that their complaints about "short" downtime equating to 1-2 hours be accepted as a matter of course. Every reasonable effort to minimize aggravation to Shrapnel users has been made, so what's the problem?

[Tifone]

wow, mister edi says things in da face http://forum.shrapnelgames.com/images/smilies/laugh.gif

[lch]

Yadda yadda yah, you guys have way too much free time on hand to post on the forums that much. http://forum.shrapnelgames.com/images/smilies/happy.gif I have read the first page and only skim-read the rest because I think that everything that can be communicated is already in there.

I think that the criticism voiced by the users modulo form/notation should be just be acknowledged and accepted without getting emotional. Just write "forums should be back within XX hours" next time, because whatever you do with them, you'll probably stop working on them when you get to bed. If you give a very cautious, long ETA, there's no problem if you don't take as long as you estimated before. But if you say "in a short time" and then it actually takes a lot of time, then of course people can object to that. I was under the impression that the users didn't criticize the actual downtime at all, just the insecurity when they should come back and look again. If they do it every couple of minutes and are being stopped at barred doors repeatedly, they could get angry.

Another thing which will curb the criticism: Give a notice of the Server downtime in advance, if possible. And at best, work on a copy of the site on another machine at all times for all developmental/testing purposes and only apply the changes that you have done on the real, live site when you're done experimenting.

So in short: I can sympathize a bit with the disapproval that has been voiced, at least consider it and don't get worked up on posts of individuals who offend you.

Ahhh, I need more tea... http://forum.shrapnelgames.com/images/smilies/tongue.gif

[NTJedi]

Edi said:
You are practically the only one who has his panties in a bunch over this and right now your posts are sounding like nothing more than a continuous whine. They come across as sanctimonious lecturing and touting your personal opinions as fact. Here's a hint: They are not.



I'm only providing advice to ShrapnelGames so some of their customers are not mislead during future forum outages. I'm providing this advice because as a common visitor the current outage message can be intrepreted as misleading.


Edi said:
As has been said previously, "short" is a relative term and the outages we have experienced here have been short given all the factors that are relevant.



I'm only advising the term "short" and "check back soon" be updated/changed to provide more accurate timeframes for the customers. Otherwise given a possible future scenario why not use "short" and "check back soon" even if the server was to be down for 2 days; 2 weeks; 2 months?? Since 2 months is short based on the lifespan of the average human.



Edi said:
This website is not a time-critical application where entire businesses are hanging in the balance and if it's down a few hours a few times ahead of a major upgrade, that's short, end of story.



A few hours is short, unfortunately the forums were down for at least 12 hours.


Edi said:
I get to deal with customers complaining about subscription services not working or having problems every day and there's a certain difference in the way paying customers and non-paying customers are handled. If a customer has purchased a service that has a 2-hour response time come Armageddon or Apocalypse, then he by damn gets that response, but he's also paying through the nose for it. If it's a normal DSL customer, time to fix the problem is 2-5 days with best effort delivered. If they don't like it, they can take their complaints and shove them, though I'm not allowed to say that to their face.



I'm not demanding anything, I'm only advising ShrapnelGames provide a more accurate outage message so it's less misleading to daily visitors.



Edi said:
This forum here falls into the goddamn services-provided-for-free category, so the users really do not have any right to expect that their complaints about "short" downtime equating to 1-2 hours be accepted as a matter of course. Every reasonable effort to minimize aggravation to Shrapnel users has been made, so what's the problem?


I don't have any expectations for a change, I'm only advising the outage message is more accurate. As I posted earlier the term "short" and "check back soon" can be intrepreted as either 1 hour OR 2 weeks depending on who reads the message.

Whether or not ShrapnelGames chooses to provide more accurate outage messages is their decision, but as a customer it's my advice. I'm not expecting or demanding anything... just my advice.

[Bwaha]

Okay guys, Play nice now. no need for profanity. Man you guys are so serious about a game. Edi, i agree with your sentiments, but disagree with the way you express them. If I find someone who I can't communicate with, I call in a coworker to deal with them. This de-escalates the whole situation. Most of the time this works rather well. Other times, well there's one rotten grape in every bunch. The trick is not to lose your temper, I feel if someone makes me loose my cool, they won and I lost. So in closing, don't let them get to you. Take the higher road and enjoy the pleasure of being in the minority of the rational beings. http://forum.shrapnelgames.com/images/smilies/biggrin.gif Bwaha
ps. if you want to give me a paid job to deal with rotten grapes, I'm all for it. I can do it for cheap. http://forum.shrapnelgames.com/images/smilies/happy.gif Bwaha