All Worlds Maintenance (Dec. 28)

[Devek]

I wonder if SE actually expects us to believe that they need a 2 hour update to fix the exceeding the maximum limit of bard songs error...honestly

Bears pointing out at this point that anytime the fix on their end requires them to reboot their servers, this is a ~2hr process, the fix itself, once found, probably took about 30 seconds, followed by a reboot which took two hours.

Thats so incredibly insane though.. I used to run a mud server and it was always kind of fun when users were like "It would be cool if there was a command to do X." and 5 seconds later I could reply "There is one!".

Beyond that fact that simple logic changes like this shouldn't require a reboot at all, the fact it takes them so long to do it just reeks of ineptness. It doesn't even make smart business sense for the process to be that way.

Imagine how many employees and how many hours are spent to plan, control, and test an update system as complicated and fucked up as to take a whole 2 hours each time?

The testing servers are usually identical to the production servers so imagine how painful it must be for the developers to make a change than have to wait 2 hours to try it out on the test server? Ouch, talk about a lot of dollars and time wasted there.. Even worse, in a 8 hour workday at the most you would only get 5 attempts at the most to fix a problem that is eluding you... a complicated problem could take weeks to fix instead of an hour.

It wouldn't be unreasonable to say that SE has lost a million dollars in time spent rebooting the servers over and over..

It would be cheaper to just hire a comp sci major from the states to go in and move all the logic to a shared library that can be reloaded at will or into a scripting language like Cython, or even simply to automate and speed up the reboot and testing process.

[MisterBob]

I wonder if SE actually expects us to believe that they need a 2 hour update to fix the exceeding the maximum limit of bard songs error...honestly

Bears pointing out at this point that anytime the fix on their end requires them to reboot their servers, this is a ~2hr process, the fix itself, once found, probably took about 30 seconds, followed by a reboot which took two hours.

Thats so incredibly insane though.. I used to run a mud server and it was always kind of fun when users were like "It would be cool if there was a command to do X." and 5 seconds later I could reply "There is one!".

Beyond that fact that simple logic changes like this shouldn't require a reboot at all, the fact it takes them so long to do it just reeks of ineptness.
It doesn't even make smart business sense for the process to be that way.

Imagine how many employees and how many hours are spent to plan, control, and test an update system as complicated and fucked up as to take a whole 2 hours each time?

The testing servers are usually identical to the production servers so imagine how painful it must be for the developers to make a change than have to wait 2 hours to try it out on the test server? Ouch, talk about a lot of dollars and time wasted there.. Even worse, in a 8 hour workday at the most you would only get 5 attempts at the most to fix a problem that is eluding you... a complicated problem could take weeks to fix instead of an hour.

It wouldn't be unreasonable to say that SE has lost a million dollars in time spent reboot the servers over and over..

It would be cheaper to just hire a comp sci major from the states to go in and move all the logic to a shared library that can be reloaded at will or into a scripting language like Cython, or even simply to automate and speed up the reboot and testing process.

There's a lot more to it than that. Testing servers are usually incredibly similar to the main environment, yes, but they are usually configured specifically to allow for quick reboots. I highly doubt SE has to spend 2 hours to reboot their test servers for every change they want to test, considering they are much smaller, and theres probably only a few of them.

In reality, over the course of 2 hours, probably 20 minutes are spent installing the patch, and another 20 are spent rebooting. However, even when you move patches into the production environment, you need to do extensive testing, unless you're retarded. However, SE's testing teams are pretty fail, or so it would seem, and they take forever to do a shitty job.

Also, as anyone should know, when you schedule a maint, you give a pessimistic timeframe. People get more pissed when you fail to meet a time than they do if you overestimate a bit. However, something always seems to go wrong at SE and they don't even make their pessimistic time assessments :(

[Devek]

There's a lot more to it than that. Testing servers are usually incredibly similar to the main environment, yes, but they are usually configured specifically to allow for quick reboots. I highly doubt SE has to spend 2 hours to reboot their test servers for every change they want to test, considering they are much smaller, and theres probably only a few of them.

In reality, over the course of 2 hours, probably 20 minutes are spent installing the patch, and another 20 are spent rebooting. However, even when you move patches into the production environment, you need to do extensive testing, unless you're retarded. However, SE's testing teams are pretty fail, or so it would seem, and they take forever to do a shitty job.

Also, as anyone should know, when you schedule a maint, you give a pessimistic timeframe. People get more pissed when you fail to meet a time than they do if you overestimate a bit. However, something always seems to go wrong at SE and they don't even make their pessimistic time assessments

If it takes 20 minutes to install the patch you have already lost.. in a clustered environment with lots of files its a matter of changing a symbolic link. This is good because if something goes wrong its always easy to jump back to the old version..

If it takes 20 min to restart you have already lost because it doesn't even take windows Vista that long.

If the test servers are identical to the production servers what is there to test exactly besides to check that everything came up already? SE should be able to run a test at any moment that tells them that instantly.

My network is infinitely more complicated than SEs because its spread out over many physical locations and I can tell you with 100% certainty that everything is up and running by just looking at my cell phone.

There are some things you can't even test unless its on production, and being able to say "ooops" and fix it instantly makes those problems less worrisome too.

[zymurgy]

The point wasn't the fact that it took 2 hours. The point was that they'd do an emergency maintenance to fix a "bug" that didn't effect anyone in any way since it required so much effect to actually do that nobody would actually seriously use it. Surely they had another reason to schedule an emergency maintenance, because they could have just added this fix to any client update patch. It certainly doesn't warrant a maintenance on its own.

[Skjie]

I remember running a RO server, and it took a long time to get everything back up and running when we made changes. The servers have to repopulate the entire world, and start up hundreds and hundreds of timers. Two hours isn't unreasonable at all. Remember, starting up your home PC is completely and utterly different than starting up a server meant to hold a MMO.

[Fhqwghads]

Going to have to agree. There must have been a ninja "fix" somewhere. Fixing something so benign sounds highly suspicious. It just doesn't make sense to bring down a gigantic network because a few extra BRD songs can be sung, especially since that bug has been around forever. They've had ample opportunities to fix this inside an update where a real "emergency" maintenance was necessary. This was hardly an emergency imo.

[Killgannon]

WoW's doing live maintenace now.... why cant FFXI<.<?? ps2limitations or some such? lol ;;

[Ushallbeasgods]

Ok, the next time you post in a thread looking for information, I'll make sure to make some joke that stopped being funny 2 years ago and see how much you enjoy it.

Hi, the servers are in Japan, why does it surprise you that server resets when they're working? Still, there have been PLENTY of maits done in NA hours. The notion that they purposefully do it to reset HNM timers at a specfic time is pretty stupid. And how is it convient for JP players to have their playing time interrupted? It's not, they have to reset servers SOMETIME and they just pick a time. Live with it, the "omg damn you SE for giving JP back fafnir" jokes are stale, old, and not funny anymore.

If you don't like what your reading its called not reading it and moving on with your life.

[inafume]

WoW's doing live maintenace now.... why cant FFXI<.<?? ps2limitations or some such? lol ;;

WOW's doing database reindexing live now. Or whatever they do to clear up their servers from going haywire after running for too long.

Little different than actual server changes.

Also: complaining about maintenance putting HNMs back in JP time is lol because JP shells don't camp so much anymore.

[Sidanoir]

WoW's doing live maintenace now.... why cant FFXI<.<?? ps2limitations or some such? lol ;;

WOW's doing database reindexing live now. Or whatever they do to clear up their servers from going haywire after running for too long.

Little different than actual server changes.

Also: complaining about maintenance putting HNMs back in JP time is lol because JP shells don't camp so much anymore.

They camp no matter when Fafnir is around on my server, be it 4 am, 1 am, 8 am during JP time...or a JP holy day even

[Janice]

omg damn you SE for giving JP back fafnir.

QFT

[Stilgar]

When I lived in japan, maintenance was always at 2 a.m. Now I live on the west coast of the US, and maintenance is lately always at 2 a.m. I guess I'm just SE's favorite player.

[Darkblood]

I wonder if SE actually expects us to believe that they need a 2 hour update to fix the exceeding the maximum limit of bard songs error...honestly

Bears pointing out at this point that anytime the fix on their end requires them to reboot their servers, this is a ~2hr process, the fix itself, once found, probably took about 30 seconds, followed by a reboot which took two hours.

Thats so incredibly insane though.. I used to run a mud server and it was always kind of fun when users were like "It would be cool if there was a command to do X." and 5 seconds later I could reply "There is one!".

Beyond that fact that simple logic changes like this shouldn't require a reboot at all, the fact it takes them so long to do it just reeks of ineptness.
It doesn't even make smart business sense for the process to be that way.

Imagine how many employees and how many hours are spent to plan, control, and test an update system as complicated and fucked up as to take a whole 2 hours each time?

The testing servers are usually identical to the production servers so imagine how painful it must be for the developers to make a change than have to wait 2 hours to try it out on the test server? Ouch, talk about a lot of dollars and time wasted there.. Even worse, in a 8 hour workday at the most you would only get 5 attempts at the most to fix a problem that is eluding you... a complicated problem could take weeks to fix instead of an hour.

It wouldn't be unreasonable to say that SE has lost a million dollars in time spent reboot the servers over and over..

It would be cheaper to just hire a comp sci major from the states to go in and move all the logic to a shared library that can be reloaded at will or into a scripting language like Cython, or even simply to automate and speed up the reboot and testing process.

There's a lot more to it than that. Testing servers are usually incredibly similar to the main environment, yes, but they are usually configured specifically to allow for quick reboots. I highly doubt SE has to spend 2 hours to reboot their test servers for every change they want to test, considering they are much smaller, and theres probably only a few of them.

In reality, over the course of 2 hours, probably 20 minutes are spent installing the patch, and another 20 are spent rebooting. However, even when you move patches into the production environment, you need to do extensive testing, unless you're retarded. However, SE's testing teams are pretty fail, or so it would seem, and they take forever to do a shitty job.

Also, as anyone should know, when you schedule a maint, you give a pessimistic timeframe. People get more pissed when you fail to meet a time than they do if you overestimate a bit. However, something always seems to go wrong at SE and they don't even make their pessimistic time assessments

Bob is alive?!

[Delekii]

I dont know if its just me, but Im BARELY able to connect to FFXI since this update.. was perfectly fine up to the point of the maintenance, and since have been getting all sorts of 0011, 0067(?) and other random error messages.

Anyone else having issues?


Edit: Just been speaking with a friend of mine who is having the same issues.. perhaps it is an Aussie problem. We are on different ISPs

[Delekii]

Tracing route to 202.67.56.123 over a maximum of 30 hops

fine until:

11 230 ms 232 ms 230 ms 203.208.168.121
12 * 317 ms * 203.208.154.94
13 * * * Request timed out.
14 * 427 ms * 203.208.154.33
15 410 ms 451 ms * 203.208.172.22
16 432 ms * 435 ms 203.208.154.22
17 637 ms 642 ms 632 ms 203.208.154.133
18 * 670 ms 669 ms 203.208.154.38
19 532 ms 534 ms * ge-5-1-0.r00.newthk01.hk.bb.gin.ntt.net [129.250
.9.1]
20 * 539 ms 542 ms ge-0-1-0.r01.newthk01.hk.bb.gin.ntt.net [129.250
.4.19]
21 540 ms * * p16-0-2-3.r21.tokyjp01.jp.bb.gin.ntt.net [129.25
0.2.233]
22 535 ms * * xe-1-0-0.a20.tokyjp01.jp.ra.gin.ntt.net [61.213.
162.234]
23 * 463 ms * ge-2-2-5.a20.tokyjp01.jp.ra.gin.ntt.net [61.120.
145.110]
24 * 447 ms 448 ms 202.67.63.45
25 446 ms 448 ms 446 ms 202.67.63.122
26 451 ms * * 202.67.56.123
27 * 447 ms * 202.67.56.123
28 447 ms 456 ms 465 ms 202.67.56.123

Trace complete.

Who broke the internet

Im building quite the collection of error messages now... 0008 0006 0011 0067 1431 1461 1341 1432.....


edit:

Browsing to Asia websites
Friday 29 December 2006, 10:55 AM
Browsing problems to China, Japan, Korea, Southeast Asia and other areas is a result of the undersea fibre-optic cables that were damaged by a powerful earthquake off the southern tip of Taiwan, Our upstream providers are working to resolve this problem.
TPG apologises for any inconvenience caused.
Outage Started: Thursday 28 December 2006, 9:07 AM AEDST
Outage Window: It is estimated to last 5 days