View previous topic :: View next topic |
Author |
Message |
Isaura
Census Taker
Joined: 08 Oct 2006
Posts: 40
Location: Netherlands
WR Updates: 1,334,593
Isaura WR Profile
|
Posted: Fri Nov 02, 2007 6:15 am Post subject: Uploads |
|
|
Is there something wrong with the uploading?
The data from yesterday and the day before yesterday aren't processed.
I update on the site, not through uniuploader or something
So I was wondering if something is wrong? |
|
Back to top |
|
 |
DM.
Census Taker

Joined: 03 Oct 2005
Posts: 1155
Location: Toronto, Canada
WR Updates: 841,833
DM. WR Profile
|
Posted: Fri Nov 02, 2007 7:24 am Post subject: |
|
|
Yup looks to be stopped,
http://www.warcraftrealms.com/dbstats.php
Last processed file was uploaded on 2007-10-31 10:34:27.
Current server time is 2007-11-02 08:23:45 _________________
Click my sig |
|
Back to top |
|
 |
Hybuir
Gear Dependent Squirrel

Joined: 06 Sep 2005
Posts: 1548
Location: Austin, TX
WR Updates: 2,883,877
Hybuir WR Profile
|
Posted: Fri Nov 02, 2007 9:08 am Post subject: |
|
|
/RollieSignal |
|
Back to top |
|
 |
Balgair
Araiceil
Joined: 30 Sep 2005
Posts: 1701
Location: UK
WR Updates: 12,856,335
Balgair WR Profile
|
Posted: Fri Nov 02, 2007 10:31 am Post subject: |
|
|
Yep noticed that myself, it's moving but very very slowly and stops for a few hours at a time. Plus even when files are processed my total updates count isn't going up - it's stayed the same for about 3 days now, and I have had a few files processed in that time  _________________ EU-Draenor:
-- Sagart - Tairbh - Balgair - Buinne - Eilnich - Ruire - Dubh - Laidir - Naomh - Falaich -- |
|
Back to top |
|
 |
Rollie
Site Admin

Joined: 28 Nov 2004
Posts: 5376
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
Posted: Fri Nov 02, 2007 10:36 am Post subject: |
|
|
Digging _________________ phpbb:phpinfo() |
|
Back to top |
|
 |
Rollie
Site Admin

Joined: 28 Nov 2004
Posts: 5376
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
Posted: Fri Nov 02, 2007 10:52 am Post subject: |
|
|
Well, looks like it's mainly just that the update process is slowing again. Moving to more powerful hardware helped out for nearly a year, but I knew it wouldn't be a permanent fix.
At first glance, I am faced with the same decision I had to make back in January.
1) I can either start pruning/archiving characters (cheap and not too hard)
2) I can move to even more powerful hardware (expensive and only a stop gap until we reach the next level)
3) I can move to server clusters that could constantly expand to meet demand (very expensive and basically out of the question)
At this point, I'm leaning towards #1. What I could potentially do is move all characters not seen in XX months into an archive table that is not looked at for updates which means smaller number of characters to check against when doing updates. I could also add maybe a once per month process that can check all latest added characters against the archive and if a match is found, it can 'un-archive' those characters. This adds some complexity rather than just archiving the data for good, but probably the best bet to be honest. _________________ phpbb:phpinfo() |
|
Back to top |
|
 |
Rollie
Site Admin

Joined: 28 Nov 2004
Posts: 5376
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
Posted: Fri Nov 02, 2007 11:03 am Post subject: |
|
|
Upon further investigation it appears that it is particular uploads from the same individual that is prematurely killing the process. Digging further.
One thing of note, if I looked to start doing archives, here's a few stats:
5M Chars not seen in 24 months
15M Chars not seen in 18 months
26M Chars not seen in 12 months
I think 12 months would be too soon, and 24 months wouldn't give enough benefit. Maybe 18 months is not a bad amount of time for a character to be dormant before being archived? _________________ phpbb:phpinfo() |
|
Back to top |
|
 |
Rollie
Site Admin

Joined: 28 Nov 2004
Posts: 5376
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
Posted: Fri Nov 02, 2007 11:08 am Post subject: |
|
|
Okay, it was WyriHaximus! Let's get him!!! Wyri, please PURGE or at least PRUNE your file. Also, can you check your UU settings as I am getting an upload from you every 30 minutes and surely the file shouldn't be changing that often? _________________ phpbb:phpinfo() |
|
Back to top |
|
 |
Balgair
Araiceil
Joined: 30 Sep 2005
Posts: 1701
Location: UK
WR Updates: 12,856,335
Balgair WR Profile
|
Posted: Fri Nov 02, 2007 11:32 am Post subject: |
|
|
Phew, was wondering if it was me at first 18 months sounds a fair compromise, especially if you can run a check now and again to make sure they haven't become active again (maybe not even as often as monthly, every 2-3 months should be enough). _________________ EU-Draenor:
-- Sagart - Tairbh - Balgair - Buinne - Eilnich - Ruire - Dubh - Laidir - Naomh - Falaich -- |
|
Back to top |
|
 |
DM.
Census Taker

Joined: 03 Oct 2005
Posts: 1155
Location: Toronto, Canada
WR Updates: 841,833
DM. WR Profile
|
Posted: Fri Nov 02, 2007 12:04 pm Post subject: |
|
|
LOL Wyri!!
And yea 18 months does seem reasonable. _________________
Click my sig |
|
Back to top |
|
 |
Hybuir
Gear Dependent Squirrel

Joined: 06 Sep 2005
Posts: 1548
Location: Austin, TX
WR Updates: 2,883,877
Hybuir WR Profile
|
Posted: Fri Nov 02, 2007 1:17 pm Post subject: |
|
|
Rollie wrote: | Okay, it was WyriHaximus! Let's get him!!! Wyri, please PURGE or at least PRUNE your file. Also, can you check your UU settings as I am getting an upload from you every 30 minutes and surely the file shouldn't be changing that often? | Wyri's secrets come out! *scrambles for UU*
Why don't we just do both prune & archive |
|
Back to top |
|
 |
DM.
Census Taker

Joined: 03 Oct 2005
Posts: 1155
Location: Toronto, Canada
WR Updates: 841,833
DM. WR Profile
|
Posted: Fri Nov 02, 2007 1:55 pm Post subject: |
|
|
Wyri's been getting a string of 0's all the time he submits..... _________________
Click my sig |
|
Back to top |
|
 |
Rollie
Site Admin

Joined: 28 Nov 2004
Posts: 5376
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
Posted: Fri Nov 02, 2007 4:18 pm Post subject: |
|
|
Because his files have been unparsable. _________________ phpbb:phpinfo() |
|
Back to top |
|
 |
WyriHaximus
Census Taker
Joined: 18 Oct 2005
Posts: 244
Location: Koedijk, Alkmaar, Noord-Holland, The Netherlands
WR Updates: 1,520,754
WyriHaximus WR Profile
|
Posted: Sat Nov 03, 2007 4:13 pm Post subject: |
|
|
Rollie wrote: | Okay, it was WyriHaximus! Let's get him!!! Wyri, please PURGE or at least PRUNE your file. Also, can you check your UU settings as I am getting an upload from you every 30 minutes and surely the file shouldn't be changing that often? | /me starts running for his life!
Anyway this is what happend: Quote: | Yeah it was a realy stupid error. Using a script to check and if changed upload the lua save file. After rollie poked me I trew away the lua file on my server and the check file with the md5 hash of the lua file. The system I'm using is kind simple. Got my WS and my server. My WS copies the lua files each minute to my server. My server checks every 15 minutes if the lua file is update by using a md5 hash. If changed the script updates the md5 check file and uploads the lua file to WCR. Somewhere along the way something went wrong (we still don't know what exactly) but by trewing away both files the problem got fixed . |
_________________

My World of Warcraft Screenshots / Map |
|
Back to top |
|
 |
xpolockx
Superior Census Taker

Joined: 10 Aug 2005
Posts: 788
Location: Lynchburg, VA
WR Updates: 4,489,012
xpolockx WR Profile
|
Posted: Sat Nov 03, 2007 7:21 pm Post subject: |
|
|
Heh Wyri, that's pretty funny Don't forget to turn on aspect of the cheetah while you run...
I think 18 months is a pretty good marker - most people that quit for longer than a year don't end up playing that same character when they come back (not everyone, obviously, but most of the people I've talked to who come back after being gone for a year or more roll new characters to get back into the game.) Of course, having a monthly check to make sure that archived characters haven't been seen recently is also a good idea. Anything to help out your poor servers  _________________ US-Whisperwind:
Kayni, Resto Shaman
Scenario, MW Monk |
|
Back to top |
|
 |
Balgair
Araiceil
Joined: 30 Sep 2005
Posts: 1701
Location: UK
WR Updates: 12,856,335
Balgair WR Profile
|
Posted: Sat Nov 03, 2007 10:43 pm Post subject: |
|
|
Also a point on archiving characters, perhaps have an even longer threshold on archiving ones on servers which consistently have very low submissions - it's perfectly possible for a server with average 1-5 submissions per month to just never see a character that only logs on for a few hours a week  _________________ EU-Draenor:
-- Sagart - Tairbh - Balgair - Buinne - Eilnich - Ruire - Dubh - Laidir - Naomh - Falaich -- |
|
Back to top |
|
 |
WyriHaximus
Census Taker
Joined: 18 Oct 2005
Posts: 244
Location: Koedijk, Alkmaar, Noord-Holland, The Netherlands
WR Updates: 1,520,754
WyriHaximus WR Profile
|
Posted: Sun Nov 04, 2007 9:46 am Post subject: |
|
|
xpolockx wrote: | Heh Wyri, that's pretty funny Don't forget to turn on aspect of the cheetah while you run... | Sprint + Preparation + Sprint + Vanish ftw ! _________________

My World of Warcraft Screenshots / Map |
|
Back to top |
|
 |
Skyfire
Trolling Enforcement
Joined: 18 Aug 2005
Posts: 746
Location: New Jersey
WR Updates: 44,279
Skyfire WR Profile
|
Posted: Sun Nov 04, 2007 9:48 pm Post subject: |
|
|
Much as I hate to say it (I would hate to lose my sig ;( ), 18 months sounds good. Of course, that leaves me set until this June, so who knows. _________________ Admin on WoWWiki
Moderator, Blogger on Wowhead |
|
Back to top |
|
 |
Rollie
Site Admin

Joined: 28 Nov 2004
Posts: 5376
Location: Austin, TX
WR Updates: 480,131
Rollie WR Profile
|
Posted: Mon Nov 05, 2007 2:28 pm Post subject: |
|
|
Well, appears the process has caught up already so after Wyr's files that were choking the system were cleaned out, we're back where we are supposed to be =) _________________ phpbb:phpinfo() |
|
Back to top |
|
 |
Balgair
Araiceil
Joined: 30 Sep 2005
Posts: 1701
Location: UK
WR Updates: 12,856,335
Balgair WR Profile
|
Posted: Mon Nov 05, 2007 2:32 pm Post subject: |
|
|
Good to know - I noticed it seemed to be catching up fast
Btw Rollie, could you take a look at this server and see if there's anything wrong with the display there, or if it genuinely is just zero entries on both sides? I'd kinda hoped it'd say 0:0 on the entries space if it was, the blank spaces leave me unsure if it's really 0:0 or just having issues  _________________ EU-Draenor:
-- Sagart - Tairbh - Balgair - Buinne - Eilnich - Ruire - Dubh - Laidir - Naomh - Falaich -- |
|
Back to top |
|
 |
|
|