.com.unity Forums
  The Official e-Store of Shrapnel Games

This Month's Specials

BCT Commander- Save $8.00
winSPWW2- Save $5.00

   







Go Back   .com.unity Forums > Shrapnel Community > Space Empires: IV & V

Reply
 
Thread Tools Display Modes
  #21  
Old July 17th, 2008, 01:41 PM
Mindi's Avatar

Mindi Mindi is offline
Lieutenant Colonel
 
Join Date: Jun 2004
Location: Lancaster, OH, USA
Posts: 1,250
Thanks: 28
Thanked 74 Times in 49 Posts
Mindi is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

To tell you the truth, I don't think it's the size of the PMs themselves that is the issue, it's more about the total number of entries in the database that is the issue. They are a relational databse and therefore has to link various different tables and entries together for each PM and that slows down the import significantly.

I mean in theory we could have a more PMs than we have currently if every user had the limit I am imposing, but in reality it was less than 200 people who had more than 100 PMs with the top 10 people on the list taking up almost 20% of the PM database alone. Some people like to save every single PM including things like read receipts and the like at if you're going to be that much of a PM hoarder, you need to copy and paste them to a word document and save them on your own machine.

I was even guilty of saving too many moderation PMs for history's sake and although they are good to keep (so I know the history we've had with certain users), they don't have to be kept in their current form, so I saved them in a document and deleted them from the forum software.
__________________
"All that is necessary for the triumph of evil is that good men do nothing."

For those looking for with this forum, please see Annette as I am retired.
Reply With Quote
  #22  
Old July 17th, 2008, 03:07 PM
douglas's Avatar

douglas douglas is offline
Major
 
Join Date: Apr 2004
Location: Atlanta, Georgia
Posts: 1,152
Thanks: 0
Thanked 0 Times in 0 Posts
douglas is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

I have no actual knowledge of how this particular database is set up, but that doesn't make much sense to me unless the database is poorly designed or certain kinds of redundant checking are involved. Is there, by any chance, a constraint that all PMs have to be unique to prevent accidental double-sending or some such? I could see that making an import take forever by forcing the database to compare each PM to all previous PMs before inserting it. If such a constraint does exist, removing it temporarily for the import should speed things up dramatically.
Reply With Quote
  #23  
Old July 17th, 2008, 03:35 PM
Mindi's Avatar

Mindi Mindi is offline
Lieutenant Colonel
 
Join Date: Jun 2004
Location: Lancaster, OH, USA
Posts: 1,250
Thanks: 28
Thanked 74 Times in 49 Posts
Mindi is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

I don't understand the full way that it is set up, I only know tid bits of what Richard has explained to me and I know UBB has a lot of backwards way of doing things so who knows. All I know is the import started out taking the better part of a whole day, Richard has done a lot of research and found a way to speed up the majority of the process, but the PMs database still takes 4-5 hours (as much as all of the other parts of the import combined) and last time it failed totally towards the end of the PM import.

Frankly we should have put some contraints around PMs before now, but it is what it is and we're dealing with it now.
__________________
"All that is necessary for the triumph of evil is that good men do nothing."

For those looking for with this forum, please see Annette as I am retired.
Reply With Quote
  #24  
Old July 17th, 2008, 03:39 PM
douglas's Avatar

douglas douglas is offline
Major
 
Join Date: Apr 2004
Location: Atlanta, Georgia
Posts: 1,152
Thanks: 0
Thanked 0 Times in 0 Posts
douglas is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

I notice you mentioned in the Dom 3 forum that about 100000 PMs were taking longer than everything else put together. Considering there are over 5 times that many posts alone, plus the user list and various other data on top of that, I can't think of any reasonable explanation for it other than an index or constraint requiring excessive redundant checking.

My first guess would be that the PM content column has a unique constraint but no index, which would force a string comparison of each and every PM with each and every other PM. This isn't that big a deal in normal use when you only have to worry about one new PM at a time, but doing 100k at once takes forever. Removing the constraint in this case would drop the import time immensely. Creating an index for it instead would greatly reduce the number of comparisons required, but the index itself would take some extra resources to maintain. Assuming my guess is correct, I would first try creating an index for the column and see how much that helps. If it still takes too long, then remove the constraint entirely before importing and then see if it can be added back in a reasonable amount of time.

Edit: Then again, I don't know that much about how most databases handle indexes on strings specifically, so it's possible that it does have an index on PM content and that removing that index could help. I doubt this is the case, but it might be possible.
Reply With Quote
  #25  
Old July 17th, 2008, 04:57 PM
Mindi's Avatar

Mindi Mindi is offline
Lieutenant Colonel
 
Join Date: Jun 2004
Location: Lancaster, OH, USA
Posts: 1,250
Thanks: 28
Thanked 74 Times in 49 Posts
Mindi is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

Like I said, I don't know all the finer points of this sort of thing. I do know what Richard did to speed up the main part of the import was to take off the indexing, but I don't think the PMs work the same way and we still haven't figured out what the total time will be once we put the index back on at the end because we haven't had a complete, successful import to try it on yet since the PM part failed last time.

I don't really like talking about all of this because it's not my area of expertise and I only know a little bit about everything Richard is doing. He's the back end, I'm the one putting in modifications, tweaking settings, importing smilies, etc.
__________________
"All that is necessary for the triumph of evil is that good men do nothing."

For those looking for with this forum, please see Annette as I am retired.
Reply With Quote
  #26  
Old July 21st, 2008, 01:22 PM
gregebowman's Avatar

gregebowman gregebowman is offline
Colonel
 
Join Date: Jun 2002
Location: tampa, fl
Posts: 1,511
Thanks: 3
Thanked 0 Times in 0 Posts
gregebowman is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

For someone who's still has one foot in the 20th century, what the heck is a PM?
Reply With Quote
  #27  
Old July 21st, 2008, 01:30 PM
Ed Kolis's Avatar

Ed Kolis Ed Kolis is offline
General
 
Join Date: Apr 2001
Location: Cincinnati, Ohio, USA
Posts: 4,547
Thanks: 1
Thanked 7 Times in 5 Posts
Ed Kolis is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

Private message
__________________
The Ed draws near! What dost thou deaux?
Reply With Quote
  #28  
Old July 21st, 2008, 01:31 PM
Fyron's Avatar

Fyron Fyron is offline
Shrapnel Fanatic
 
Join Date: Jul 2001
Location: Southern CA, USA
Posts: 18,394
Thanks: 0
Thanked 12 Times in 10 Posts
Fyron is an unknown quantity at this point
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

"Private Messages" existed for much of the latter decade of the 20th century.
__________________
It's not whether you win or lose that counts: it's how much pain you inflict along the way.
--- SpaceEmpires.net --- RSS --- SEnet ModWorks --- SEIV Modding 101 Tutorial
--- Join us in the #SpaceEmpires IRC channel on the Freenode IRC network.
--- Due to restrictively low sig limits, you must visit this link to view the rest of my signature.
Reply With Quote
  #29  
Old July 21st, 2008, 01:52 PM
narf poit chez BOOM's Avatar

narf poit chez BOOM narf poit chez BOOM is offline
Shrapnel Fanatic
 
Join Date: Mar 2003
Location: CHEESE!
Posts: 10,009
Thanks: 0
Thanked 7 Times in 1 Post
narf poit chez BOOM is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

...So less than 10%?
__________________
If I only could remember half the things I'd forgot, that would be a lot of stuff, I think - I don't know; I forgot!
A* E* Se! Gd! $-- C-^- Ai** M-- S? Ss---- RA Pw? Fq Bb++@ Tcp? L++++
Some of my webcomics. I've got 400+ webcomics at Last count, some dead.
Sig updated to remove non-working links.
Reply With Quote
  #30  
Old July 23rd, 2008, 10:25 PM
Azselendor's Avatar

Azselendor Azselendor is offline
First Lieutenant
 
Join Date: May 2003
Location: Gettysburg Sector
Posts: 785
Thanks: 7
Thanked 5 Times in 5 Posts
Azselendor is on a distinguished road
Default Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!

Oh, you are all very very wrong.

Private Messages have existed for as long as there has been children passing private messages in class. This of course was always accompanied by the Department of Homeland Security - formerly known as Teachers - whom would intercept and read them randomly and to your embarrassment.
__________________
@Azselendor #BoldlyGoing
/ Space Empires // Orlando Pest Control
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump


All times are GMT -4. The time now is 06:16 AM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©1999 - 2024, Shrapnel Games, Inc. - All Rights Reserved.