• This is a second notice. One notice is not enough. I need to annoy you with more. But this one can at least be dismissed. And it has my avatar in it. Lots of options.
  • The last one. I wanted to see how they stack in the same place.

Upgrade Status

More threads by GrnEyedDvl

This thing rocks on 64 cores. It imported all our users in about 10 minutes. It did crash once, though I think it got ahead of itself and got into a race condition. when I came back in it was stopped. I restarted it and it picked right up but I think it counted some twice. Hopefully it didn't insert them twice.

Current import

newimport.webp




The very first import when I did not know you could run it on multiple cores. Thats a HUGE difference.
import9.webp
 
Private Messages finished in 17 minutes, that was an hour on 16 cores. And all 64 cores are running about 60ish percent. Plenty of RAM left, its only using 40ish. gigs.

I am going to bed because I know that it will still be running for hours. I have been up 24 hours now. Hopefully it doesn't get stuck again.
 
It is. That is the advantage of a 64 core machine. I knew it would be faster I just didn't want to throw a number out there. It took me longer than I wanted to get Odin configured for base services, I had a pretty good list of stuff put together when I loaded Thor. And I still have one issue on Odin that was no problem on Thor. In fact it was so easy I did not document it. Now I cant get it to load properly on Odin. Irritating. I will get it figured out and its not a hugely important issue. Most people would have no idea unless I told them anyways.

But the actual data import is going faster. So I am ahead a bit. I think we will be open right at about the 24 hour mark, which is pretty damned good considering the first import I ran took 19 hours just for that. This time I had to do a full server wipe and install from scratch.
 
I think I understand the difference between Import and Finalize, at least partly. It's not documented anywhere that I can find. But because I am an idiot that had to go poke around while its still running and threw an error it gave me a little insight.

Post info, message info, whatever, is not stored in a single table. I think there are 260ish tables. vBulletin has 300. I think when it imports something like Posts it writes that table, but a table that is JOINED to posts does not get written at that time. post_edit is another table and it processes them separately. So even though Posts are imported, it has to finalize stuff after it processes post_edits. Just one example. I'm sure there are many in a program this complex. So it has to loop through the various tables not once, but many times.
 
I also cannot fix rewrites yet, because you have to restart web services to do that and the importer runs on a php command line process. So when you read a thread that goes to twcenter.net/forums you get a not found. I think I have to write a bunch of custom rules. They have a ruleset for Apache, but we do not use Apache.
 
I am just going to redo this.

Code: 
ℹ  * InnoDB migration request for `totalwar_vb`.`searchgroup_text` Table: ALTER TABLE `totalwar_vb`.`searchgroup_text` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`blog_text` Table: ALTER TABLE `totalwar_vb`.`blog_text` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`pt_issue` Table: ALTER TABLE `totalwar_vb`.`pt_issue` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`searchcore_text` Table: ALTER TABLE `totalwar_vb`.`searchcore_text` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`vboptimise_cdn` Table: ALTER TABLE `totalwar_vb`.`vboptimise_cdn` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`socialgroup` Table: ALTER TABLE `totalwar_vb`.`socialgroup` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`language` Table: ALTER TABLE `totalwar_vb`.`language` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`groupmessage` Table: ALTER TABLE `totalwar_vb`.`groupmessage` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`pt_issuenote` Table: ALTER TABLE `totalwar_vb`.`pt_issuenote` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`blog` Table: ALTER TABLE `totalwar_vb`.`blog` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`dl2_votes` Table: ALTER TABLE `totalwar_vb`.`dl2_votes` ENGINE=InnoDB;
ℹ  * InnoDB migration request for `totalwar_vb`.`vboptimise` Table: ALTER TABLE `totalwar_vb`.`vboptimise` ENGINE=InnoDB;

Something happened and I might have an idea what, but I cannot restart from this point. The site works, mostly. But lots of weird things going on. And a pile of errors on certain kinds of content.

I think I am going to lock myself out of it too so I do not get anxious lol
 
Import on Odin, thats 24.9 million entries.
Code: 
MariaDB [totalwar_xf]> SELECT COUNT(*) FROM import_log_vbulletin4withblog_1;
+----------+
| COUNT(*) |
+----------+
| 24992418 |
+----------+
1 row in set (1 min 9.789 sec)


That's 18 million entries. I do not get it. Odin should be a little bigger. Not anything like another 50%.
Code: 
MariaDB [totalwar_xf]> SELECT COUNT(*) FROM import_log_vbulletin4withblog_1;
+----------+
| COUNT(*) |
+----------+
| 18305793 |
+----------+
1 row in set (2 min 23.231 sec)
 
It's running again. We will see what happens I guess.

btw I did keep all the drives from Odin intact. I can easily reinstall them and boot back up to vBulletin if needed but I really do not want to do that.
 
Ok this is running smoothly And it did not add or count 200,000 users twice. I still am not sure why it did that. If you look at the first import I ran vs the one that crashed and restarted it has a lot more users. Weird.

newimport.webp
 
And here I was bothered those days about my 8086 machine taking so long to read my accounting floppy disks....
 
Well something has definitely changed. I am not so sure they developed this importer with 64 cores in mind. After thinking that maybe this was a race condition as I posted above, and the differences in RAM per core, when I ran this import I still used 64 cores but I dropped the RAM down to 48 gigs vs 60. I think that is forcing the cores to slow down a bit because load is only about 8 or 9 with all 64 cores running at 10%. They were running at 60% earlier.

That sounds a bit counterproductive, but it is actually working out better I think, as long as it doesn't crash again. All 14.8 million posts processed in under 3 hours. The first import on Thor posts took 9 hours. The first import on Odin last night posts took a little under 6. Check out these numbers and compare them to the earlier images.

newport3.webp
 
btw the guy I am going to have handle the Downloads import quoted me a price of $650 I think, to do the import on the forums and another $100 for Downloads. I will go grab that conversation. He claims to have built his own importer that is both faster and supports incremental imports, so you can import the site in chunks instead of all at once. That would be very useful in our situation.

In 2014 I bought a copy of XenForo 1 and it completely choked on our vBulletin setup. I did not pursue it any further because vBulletin 5 had been announced, and silly me I assumed it would at least have the functions of vBulletin 4. Now there are enough people switching that this guy has spent the time to build his own importer and charges big money for it. Good for him.

He does this often enough he probably knows a LOT of things I do not know about it. I am going to do this one time and one time only and am learning as I go. He does 3 or 4 a week. If this does not go fairly smoothly after today he and I might be having another conversation.

quote.webp
 
dont-touch-it-liz-birdswort.gif
 
Ugh. Still at 1.6 millions Direct messages. processed. Out of 2 million. On the second import I did for the test site I did not import private messages. Now I remember why. They take bloody forever.
 
I am starting to get really frustrated with this. It now has processed 2.3 million private messages. Which is 200,000 more than existed in the import on April 3rd. I do not see us adding that many in 3 weeks. Maybe I am wrong and everything is fine.

I missed the summary because I told it to finalize with no input. That would have shown me then how many of everything and how much time was involved. I can probably go dig it up but I do not want to touch it.
 
Well I found this.

It's common for the XenForo finalize import process to take longer than the actual import itself. This is because the finalize step involves tasks like indexing, rebuilding caches, and ensuring data integrity after the raw import data has been ingested. Factors like server performance, database size, and the complexity of the imported data can all influence the finalize time
Click to expand...

And this, both from posts a couple of years old. They should document this stuff instead of letting it rot at the bottom of a forum. For guys that do this all the time its common knowledge. For those of us who do it once, not so much.
The lack of batching in XenForo is what makes the innodb_flush_log_at_trx_commit=2 change such a massive performance improvement, as the default value prevents MySQL & the operating system from batching writes in the background.
Click to expand...
 
Still working on threads (these have a weird number too, way bigger than import #1) and I do not dare stop it. I have been reading a LOT about performance issues with installs. Now Odin is not setup with default php or mysql settings, and neither is Thor. But its certainly possible that I tweaked something on one and not on the other. The only thing I know of that I did not do is a vhost setting that should not impact this.

A few people report cases like this with a fresh install, not an import. I will be testing this later on another machine.
I first ran the installation with all default server settings, installation time was 18min:11s.
After your suggested optimizations, I ran the installation again, this time installation was 1m:36s.
Click to expand...
 
You must log in or register to reply here.

Staff online

Members online

Total: 3 (members: 2, guests: 1)

Site News

Site Polls

Total War Center is

  • An Awesome forum

    Votes: 0 0.0%

  • Terrible a waste of your time, don't bother

    Votes: 1 16.7%

  • Well, that GED guy is a nut

    Votes: 2 33.3%

  • Tacos

    Votes: 3 50.0%
See comments…
Back
Top