[CONVERT] Crawler Converters (Forumer, ZetaBoards, etc.)

Converting from other board software? Good decision! Need help? Have a question about a convertor? Wish to offer a convertor package? Post here.
Scam Warning
nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sat Nov 14, 2009 7:02 am

@eatingmashin: Try the resync_bbcode.php: http://www.startrekguide.com/community/ ... 648#p47648

@Arrnea: You could also have e-mailed me: nneonneo [at] gmail <dot> com.

EDIT: Ok, Arrnea, I just ran a test conversion. Since I don't have many permissions on the forum, I can only see a few topics, but of those topics I can see, the converter converted all of them.

The result is a large number of "Topic is empty!" messages with the occasional "Completed! Added <n> posts". However, it seems to me that it is working. Remember, NUMTOPICS should be greater than the highest topic ID, not the total number of topics.

EDIT2: So it apparently isn't just my good luck: there's a minor change that needs to be made. iPBfree apparently started checking the user agent of the browser for security reasons, and my script has been logging on with a different user agent than the one it actually uses to crawl the forum.

This is easy to fix: in common.py, change

Code: Select all

req=urllib2.Request(url,headers={'COOKIE':cookiedata,'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'})
to

Code: Select all

req=urllib2.Request(url,headers={'COOKIE':cookiedata})
and run the get_cookie script again. This should allow all the threads to be properly downloaded.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

Arrnea
Registered User
Posts: 9
Joined: Fri Nov 13, 2009 5:39 pm

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by Arrnea » Sat Nov 14, 2009 8:16 am

Thanks for the quick response. I'll try that soon and let you know if it works.

Edit: It seems to be working fine now. The odd topic returns an empty topic error, but those are topics that have been deleted at one point or another. ;)

Edit again: Using relative timestamps (eg: Yesterday, 2:10pm, 53 minutes ago etc) also seems to cause an error. I've turned them off, should take care of the error, but it might be something worth adding to the instructions of things to turn off.

Arrnea
Registered User
Posts: 9
Joined: Fri Nov 13, 2009 5:39 pm

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by Arrnea » Sat Nov 14, 2009 9:01 am

Ok, getting this when I try and import it into my MySQL database:

Code: Select all

SQL query:

INSERT INTO `phpbb_forums` ( forum_id, cat_id, forum_order, forum_name, forum_desc )
VALUES ( 2, 3, 20, 'SOS Brigade Archives', '' ) ;

MySQL said: Documentation
#1054 - Unknown column 'cat_id' in 'field list' 
Edit: Oh, I know why this is happening. It's because I'm trying to import into phpBB3 and this script is designed for phpBB2. Curses.
I'll manually edit the references to categories out of the generated sql file and see if it works then.

Edit again: Meh, I'll just install phpbb2 then upgrade to phpbb3 from the existing installation.

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sat Nov 14, 2009 4:12 pm

Thanks for the tip about relative timestamps. I'll add that to the instructions.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

Shomiku
Registered User
Posts: 2
Joined: Tue Dec 08, 2009 1:01 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by Shomiku » Tue Dec 08, 2009 1:08 am

I downloaded the converter and python and read the instructions. . .
But i cant figure out how i start it,
do you think you could help. . .?
Icase it helps i am trying to convert Poboards to invisionfree.



Also, when i get it working, do all the threads/posts/etc stay on the old one as well?

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Tue Dec 08, 2009 1:52 am

Did you say you wanted to convert Proboards to Invisionfree? I'm afraid that's not possible...the converter can only take Proboards to phpBB (or SMF), or Invisionfree to SMF.

Posts, etc. will remain on the old forum after conversion.

Can you state more specifically what your problem is?
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

Shomiku
Registered User
Posts: 2
Joined: Tue Dec 08, 2009 1:01 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by Shomiku » Tue Dec 08, 2009 2:26 am

The problem is i dont knw how to run it, like, do i put in the site URL somwhere?


nneonneo wrote:Did you say you wanted to convert Proboards to Invisionfree? I'm afraid that's not possible...the converter can only take Proboards to phpBB (or SMF), or Invisionfree to SMF.

Posts, etc. will remain on the old forum after conversion.

Can you state more specifically what your problem is?
Oh. . . well, can phpBB be converted into anything other then proboards?

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Tue Dec 08, 2009 3:52 am

I believe the instructions clearly state that you put the site URL and other settings in the settings.py file. If you are confused, please read them again, carefully.

Erm, are you asking if Proboards can be converted into something other than phpBB, or the other way around? I don't believe it is possible to import data into Proboards or Invisionfree, so there is no way to convert anything to these two, but with my crawler, you can convert *from* them *to* other software, like phpBB.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

fantanoice
Registered User
Posts: 6
Joined: Sat Nov 07, 2009 6:58 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by fantanoice » Sat Dec 12, 2009 5:41 am

nneonneo wrote:You're in luck, then! It turns out that the iPBfree/Invisionfree converter, with minor changes, can convert Forumer boards. Attached is a converter package that should handle Forumer forums in IPB mode. Let me know how it goes.
Now, this seems to be working alright, but I have a topic with 3,000+ posts which tends to devastatingly slow down at about the 70-page mark (out of 178). Is there a way to speed this up?

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sat Dec 12, 2009 6:12 am

I don't know of any way to speed up the conversion if it is slow on a download. If it consistently slows down at the same spot, it's possible that Forumer is applying rate-limiting (throttling), or that this particular thread (at that spot) just takes a long time to load. Does the slowness persist after the thread is completed?
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

fantanoice
Registered User
Posts: 6
Joined: Sat Nov 07, 2009 6:58 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by fantanoice » Sat Dec 12, 2009 7:27 am

It just seems to gradually slow down after each page. It's still downloading (page 112/178), so I'm not sure whether it's still slow after it has been completed.

Every other thread up to this point converted pretty quickly, but they maybe had, at max, 50 posts to convert (about 2-3 pages). 178 pages is, well, a LOT more than that. xD

Though I still have 1,500+ topics to convert after this one, though this is the forum's largest topic to it should be 'okay' to convert the others if it gets through this.

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sat Dec 12, 2009 8:00 am

How much is being downloaded each time? It's possible that the total size of the topic is so large that it is causing problems for the converter.

To check, see if the "python" process on your computer is taking a lot of CPU time. If it is, then it is almost certainly the aforementioned efficiency problem; if not, then it's a slow download.

The fix is as follows:
Change the lines

Code: Select all

        pagedata=download_page(URL+'?showtopic='+str(topicid),progressline, COOKIEDATA)
        pages=re.findall(re_pages,pagedata)
to

Code: Select all

        pagedata=[download_page(URL+'?showtopic='+str(topicid),progressline, COOKIEDATA)]
        pages=re.findall(re_pages,pagedata[0])
(be sure to preserve the indentation)
and change

Code: Select all

                pagedata+=download_page(URL+'?showtopic='+str(topicid)+'&st='+str(offset*perpage),progressline, COOKIEDATA)
to

Code: Select all

                pagedata.append(download_page(URL+'?showtopic='+str(topicid)+'&st='+str(offset*perpage),progressline, COOKIEDATA))
        pagedata = ''.join(pagedata)
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

damienlucifera
Registered User
Posts: 1
Joined: Sun Dec 27, 2009 11:33 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by damienlucifera » Sun Dec 27, 2009 11:39 am

Hi,
I'm having trouble converting posts and threads from Proboards. Everything else seems fine, but whenever it starts converting posts and threads, it comes up with error messages telling me to refer to the convert_error.txt

Here's one of the errors:

Topic ID 257 in [BOARD NAME]
Traceback (most recent call last):
File "[FILE DIRECTORY]\Proboards\members_topics_posts_smf.py", line 496, in <module>
post[0]=parsetime(ln[4])
File "[FILE DIRECTORY]\Proboards\members_topics_posts_smf.py", line 154, in parsetime
,fmt)
File "[FILE DIRECTORY]\Python\lib\_strptime.py", line 454, in _strptime_time
return _strptime(data_string, format)[0]
File "[FILE DIRECTORY]\Python\lib\_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '11/11/09 at 3:25am' does not match format '%m/%d/%Y at %I:%M%p'

Obviously I've taken out certain info on the board name and file directory, but I'm hoping whatever I've provided is enough for someone to let me know where I'm going wrong?

Please help!

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sun Dec 27, 2009 8:31 pm

You selected the wrong date format in your preferences; it should be the one that looks like "12/28/2004 at 11:11pm", not "12/28/04 at 11:11pm" ("2004" instead of "04").
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

mbird_2005
Registered User
Posts: 18
Joined: Thu Dec 31, 2009 12:26 pm

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by mbird_2005 » Thu Dec 31, 2009 2:10 pm

Arrnea wrote:Hi, I am attempting to use your IPBfree crawler converter to move my IPBfree forums to phpBB.
However, when I attempt to do so, I find the following error upon executing the topics_posts_phpBB.py file:

Code: Select all

Topic ID 1 ... Error: Topic is empty!                                          
Topic ID 2 ... Error: Topic is empty!                                          
Topic ID 3 ... Error: Topic is empty!                                          
Topic ID 4 ... Error: Topic is empty!                                          
Topic ID 5 ... Error: Topic is empty!                                          
Topic ID 6 ... Error: Topic is empty!                                          
Topic ID 7 ... Error: Topic is empty!                                          
Topic ID 8 ... Error: Topic is empty!                                          
Topic ID 9 ... Error: Topic is empty!                                          
Topic ID 10 ... Error: Topic is empty!                                         
Topic ID 11 ... Error: Topic is empty!                                         
Topic ID 12 ... Error: Topic is empty!                                         
Topic ID 13 ... Error: Topic is empty!                                         
Topic ID 14 ... Error: Topic is empty!                                         
Topic ID 15 ... Error: Topic is empty!                                         
Topic ID 16 ... Error: Topic is empty!                                         
Topic ID 17 ... Error: Topic is empty!                                         
Topic ID 18 ... Error: Topic is empty!                                         
Topic ID 19 ... Error: Topic is empty!
This continues through all of the Topic IDs until the cap specified in settings.py is reached.
Any ideas what is causing this?

I'm running python2.6 on Ubuntu 9.10 'Karmic', by the way.

i have the same problem here

any one can help me plz ??

Locked

Return to “[3.0.x] Convertors”