[CONVERT] Crawler Converters (Forumer, ZetaBoards, etc.)

Converting from other board software? Good decision! Need help? Have a question about a convertor? Wish to offer a convertor package? Post here.
Get Involved
Locked
nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

[CONVERT] Crawler Converters (Forumer, ZetaBoards, etc.)

Post by nneonneo » Sun Aug 30, 2009 9:45 pm

Since the phpBB2 support forums have been closed, I have been unable to update the converters on this thread. Hence, I will be posting updated converters here.

The Proboards and Forumotion converters can be found at their respective topics:
Proboards converter, Forumotion converter.

Note that this converter has been re-uploaded several times. Thus, the download count may not be accurate. As of time of writing, it has been downloaded over 2100 times.

Please note: I provide no warranty on this software, but I will be happy to provide support in this thread.

Converters for Forumer, ZetaBoards, SuddenLaunch and several other forums are provided. Please read the Instructions.txt file in the corresponding folder. I've tried my best to make these understandable, but any comments or suggestions on any part of the process are welcome.

The phpBB2 converters produce SQL which should be compatible with phpBB 2.0.22.
The phpBB3 converters produce SQL which should be compatible with phpBB 3.0.7.

The converters are NOT written in PHP, they are written in Python. This means that Windows users will need a current distribution of the 2.x Python programming framework, available at http://python.org (the Windows Installer link on the left is the most direct way to get it). Linux and Mac OS X should come with Python interpreters pre-installed. Please note that the converters are not yet compatible with Python 3.x.

Converters included:
  • ForumerIPB (phpBB2, SMF)
  • InvisionFree (phpBB2, SMF)
  • phpBB2 crawler (phpBB2 only)
  • phpBB3 crawler (phpBB3 only)
  • Suddenlaunch (phpBB2, phpBB3, SMF)
  • Yuku (phpBB2, SMF)
  • ZetaBoards (phpBB2, SMF)
ActiveBoards and ChatArea converters are SMF-only, and can be found on the SimpleMachines forum: http://www.simplemachines.org/community ... msg1215281. They are not included here because of attachment size limits.
Proboards and Forumotion converters can be found at their respective topics: http://www.phpbb.com/community/viewtopi ... 5#p4371825 for Proboards, http://www.phpbb.com/community/viewtopi ... 5#p4914295 for Forumotion.

Forumer can be converted using the corresponding converter for your board type (ForumerIPB, phpBB2 or phpBB3). Most phpBB2 and phpBB3 forums can be converted using the corresponding crawler.

General instructions: You will need Python 2.5, 2.6 or 2.7 (3.0 and 3.1 will not work; the usual symptom is that the converter starts up and immediately quits). Unzip the entire package, and run the converter you want. Alternately, you can unzip the converter you want (e.g. Proboards), then copy the common.py file into that converter directory.

Another utility split_sql.py is provided to split large SQL files into smaller ones; use this if your SQL import fails because the file is too large.
Attachments
Crawler Converters.zip
Crawler Converters [May 14 2011]
(200 KiB) Downloaded 2560 times
Last edited by nneonneo on Sat May 14, 2011 8:10 pm, edited 18 times in total.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

eatingmashin
Registered User
Posts: 11
Joined: Sat Sep 05, 2009 6:04 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by eatingmashin » Sat Sep 05, 2009 6:09 am

Hi! I'm trying to convert IF -> PhpBB2 -> Phpbb3.

But in IF -> PhpBB2, I'm having problems getting past step 4 (Execute the SQL batch files topics_posts.sql and categories_forums.sql in the forum db).

categories_forums.sql executes without problems but when I try to run topics_posts.sql, I get this error in MySql:
"Incorrect integer value: ' ' for column 'post_edit_time' at row 1"

What date/time format should I use for my IF? mySQL db? PhpBB2? Is my problem related to the date/time format?

Thanks!

EDIT: I got over the error by reconfiguring the mySQL database to not be in strict mode.. :)

eatingmashin
Registered User
Posts: 11
Joined: Sat Sep 05, 2009 6:04 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by eatingmashin » Sun Sep 06, 2009 9:55 am

Now I got a new error..

running topic_posts_phpBB.py went really well..

when I ran members_phpBB.py, it also went well except for this members_convert_error_log entry:
Member ID 1643
Traceback (most recent call last):
File "members_phpBB.py", line 47, in <module>
userdata=conn.read()
File "C:\Python26\lib\socket.py", line 327, in read
data = self._sock.recv(rbufsize)
File "C:\Python26\lib\httplib.py", line 517, in read
return self._read_chunked(amt)
File "C:\Python26\lib\httplib.py", line 578, in _read_chunked
value += self._safe_read(chunk_left)
File "C:\Python26\lib\httplib.py", line 617, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "C:\Python26\lib\socket.py", line 351, in read
data = self._sock.recv(left)
socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host
Any ideas?

hamsterbacke
Registered User
Posts: 2
Joined: Mon Oct 05, 2009 12:56 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by hamsterbacke » Mon Oct 05, 2009 1:10 am

i've got a problem with converting a yuku forum. it parses all the boards but one... and the on is the last.
it hangs with "finding topics". i do a strg+c and look in the critical_error.log and it says:
Traceback (most recent call last):
File "members_topics_posts_phpBB.py", line 229, in <module>
added_topics=re.findall(thread_search,pagedata)
File "/usr/lib/python2.5/re.py", line 175, in findall
return _compile(pattern, flags).findall(string)
KeyboardInterrupt

so it seems to hang on a regex? maybe it's because the last board has got a special character in it? my locale is set to de_DE.UTF-8... maybe i have to pass that to python?
i have pyhton version 2.5.4.

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Mon Oct 05, 2009 4:17 am

Hanging on a regex usually means a bug in the regex-based parser for the crawler.

Would you mind sending me the settings.py so I can take a look? My e-mail is [nneonneo [AT] gmail [dot] com].
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

hamsterbacke
Registered User
Posts: 2
Joined: Mon Oct 05, 2009 12:56 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by hamsterbacke » Sun Nov 01, 2009 2:39 pm

i've written a bash script that makes automatically backups. i run it once a month with a cronjob to make a backup of our yuku board. it logins with a specific username and extracts the cookie data and inserts it into the settings.py
maybe someone can use it.

Code: Select all

#!/bin/bash
## script that automatically downloads a yuku board and converts it to a
## phpBB2 board. it writes the data in sql dump files
## they can then be added to the mysql database with something like:
## mysql -u dbuser -p dbname < name.sql
## don't forget to remove/uncomment the "rawinput" from the members_topics_posts_phpBB.py

USERNAME=:)
PASS=:)
AGENTIDENTIFIER="Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
LOGINURL=http://:).yuku.com
COOKIENAME=cookie.txt
EXECUTIONDIR=/home/:)/Yuku

## login to yuku and use the cookie data to convert the board
## cookie data has to be written in settings.py
## variable name is COOKIEDATA

# write cookie to $EXECUTIONDIR/cookie.txt
curl -s -A "$AGENTIDENTIFIER" -c $EXECUTIONDIR/cookie.txt -d "login=$USERNAME" -d "password=$PASS" -d "submit=Log\ In" -L $LOGINURL >/dev/null

# extract the cookie file and format it
PHPSESSID=`cat $EXECUTIONDIR/cookie.txt | grep PHPSESSID | awk '{print $7}'`
LOGIN_AID=`cat $EXECUTIONDIR/cookie.txt | grep login_aid | awk '{print $7}'`
YUG=`cat $EXECUTIONDIR/cookie.txt | grep YUG | awk '{print $7}'`

# write new cookie data in settings.py
sed "s/COOKIEDATA.*/COOKIEDATA=\'login_aid=$LOGIN_AID; PHPSESSID=$PHPSESSID; YUG=$YUG;\'/" < $EXECUTIONDIR/settings.py > $EXECUTIONDIR/settings.new.py
rm $EXECUTIONDIR/settings.py
mv $EXECUTIONDIR/settings.new.py $EXECUTIONDIR/settings.py

# do all the conversations....
python $EXECUTIONDIR/members_topics_posts_phpBB.py

# make dir from current date and move the sql files in there
DATE=`date +%d.%m.%Y`
mkdir $DATE
mv *.sql $DATE

# final step! zip it
7za a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on -p:) $DATE.7z $DATE
rm -r $DATE 

fantanoice
Registered User
Posts: 6
Joined: Sat Nov 07, 2009 6:58 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by fantanoice » Sat Nov 07, 2009 7:17 am

Cheers for this guys. the Zeta->SMF one seems to work fine.

Are there any plans for a Forumer converter? It would be fantastic if you could make a Forumer->SMF as well. :) I lost about 60,000 posts when I moved my forum to from Forumer to Zeta, but if I get can back my 20,000 from Zeta AND my 60,000 from Forumer, then that would be just amazing. ^_^

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sat Nov 07, 2009 6:33 pm

What Forumer mode are you using (phpBB2, phoBB3, IPB)?
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

fantanoice
Registered User
Posts: 6
Joined: Sat Nov 07, 2009 6:58 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by fantanoice » Sun Nov 08, 2009 6:39 am

I was using IPB.

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sun Nov 08, 2009 6:35 pm

You're in luck, then! It turns out that the iPBfree/Invisionfree converter, with minor changes, can convert Forumer boards. Attached is a converter package that should handle Forumer forums in IPB mode. Let me know how it goes.
Attachments
Forumer.zip
Forumer converter
(26.61 KiB) Downloaded 413 times
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

fantanoice
Registered User
Posts: 6
Joined: Sat Nov 07, 2009 6:58 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by fantanoice » Mon Nov 09, 2009 10:43 am

Cheers. I'll give it a crack. :)

Arrnea
Registered User
Posts: 9
Joined: Fri Nov 13, 2009 5:39 pm

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by Arrnea » Fri Nov 13, 2009 5:42 pm

Hi, I am attempting to use your IPBfree crawler converter to move my IPBfree forums to phpBB.
However, when I attempt to do so, I find the following error upon executing the topics_posts_phpBB.py file:

Code: Select all

Topic ID 1 ... Error: Topic is empty!                                          
Topic ID 2 ... Error: Topic is empty!                                          
Topic ID 3 ... Error: Topic is empty!                                          
Topic ID 4 ... Error: Topic is empty!                                          
Topic ID 5 ... Error: Topic is empty!                                          
Topic ID 6 ... Error: Topic is empty!                                          
Topic ID 7 ... Error: Topic is empty!                                          
Topic ID 8 ... Error: Topic is empty!                                          
Topic ID 9 ... Error: Topic is empty!                                          
Topic ID 10 ... Error: Topic is empty!                                         
Topic ID 11 ... Error: Topic is empty!                                         
Topic ID 12 ... Error: Topic is empty!                                         
Topic ID 13 ... Error: Topic is empty!                                         
Topic ID 14 ... Error: Topic is empty!                                         
Topic ID 15 ... Error: Topic is empty!                                         
Topic ID 16 ... Error: Topic is empty!                                         
Topic ID 17 ... Error: Topic is empty!                                         
Topic ID 18 ... Error: Topic is empty!                                         
Topic ID 19 ... Error: Topic is empty!
This continues through all of the Topic IDs until the cap specified in settings.py is reached.
Any ideas what is causing this?

I'm running python2.6 on Ubuntu 9.10 'Karmic', by the way.

nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo » Sat Nov 14, 2009 12:07 am

Have you changed the settings in your profile and switched to the default iPBfree skin?

If that doesn't work, PM me the board address; because of local variations among boards, I cannot recommend a general fix for this problem.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.

eatingmashin
Registered User
Posts: 11
Joined: Sat Sep 05, 2009 6:04 am

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by eatingmashin » Sat Nov 14, 2009 1:23 am

Hi again!

After successfully using your converter (IF->PhpBB), my old smilies from the previous board does not show as they should. It just shows text that represents the smilies. I've recreated the smilies and corresponding texts after migrating..

I tried resynchronizing the category then by forum, purging the cache but still nothing..

New smilies are working just fine..

Help! :(

Arrnea
Registered User
Posts: 9
Joined: Fri Nov 13, 2009 5:39 pm

Re: [CONVERT] Proboards/Invisionfree/ActiveBoards converters

Post by Arrnea » Sat Nov 14, 2009 4:25 am

nneonneo wrote:Have you changed the settings in your profile and switched to the default iPBfree skin?

If that doesn't work, PM me the board address; because of local variations among boards, I cannot recommend a general fix for this problem.
Yes, I've changed the settings in the profile and switched the skin.

I will PM you the board address now.

Edit: Argh, can't send a PM because I don't have five posts. I'll have to gather those up before I can PM the board URL to you.

Edit (again): Finally got up to five posts. Silly rule, that. Will PM the URL now.
Edit (again #2): What the? I have five posts, and it still says I can only PM team members. :evil:
I guess I will just have to risk putting it in this topic. Link no longer needed, removed.
Last edited by Arrnea on Sat Nov 14, 2009 8:26 am, edited 2 times in total.

Locked

Return to “[3.0.x] Convertors”

Who is online

Users browsing this forum: oBot and 10 guests