Proboards/Invisionfree/ActiveBoards converters

This is an archive of the phpBB 2.0.x convertors forum. Support for phpBB2 has now ended.
Forum rules
Following phpBB2's EoL, this forum is now archived for reference purposes only.
Please see the following announcement for more information: viewtopic.php?f=14&t=1385785
User avatar
PPV
Registered User
Posts: 57
Joined: Tue Sep 30, 2008 3:08 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by PPV »

Ok. I got a problem with the import of thread. I get memory errors on the bigger ones (500 pages + at 90 posts per pages)

Is there some way that we could tweak the script so it does one specific thread to be reincorporated after?

I have to mention that IF is being a twat and not letting us splitting those threads... each has more than 45000 posts in them...
nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo »

Is Python giving memory errors? Or is it the SQL import?

If it is the SQL import, then it is not hard to fix; use the split_sql.py program (put it in the same place as the .sql files) to split the file into smaller chunks.

If it is Python, then I would be surprised, since I've seen the converter handle 900,000 post forums.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.
User avatar
PPV
Registered User
Posts: 57
Joined: Tue Sep 30, 2008 3:08 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by PPV »

nneonneo wrote:Is Python giving memory errors? Or is it the SQL import?

If it is the SQL import, then it is not hard to fix; use the split_sql.py program (put it in the same place as the .sql files) to split the file into smaller chunks.

If it is Python, then I would be surprised, since I've seen the converter handle 900,000 post forums.
The SQL Import goes fine, It is really python that is giving these errors. It only chokes on those threads (the 45 000 + posts ones). It downloads everything fine, it's in the processing that the errors appears. I'll post the error log a bit later. This as been tried on two different computers, with differents providers. One system had 2 gigs mem and a 4gig swap, just to be on the safe side. We use the latest stable Python builds, 2.5.2 and 2.6
Last edited by PPV on Thu Oct 02, 2008 1:19 pm, edited 2 times in total.
User avatar
PPV
Registered User
Posts: 57
Joined: Tue Sep 30, 2008 3:08 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by PPV »

Here is the error it gives:
Topic ID 163
Traceback (most recent call last):
File "C:\Converters\InvisionFree\topics_posts_phpBB.py", line 260, in <module>
processed_data=re.sub(search,repl,processed_data)
File "C:\Python25\lib\re.py", line 150, in sub
return _compile(pattern, 0).sub(repl, string, count)
MemoryError
It fails ine the middle of processing.
nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo »

Huh.

Well, I suppose this is probably because of the aforementioned giant threads. At 100KB per page, you'd probably be looking at a monstrous 50MB thread to process.

Here's a possible fix, but I have not tested this thoroughly.

Replace

Code: Select all

        pagedata=download_page(URL+'?showtopic='+str(topicid),progressline, COOKIEDATA)
        pages=re.findall(re_pages,pagedata)
        if pages:
            perpage=int(pages[0][0])
            np=int(pages[0][1])
            for offset in range(1,np):
                progressline=statusline+"Downloading Extra Page %i of %i - "%(offset,np-1)
                pagedata+=download_page(URL+'?showtopic='+str(topicid)+'&st='+str(offset*perpage),progressline, COOKIEDATA)
        processed_data=pagedata
        cn=0
        for search,repl in re_repstrings:
            cn+=1
            printline(statusline+"Processing Step %i of %i"%(cn,len(re_repstrings)))
#           file("%i_%i.txt"%(topicid,cn),"w").write(processed_data)
            processed_data=re.sub(search,repl,processed_data)
by

Code: Select all

        pagedata=download_page(URL+'?showtopic='+str(topicid),progressline, COOKIEDATA)
        processed_data=pagedata
        cn=0
        for search,repl in re_repstrings:
            cn+=1
            printline(statusline+"Processing Step %i of %i"%(cn,len(re_repstrings)))
            processed_data=re.sub(search,repl,processed_data)
        pages=re.findall(re_pages,pagedata)
        if pages:
            perpage=int(pages[0][0])
            np=int(pages[0][1])
            for offset in range(1,np):
                progressline=statusline+"Downloading Extra Page %i of %i - "%(offset,np-1)
                pagedata=download_page(URL+'?showtopic='+str(topicid)+'&st='+str(offset*perpage),progressline, COOKIEDATA)
                temp=pagedata
                cn=0
                for search,repl in re_repstrings:
                    cn+=1
                    printline(statusline+"Processing Step %i of %i"%(cn,len(re_repstrings)))
                    temp=re.sub(search,repl,temp)
                processed_data+=temp
This will process each page as it comes in. Since processing strips a ton of HTML, it is likely that this will reduce the size of the string and hopefully allow the conversion to succeed.

Please let me know if this hack works, so I can implement it to the converters (since it is a serious enough problem that it should be fixed)
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.
User avatar
PPV
Registered User
Posts: 57
Joined: Tue Sep 30, 2008 3:08 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by PPV »

Thanks! will try this and keep you posted. 8-)
User avatar
PPV
Registered User
Posts: 57
Joined: Tue Sep 30, 2008 3:08 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by PPV »

The dump reported no problem whatsoever. Am currently importing the database (wich is now 4 times the size) to see if everything went smothly.

So far, so good! 8-)
User avatar
PPV
Registered User
Posts: 57
Joined: Tue Sep 30, 2008 3:08 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by PPV »

Seems to have worked like a charm, I found nothing missing! 8-) Thank you very much for your help!
Skipper29
Registered User
Posts: 1
Joined: Fri Oct 03, 2008 6:55 pm

Re: Proboards/Invisionfree/ActiveBoards converters

Post by Skipper29 »

Hi,

i have a proboards account, and am wanting it to go on to a PHPbb 2 forum. it all looks rather complicated for me. anyone want to earn some $$/££ to do it for me?

removed
Last edited by stickerboy on Sun Oct 05, 2008 9:55 pm, edited 1 time in total.
Reason: Removed email
User avatar
stickerboy
Former Team Member
Posts: 7349
Joined: Mon Mar 29, 2004 2:27 pm
Location: Scotland
Name: Kenny Cameron
Contact:

Re: Proboards/Invisionfree/ActiveBoards converters

Post by stickerboy »

Skipper29 wrote:Hi,

i have a proboards account, and am wanting it to go on to a PHPbb 2 forum. it all looks rather complicated for me. anyone want to earn some $$/££ to do it for me?
We don't allow solicitation on these forums, as per our forum rules. If you have any specific question or queries, we are happy to help you :)
I'm a web-designing code-decrypting tech-support musician
|| Twitter || Flickr || phpBB Snippets ||
Formerly known as cherokee red
sienile
Registered User
Posts: 2
Joined: Tue Oct 07, 2008 4:44 am

Re: Proboards converter

Post by sienile »

I'm trying to convert a ProBoards forum to a phpBB forum. I downloaded the converter.zip file, but I don't see a converter that says it's for ProBoards... am I just missing it or do I need a different file?

Also, I'm new to both forum formats... (I'm doing this conversion for a friend. And I only have experience with a ASP/Access DB forum.) ...Is there anything I should know before I start this conversion?
sienile
Registered User
Posts: 2
Joined: Tue Oct 07, 2008 4:44 am

Re: Proboards converter

Post by sienile »

I managed to get ahold of the ProBoards converter. I ran the code and it works great with one very serious, but probably easily fixed, problem.

The topic titles do not download. I'm converting to phpBB2.

I checked the SQL files to make sure that the topic titles were not in there. Some how it's not fetching them from the ProBoards forum. Any ideas on what's causing this and how I can fix it? Without the topic titles I can't view the topics unless they have multiple pages.
nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo »

BOARDNAMEs with unusual (i.e. non-alphanumeric characters or spaces) may fail to be correctly recognized (the converter pulls topic titles from the <title> of the page).

If that happens, then you should change the line

Code: Select all

("""<title>%s - ([^<]+?)</title>"""%BOARDNAME,"""<^>TITLE<|>\\1</^>"""),
to

Code: Select all

("""<title>%s - ([^<]+?)</title>"""%re.escape(BOARDNAME),"""<^>TITLE<|>\\1</^>"""),
in the converter and try again. I would appreciate it if you could PM me the BOARDNAME so I can fix the converter.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.
psywildfire
Registered User
Posts: 2
Joined: Mon Nov 24, 2008 11:55 pm

Re: Proboards/Invisionfree/ActiveBoards converters

Post by psywildfire »

I'm looking for some assistance with the Zetaboard converter.

Running the scripts seem to be successful, but it only seems to be capturing about a tenth of the topics.

I'm currently trying to get this to load correctly on my local computer so I can proceed with the conversion elsewhere. Basically doing a dry run.

I will provide board address upon request.

Thanks in advance!

-WF
nneonneo
Registered User
Posts: 549
Joined: Sun Apr 30, 2006 1:42 am

Re: Proboards/Invisionfree/ActiveBoards converters

Post by nneonneo »

I've never been able to test the Zetaboards converter on a "real", non-test board, so there will probably be bugs. PM me your board address (and preferably also an account to test with) and I will try it out.
Need a conversion from another forum, but they won't give you the database? Try a crawler converter. If your converter isn't listed, feel free to post in that thread to ask for one.
Locked

Return to “[2.0.x] Convertors”