Generic message import

Converting from other board software? Good decision! Need help? Have a question about a convertor? Wish to offer a convertor package? Post here.
Anti-Spam Guide
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

prototech wrote:Did you follow the DIY instructions in install.xml?
Navigate to:
Administration Control Panel > System > Module Management > Administration Control Panel > Posting > Messages
Select “Import Messages” from the Add Module drop-down menu and click “Add module”.
Then click “Enable” on the Module.
Browse to ACP > Posting > Messages > Import Messages and confirm install!
Yes. The "import messages" option isn't there in ACP > Posting > Messages.
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

Oyabun1 wrote:There are a few errors in the install.xml file. If you are not seeing the new menu item on the POSTING tab then you probably haven't installed the module, as explained in the DIY instructions.

To install the module you have 2 options
  1. Delete includes/acp/info/acp_import_users.php and replace it with root/includes/acp/info/acp_import_messages.php. The file was misnamed in the install file so AutoMOD will have renamed it. Then
    import_messages_readme.txt wrote:Go to: Administration Control Panel > System > Module Management > Administration Control Panel > Posting > Messages
    From the Add Module dropdown, select Import Messages
    Click Add Module
    Click Yes
    Click Back to previous page
    Enable the added module
  2. I've created a new install_mod.xml file with some updates and corrections in it as well as an install file that can be run to install the module. Uninstall the MOD using AutoMOD then replace the MOD files with the ones in the zip file. Install with AutoMOD and at the end there will be a link to the install file, click on the link then on a few more buttons and the module will be installed and the cache purged.
Thanks, tried both:
Ad 1:
Delete includes/acp/info/acp_import_users.php is not there. includes/acp/info/acp_import_messages.php is.
Ad 2:
Deleted the MOD, got a not found error on one of the files. But the MOD showed uninstalled. Deleted it.
Next tried to install your new zip. It uploads fine in AutoMOD and AutoMOD says it is ready to install. But when I return to AUTOMOD, the module is not listed - "No uninstalled MODs detected".

I'll try again copying and installing manually.
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

The manual copy and install (of 1.01) worked.
I did a small test run. Turns out the post date can't be today. But it worked!

Thanks everyone for your help. It is greatly appreciated!

One question remains: Is it possible to mark all imported messages as seen for all members?
The reason being I am importing an archived copy of a defunct BBS. There is absolutely nothing "new" in there. :)
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

I am also running into the "invalid integer" SQL error when attempting to import a message from an unknown userid.
Preview says the userid will be treated as 'Anonymous' but that doesn't work.
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

Had to solve the unknown user problem manually by importing all users mentioned in all messages to be imported manually. And changing "guest" in "Anonymous".

The user import will skip users already known so it is an ease way to add those that have posted messages to be imported, but aren't registered for one or another reason.
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

I'm almost done importing 40.000 messages.
There is one annoyance in this otherwise excellent tool: More often than not, it doesn't finish. The import will be done, but the ACP page remains waiting forever. It never comes back with the "Success" page.

My way to work around this has been to open a 2nd browser and refresh the index page of the forum being imported. When the counters for topics and posts don't change any longer, the import is done and you can proceed with the next batch.

I experimented with different batch sizes but that didn't really make a difference. I set the max_execution_time in php.ini to 1800 seconds so it definitely is not running out of time. I can't figure out why it won't return most of the times. Sometimes it does. With the preview, it always does. Perhaps the authors have an idea.

As a final suggestion, I would have been much happier with the tool if it had been documented how to repeatedly invoke the import routine from e.g. a batch file, or a small php program. As it is now, you have to manually key in the file name of each batch and click Submit. That becomes boring rather quickly when you have > 100 batches in one forum.

Please take the above remarks in the way they were meant, as positive criticism in the hope that the authors see these as possible room for improvement.

Thanks very much for the tool, it has made my members very happy to get the archive of their old board back.
User avatar
pieterhuizinga
Registered User
Posts: 48
Joined: Tue Jan 08, 2013 5:28 pm
Name: Pieter Huizinga

Re: Generic message import

Post by pieterhuizinga »

Oh, one last suggestion. The topic counter of rejected topics doesn't increase.
515ae4a7.jpg
515ae4a7.jpg (188.99 KiB) Viewed 18243 times
This probably should be very easy to fix.
TrevorH
Registered User
Posts: 1
Joined: Fri Aug 02, 2013 1:20 pm

Re: Generic message import

Post by TrevorH »

I've been using the 1.0.2 version that someone attached and have fixed a couple of bugs. When an unknown username is found in the import then it blows up with a SQL error due to passing the first letter of the username to the insert instead of a numeric id. Also, if the username is unknown it was trying to set it to anonymous and failing, causing a similar error. This patch fixes both problems in two separate places and also another one where the postcount was trying to increment but getting a php error if the postcount didn't exist. This now checks if it it exists and if not, initialises it to 1.

Code: Select all

--- /home/trevor/import_messages_1_0_2/root/includes/acp/acp_import_messages.php        2013-03-23 22:10:14.000000000 +0000
+++ ../includes/acp/acp_import_messages.php     2013-08-02 03:15:05.179597006 +0100
@@ -701,9 +701,11 @@
                $forum_time  = $topic_elm->message[0]['date'];
                $poster_name = (string)$topic_elm->message[0]['by'];
                $poster_data = &$this->poster_ary[$poster_name];
-               if ($poster_data === false)
+               if (($poster_data === false) || !is_numeric($poster_data['user_id']))
                {
-                       $poster_data = $anonymous;
+                       $poster_data = array('user_id'  => ANONYMOUS,
+                                        'username' => '',
+                                        'colour'   => '');
                        $poster_name = '';
                }
 
@@ -737,9 +739,11 @@
                        $msg_date    = $msg_data['date'];
                        $poster_name = (string)$msg_data['by'];
                        $poster_data = &$this->poster_ary[$poster_name];
-                       if ($poster_data === false)
+                       if (($poster_data === false) || !is_numeric($poster_data['user_id']))
                        {
-                               $poster_data = $anonymous;
+                               $poster_data = array('user_id'  => ANONYMOUS,
+                                                'username' => '',
+                                                'colour'   => '');
                                $poster_name = '';
                        }
                        $sql_data = array(
@@ -780,7 +784,14 @@
                        {
                                if ($post_count)
                                {
-                                       ++$poster_data['post'];
+                                       if (isset($poster_data['post']))
+                                       {
+                                               $poster_data['post'] = 1;
+                                       }
+                                       else
+                                       {
+                                               ++$poster_data['post'];
+                                       }
                                }
                                if ($msg_date > $poster_data['lastpost_time'])
                                {
Patch attached too in case tabs are translated in the inline version.
acp_import_message.patch
(1.51 KiB) Downloaded 702 times
dimuse_dioplut
Registered User
Posts: 1
Joined: Thu Aug 29, 2013 7:10 pm

Re: Generic message import

Post by dimuse_dioplut »

Thanks a lot for your Mod.
Small addition.
If you use (for example) Russian code page to yours topic or forum names, better and, I think, right to use in acp_import_messages.php:

Code: Select all


			if (mb_strlen($topic_title,'UTF-8') > 60)
			{
				$this->errors[] = sprintf($user->lang['MSG_TITLE_TRUNCAT'], $topic_title);
				$message_elm[0]['title'] = mb_substr($topic_title, 0, 60,'UTF-8');
                         }
instead of

Code: Select all


			if (strlen($topic_title) > 60)
			{
				$this->errors[] = sprintf($user->lang['MSG_TITLE_TRUNCAT'], $topic_title);
				$message_elm[0]['title'] = substr($topic_title, 0, 60);
			}

jtaubman
Registered User
Posts: 17
Joined: Thu Aug 09, 2012 3:16 pm

Re: Generic message import

Post by jtaubman »

Just in case anyone comes along and has been using the "WebApp Automated Perl Portal", I did a couple of scripts to convert forums to the XML format for this converter.

cvtforum

Change the $filelist array to include all the forum files you need to import.

Code: Select all

<?php
$header = '<?xml version="1.0" encoding="utf-8" standalone="yes"?><forum-message-list date-format="%m/%d/%Y - %H:%M:%S">';
$footer = "\n".'</forum-message-list>';
$dummy = '<topic forum-name="announcements" title="dummy"><message posted="08/30/13 - 09:00:00" by="jane" ip="0.0.0.0"><![CDATA[ deletethis ]]></message></topic>';

$topic = "\n".'<topic forum-name="%s" title="%s">';
$topicfooter = "\n".'</topic>';
$message = "\n".'<message posted="%s" by="%s" ip="%s">%s</message>'."\n";
$exportfile = $header.$dummy;
$oldforumpath = '../cgi-bin/db/forum/'; 
$storepath = "store/";


$filelist = array(
"forumname.txt",
);

foreach ($filelist as $orgfile) {
$forum = basename($orgfile,'.txt');
$fileforum = $forum;
if (substr($orgfile,0,3) == 'arc') {
 $fileforum = 'arc'.$forum;}
$file =  $oldforumpath.$orgfile;
echo 'from '.$orgfile.'=>'.$forum.'<br/>';
$meslist = file($file);
$exportfile = $header.$dummy;
foreach($meslist as $key => $value) {
  $msgheader = explode('|',$value);
  $title = str_replace('"','\'',$msgheader[1]);  
  $title = str_replace('&quot','\'',$msgheader[1]);  
  $title = str_replace('&','',$title);  

  $exportfile = $exportfile.sprintf($topic,$forum,$title);
  $messagefile = $oldforumpath.'/messages/'.$msgheader[0].'.txt';
  if (file_exists($messagefile)) {
  $posts = file($messagefile);
  foreach($posts as $key2 => $post) {
    $postfield = explode('|',$post); 
    $text = str_replace('<br>',"\n",$postfield[7]) ;
    $text = str_replace('&nbsp;',' ',$text);
    $text = str_replace('<pipe>','|',$text);
    $text = str_replace('"','\'',$text);

    if ($key2 == 0) {
       $text = $text."\nID:".$msgheader[0];
    }
    if   ($postfield[4] == 'admin') {$postfield[4] = 'jane'; };
    $exportfile = $exportfile.sprintf($message,$postfield[3],$postfield[4],$postfield[6],$text);
  }
  } else {echo 'Error:'.$value.'<br />';}
  $exportfile = $exportfile.$topicfooter;
}
$exportfile = utf8_encode($exportfile.$footer);
echo $storepath.$forum.'.xsd'.' file exported<br/>';
 file_put_contents($storepath.$fileforum.'.xsd',$exportfile);
 }
?>
You also need the fixes for missing users.

To do the members I used the old memberlist.pl script to export a CSV file and used the user import option to bring them in.
Inabinet
Registered User
Posts: 1
Joined: Thu Sep 05, 2013 3:47 am

Re: Generic message import

Post by Inabinet »

I just successfully converted around 1,000 messages with this. Thank you very much!
I love the smell of sun
usju
Registered User
Posts: 13
Joined: Sun Nov 30, 2014 8:12 pm

Re: Generic message import

Post by usju »

Great tool! Thank you Giles314 and everyone that made updates and posted useful instructions and examples :D

A little tip of my own (although it might have been mentioned before somewhere):
In order to avoid the import timeouts after around ~30 seconds you can temporarily edit your php.ini file (/etc/php5/apache2/php.ini) and change 'max_execution_time' from 30 to something silly like 3000. I did this on a test machine. It's probably not a good idea to do this on a production machine.
I was able to import a complete subforum with up ~1600 topics and > 22.000 posts in one go.

Unfortunately I did have to use the Support Tool Kit (as mentioned in this thread) to re-parse the BBcode. That works but it seems after I run the re-parse function from the STK all topic authors are set to the name of the last poster in the thread (the topic start date and time remain untouched).

Did anyone experience this issue as well? Is it a bug in the import tool, my input XML or the STK?
Is there anyway to avoid having to use the STK to re-parse BBcode by changing the XML input.

Edit: I think I was able to solve my issue by running:

Code: Select all

UPDATE phpbb_topics SET topic_first_poster_name = (SELECT username FROM phpbb_users WHERE user_id = topic_poster);
from PHPMyAdmin on my test machine.
al4141
Registered User
Posts: 2
Joined: Mon Dec 01, 2014 4:02 pm

Re: Generic message import

Post by al4141 »

usju wrote: it seems after I run the re-parse function from the STK all topic authors are set to the name of the last poster in the thread (the topic start date and time remain untouched).

Did anyone experience this issue as well? Is it a bug in the import tool, my input XML or the STK?

Is there anyway to avoid having to use the STK to re-parse BBcode by changing the XML input.
I know I'm replying to a post that's a year old (exactly a year old!) but I thought it might be helpful to say that I had the same problem. The import of some 28,000 posts, thankfully to a test forum, went well after a lot of XML fiddling but I noticed that many of the BBcodes weren't displaying properly. If I "edited" a post, saving it with no changes, that seemed to fix the issue but I didn't fancy doing that on 28,000 and since new posts worked fine I decided to live with it.

Then I ran the STK re-parse BBcode (before reading this thread!) and saw, as did usju, that each topic's author appeared to be set to the name of the last poster when displayed on the viewforum page (I've added an Author column) although the incorrect author name links to the correct author's profile when clicked.

It is a puzzle and I won't be running the STK re-parse on the live forum.

One other 'fault' seems to be that the number of posts and number of replies in a given topic are displayed incorrectly. For instance, a topic with three posts is shown on the viewforum page as having three replies and the top of the viewtopic page reads: "4 posts". The topic itself displays correctly with the original post and the two replies.
usju
Registered User
Posts: 13
Joined: Sun Nov 30, 2014 8:12 pm

Re: Generic message import

Post by usju »

Thanks for your confirmation. I reported the issues in the STK sub-forum*, but no reply as of yet. The STK has had a related bug with the BBcode re-parce function where it would set the topic author to the 'Anonymous' user. As mentioned, I was able to fix my problem by running the SQL statement posted in my previous post (adapted from the work-around for the 'Anonymous' user problem

I did not notice the problem with the post count, I will check my forum again though.

Btw you did not reply to a year old post but merely a few hours after my original post :D

*) https://www.phpbb.com/customise/db/offi ... pic/133676
usju
Registered User
Posts: 13
Joined: Sun Nov 30, 2014 8:12 pm

Re: Generic message import

Post by usju »

al4141 wrote:One other 'fault' seems to be that the number of posts and number of replies in a given topic are displayed incorrectly. For instance, a topic with three posts is shown on the viewforum page as having three replies and the top of the viewtopic page reads: "4 posts". The topic itself displays correctly with the original post and the two replies.
I think you can fix the reply count like this:

Code: Select all

UPDATE `phpbb_topics` SET `topic_replies` = (`topic_replies` - 1);
UPDATE `phpbb_topics` SET `topic_replies_real` = (`topic_replies_real` - 1);
This will only work on a clean board with only newly imported posts! If you have already tests posts where the post reply value is already correctly set at '0' the SQL will fail when trying to insert a negative number.


Massive disclaimer: I have no idea what I'm doing. Use at your own risks and please make lots of backups!
Last edited by usju on Fri Dec 12, 2014 7:44 am, edited 2 times in total.
Locked

Return to “[3.0.x] Convertors”