Generic message import

[pieterhuizinga]

Did you follow the DIY instructions in install.xml?

Navigate to:
Administration Control Panel > System > Module Management > Administration Control Panel > Posting > Messages
Select “Import Messages” from the Add Module drop-down menu and click “Add module”.
Then click “Enable” on the Module.
Browse to ACP > Posting > Messages > Import Messages and confirm install!

Yes. The "import messages" option isn't there in ACP > Posting > Messages.

[pieterhuizinga]

There are a few errors in the install.xml file. If you are not seeing the new menu item on the POSTING tab then you probably haven't installed the module, as explained in the DIY instructions.

To install the module you have 2 options

1. Delete includes/acp/info/acp_import_users.php and replace it with root/includes/acp/info/acp_import_messages.php. The file was misnamed in the install file so AutoMOD will have renamed it. Then

   Go to: Administration Control Panel > System > Module Management > Administration Control Panel > Posting > Messages
   From the Add Module dropdown, select Import Messages
   Click Add Module
   Click Yes
   Click Back to previous page
   Enable the added module

2. I've created a new install_mod.xml file with some updates and corrections in it as well as an install file that can be run to install the module. Uninstall the MOD using AutoMOD then replace the MOD files with the ones in the zip file. Install with AutoMOD and at the end there will be a link to the install file, click on the link then on a few more buttons and the module will be installed and the cache purged.

Thanks, tried both:
Ad 1:
Delete includes/acp/info/acp_import_users.php is not there. includes/acp/info/acp_import_messages.php is.
Ad 2:
Deleted the MOD, got a not found error on one of the files. But the MOD showed uninstalled. Deleted it.
Next tried to install your new zip. It uploads fine in AutoMOD and AutoMOD says it is ready to install. But when I return to AUTOMOD, the module is not listed - "No uninstalled MODs detected".

I'll try again copying and installing manually.

[pieterhuizinga]

The manual copy and install (of 1.01) worked.
I did a small test run. Turns out the post date can't be today. But it worked!

Thanks everyone for your help. It is greatly appreciated!

One question remains: Is it possible to mark all imported messages as seen for all members?
The reason being I am importing an archived copy of a defunct BBS. There is absolutely nothing "new" in there.

[pieterhuizinga]

I am also running into the "invalid integer" SQL error when attempting to import a message from an unknown userid.
Preview says the userid will be treated as 'Anonymous' but that doesn't work.

[pieterhuizinga]

Had to solve the unknown user problem manually by importing all users mentioned in all messages to be imported manually. And changing "guest" in "Anonymous".

The user import will skip users already known so it is an ease way to add those that have posted messages to be imported, but aren't registered for one or another reason.

[pieterhuizinga]

I'm almost done importing 40.000 messages.
There is one annoyance in this otherwise excellent tool: More often than not, it doesn't finish. The import will be done, but the ACP page remains waiting forever. It never comes back with the "Success" page.

My way to work around this has been to open a 2nd browser and refresh the index page of the forum being imported. When the counters for topics and posts don't change any longer, the import is done and you can proceed with the next batch.

I experimented with different batch sizes but that didn't really make a difference. I set the max_execution_time in php.ini to 1800 seconds so it definitely is not running out of time. I can't figure out why it won't return most of the times. Sometimes it does. With the preview, it always does. Perhaps the authors have an idea.

As a final suggestion, I would have been much happier with the tool if it had been documented how to repeatedly invoke the import routine from e.g. a batch file, or a small php program. As it is now, you have to manually key in the file name of each batch and click Submit. That becomes boring rather quickly when you have > 100 batches in one forum.

Please take the above remarks in the way they were meant, as positive criticism in the hope that the authors see these as possible room for improvement.

Thanks very much for the tool, it has made my members very happy to get the archive of their old board back.

[pieterhuizinga]

Oh, one last suggestion. The topic counter of rejected topics doesn't increase.

515ae4a7.jpg

This probably should be very easy to fix.

[TrevorH]

I've been using the 1.0.2 version that someone attached and have fixed a couple of bugs. When an unknown username is found in the import then it blows up with a SQL error due to passing the first letter of the username to the insert instead of a numeric id. Also, if the username is unknown it was trying to set it to anonymous and failing, causing a similar error. This patch fixes both problems in two separate places and also another one where the postcount was trying to increment but getting a php error if the postcount didn't exist. This now checks if it it exists and if not, initialises it to 1.

Code: Select all

--- /home/trevor/import_messages_1_0_2/root/includes/acp/acp_import_messages.php        2013-03-23 22:10:14.000000000 +0000
+++ ../includes/acp/acp_import_messages.php     2013-08-02 03:15:05.179597006 +0100
@@ -701,9 +701,11 @@
                $forum_time  = $topic_elm->message[0]['date'];
                $poster_name = (string)$topic_elm->message[0]['by'];
                $poster_data = &$this->poster_ary[$poster_name];
-               if ($poster_data === false)
+               if (($poster_data === false) || !is_numeric($poster_data['user_id']))
                {
-                       $poster_data = $anonymous;
+                       $poster_data = array('user_id'  => ANONYMOUS,
+                                        'username' => '',
+                                        'colour'   => '');
                        $poster_name = '';
                }
 
@@ -737,9 +739,11 @@
                        $msg_date    = $msg_data['date'];
                        $poster_name = (string)$msg_data['by'];
                        $poster_data = &$this->poster_ary[$poster_name];
-                       if ($poster_data === false)
+                       if (($poster_data === false) || !is_numeric($poster_data['user_id']))
                        {
-                               $poster_data = $anonymous;
+                               $poster_data = array('user_id'  => ANONYMOUS,
+                                                'username' => '',
+                                                'colour'   => '');
                                $poster_name = '';
                        }
                        $sql_data = array(
@@ -780,7 +784,14 @@
                        {
                                if ($post_count)
                                {
-                                       ++$poster_data['post'];
+                                       if (isset($poster_data['post']))
+                                       {
+                                               $poster_data['post'] = 1;
+                                       }
+                                       else
+                                       {
+                                               ++$poster_data['post'];
+                                       }
                                }
                                if ($msg_date > $poster_data['lastpost_time'])
                                {

Patch attached too in case tabs are translated in the inline version.

acp_import_message.patch

[dimuse_dioplut]

Thanks a lot for your Mod.
Small addition.
If you use (for example) Russian code page to yours topic or forum names, better and, I think, right to use in acp_import_messages.php:

Code: Select all

			if (mb_strlen($topic_title,'UTF-8') > 60)
			{
				$this->errors[] = sprintf($user->lang['MSG_TITLE_TRUNCAT'], $topic_title);
				$message_elm[0]['title'] = mb_substr($topic_title, 0, 60,'UTF-8');
                         }

instead of

Code: Select all

			if (strlen($topic_title) > 60)
			{
				$this->errors[] = sprintf($user->lang['MSG_TITLE_TRUNCAT'], $topic_title);
				$message_elm[0]['title'] = substr($topic_title, 0, 60);
			}

[jtaubman]

Just in case anyone comes along and has been using the "WebApp Automated Perl Portal", I did a couple of scripts to convert forums to the XML format for this converter.

cvtforum

Change the $filelist array to include all the forum files you need to import.

Code: Select all

<?php
$header = '<?xml version="1.0" encoding="utf-8" standalone="yes"?><forum-message-list date-format="%m/%d/%Y - %H:%M:%S">';
$footer = "\n".'</forum-message-list>';
$dummy = '<topic forum-name="announcements" title="dummy"><message posted="08/30/13 - 09:00:00" by="jane" ip="0.0.0.0"><![CDATA[ deletethis ]]></message></topic>';

$topic = "\n".'<topic forum-name="%s" title="%s">';
$topicfooter = "\n".'</topic>';
$message = "\n".'<message posted="%s" by="%s" ip="%s">%s</message>'."\n";
$exportfile = $header.$dummy;
$oldforumpath = '../cgi-bin/db/forum/'; 
$storepath = "store/";


$filelist = array(
"forumname.txt",
);

foreach ($filelist as $orgfile) {
$forum = basename($orgfile,'.txt');
$fileforum = $forum;
if (substr($orgfile,0,3) == 'arc') {
 $fileforum = 'arc'.$forum;}
$file =  $oldforumpath.$orgfile;
echo 'from '.$orgfile.'=>'.$forum.'<br/>';
$meslist = file($file);
$exportfile = $header.$dummy;
foreach($meslist as $key => $value) {
  $msgheader = explode('|',$value);
  $title = str_replace('"','\'',$msgheader[1]);  
  $title = str_replace('&quot','\'',$msgheader[1]);  
  $title = str_replace('&','',$title);  

  $exportfile = $exportfile.sprintf($topic,$forum,$title);
  $messagefile = $oldforumpath.'/messages/'.$msgheader[0].'.txt';
  if (file_exists($messagefile)) {
  $posts = file($messagefile);
  foreach($posts as $key2 => $post) {
    $postfield = explode('|',$post); 
    $text = str_replace('<br>',"\n",$postfield[7]) ;
    $text = str_replace('&nbsp;',' ',$text);
    $text = str_replace('<pipe>','|',$text);
    $text = str_replace('"','\'',$text);

    if ($key2 == 0) {
       $text = $text."\nID:".$msgheader[0];
    }
    if   ($postfield[4] == 'admin') {$postfield[4] = 'jane'; };
    $exportfile = $exportfile.sprintf($message,$postfield[3],$postfield[4],$postfield[6],$text);
  }
  } else {echo 'Error:'.$value.'<br />';}
  $exportfile = $exportfile.$topicfooter;
}
$exportfile = utf8_encode($exportfile.$footer);
echo $storepath.$forum.'.xsd'.' file exported<br/>';
 file_put_contents($storepath.$fileforum.'.xsd',$exportfile);
 }
?>

You also need the fixes for missing users.

To do the members I used the old memberlist.pl script to export a CSV file and used the user import option to bring them in.

[Inabinet]

I just successfully converted around 1,000 messages with this. Thank you very much!

[usju]

Great tool! Thank you Giles314 and everyone that made updates and posted useful instructions and examples

A little tip of my own (although it might have been mentioned before somewhere):
In order to avoid the import timeouts after around ~30 seconds you can temporarily edit your php.ini file (/etc/php5/apache2/php.ini) and change 'max_execution_time' from 30 to something silly like 3000. I did this on a test machine. It's probably not a good idea to do this on a production machine.
I was able to import a complete subforum with up ~1600 topics and > 22.000 posts in one go.

Unfortunately I did have to use the Support Tool Kit (as mentioned in this thread) to re-parse the BBcode. That works but it seems after I run the re-parse function from the STK all topic authors are set to the name of the last poster in the thread (the topic start date and time remain untouched).

Did anyone experience this issue as well? Is it a bug in the import tool, my input XML or the STK?
Is there anyway to avoid having to use the STK to re-parse BBcode by changing the XML input.

Edit: I think I was able to solve my issue by running:

Code: Select all

UPDATE phpbb_topics SET topic_first_poster_name = (SELECT username FROM phpbb_users WHERE user_id = topic_poster);

from PHPMyAdmin on my test machine.

[al4141]

it seems after I run the re-parse function from the STK all topic authors are set to the name of the last poster in the thread (the topic start date and time remain untouched).

Did anyone experience this issue as well? Is it a bug in the import tool, my input XML or the STK?

Is there anyway to avoid having to use the STK to re-parse BBcode by changing the XML input.

I know I'm replying to a post that's a year old (exactly a year old!) but I thought it might be helpful to say that I had the same problem. The import of some 28,000 posts, thankfully to a test forum, went well after a lot of XML fiddling but I noticed that many of the BBcodes weren't displaying properly. If I "edited" a post, saving it with no changes, that seemed to fix the issue but I didn't fancy doing that on 28,000 and since new posts worked fine I decided to live with it.

Then I ran the STK re-parse BBcode (before reading this thread!) and saw, as did usju, that each topic's author appeared to be set to the name of the last poster when displayed on the viewforum page (I've added an Author column) although the incorrect author name links to the correct author's profile when clicked.

It is a puzzle and I won't be running the STK re-parse on the live forum.

One other 'fault' seems to be that the number of posts and number of replies in a given topic are displayed incorrectly. For instance, a topic with three posts is shown on the viewforum page as having three replies and the top of the viewtopic page reads: "4 posts". The topic itself displays correctly with the original post and the two replies.

[usju]

Thanks for your confirmation. I reported the issues in the STK sub-forum*, but no reply as of yet. The STK has had a related bug with the BBcode re-parce function where it would set the topic author to the 'Anonymous' user. As mentioned, I was able to fix my problem by running the SQL statement posted in my previous post (adapted from the work-around for the 'Anonymous' user problem

I did not notice the problem with the post count, I will check my forum again though.

Btw you did not reply to a year old post but merely a few hours after my original post

*) https://www.phpbb.com/customise/db/offi ... pic/133676

[usju]

One other 'fault' seems to be that the number of posts and number of replies in a given topic are displayed incorrectly. For instance, a topic with three posts is shown on the viewforum page as having three replies and the top of the viewtopic page reads: "4 posts". The topic itself displays correctly with the original post and the two replies.

I think you can fix the reply count like this:

Code: Select all

UPDATE `phpbb_topics` SET `topic_replies` = (`topic_replies` - 1);
UPDATE `phpbb_topics` SET `topic_replies_real` = (`topic_replies_real` - 1);

This will only work on a clean board with only newly imported posts! If you have already tests posts where the post reply value is already correctly set at '0' the SQL will fail when trying to insert a negative number.


Massive disclaimer: I have no idea what I'm doing. Use at your own risks and please make lots of backups!