[CDB] Spam remover

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Scam Warning
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

[CDB] Spam remover

Post by MarkDHamill »

This extension uses the Akismet web service to find, flag and remove spam on your phpBB board. Akismet is mostly used to moderate comments on WordPress sites. Note that there is a phpBB Akismet extension already written by Senky. However, its use is strictly to handle new spam using Akismet. In contrast, this extension finds and lets you remove old and embedded spam.

phpBB has a number of spambot countermeasures, but it has no ability to clean up old spam other than depending on users to flag it and moderators to delete it. Particularly on boards that have a lot of old legacy spam, it’s nearly impossible to find, flag and remove all the spam.

Enter this extension: Spam Remover. It can go through all your posts, as well as private messages, and flag the spam as Akismet judges it. You can review the spam and selectively unflag posts and private messages you judge not to be spam. Afterward, or instead of this, you can use the bulk remove spam feature to quickly delete the spam.

Use of the Akismet web service is not necessarily free. It can be used for free on a personal board by selecting $0 for the price, but they will encourage you to make a donation. Otherwise, you have to pay a fee, either monthly or annually, to use the service. So although this extension is free to use, the Akismet service isn’t necessarily free to use. You must obtain a license key from Akismet first, and enter the key on the extension’s settings page before you can find and remove spam with this extension.

If you using Akismet already for a WordPress instance, you can use that same key here. If you have to pay to use Akismet, you can pay for only one month. Or you may want to pay for an annual subscription and use Senky’s extension to find and remove newly submitted spam.

Status

This is currently a release candidate extension. As such it should not be used on a live board. It is highly advised that you make a backup of your entire database before removing any spam from your board.

Screenshots

You can tour the user interface by looking at the screenshots below. Note that on the screenshot for the private messages details report, the private message text has been blurred out. An administrator can read the private message text on this screen.
Settings
Settings
Overview
Find spam
Find spam
Find progress meter
Find progress meter
Spam summary
Spam summary
Spam posts details report
Spam posts details report
Spam private messages details report
Spam private messages details report
Bulk spam removal
Bulk spam removal
Reset spam data
Reset spam data
Privacy considerations

In order to make a spam determination, the post or private message content must be sent to Akismet. Currently this extension does not distinguish between content in public and private forums. This feature may be added in the future. I assume but don’t know for certain that Akismet won’t include in its database any content not considered spam that you send it.

In addition, if a private message is flagged as spam, you have the ability to read the private message content.

Functions

There are seven modules of discrete functionality with this extension:

If any of these concerns are deal breakers, you should not use this extension.
  • Settings. Here you enter your license key and control some basic settings.
  • Find spam. Your selected posts and/or private messages are sent to the Akismet service for a spam judgment. You can constrain the time range of posts and private messages to scan in the user interface, or opt to scan only posts, or only private messages. The screen will refresh after a batch of posts or private messages are processed. It can take hours, or in some cases, days to find all the spam on a very large board. Akismet will process about four posts or private messages per second. Consequently, you often will want to scan for posts from a date when you started noticing they were occurring. On a test board with 58,000 posts and 300 private messages, it took about 4.25 hours for all the content to be processed.
  • Spam summary. After you run the find spam function, this will provide a summary of the spam found. If you didn’t have all your posts and private messages scanned, it will only show statistics for the range of posts and private messages that you had Akismet scan. Akismet distinguishes between probable spam and blatant spam. Probable spam should be manually reviewed in case Akismet made an error. Akismet says that blatant spam is safe to remove without review.
  • Spam posts details report. You can review the spam posts that were found, and filter them based on whether you want to review just probable or blatant spam. The post text is shown to help in your assessment. There are a number of sort options as well. When you flag a post as “ham” (not spam), Akismet is notified to improve the accuracy of its database.
  • Spam private messages details report. The interface is the same as for the spam posts details report. Note that the private message text can be seen by the administrator. This is necessary to judge whether it is spam or not.
  • Bulk spam removal. This interface removes anything that was flagged as spam and which you did not flag as ham. You can tell it to remove the blatant spam only. If the post is the first post of a topic, all its replies are removed as well. In addition, if after deletion the poster has no approved posts, their account is deleted as well.
  • Reset spam data. This lets you quickly clear any spam judgments made by Akismet. Afterward, you might want to run the find spam function again.
Downloading the extension

The extension can be downloaded from the extension's page on my website or from the GitHub release branch page. If you download it from Github, place the extension in the /ext/phpbbservices/spamremover folder.
Last edited by MarkDHamill on Sat Sep 18, 2021 7:15 pm, edited 12 times in total.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: [3.3][DEV] Spam remover

Post by 3Di »

🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][DEV] Spam remover

Post by MarkDHamill »

It's a fine extension, it just can't find legacy spam. I've suggested it as a feature but last I checked it hadn't been added. I regularly have clients that need to get rid of old, legacy spam.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
User avatar
RMcGirr83
Former Team Member
Posts: 22016
Joined: Wed Jun 22, 2005 4:33 pm
Location: Your display
Name: Rich McGirr

Re: [3.3][DEV] Spam remover

Post by RMcGirr83 »

3Di wrote: Sun Jan 24, 2021 4:10 pm What's wrong with this? https://www.phpbb.com/customise/db/extension/akismet/
To apply your own responses that you have posted in topics, please read the first post and try to comprehend it.
Note that there is a phpBB Akismet extension already written by Senky. However, its use is strictly to handle new spam using Akismet. In contrast, this extension finds and lets you remove old and embedded spam.
Former Modifications/Extensions Team Member | My extensions | github | All requests for support via PM will be ignored
Appreciate the extensions/mods/support then buy me a beer Image
User avatar
GTI
Registered User
Posts: 300
Joined: Mon Aug 24, 2020 4:59 pm
Location: West Sussex
Name: Laura

Re: [3.3][DEV] Spam remover

Post by GTI »

RMcGirr83 wrote: Sun Jan 24, 2021 10:26 pm
3Di wrote: Sun Jan 24, 2021 4:10 pm What's wrong with this? https://www.phpbb.com/customise/db/extension/akismet/
To apply your own responses that you have posted in topics, please read the first post and try to comprehend it.
Note that there is a phpBB Akismet extension already written by Senky. However, its use is strictly to handle new spam using Akismet. In contrast, this extension finds and lets you remove old and embedded spam.
Excuse me for butting in but I believe MarkDHamill added that note after 3Di asked the question.

I read this topic with great interest not long after it was posted and I'm sure the Akismet extension wasn't mentioned originally.
Most of us will enjoy Christmas in the comfort of our own homes this year, however, sadly, that can't be said for people in the Ukraine who are suffering day after day, due to the Russian invasion.

If you would like to make a small donation to help Ukrainians in their time of need, you can donate to help all Ukrainians via BritishRedCross or you can donate to help the Ukrainian children via Unicef. 🇺🇦 Thank You 🇺🇦
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][DEV] Spam remover

Post by MarkDHamill »

GTI wrote: Sun Jan 24, 2021 10:53 pm I read this topic with great interest not long after it was posted and I'm sure the Akismet extension wasn't mentioned originally.
I copied and pasted this from my web page, so yes it was there originally.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
User avatar
GTI
Registered User
Posts: 300
Joined: Mon Aug 24, 2020 4:59 pm
Location: West Sussex
Name: Laura

Re: [3.3][DEV] Spam remover

Post by GTI »

MarkDHamill wrote: Sun Jan 24, 2021 11:16 pm
GTI wrote: Sun Jan 24, 2021 10:53 pm I read this topic with great interest not long after it was posted and I'm sure the Akismet extension wasn't mentioned originally.
I copied and pasted this from my web page, so yes it was there originally.
My mistake, sorry for spamming your topic,

Nonetheless this looks like a good extension, will there be a version that doesn't send the content of PM's to Akismet? My forum members use PM's a lot so if they realised the PM content was going to be sent they may run a mile, great way for removing the spam in the public forums though.
Most of us will enjoy Christmas in the comfort of our own homes this year, however, sadly, that can't be said for people in the Ukraine who are suffering day after day, due to the Russian invasion.

If you would like to make a small donation to help Ukrainians in their time of need, you can donate to help all Ukrainians via BritishRedCross or you can donate to help the Ukrainian children via Unicef. 🇺🇦 Thank You 🇺🇦
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][DEV] Spam remover

Post by MarkDHamill »

The interface already allows you not to check private messages. Spam is not removed until checking is complete so if you don't check private messages, none will be removed.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
User avatar
ViolaF
I've Been Banned!
Posts: 1609
Joined: Tue Aug 14, 2012 11:52 pm

Re: [3.3][DEV] Spam remover

Post by ViolaF »

Thanx 2 Mark for this Great One :D
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][DEV] Spam remover

Post by MarkDHamill »

I'm a little surprised that there's been virtually no discussion. But this does require very special test conditions and spam is less of a problem now that CAPTCHAs have improved. Getting rid of legacy spam has been a problem for a lot of my clients, and there aren't any tools to do a comprehensive spam find other than manual inspection.

But I did come across one error. In /controller/acp_controller.php line 790 is:

Code: Select all

								enable_bbcode, 	enable_smilies, enable_magic_url, topic_first_post_id,
and should be:

Code: Select all

								enable_bbcode, enable_smilies, enable_magic_url, topic_first_post_id, bbcode_uid, bbcode_bitfield,
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
Midway
Registered User
Posts: 76
Joined: Tue Dec 09, 2014 8:02 pm

Re: [3.3][DEV] Spam remover

Post by Midway »

Hello,

I think this extension is good but to get a free license you also have to pay which is a shame.
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][DEV] Spam remover

Post by MarkDHamill »

If the use is for a personal board, it should be possible to get a free license. In addition, you should be able to pay for only a single month during which you can deal with the spam issue.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
User avatar
3Di
I've Been Banned!
Posts: 17538
Joined: Mon Apr 04, 2005 11:09 pm
Location: I'm with Ukraine 🇺🇦
Name: Marco
Contact:

Re: [3.3][DEV] Spam remover

Post by 3Di »

Midway wrote: Mon May 17, 2021 3:30 pm Hello,

I think this extension is good but to get a free license you also have to pay which is a shame.
2021-05-18 19_50_19-Select an Akismet Subscription.png
🆓 Free support for our extensions also provided here: phpBB Studio
🚀 Looking for a specific feature or alternative option? We will rock you!
Please PM me only to request paid works. Thx. Buy me a coffee -> Image
My development's activity º PhpStorm's proud user º Extensions, Scripts, MOD porting, Update/Upgrades
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][RC] Spam remover

Post by MarkDHamill »

A new version 1.0.1 is now available. See the first post for links.

There is no new functionality in this release aside from adding a "Go" button to the spam posts details report and the spam private messages details report. This actually fixes a bug, as it's not new functionality, just functionality that was missing before. Without it to sort rows on these reports you had to click on the submit button which had the effect of taking actions on any posts or private messages you marked as ham.

I've tested in carefully and judge it to be a release candidate. It will be submitted for the extension review team for approval.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
User avatar
MarkDHamill
Registered User
Posts: 4885
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [3.3][RC] Spam remover 1.0.2 released

Post by MarkDHamill »

This extension has had its first review by the extensions review team. It didn't pass, but a lot of suggestions were made. Among the relevant changes made in version 1.0.2:
  • Test mode can now be toggled on or off on the settings page. Before it was a constant. When on, no commit-like actions happen. You still get a spam judgment from Akismet, but when you report spam or ham, Akismet won't be notified. No posts, private messages, topics or users are deleted. This allows the extension to be toured and tested more easily.
  • Bug fix: searches for private messages now use the private message date range specified, not the range for posts.
  • The version's extension is now pulled dynamically from the composer.json file and is used in a user agent identification string sent to Akismet.
  • Bug fix: the log entry LOG_ACP_SPAMREMOVER_BULK_REMOVE_RAN now properly formats information for the log file.
  • Certain database queries were optimized to fetch the data just once.
A full list of file changes can be seen here.

Download links are in the first post.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. Covers through phpBB 3.3.7. eBook and paper versions available.
Locked

Return to “Extensions in Development”