[RELEASE] Usenet Gateway for vB 2.01

[fastforward]

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.01

Current version is 20010805 for vB2.03

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[george_proost]

Thanks FF

Little Problem with SEARCH.PHP

The new code for ...newusenet search does not seem to work.

I get redirected to the standard search instead of displaying the no results forund.

Any help appreciated ... I'm still looking


==========================================

• 
• download it    --- right click...save target as

[orca]

Great work. Installation went very smoothly. Binary attachments aren't that important for me. I'm still playing abit around.
A great feature would be, that Usenet posters would be given the title Usenet guest or so automatically.

[george_proost]

Bugs !

I seem to be having some trouble. I have upgraded as per the instructions. I seem to be loading quite a bit of data, but no population of the threads is observed. (sometimes... 1 or 2 pop up )

I also restarted from scratch some loads and after a while I get the following :

sh-2.04$ perl www/admin/newnews.pl
Connecting to news.newzpig.com... Connected
Sending authentication info... Authenticated and logged in
Processing outgoing posts
Getting article batch from bit.listserv.ispf-l
No new messages in bit.listserv.ispf-l
inserting new threads from bit.listserv.ispf-l
Processing outgoing posts
Processing outgoing posts
Getting article batch from bit.listserv.tsorexx
No new messages in bit.listserv.tsorexx
inserting new threads from bit.listserv.tsorexx
Processing outgoing posts
Processing outgoing posts
Getting article batch from comp.databases.ibm-db2
Fetching headers of articles 46866 to 46915... 
[B][COLOR=RED]I 46866-46915 = 49 ( I requested 50 ) does this mean that if I request 1, nothing will happen ?
[/COLOR][/B][color=green] yip request one get none[/color]
Processing headers for article 46867... 
[B][COLOR=RED]Where is  46866 ?  Hmm leaves 48 now to process[/COLOR][/B]
Processing headers for article 46868... 
Processing headers for article 46869... 
Processing headers for article 46870... 
Processing headers for article 46871... 
Processing headers for article 46872... 
Processing headers for article 46873... 
Processing headers for article 46874... 
Processing headers for article 46875... 
Processing headers for article 46876... 

<snip>

Processing headers for article 46892... 
Processing headers for article 46893... 
Processing headers for article 46894... 
Processing headers for article 46895... 
Processing headers for article 46896... 
Processing headers for article 46897... 
Processing headers for article 46898... 
Processing headers for article 46899... 
Processing headers for article 46900... 
Processing headers for article 46901... 
Processing headers for article 46902... 
Processing headers for article 46903... 
Processing headers for article 46904... 
Processing headers for article 46905... 
Processing headers for article 46906... 
Processing headers for article 46907... 
Processing headers for article 46908... 
Processing headers for article 46909... 
Processing headers for article 46910... 
Processing headers for article 46911... 
Processing headers for article 46912... 
Processing headers for article 46913... 
Processing headers for article 46914... 
Processing headers for article 46915... 
Fetching article body 46867... OK 
[B][COLOR=RED]Where is  46866 ?  Hmm leaves 48 now to process[/COLOR][/B]
Fetching article body 46868... OK
Fetching article body 46869... OK
Fetching article body 46870... OK
Fetching article body 46871... OK
Fetching article body 46872... OK
Fetching article body 46873... OK
Fetching article body 46874... OK
Fetching article body 46875... OK
Fetching article body 46876... OK
<snip>
Fetching article body 46901... OK
Fetching article body 46902... OK
Fetching article body 46903... OK
Fetching article body 46904... OK
Fetching article body 46905... OK
Fetching article body 46906... OK
Fetching article body 46907... OK
Fetching article body 46908... OK
Fetching article body 46909... OK
Fetching article body 46910... OK
Fetching article body 46911... OK
Fetching article body 46912... OK
Fetching article body 46913... OK
Fetching article body 46914... OK
Fetching article body 46915... OK
Processing article batch...
Requested 49 messages... 0 not available or rejected. 
[B][COLOR=RED]I REQUESTED 50 actually records 48[/COLOR][/B]
inserting new threads from comp.databases.ibm-db2
Processing outgoing posts
Processing outgoing posts
Getting article batch from bit.listserv.asm370
No new messages in bit.listserv.asm370
inserting new threads from bit.listserv.asm370
Processing outgoing posts
Processing outgoing posts
Getting article batch from comp.lang.apl
No new messages in comp.lang.apl
inserting new threads from comp.lang.apl
Processing outgoing posts
Processing outgoing posts
Getting article batch from comp.lang.cobol
No new messages in comp.lang.cobol
inserting new threads from comp.lang.cobol
Processing outgoing posts
Processing outgoing posts
Getting article batch from comp.lang.fortran
No new messages in comp.lang.fortran
inserting new threads from comp.lang.fortran
Processing outgoing posts
Processing outgoing posts
Getting article batch from comp.lang.rexx
No new messages in comp.lang.rexx
inserting new threads from comp.lang.rexx
Processing outgoing posts
Processing outgoing posts
Getting article batch from comp.lang.c++.moderated
No new messages in comp.lang.c++.moderated
inserting new threads from comp.lang.c++.moderated
Processing outgoing posts
Clean disconnection from news.newzpig.com
Purging usenet articles...
Purging usenet threads older than 356 days from bit.listserv.ibm-main... OK
Purging usenet threads older than 356 days from bit.listserv.vse-l... OK
Purging usenet threads older than 356 days from bit.listserv.ispf-l... OK
Purging usenet threads older than 356 days from bit.listserv.tsorexx... OK
Purging usenet threads older than 356 days from comp.databases.ibm-db2... OK
Purging usenet threads older than 356 days from bit.listserv.asm370... OK
Purging usenet threads older than 356 days from comp.lang.apl... OK
Purging usenet threads older than 356 days from comp.lang.cobol... OK
Purging usenet threads older than 356 days from comp.lang.fortran... OK
Purging usenet threads older than 356 days from comp.lang.rexx... OK
Purging usenet threads older than 356 days from comp.lang.c++.moderated... OK
Purge completed.

[george_proost]

I needed to do this to the code to be able to understand it ..
it sorts out the counting problem
(fetchlimit=1 also works)
... the console output I added '+1' text to confirm...and now reads
Fetching headers of articles 46866+1 to 46915...

I still get NO threads populated
the above '+1' issue needs to be resolved.

post_outgoing_t($c,\$newsgroup);

my ($svrfirstmsg,$svrlastmsg,$tmp) = $c->group($newsgroup->{newsgroup});

if ($newsgroup->{lastmsg} eq 0) { 
   $newsgroup->{lastmsg} = $svrfirstmsg; }

elsif ($newsgroup->{lastmsg} > $svrlastmsg) {
   $newsgroup->{lastmsg} = $svrlastmsg; }

$batch_end = $newsgroup->{lastmsg}+$config{batch_limit};
      
if ($batch_end > $svrlastmsg) {
   $batch_end = $svrlastmsg; }

console("Getting article batch from $newsgroup->{newsgroup}\n");
      
if ($svrlastmsg > $batch_end) .......

I'm off to bed

[fastforward]

Those message numbers are not guaranteed to be incremental. Articles are requested from the server based on the current pointer within the group. The server will return the next available message number, NOT the next sequential message number.

As far as the other problem, hold off posting reams of logs and give me a chance to get home and have a look at it.

I'd advise everyone else to wait a few hours before installing... unless it's too late

[george_proost]

update

+notify

[fastforward]

Re: update

Originally posted by george_proost
+notify

What's that supposed to mean?

Also, using big red text demanding that things need resolving doesn't really motivate me to go out of may to fix this.

[fastforward]

Bugfix!

This release fixes one major bug and suppresses the new annoying feature of the new DBD::MySQL module.

v20010713 (for 2.01)

• Posts not getting loaded bug fix. This turned out to be just a misplacement of the next() method. The pointer was never moving to the next message because the code was placed outside the loop.
  
• Suppressed DBI dup key warning messages. I upgraded my modules and noticed a whole bunch of informational warning messages start to appear whenever a duplicate key was violated. The violations are supposed to happen. The script lets MySQL do the work of checking the constaints in the word table. This is far more efficient than trying to figure it out in code (especially on large tables). The messages have been suppressed and you don't need to know about them . This is probably what some people were assuming to be errors. There was no need to worry and indexing need not have been disabled.
  
• Fixed Option description in CP. The 'Suppress Usenet Posts in Stats' is back to front. Without a code change, they will be shown. With the code change, this option acts as a toggle.

If you installed v20010712 last night or today, you just need to replace the newnews.pl and run the SQL in the upgrade script to fix the control panel typo.

Download Here

[fastforward]

Bugfix!

Another couple of little bugfixes.

v20010714 (for 2.01)

• Fix to prevent duplicates being loaded into forums
  
• Fix to stop hashes being displayed in place of angled brackets around address.

If you installed v20010713 today, you just need to replace the newnews.pl.

Download Here

[george_proost]

Dear Paul (FF),

The intention is not to rattle your cage. I'd never do that.
All I wanted to do was make notes etc.. while I went along.
Be assured, you are doing a tremendously good job here.
I want to cooperate.

The red, ... yea is red ... for me outstanding issues
the green, things that seem fixed. due to my muddling around.

It's looking better everey day...

The sad part is that I have been told to move my server, I
apparently crashed a shared server out 3-4 times, a 'kernel panic' they said... whatever that means.

Any ideas for a good ISP.

[orca]

George, try http://www.venturesonline.com

I run the gateway there now for 3 days. No complaint yet. They even installed the modules for me without any problems. I only had to ask for them .

[george_proost]

it is www.venturesonline.com that are dumping me ...

[fastforward]

HostPro/vservers are very good. I used their Freedom servers (semi dedicated) before I moved to a dedicated at Dialtone.

George... just ignore my outburst earlier. I was having a bad day at work (no change there) I apologize.

Has anybody installed the latest with bug fixes yet?

[orca]

I did and I don't seem to have any problem yet. Works smoothly. Well, I don't have console output on anymore. Used to have many "duplicate entry" if there were new posts.

[edit]
Just checked, everything ok.
[/edit]