[RELEASE] Usenet gateway

[fastforward]

Originally posted by TechTalk
Every time i have tried to view the examples or download the file, both http://britishexpats.com and http://www.dbforums.com are down :(
eh? When?

According to my internetseer reports, BritishExpats.com has been down once over the past 2 months sometime between 9am and 10am a few days ago.

How often have you tried? Where from and when?

[chrispadfield]

at this rate you are getting very close to running out of possible features to add :)

[TechTalk]

Im trying both as we speak.

C:\>tracert BritishExpats.com
Unable to resolve target system name BritishExpats.com.

Thats a trace from 24.217.12.87 (home)


[cmiller@server1 cmiller]$ /usr/sbin/traceroute BritishExpats.com
traceroute: unknown host BritishExpats.com


From my dedicated box 209.61.188.107 (other end of the country)

[TechTalk]

About 2 mins after i started writing the above post, both sites became available.

~Chris

[Gilby]

Originally posted by chrispadfield
at this rate you are getting very close to running out of possible features to add :)

There are always new features to be added.

Here are a couple that are on the top of my list:
Support for multi part messages, so only the text part is inserted into the post, and the html version is tossed.
Attachments, if from a newsgroup, then it'll make it a regular attachment in vB. If posted on the forums, a link at the end of the newgroup post is given for where people can download it.
Better text formatting int he newsgroup. (Possibly take the post, and parse the vB code into html and then use the html to text perl module). Good for stuff like lists.

[fastforward]

Originally posted by Gilby
Better text formatting int he newsgroup. (Possibly take the post, and parse the vB code into html and then use the html to text perl module). Good for stuff like lists.
[/list]
I take it you mean the outgoing posts? This is something I haven't really spent much time on yet. But I agree, it's something that needs improving. I take the easy option at the moment and disable BB code & html for outgoing posts in the forum :)

As for incoming posts; there's not much else we can do with that. The posts that look bad in the forums look just as bad on usenet due to crap news clients or people posting using MS Outlook. I'm sure it wouldn't be too hard to handle the dual format messages correctly though. I'll get on it :)

[Gilby]

Originally posted by fastforward

I take it you mean the outgoing posts? This is something I haven't really spent much time on yet. But I agree, it's something that needs improving. I take the easy option at the moment and disable BB code & html for outgoing posts in the forum :)

Yup, I think it'd be cool if the posts that come out of the forum would have good formatting and are better than any other formatting of plain text documents.


As for incoming posts; there's not much else we can do with that. The posts that look bad in the forums look just as bad on usenet due to crap news clients or people posting using MS Outlook. I'm sure it wouldn't be too hard to handle the dual format messages correctly though. I'll get on it :)

Most the html posts that make it on my forums are from AOL users, now that AOL has AOL 6.0 out, they now send html formatted posts instead of plain text out to non-AOL users by default. Of course there is the occasional html post from other people as well.

[tamarian]

Hi Fastforward,

When I switched autopurge on, I got an error from newnews.pl, for invalid isreg field?

Looking up the ugrade sql statements, this field was not added. Shoul it have been there, or is it from previous versions?

[fastforward]

Sorry. my fault. It's the same in mine but I didn't notice :)

You need to change
isreg
in newnews.pl to
regpost

There is only one reference to it on line 856.

It's referring to a column in the thread table. If it's set to 1, it means the thread has been posted in be a registered member. It gets updated in newthread.php and newreply.php. Those files seem to be OK though, so you only need to change the one occurence in newnews.pl.

[tamarian]

Now it purges fine, thanks.

[fastforward]

DO NOT turn on autopuge. Tamarian has uncovered a serious bug that purges non-usenet groups aswell as the usenet groups. This is due to a left join being used to get the forumids.

The fix will be released ASAP.

[fastforward]

The auto-purge bug is now fixed. It will now wait until all news has been collected and then purge ONLY groups that are in the usenet_group table. The links have been updated to point to the new version 2.7.2.

To upgrade, you only need to replace the newnews.pl script.

[Stephan Whelan]

Just installed 2.7.2 and have to say that it all seems to work great...auto-purge as well.

Congratulations again fastforward!

:cool:

[TechTalk]

You dont know how bad this is needed in vb 2.1!

[Stephan Whelan]

fastforward,

well it looks like I was too quick in saying that the auto-purge worked.

I am running v2.7.2 of your hack but it seems that the auto-purge still doesn't work the way it's supposed to.

It is currently deleting anything older than 28 days(what I set in the USENET settings) from all forums - not just the usenet ones.

[fastforward]

It's working on mine.

Check the article_purge function in newnews.pl. It's at the bottom starting at line 851.

It should like exactly like this:

sub article_purge {
my $forums = db_fetch("SELECT forum, newsgroup FROM usenet_group");
while (my($forumid,$newsgroup) = $forums->fetchrow_array) {
my $purge = time() - ($config{auto_purge_days} * 86400);
console("Purging usenet threads older than $config{auto_purge_days} days from $newsgroup...");
my $threads = db_fetch("SELECT threadid FROM thread WHERE forumid=$forumid AND lastpost < $purge".(!$config{auto_purge_reg}?" AND regpost = 0":""));
while (my($threadid) = $threads->fetchrow_array) {
my $posts = db_fetch("SELECT postid FROM post WHERE threadid = $threadid");
while (my($postid) = $posts->fetchrow_array) {
db_execute("DELETE LOW_PRIORITY FROM searchindex WHERE postid = $postid");
}
db_execute("DELETE LOW_PRIORITY FROM post WHERE threadid=$threadid");
}
db_execute("DELETE LOW_PRIORITY FROM thread WHERE forumid = $forumid AND lastpost < $purge");
console(" OK\n");
}
console("Purge completed.\n");
}

[Stephan Whelan]

Here is the section in my newnews.pl...doesn't seem to be exactly the same...


sub article_purge {
my $forums = db_fetch("SELECT forum, newsgroup FROM usenet_group");
while (my($forumid,$newsgroup) = $forums->fetchrow_array) {
my $purge = time() - ($config{auto_purge_days} * 86400);
console("Purging usenet threads older than $config{auto_purge_days} days from $newsgroup...");
my $threads = db_fetch("SELECT threadid FROM thread WHERE forumid=$forumid AND lastpost < $purge".(!$config{auto_purge_reg}?" AND regpost = 0":""));
while (my($threadid) = $threads->fetchrow_array) {
my $posts = db_fetch("SELECT postid FROM post WHERE threadid = $threadid");
while (my($postid) = $posts->fetchrow_array) {
db_execute("DELETE LOW_PRIORITY FROM searchindex WHERE postid = $postid");
}
db_execute("DELETE LOW_PRIORITY FROM post WHERE threadid=$threadid");
db_execute("DELETE LOW_PRIORITY FROM thread WHERE forumid = $forumid AND lastpost < $purge");
}
console(" OK\n");
}
console("Purge completed.\n");


I'm going to change it to your code.

[fastforward]

You must have downloaded that within seconds of me uploading the version!

I replaced it within 2 or 3 minutes of initial upload.

Sorry about that. I didn't think anyone had got to it first. Did you lose any posts?

[fastforward]

This release just updates things for vB 2.0 RC1. There were very few changes made. The main changes were the control panel, to make it look like the new vB style, and the line numbers in the documentation.

Download (http://britishexpats.com/download/usenet_gateway.tar.gz) latest version for vB 2.0 RC1

The latest version of this hack will always be in this first post of the thread.

[Stephan Whelan]

fastforward,

OK - it now is working fine - that will teach me to be too quick at downloading updates!! I didn't lose too much as we only changed to vB from UBB in the last month so just re-imported old posts from the old forums.

Anyway, I do have one question though (I feel you'll learn to hate me:D ) :

- How long does it take you to process the newsfeeds. I have 12 newsgroups running and I have to set my cron schedule to every 2 hours (I might put it back to every 3 hours) as it takes on average around 80 minutes to complete a run at processing all posts. I have around 500 posts in the usenet_article table and the expiry length is set to 2 days in my CP. I'm not to fussed in some respects, but I am a bit concerned that there is almost a constant load on the server...

What are other peoples experiences with this?

[fastforward]

Originally posted by Stephan Whelan
fastforward,
- How long does it take you to process the newsfeeds. I have 12 newsgroups running and I have to set my cron schedule to every 2 hours (I might put it back to every 3 hours) as it takes on average around 80 minutes to complete a run at processing all posts. I have around 500 posts in the usenet_article table and the expiry length is set to 2 days in my CP. I'm not to fussed in some respects, but I am a bit concerned that there is almost a constant load on the server...

Woooah!!

You have a problem! I pull, process and load 70 groups in 5 minutes! When it was on a shared virtual server it used to take up to 15 minutes.

Even when pulling history with batch limit set to 150 it only takes 10-20 minutes! That's over 10,000 posts.

What version of Perl are you using?

[Stephan Whelan]

fastforward,

Perl is 5.005_03.

I've watched the server pull newsgroups stuff through and it pulls the articles very quickly, the problem seems to be when it reaches "Checking for Orphans" and "Loading x Replies" stage.

I've followed details on optimising MySQL from various threads in these forums but doesn't seem to help very much.

It isn't a shared server, it is a cobalt raq 4i with 512 RAM.

[c0bra]

Since trying to apply this hack our forums have been down with the following error:

Parse error: parse error in /home/sites/site3/web/forums/admin/functions.php on line 1729

Fatal error: Call to undefined function: vbsetcookie() in /home/sites/site3/web/forums/admin/sessions.php on line 198

I followed the directions exactly as specified in the INSTALL.TXT file. Modified the required files and then manually uploaded them to the server.

Any ideas on the problem? A prompt solution would be much appreciated since our boards are down at this moment. :eek:

[chrispadfield]

first- replace functions.php with the backup you made to get forums working again.

then look inside functions.php the error might be beginning here:

function vbsetcookie ...


basically you have done something wrong in fuctions.php, missed of a closing bracket would be my guess

[c0bra]

I uploaded the original functions.php and indeed the forum started working again. I then proceeded to go through the process of modifying the functions.php file again double and triple checking eachtime that the changes I made were absolutely correct. However, when I uploaded the file again the error I posted above immediately reappeared.

Maybe someone could attach a copy of their working modified functions.php file so I can take a look?

[chrispadfield]

i havn't done it yet so can't help.

You are using the correct version of vb for the hack, release candidate 1. I could not tell from your forums as the version number is not there.

[fastforward]

Originally posted by Stephan Whelan
fastforward,
Perl is 5.005_03.
I believe this may be your problem. The email::find module requires version 5.005_63 to work at it's optimum. Although having said that I was using it with and older version with no problems. Here is what the module author states:

This module requires 5.005_63 or higher!
This module runs so slow as to be unusable with 5.005 stable. I'm not sure, but it might be because I build up my search regex using lots of compiled regexes. Either way, it runs orders of magnitude faster under 5.005_63.

To confirm my suspicion, try commenting out line 828 of newnews.pl.
It currently reads

find_emails($from, sub { my($e,$orig) = @_; $email = $e->format; } );

It won't make any difference to the functionality unless you are importing usenet users.

Let me know what happens.

[fastforward]

Originally posted by c0bra
Since trying to apply this hack our forums have been down with the following error:

Parse error: parse error in /home/sites/site3/web/forums/admin/functions.php on line 1729

Fatal error: Call to undefined function: vbsetcookie() in /home/sites/site3/web/forums/admin/sessions.php on line 198

This indicates there is a brace, semi colon or some other syntactical error. I'll have a look at my instructions again, although it worked when I did mine using my own instructions.

[fastforward]

I've checked the instructions and it works for me. Not what you wanted to hear but it does :)

You might want to re-check the two edits in the deletethread() and deletepost() functions.

The edits replace the closing brace with another line of code AND a closing brace. Make sure you don't have two closing braces in there.

[c0bra]

Thanks your solution worked for me. :D I created a new category forum called "usenet". Created the groups I wished to display on our VB. Ran the perl script, it connected fine, reported downloading all the messages fine, and disconnected fine. However, when I view our forum homepage the newsgroups are not showing up at all.

Should the newsgroup automatically appear under the chosen category when it's created in admin?

Any ideas?

[fastforward]

You have to create a forum using the vB functions for every usenet group you want. Then you also have to enter the newsgroup and assign it to the vB forum using the usenet control panel page.

If you did that and they're still not showing, make sure the messages you pulled are current and not just hidden from view by the forum settings :)

Be careful that you assign the correct forum to the newsgroup or you may end up with usenet messages in a real forum!

[Stephan Whelan]

Fastforward,


To confirm my suspicion, try commenting out line 828 of newnews.pl.
It currently reads

find_emails($from, sub { my($e,$orig) = @_; $email = $e->format; } );



Hmm - I commented out that line, there seems to be a slight increase in speed but not really anything significant.

I've been watch what is going on with my server and i'm wondering if it is something to do with the way mysql is setup on my system...as i've said i'm running it on a Cobalt RAQ4i that people in these forums seem to have mixed feelings about...whilst studying what is going on with the mysql processing during the long periods the following processes seem to spend a long time:


INSERT LOW_PRIORITY IGNORE INTO usenet_article_loader SELECT DISTINCT a.newsgroup, a.forum, a.msgid


The state it seems to be in a lot is:


Copying to tmp table


Which leads me to believe that mysql isn't perfoming well. This is strange as I have 512 Mb of RAM with 30 GB HDD - I followed instructions on changing some of the sartup parameters for mysql but nothing really seems to have a great impact.

Any suggestions?

[fastforward]

That query probably will use a fair bit memory as it's joining to the post table. This was added to eliminate the need for a unique constraint on the msgid column in the post table. The unique key had a potential to cause problems with vB upgrades and uninstalling the hack etc.

You can remove the DISTINCT clause from the query. That is causing a sort when it's not really required as we're joing on msgid which should be unique in both tables. See if that stops it creating a temp table.

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.

Has the performance always been the same?
If that still doesn't work, you might try removing the LOW_PRIORITY from the statement. It shouldn't make any difference, but you never know.

[Stephan Whelan]

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.


OK - I think I hit gold here...I double checked the indexes on those tables and found the post msgid index missing...I added that and performance has increase by about 1000%! Look like that is the major problem - must have missed that during an upgrade.

I didn't do any other changes...i'm also going to reinstate the line you recommended to comment out of newnews.pl.

[fastforward]

I just noticed the Spam Control page in the control panel isn't refreshing properly after the cosmetic changes made.

I've put a revised usenet.php in the package and made the version 2.8.1.

Only the usenet.php changed so you can just download this one file here and overwrite yours.

The code in here is a total mess and really needs sorting out. I still get the 'headers already sent error' ob the spam page, but it works. I'll look at tidying it up later (when I learn some more PHP :) )

[Mark Hewitt]

I would really like to be able to use this hack on my forums but I'm concerned that it would eat through the bandwidth (data transfer) on my site and cost me a lot of money.

Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?

If I could put it on my site and it would only take (say) 1 or 2GB per month then that would be great but if it was to take (say) 20GB that would be impossible for me.

Thanks

[Stephan Whelan]

Mark,


Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?


Well, I haven't got any firm figures but here are my general thoughts on this:

- First reaction is not much!
- Biggest drain will be when you do your first batch pull as this is getting a lot of posts for history
- Depends on how many newsgroups you pull...
- ...and how active they are
- Since the perl script ignores Binary Posts (or posts above a certain amount of lines) these are not a drain on your bandwidth

Overall I don't think it should be too bad, but i'm sure others have some sort of analysis on this.

[chrispadfield]

i would have thought hardly anything if you are only pulling a text database. All it you are doing is pulling some newsposts and getting the article index, i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

[Mark Hewitt]

Originally posted by chrispadfield
i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

Really! I was expecting much more than that!. Outstanding news if it's accurate. And yes I was thinking about doing about 5 text newsgroups.

PS Does this hack work ok on BETA5? (sorry if thats already covered)

[chrispadfield]

it is only an estimation but if you think about it what other data is there going to be? all the data you are pulling is the actual text posts and they go straight into your db. If you were pulling 1GB of data a month the worry would not be the bandwidth problem but the size of your database! I think matt said recently at sitepoint that a 1,300,000 post database was about 1.5GB or something which should give you an indication... no way will you pull anything near like a GB per month i am pretty certain of it.

About 1 more week i recon until i have another go with this, it has changed a lot since i first installed it on v 1.5 ;)

[fastforward]

Originally posted by Mark Hewitt
PS Does this hack work ok on BETA5? (sorry if thats already covered)
It almost works on beta 5

You may have problems with the usenet.php page that generates the control panel as that changed in RC1. Also the line numbers in the instructions for code changes will ahve changed in many places.

What you can do is download this old version for beta 5 (http://britishexpats.com/download/usenet_gateway_v2.7.2__20b5.tar.gz) but use the newnews.pl from the latest version (http://britishexpats.com/download/usenet_gateway.tar.gz).

NOTE: It's imperitive that you DON'T use the old newnews.pl script from the beta 5 package as it has a serious bug in the auto-purge department! (ask Tamarian... he'll tell you!). Everything except newnews.pl should be used from the old package for beta 5.

As for bandwidth, Chris is right. You can use the the size of the database as a rough guide. I haven't actually looked, but my vserver account only allows 20GB/month and I have 2 sites with 75,000 page views pulling about 80 newsgroups between them and I haven't had a problem. You'll be using megabytes rather than gigabytes.

[Gilby]

I am using multiple servers and posts to one of the servers does not make it to the newsgroup. Looking at the output of running the newnews.pl script, all outgoing posts are trying to be posted on the first newsgroup it processes which for posts that are going to a newsgroup that is not on that server, it means it doesn't make it to the newsgroup but says it was sent. Disabling all the newgroups on one server makes it where the posts go to the right server (but tries to send the messages for the disabled newsgroups too). It does the same for inserting replies where all of them are done on the first newsgroup, but they go to the correct forum, so that's not a big deal.

This has happened since I upgraded to the vb2.0b5 version and I just tried it with the newest newnews.pl script and it occurs with this one too.

I have not investigated further... I would assume it's a coding problem or a problem with the database tables (something not upgraded properly). Does this happen for others that are using multiple servers?

Here is the output from newnews that I get:
Connecting to news.tc.umn.edu... Connected
Sending authentication info... Authenticated and logged in
Getting article batch from rec.sport.unicycling
No new messages in rec.sport.unicycling
inserting new threads into forums
finding replies...
checking for orphans...
Processing outgoing messages
Posting message by Gilby to webdiscuss.test... Sent

You can see that it is trying to post to "webdiscuss.test" while processing for "rec.sport.unicycling". The "webdiscuss.test" newsgroup is disabled and also is on another server. This post did not make it to the newsgroup even though it says "Sent".

[fastforward]

Originally posted by Gilby
I am using multiple servers and posts to one of the servers does not make it to the newsgroup. Looking at the output of running the newnews.pl script, all outgoing posts are trying to be posted on the first newsgroup it processes which for posts that are going to a newsgroup that is not on that server, it means it doesn't make it to the newsgroup but says it was sent.
I've had a quick look and can't see anything obvious, so I'll need to spend some time trying to find out what's wrong. I should have something by tonight.

One point to note though. The sent message only means the server accepted the message. The server will not check to see if it's a valid group first. That validation should really be handled by the script. However, as the groups are pre-defined within vB the chance of sending to an invalid group are practically zero... except in this case :rolleyes:

[fastforward]

I think its due to misplaced parenthesis in the WHERE clause of the post_outgoing() function. As it is at the moment, it will:

either (get messages with no refs) OR (get aanything with a subject that doesn't start with Re: and is in this group)
Therefore it will match any message that has no refs. regardless of whether it's in the right group.

What it should be doing is:

(get messages with no refs or ones that have a subject that doesn't start with Re:) AND (also make sure they are in the newsgroup that is currently being processed)

Here's the fix:

Replace line 750 which currently reads:

my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body,threadid,postid FROM usenet_outgoing WHERE (refs <> ' ') OR subject NOT LIKE 'Re: %' AND newsgroup = ". $dbh->quote($$newsgroup->{'newsgroup'}));

With this line:
my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body,threadid,postid FROM usenet_outgoing WHERE (refs <> ' ' OR subject NOT LIKE 'Re: %') AND newsgroup = ". $dbh->quote($$newsgroup->{'newsgroup'}));

Let me nkow if that fixes it. I can't actually test it at the moment coz I'm busy at work. :p

[fastforward]

The fix mentioned above is now packaged up and the release number is 2.8.2. There was also another little problem with the logic that handled re-importing of forum generated posts that didn't exist in the forum. Unless you had the option to re-import turned on, you wouldn't have noticed this.

[Gilby]

Originally posted by fastforward
Let me nkow if that fixes it.

Works great now! Thanks.

[Mark Hewitt]

<problem solved post removed>

[Mark Hewitt]

When upgrading the board I need to be able to take backups of the database in case things go wrong.

I tried this on moving to a test database but got this

ERROR 1064 at line 2848: You have an error in your SQL syntax near 'ord(ord), KEY isusenetpost(isusenetpost)
) TYPE=MyISAM' at line 26


This is a serious error because it means I can no longer take backups of my database :(

Can anyone tell me a fix or point me to the instructions as to how to fully uninstall the hack from the database? I've got the empty_usenet.pl script but want to make sure it will run with the latest version of the hack, since I can't backup my database :(

[fastforward]

Originally posted by Mark Hewitt
When upgrading the board I need to be able to take backups of the database in case things go wrong.

I tried this on moving to a test database but got this

ERROR 1064 at line 2848: You have an error in your SQL syntax near 'ord(ord), KEY isusenetpost(isusenetpost)
) TYPE=MyISAM' at line 26


This is a serious error because it means I can no longer take backups of my database :(

Can anyone tell me a fix or point me to the instructions as to how to fully uninstall the hack from the database? I've got the empty_usenet.pl script but want to make sure it will run with the latest version of the hack, since I can't backup my database :(
What exactly are you trying to do? What code generated that error? I can still take backups of my database.

To uninstall the hack, simply upload all your original vB files and drop the columns and table that were added to the vB tables. You can see what these were in the installation scripts.

[fastforward]

This release affect newnews.pl only. It fixes a few minor errors.
[list=1]
Problem with email notification not using the vB templates correctly.
Fixed autopurge option to correctly clean out the searchindex table.
Added additional checks to all queries that delete or update data to ensure that only usenet groups are affected.
[list]
You can just overwrite your newnews.pl with the attached file. The new file is also in the full package.

The next release will have a lot of changes that eliminate the need for many of the code edits to vB files. I have already got it working without the need to edit newreply and newthread.php which also means move,copy,split etc will work without problems for any thread.

The other major change is the removal of the option to import usenet users. I think this is a waste of time and space and serves no real purpose except the ability to search by username. For the extra code edits and problems that this introduces, I don't think it's worth it. Anybody have a problem with that?

My aim is to limit the code changes to a state that allows vB upgrades to happen with no loss in functionality to either vB or the hack (except maybe some inevitable display issues such as count of usenet posts etc)

[tamarian]

Originally posted by fastforward

The next release will have a lot of changes that eliminate the need for many of the code edits to vB files. I have already got it working without the need to edit newreply and newthread.php which also means move,copy,split etc will work without problems for any thread.

Can't wait for this! :)

[Mark Hewitt]

Originally posted by fastforward
What exactly are you trying to do? What code generated that error? I can still take backups of my database.
[quote]

Just dumping the database and attempting to reupload it to an empty database.

[quote]
To uninstall the hack, simply upload all your original vB files and drop the columns and table that were added to the vB tables. You can see what these were in the installation scripts.

Thanks, I'll do that.

I did notice one bug which you may or may not be aware of, when I tried to 'update counters' for last posters etc, it threw a database error when it found a usenet post with a / or a ' character.

I can't wait until your next version of this hack :cool:

[TechTalk]

If you make a version of this that can run without editing any vbfiles (other than templates) i will bow to you.

I really want to use it but I refuse to edit vb files so when something goes wrong their the only ones to blame :)

Cant wait for the next version :)

[fastforward]

This is a first release of the new newnews.pl lite :)

RC2 came out a bit sooner than expected, so I rushed out an early version.

The good news:

Only one vB code edit to make! And that is just to add a link for the control panel on the admin/index page.
Discovered and fixed some more bugs and efficiency issues
Reduced the number of additional columns added to vB tables


The bad news:

You have to work with your templates to hide unnecessary fields such as post counts and location etc. As there are no conditional statements or constructs alowed in templates you will have to create a new styleset and assign it to the usenet forums. This could actually be seen as an advantage.
Due to the unexpected release of RC2, this release doesn't provide a way to display a seperate count of usenet posts on the front page. Neither does it allow posts to be hidden from the 'new posts' search. I will try to figure a way to do this without editing the code.
Finally, there is no extra permission flag to prevent posts propogating to usenet. This option was a little pointless anyway and could be confusing. If you don't want people posting to usenet then just don't let them post in the usenet forum. Use the vB permissions for it.


More good news: I believe it's possible to have a fully featured usenet gateway with zero code edits to vB files.

Once again, this release was a little rushed. I believe the posting and threading are working correctly, but there may be some other issues I've overlooked so use it with care. It shouldn't really affect any of the vB functionality as we don't touch the code anymore.

One more thing, as mentioned in an earlier post, the 'import user option' is gone.

The package has been updated in the first post of this thread or you can get it here (http://britishexpats.com/download/usenet_gateway.tar.gz).

If you want to continue using the old one with all the code edits, you can still get it by calling up a directory listing at http://britishexpats.com/download and picking the version you want.

[Mark Hewitt]

Since this doesn't edit vbulletin code anymore is the last thread title "bonus" still included (I assume not).

I use this on my board, is there anywhere I can get instructions as to the code changes needed to implement this on it's own?

[fastforward]

Originally posted by Mark Hewitt
Since this doesn't edit vbulletin code anymore is the last thread title "bonus" still included (I assume not).

I use this on my board, is there anywhere I can get instructions as to the code changes needed to implement this on it's own?
Ah.. I didn't think anyone was using this. I'l get the instructions out later today. It's quite easy and only needs editing in one place (indexpost()) if I remember correctly.

[tamarian]

Fastforward, great job. I just looked up the instructions. This is virtually no code change? If I use phpMyAdmin to enter my news server info and options, or if the settings are entered in newnews.pl, no code changes are necessary?

[fastforward]

Originally posted by tamarian
If I use phpMyAdmin to enter my news server info and options, or if the settings are entered in newnews.pl, no code changes are necessary?
Yep. That's right. I did actually have a go at changing the config page to work standalone which would mean you wouldn't even have to make the one little change left, but it was getting late and I broke it :) I'll give it another go if I get time before RC3 or vB final is released.

I've figured an easy way to get the usenet post counts and any other config setting you make in the usenet control panel into a global variable that can be referenced in templates. I can just append them to the options template during each news pull. The only drawback to that is if you update the vB config using in between news pulls, the usenet variables will be overwritten until the next news pull set them again. It won't be a problem as long as you remember to resubmit the usenet config straight after you update the vB config. And if you really can't remember to do that, it would only be one little optional code change to functions.php to automate it. I'll try to find another way, but that method is favourite at the moment.

I still haven't figured out a way to eliminate the usenet posts from 'new post' searches without code edits. I'm sure there must be a way though :)

[Mark Hewitt]

I've tried the new version but all the posts have the date/time of when they were inserted into the database rather than the correct date when they were posted.:(

[fastforward]

Originally posted by Mark Hewitt
I've tried the new version but all the posts have the date/time of when they were inserted into the database rather than the correct date when they were posted.:(
That is how it's supposed to work. Otherwise, they won't be threaded correctly when local posts are in the same thread as usenet posts.

If you want it the old way then you will have to go back to the old method of editing the vB code in newreply.php, newthread.php, postings.php and functions.php. Which would you prefer?

The time of posting is not very reliable anyway. It's not even the NNTP time. It's based on the users PC from which the post was made. With this method, the post times are actually going to be more consistent. Plus. if you're pulling news 2 or three times an hour, there won't be much discrepency between the original post (except when the original post was wrong).

[Gilby]

Originally posted by fastforward
That is how it's supposed to work. Otherwise, they won't be threaded correctly when local posts are in the same thread as usenet posts.

I like the old way better, where the date of the post is the same as the date in the message on the newsgroup. Maybe some date checking would be appropriate such as making sure the date isn't in the future and that it's not older than a few hours or since the last time newnews was run, whichever is older.

Let's say that either the news server or your own server was down for a while. You don't want to have the messages that were not loaded earlier being the same date as all the others. Sometimes it takes a while for messages to make it to the a particular news server and they might come out of order too.

[fastforward]

With this method, posts will appear just as they do using a newsclient against a newserver. The news server simply displays them as it receives them.

If the date of the news post is used and placed into the dateline field, that is how vB orders the posts. This means they will NOT be shown in the correct order due to the time taken for news to propogate across usenet. You will get usenet posts coming in that are placed in the middle of threads. Also, they may not show up as new posts as they may be dated before your last visit. It becomes impossible to keep track of long threads.

The other problem is when a local reply is posted in a usenet thread. You will also get a whole bunch of usenet posts placed above it when the next run occurs. The local poster gets an email telling them someone has replied and can't see any posts after their one. Very confusing and frustrating.

These example aren't isolated occurances. It happens all the time. In fact, the very first version of this hack did exactly what you are asking for. It was changed for the reasons above.

The way it was handled in the more recent versions was by using the 'seq' (sequence) field. The thread was orderd using this column which meant everything was shown in the right sequence (although the message dates were not necessarily in order). This was achieved by editing the showthread, newreply, newthread and posting.php.

As far as checking goes, this was also done in the previous version to minimize the message dates appearing strange and out of sequence, and also to make sure posts were shown as new in a timely fashion. But, it's only of use when combined with the vB code changes.

Basically, you have a 3 choices.

Use the news post date with no vB code changes and have posts that you may never see as they will be inserted in the middle of threads and not be marked as new.
Use the method of setting the date to the date they are inserted into the forums. (Just as a real posts do). The posts are in the correct order, but have a 'wrong' date.
Use the newspost date along with all the code changes that are necessary to support it and the inability to upgrade to newer vB versions without breaking the hack, vB, or both.

[chrispadfield]

trust me, you want to stick with this way! getting posts going above native VB posts is incredibly confusing and as fastfoward says it happens all the time. Newsgroups themselves don't sort by date/time, VB shouldn't do either for newsgroup posts and threads.

[Mark Hewitt]

Use the news post date with no vB code changes and have posts that you may never see as they will be inserted in the middle of threads and not be marked as new.

That is actually how I would like it to work, would it be possible to set this as an option in the usenet.php control panel as I really don't like the way it is done now.


OK, that "bug" now found to be a "feature" :). I have another one for you :rolleyes: !

The names don't appear in the individual posts. Even though they do appear in the thread starter and last poster columns on forumdisplay.

Also I don't know if this is to do with your code but threading on outgoing posts doesn't seem to be working correctly. There was a long thread with 5 replies and I clicked reply and replied to the message. I expected the post to be in reply to post number 5 instead it appeared in reply to the first post.

[fastforward]

Originally posted by Mark Hewitt
Also I don't know if this is to do with your code but threading on outgoing posts doesn't seem to be working correctly. There was a long thread with 5 replies and I clicked reply and replied to the message. I expected the post to be in reply to post number 5 instead it appeared in reply to the first post.
This isn't possible without some major code changes. vB is not a threaded bulletin board. If vB doesn't know or care which post is being replied to, it's impossible for an external script to figure it out.

It would be possible to figure out if an individual post was being replied to by using the quote feature, but that would mean more vB code changes and it would only work if the quote was used.

This hack has always only 'threaded' incoming messages by placing them in the correct sequence and a logical order. Outgoing messages have always been in reply to the first post.

As for the name not appearing, I'll have to look into that some more. They seem to be showing up on mine.

[Mark Hewitt]

This isn't possible without some major code changes. vB is not a threaded bulletin board. If vB doesn't know or care which post is being replied to, it's impossible for an external script to figure it out.


Ah right, that's a pity because it will look quite strange on usenet. (There quite a fickle lot over on the newsgroups). Can't newreply.php be modified to include a "in reply to" tag which would handle this. (code changes aren't that bad really!).


As for the name not appearing, I'll have to look into that some more. They seem to be showing up on mine.

Thanks, see example here -
http://www.motorsportforum.com/vb22/upload/showthread.php?s=&threadid=235

[fastforward]

Originally posted by Mark Hewitt
Can't newreply.php be modified to include a "in reply to" tag which would handle this. (code changes aren't that bad really!).

It could. But the whole point of this 'new' usenet hack is to eliminate all code changes to ease support issues for everyone and allow timely upgrades to new vB versions. It also means I don't have to stay up until 4am the day Jelsoft releases a new version :)

Remember this new version is still in it's infancy after stripping out all the code changes. A lot of the missing features will be readded... and still with no code changes. I just need time to figure out how.

By the way, I can add an option for the usenet post date issue. But remember; new posts may be placed before recent posts. The dates will obviously be in sequence, but you may receive the newer post first. The real problem is when local posts are entered into a thread along with usenet posts. You will nearly always get usenet posts placed before it. I'll add the option with a disclaimer.

[tamarian]

Fastforward,

Here's a suggestion. No vb code changes, and minimal usenet customization can be the basic usenet hack. A list of mods to the hack can be updated, by you and other experienced users of the hack. Like a text in the hack zip with a format like:

1. Search: To exclude usenet posts from being displayed in view new, find this in search.php, replace with, etc.

2. Stats to eclude from stats, do this...

bla bla. Each needed enhancement can be hacked by itself, while the simplest usenet hack won't need any code changes.

I can see this hack taking a lide of it's own!

[Mark Hewitt]

I would be in favour of that but I doubt fastforward would since he is doing this for free after all!

However another thing I found was that wereas previously the usenet posts didn't appear in my overall post total they do now, this is bad.

[tamarian]

Originally posted by Mark Hewitt
I would be in favour of that but I doubt fastforward would since he is doing this for free after all!


We probably won't go through another repetitive upgrade cycle like vb1.6-vb2.0 for a while. Some mods to the Usenet hack are quite simple, especially if you hard code them instead of using the CP. Like adding "AND forumid <>" in search to control search.

I actually use the 2.8.2 of the hack on top of RC2, so it still behaves like it used to, and will install 2.9 when vb2 is released, with mods for stats msg count and seach behaviour on the usenet messages.

[fastforward]

Tamarians suggestion isn't too bad actually. It will still only be one version. It will just be up to the user how many of the optional code changes they wany to make to integrate it with vB. It will make the newnews.pl a bit more complicated due to the additional if/elses etc but it shouldn't be a big deal. I still intend to drop the 'import user' function.

Does that sound like a resonable solution?.

[tamarian]

Originally posted by fastforward
Tamarians suggestion isn't too bad actually. It will still only be one version. It will just be up to the user how many of the optional code changes they wany to make to integrate it with vB. It will make the newnews.pl a bit more complicated due to the additional if/elses etc but it shouldn't be a big deal. I still intend to drop the 'import user' function.

Does that sound like a resonable solution?.

VERY much so!

Thanks :)

[Mark Hewitt]

More than I ever hoped for :D

[Mark Hewitt]

Theres only I few things that I would need in the next version before I can implement it on my board.

Fix the blank name on posts bug
Option to show date as when posted, rather than imported
A way to exclude usenet posts from the total post count. (and from pretty much everything else as well, but the post count is most important).

Cheers :cool:

[fastforward]

Originally posted by Mark Hewitt
Theres only I few things that I would need in the next version before I can implement it on my board.

Fix the blank name on posts bug
Option to show date as when posted, rather than imported
A way to exclude usenet posts from the total post count. (and from pretty much everything else as well, but the post count is most important).

Should be all done by tonight.

[fastforward]

This is the first release of the new style usenet gateway that works without any code edits but allows you to optionally integrate it more tightly into vB by making code edits. You can make as many or as few code edits as you like. The script will still work.

This release details enough edits to bring back the functionality that was stripped out in the last release. More integration/options will be added as and when people request them. Two of my next features will be the ability to download from the binary groups and provide a link to the file from the description usually found in the '01' section of the usenet article and an option to recieve email from usenet via the PM system.

Here's what this one can do:

Without code changes:

full control panel integration
email notification to usenet replies
emoticon translation into vb icons
vb code removal or conversion prior to posting to usenet
hyperlinked urls in messages & color coded and italicized posts
vB style quote to usenet style quote conversions for outgoing posts
logging of outgoing posts
support for multiple news servers
support for seperate footers per forum in outgoing posts
multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
flexible spam control and replacement variable options for both incoming and outgoing messages
configurable auto-purge option
global option for users to show their email address if user has selected that his/her email be displayed normally
global option to enable/disable user signatures on outgoing posts.
referencing usenet post count in templates (total posts will also include usenet posts)
optional time source for usenet posts (time entered into forum vs time user posted to usenet)
option to turn off indexing for usenet posts


With replacement variables and/or template changes:

user selectable email address in their outgoing usenet posts (may be different than normal one in profile)
suppression of unnecessary info in usenet posts, ie location, post count etc.
configurable indented quotes in incoming usenet posts


With code changes:

suppression of usenet post count from totals in templates
auto refresh of global usenet variables to ensure post counts etc are always up to date
optional suppression of usenet post count from totals in stats
better ordering and sequencing of usenet posts when using the NNTP posting date as time source
the 'last active' thread hack a la UBB.


Download the latest version (http://britishexpats.com/download/usenet_gateway.tar.gz)

Once again it's late and I'm tired, so treat this release with care. Let me know of any bugs asap and I can release a true 'release version' to coincide with the vB final. You'll probably notice I've changed the version number to date. With the frequency of updates this makes more sense and is easier to track... maybe :)

I also intend to move this hack and all discussion over to vbdev.org at the next release.

The latest version of this hack will always be in this first post of the thread.

[tamarian]

As always, you rock, fastforward :)

vbdev.org? Did you mean:

http://www.vbulletin.org

Search, and view new posts disabling? I personally hard code it, since I'm only interested in one usnet group.

I love it, and will install this weekend and report bugs (if any) :)

[fastforward]

Originally posted by tamarian
vbdev.org? Did you mean:

http://www.vbulletin.org

yeah... whatever :) I must have got confused with ubbdev :eek:

Search, and view new posts disabling? I personally hard code it, since I'm only interested in one usnet group.

ooops. I forgot about that one. I'll add that tomorrow.

[Mark Hewitt]

I can run newnews.pl and it appears to run through correctly. However after it does I get these errors on every page of my forums


Parse error: parse error in ./global.php(55) : eval()'d code on line 63

Warning: Cannot add header information - headers already sent by (output started at ./global.php(55) : eval()'d code:63) in ./functions.php on line 1172

Warning: Cannot add header information - headers already sent by (output started at ./global.php(55) : eval()'d code:63) in ./global.php on line 131


When I load my forums home page it gives the above errors and only displays the categories and no forums. It also gives a database error when trying to run showthread.


Database error in vBulletin: Invalid SQL:
SELECT IF(votenum>=,votenum,0) AS votenum,
IF(votenum>= AND votenum > 0,votetotal/votenum,0) AS voteavg, icon.title as icontitle,icon.iconpath,
thread.threadid,thread.title,lastpost, forumid,pollid,open,replycount,postusername,postus erid,
lastposter,thread.dateline,views,thread.iconid,not es,thread.visible,sticky,votetotal,attach
FROM thread
LEFT JOIN icon ON (icon.iconid = thread.iconid)

WHERE thread.threadid IN (0)
ORDER BY sticky DESC, lastpost DESC

mysql error: You have an error in your SQL syntax near 'votenum,0) AS votenum,
IF(votenum>= AND votenum > 0,votetotal/v' at line 2
mysql error number: 1064
Date: Wednesday 02nd of May 2001 06:35:06 AM
Script: /forums/forumdisplay.php?s=&forumid=3
Referer: http://www.motorsportforum.com/forums/forumdisplay.php?s=&forumid=33


However if I go to /admin/options.php and resubmit the options everything goes back to normal :confused:

[fastforward]

What code changes did you make? Did you go through and re-check the tables to make sure all columns and indexes are as specified in the new versions. Alao, did you make sure you removed any existing code changes you may have made?

[Mark Hewitt]

Originally posted by fastforward
What code changes did you make?


I implemented all the code changes you specified.


Did you go through and re-check the tables to make sure all columns and indexes are as specified in the new versions.


Yes, I deleted all my current usenet forums and removed the usenet_ tables. I then made sure the extra feilds matched up. Think it was only regpost I needed to add. I then created and populated the new usenet tables.


Also, did you make sure you removed any existing code changes you may have made?

I implemented it from a clean copy of RC2 which I was previously running the last version of the hack on.

Hate to heap more problems on you but the NNTP posting time option doesn't seem to work, still uses local time.

Apart from that it works a treat :D

[fastforward]

I'm working on it. I'll get back to you.

:rolleyes: If only there were no users. life would be so much easier for developers :D

[Mark Hewitt]

If only there were no users. life would be so much easier for developers :D

I hear ya. :cool: Especially ones that don't know how to run a Java class (long story)

[fastforward]

OK...

The parse errors may have been because it was the first time the script was run. The way it works now is it appends any usenet variables that need to be referenced by vB to the 'options' template. This means they can be accessible through layout templates without code changes. The variables causing problems are the post counts which were not quoted. If these are NULL, then it will generate a parse error as the resulting code would like this: ' $usenettotalthreads= ' instead of ' $usenettotalthreads="" '. That's easy to fix.

I can not reproduce your time problem. When I set NNTP in the options all my times show up correctly as the original posting date. Can you reconfirm that this is happening by comparing it against your newsreader? Or have you already done that?

[Mark Hewitt]

The parse errors may have been because it was the first time the script was run. The way it works now is it appends any usenet variables that need to be referenced by vB to the 'options' template.

This sounds about right, especially as everything was restored when I resubmitted the options. However I did run the script twice and the same problem happend both times.


I can not reproduce your time problem. When I set NNTP in the options all my times show up correctly as the original posting date. Can you reconfirm that this is happening by comparing it against your newsreader? Or have you already done that? [/B]

Looking back on it, it's possible the option wasn't set on the second time I ran it. As half the posts seem to be okay. So ignore that part :)

[fastforward]

A few bugfixes and the addition of the 'exclude usenet posts from new post search' option.

Take note of item number 4. If you have already installed the last version, you MUST change this column type to SIGNED for correct handling of outgoing posts.

Added option to exclude usenet posts from 'Get New Posts' search.
Fixed bug causing parse errors during first run.
Fixed bug causing newnews.pl errors 'e.field invalid column' error if custom profile field for outgoing email isn't added.
Fixed incorrect column type for isusenetpost in post and thread tables. They should be TINYINT(2) SIGNED.

This should be one step closer to a bugfree version. Time is running out, vB 2.0 will be out any day :)

I'll delay the move to vbulletin.org until the final release.

Download latest version (http://britishexpats.com/download/usenet_gateway.tar.gz)

[Mark Hewitt]

Originally posted by fastforward
[B] Fixed bug causing parse errors during first run.

:( I'm still getting the same errors after a run that I detailed above :(

[Mark Hewitt]

I've looked at the options template and after it has just run it has

$totalusenetposts=5902;
$totalusenetthreads=18;

After I go back and resumit the vB options page is changes to

$totalusenetposts="5902";
$totalusenetthreads="18";

i.e. it now has quotes

For temporary fix I've commented out generateoptions(); as I don't use total threads/posts anyway, and now all seems well.

(another edit). The NNTP post time seems to work for the first few posts then gives up and goes back to local time.

[fastforward]

I think caching causes problems for this template update function aswell. I think the best solution will be if I remove it for the threadcounts and you will have to make a code edit to use them. It's only a one liner anyway.

I still can't reproduce your posting date problem. It's actually only 1 little line in the script that decides the date. When the script starts up, it reads all the config from the tables and then when it gets to the load forums bit it checks to see whether it should replace the nntp date with the local date. So the default is actually NNTP date. I'll keep trying to reproduce it though.

[fastforward]

I've just looked at the code and the quotes are gone again! I think this because I started making the the edits on my test board while at work and then not transfer them to the distribution copy when I got home. All the other fixes I mentioned were still there, it's just the quuote issue around the post counts (which I'm removing anyway).

[fastforward]

OK... sorry about all these posts. I have just found the problem with the date thing. The problem is obvious and I have no idea how I missed it! The conversion is only done when a thread starter is loaded. For replies I still had a hard code of '$dtm=time();' in there... ooops

Sorry about that.

[Mark Hewitt]

Ah right :cool: good to hear it's not my misconfiguration again :).

[fastforward]

Here's another one that attempts to fix the recent issues.

Eliminated parse errors by removing the function that appended to the vB templates. It was too unreliable and prone to error. In order to display the usenet counts on template pages you must now make a code edit. There is also a requirement to make one common code change if any of the optional code edits are made.
Fixed the NNTP/LOCAL time issues. (really this time... honest!)
There is also an addition to the code change instructions if you are using the lastactivethread hack.


Download v20010504 for RC2 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[fastforward]

I messed up again... I forgot to include the new usenet.php file in the package! I've edited the previous post and changed the version and download link. All should be OK now.

[Mark Hewitt]

When I run newnews.pl I get this


Fetching article body 64980... OK
Processing article batch...
Use of uninitialized value in numeric gt (>) at newnews.pl line 216.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.

Query failed:
INSERT LOW_PRIORITY IGNORE INTO usenet_ref (msgid,ref,cnt,dtm) VALUES ('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<99pjhv$6tt$1@dyfi.aber.ac.uk>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<%k2w6.3328$W51.2310054@e3500-chi1.usenetserver.com>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<3AC0F88A.312BF9B7@crosslink.net>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<3AC3FEDC.4043@i-plus.net>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<99vgrr$g0k$2@dyfi.aber.ac.uk>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<290320010806569457%scott@nospam.winders.com>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<9a9ehk$5fu$5@dyfi.ab',6+1,)

DBD::mysql::db do failed: You have an error in your SQL syntax near '),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<%k2w6.3328$W51.2310054@e3500-chi1.usenetse' at line 1 at newnews.pl line 576.


I try running it again until it gets to that particular post and it fails again :( Previous to that though I had imported some posts and was working correctly.

[fastforward]

What's the size and type of your ref column in the usenet_ref table. Try making it bigger.

[Mark Hewitt]

It was varchar(128).

I've tried changing it to varchar(255) but get the same error.

[Mark Hewitt]

If you need to test it the import was

Getting article batch from rec.motorcycles.racing
Fetching headers of articles 64663 to 64980...

[fastforward]

It' the usenet date/time. It either doesn't exist (unlikely) or it's malformed in the usenet post header. This is one of the reasons I went to local time. Have a look at the article on the server and see what the date looks like. I can't see it as your article numbers mean nothing to my server. As a workround you can add the following code at line immediately above line 216 that reads: if ($dtm > time()+600) { $dtm = time(); }

if (!$dtm) { $dtm = time(); }

That is an addition, NOT a replacement.

[fastforward]

Note the little change I just made to that change. It should be !$dtm.

[annams]

First of all, this is a wonderful hack!!

Now, on my web server I did not have Email::Find and MIME::WordDecoder and I installed both the modules. After that I am encountering the following error:

defined(@array) is deprecated at /usr/home/dsw/local/lib/perl5/site_perl/5.6.0/Net/DNS.pm line 137.
(Maybe you should just omit the defined()?)

Please help!!!

[fastforward]

Originally posted by annams
Now, on my web server I did not have Email::Find and MIME::WordDecoder and I installed both the modules. After that I am encountering the following error:

defined(@array) is deprecated at /usr/home/dsw/local/lib/perl5/site_perl/5.6.0/Net/DNS.pm line 137.
(Maybe you should just omit the defined()?)

That error is actually being generated from the Net::DNS module. That module is not part of the hack. I believe the DNS module is patr of the libnet bundle. Try upgrading the libnet bundle using CPAN.

If that fails, you may want to try your suggestion. Find the line in DNS.pm that it talks about and change it there. Try just declaring it as @array; and see what happens.

In the meantime, I'll see if I can find any reference to this problem anywhere else.

[annams]

That was just a warning! Since no processing was happening after that I thought it was an error. But then I found that I had both groups disabled. Now, I got everything working fine.

Thanks a bunch for the hack!!

Btw, I had to take a paid registration with usenet.com to get the newsfeed. Though I like the fact that it has a spamfree news server. Is there a free or cheaper newsfeed?

[fastforward]

Originally posted by annams
That was just a warning! Since no processing was happening after that I thought it was an error. But then I found that I had both groups disabled. Now, I got everything working fine.

If you remove the -w switch from the shebang line in newnews.pl that warning will go away. At least you won't have to look at it each time :)

Btw, I had to take a paid registration with usenet.com to get the newsfeed. Though I like the fact that it has a spamfree news server. Is there a free or cheaper newsfeed?
There are free ones around, but you have to look hard. They are usually unreliable, slow and carry a limited set of groups. You might be able to find decent $5.95 one for just the text groups but $10 seems the norm for a decent account.

[Mark Hewitt]

I've made all the changed you specified and in addition I've switched the option to use local server time. However the same error still results :confused:

[Mark Hewitt]

I removed the newgroup that was causing the trouble but it just happened on another group.


Fetching article body 27334... OK
Processing article batch...
Use of uninitialized value in numeric gt (>) at newnews.pl line 216.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.

Query failed:
INSERT LOW_PRIORITY IGNORE INTO usenet_ref (msgid,ref,cnt,dtm) VALUES ('<philip.
remove.orr.629.011B100A@bigfoot.com.spiced.ham.in. a.can.au>','<3ab213a3@news.ipr
imus.com.au>',2+1,),('<philip.remove.orr.629.011B100A@bigfoot.com.spiced. ham.in.
a.can.au>','<3ab21537@occy.pnc.com.au>',2+1,),('<philip.remove.orr.629.011B100A@
bigfoot.com.spiced.ham.in.a.can.au>','<3ab2c1ce@grissom>',2+1,)

DBD::mysql::db do failed: You have an error in your SQL syntax near '),('<philip
.remove.orr.629.011B100A@bigfoot.com.spiced.ham.in .a.can.au>','<3ab21' at line 1
at newnews.pl line 576.
bash$

[fastforward]

There's no need to set it to local time. The change just checked whether the $dtm field was populated with the nntp time and if it wasn't use the local time.

There is no doubt the error is being caused by the $dtm field not being populated. The reason this happens is because the time in the header is missing or malformed. The warning: 'Use of uninitialized value in numeric gt (> ) at newnews.pl line 216' is due to the $dtm field being null. (an advantage of 'use strict' and the -w flag) confirms this. Also you can see the dtm field is missing in the sql statement.

What news server are you using and what groups are giving the most problems. I'll try them and add some more error checking.

To confirm the problem, do this: Find line 299 that currently reads:

Add the following line directly ABOVE it
[code]
if (!$d) { $d=time(); }

So the resulting block of code around there will be:

if (/^Lines: (.*)/) { $l = $1; }
}
if (!$d) { $d=time(); }
($p,$e) = parse_from($f);
push (@a,{ from => $f,

Another quick fix is just remove the NOT NULL constraint from the usenet_ref table. This may have other unwanted side effects though :)

[maxc]

Hi....fastforward, Thanks for your hack...
But I have some questions, When I used spam control,it always appear a message as below:

Warning: Cannot add header information - headers already sent by (output started at /home/httpd/html/forum/admin/adminfunctions.php:18) in /home/httpd/html/forum/admin/usenet.php on line 231

how to slove this problems.please...

and in Options -> Character set for MIME decoder, it appear

Parse error: parse error, expecting `']'' in /home/httpd/html/forum/admin/usenet.php(55) : eval()'d code on line 2

Thanks for your help.... :) :) :)

[maxc]

Originally posted by fastforward

To confirm the problem, do this: Find line 299 that currently reads:

($p,$e) = parse_from($f);

Add the following line directly ABOVE it

if (!$d) { $d=time(); }

So the resulting block of code around there will be:

if (/^Lines: (.*)/) { $l = $1; }
}
if (!$d) { $d=time(); }
($p,$e) = parse_from($f);
push (@a,{ from => $f,

Another quick fix is just remove the NOT NULL constraint from the usenet_ref table. This may have other unwanted side effects though :)

I have already patch my newnews.pl with this method, But it still show the error message below:

Query failed:
INSERT LOW_PRIORITY IGNORE INTO usenet_article (newsgroup,forum,msgid,dtm,subject,poste
r,email,refs,body,msgnum,nntpposter,ord,threadid,p ostid) VALUES ('tw.bbs.comp.hardware'
,7,'<3f0Mhc$Zy5@bbs.cis.nctu.edu.tw>',982109749,'Re: ½Ð±Ð¦U¦ì°ª¤â¡ã¦³Ãöwindow2000ªººô¸ô
³]©w°ÝÃD¡ã«æ¡ã¡ã','ªA´À¥N§Ðªº¤p³\\','wuuchung.bbs@bb s.cis.nctu.edu.tw',NULL,'==> ¦b ens
.bbs@zoo.ee.ntu.edu.tw (¤S*n¼Z¸¨¤F) ªº¤å³¹¤¤´£¨ì:\n> ´N¬O§Ú*̲oADSL\n> µM«á¥D¾÷¾A¥ÎW
IN98¨t²Î\n> §Ú¬O¥ÎWIN2000¨t²Î\n> §Q¥Îsygate¨Ó¤Wºô\n> ¦ý¬O§Ú²{¦b¥u¯à¤WB
BS\n> ¦Ó¤£¯à³s¤WWWW\n> ½Ð°Ý*n¦p¦ó³]©w¤~¯à¸Ñ¨M¡H¡H¡H\n>
£³Ó·P¿E¡ã¡ã¡ã¡ã\n\n À³¸Ó¬ODNS¨S¦³³]©w¦n.......\n--\n\n §Ú¬O²Ä¤T±è´À¥N§Ð§Ð¨k,²{¦b¦b¥x?
_¿¤·s²ø¥«ªA®ø¨¾´À¥N§Ð,¦³¥ô¦ó¦³Ãö´À¥N§Ð.\n\n ªÑ²¼.´Á³f.ÅvÃÒ.¹q¸£µwÅé.µ´¹ïªZ¤OCSªº°ÝÃD,?
Ú³?wªï§A*̨ӻP§Ú¤@°_°Q½×........\n--\n * Origin: ¡¹ ¥æ³q¤j¾Ç¸ê°T¬ì¾Ç¨t BBS ¡¹ <bbs.cis
.nctu.edu.tw: 140.113.23.3',245055,'wuuchung.bbs@bbs.cis.nctu.ed u.tw (ªA´À¥N§Ðªº¤p³\\)'
,-1+1,0,0)

DBD::mysql::db do failed: You have an error in your SQL syntax near 'wuuchung.bbs@bbs.c
is.nctu.edu.tw',NULL,'==> ¦b ens.bbs@zoo.ee.ntu.edu.tw (¤S*n¼Z' at line 1 at ./newnews.
pl line 579.

bash$

[fastforward]

Originally posted by maxc
Hi....fastforward, Thanks for your hack...
But I have some questions, When I used spam control,it always appear a message as below:

Warning: Cannot add header information - headers already sent by (output started at /home/httpd/html/forum/admin/adminfunctions.php:18) in /home/httpd/html/forum/admin/usenet.php on line 231

That warning will always be there until I get around to rewriting the control panel page. It's on the bottom of my to do list. It doesn't affect the functionality in any way.

and in Options -> Character set for MIME decoder, it appear

Parse error: parse error, expecting `']'' in /home/httpd/html/forum/admin/usenet.php(55) : eval()'d code on line 2

I have no idea. I can't reproduce this. Did you change the character set from the default? I'll look into it.

[fastforward]

For those of you getting the error when inserting into the usenet_ref table; check your my.cnf setting for MAX_ALLOWED_PACKET.

I'll lookat rewriting that section to loop and insert each ref individually. A little less efficient, but it should provide better error trapping and correction.

It must be a configuration issue as I can't reproduce it and I collect over 60 groups on dbforums.com

[fastforward]

maxc:

I just cut and pasted your SQL statement that was giving you and error and it loaded into my database without problems.

Post the structure of your usenet_article table and give me the mysql version.

[fastforward]

Originally posted by Mark Hewitt
I removed the newgroup that was causing the trouble but it just happened on another group.
Query failed:

Looking at the SQL statement, you can see it still isn't picking up the date. Give me the newsgroup name and the date of the dodgy article and I'll have look at it. This error is in no way related to maxc's error by the way.

[maxc]

Excuse me,I am a Taiwanese use Tranditional Chinese,
thus,I surmise I need to edit defalt language character,can
I add Big 5 8bit(Chinese language code) into program,will
it work If I added.it.......

[fastforward]

Originally posted by maxc
Excuse me,I am a Taiwanese use Tranditional Chinese,
thus,I surmise I need to edit defalt language character,can
I add Big 5 8bit(Chinese language code) into program,will
it work If I added.it.......
Unfortunately not. The module I use only supports the ISO 8859 single-byte coded (8bit) character sets for the following:

Latin1 (West European)
Latin2 (East European)
Latin3 (South European)
Latin4 (North European)
Cyrillic
Arabic
Greek
Hebrew
Latin5 (Turkish)
Latin6 (Nordic)

Simply adding to this list will not achieve anything. I believe I need unicode support for Chinese characters (or a module that supports more of the ISO 8859 sets). I'm not sure whether more sets exist yet although I know they are planned.

Sorry about that. I will look around for another module that supports it.

[maxc]

Originally posted by fastforward
maxc:

I just cut and pasted your SQL statement that was giving you and error and it loaded into my database without problems.

Post the structure of your usenet_article table and give me the mysql version.

Ok... I am using MySQL v3.23.37, and My structure of usenet_article table is in the attach file.

Thanks for your help....

[fastforward]

One more thought about the Chinese cahracters:
Try just disabling the worddecoder routine. Do this:

Change line 190 that currently reads

$$art{body}=$wd->decode(join("",@$b));

to this

$$art{body}=join("",@$b);

[maxc]

Originally posted by fastforward

Simply adding to this list will not achieve anything. I believe I need unicode support for Chinese characters (or a module that supports more of the ISO 8859 sets). I'm not sure whether more sets exist yet although I know they are planned.

Sorry about that. I will look around for another module that supports it.

Just Ok! It didn't effect of my forum, Thank you very much! ;)

[fastforward]

Minor release to change the way the custom email address works. This way should be more logical and prevent any user inadvertently setting their options in such a way that will display their email address in outgoing usenet posts.

It also adds some additional error checking in places.

There have been a lot of major changes in the last few versions. This includes changes to table structures and definitions. If you have remnants of installations before the version numbers were changed to a date, you should carefully go through each table and index and check the settings against the instructions. Where possible, you should drop and re-create the table.

This version has been tested on over 200 newsgroups pulling articles from 1 week ago. No errors were seen and not once was there an instance of the fields in article headers not being found. The servers used were onlynews.com and a local DNEWS server.

Download v20010505 for RC2 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[Mark Hewitt]

I've changed to the new version and I'm not getting the errors I was getting before.

However some newsgroups don't seem to be fetching the articles properly.

There are some where I know for a fact there are many many new posts and yet newnews.pl is saying No new messages, together with other assorted errors.


Use of uninitialized value in addition (+) at newnews.pl line 95.
Use of uninitialized value in numeric gt (>) at newnews.pl line 95.
Getting article batch from rec.autos.sport.f1.moderated
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
No new messages in rec.autos.sport.f1.moderated
inserting new threads from rec.autos.sport.f1.moderated
checking for orphans...
Processing outgoing posts
Processing outgoing posts

[fastforward]

Originally posted by Mark Hewitt
I've changed to the new version and I'm not getting the errors I was getting before.

Use of uninitialized value in addition (+) at newnews.pl line 95.
Use of uninitialized value in numeric gt (>) at newnews.pl line 95.
Getting article batch from rec.autos.sport.f1.moderated
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
No new messages in rec.autos.sport.f1.moderated
inserting new threads from rec.autos.sport.f1.moderated
checking for orphans...
Processing outgoing posts
Processing outgoing posts

You have something very wrong with your installation. Those errors are telling you it can't find the last message number from the config. Those Particular lines have not been changed since the early versions. The same goes for the date issues you were getting. The reason you don't get date errors now is because it is forced to local time if it's still NULL after trying to get the NNTP date.

I can't reproduce the errors and I've pulled countless newsgroups including rec.motorcycles.racing that was giving you problems.

Have you changed Perl versions recently? Did you do a *complete* fresh install? Is anyone else having these problems.

The only way to troubleshoot your problem would be if I had access to you machine and newserver.

Let me try anaother fresh install on another server and I'll see if I can re-produce the errors. Give me a few hours.

[fastforward]

OK.. Here's another one. I rechecked the entire code and errorchecking. I installed it on a fresh vB on two different servers

FreeBSD with an old version of MySQL (3.22.32) and Perl 5.005_03
Linux 6.1 with the latest MySQL (3.23.37) and Perl 5.005_03


Both installations had the latest required modules from CPAN.
Two newservers were used, onlynews.com and DNEWS. DNEWS is used by a lot of the major news providers including Supernews.

There were no errors found with either configuration on either server.

There realy isn't much else I can do. All I suggest is you make a completely fresh install, DO NOT make any code changes except the bit to add the control panel. Add the basic configuration settings, but leave most of the settings as the default. Then run the script. Change the settings and/or and add the code changes one by one if you need them and see where the errors start occuring.

Download v20010506 for RC2 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[SNA]

Just reinstalled it... it loops after reading in the headers...


Connecting to #######... Connected
Processing outgoing posts
Getting article batch from ######
Fetching headers of articles 1 to 23...
Processing headers for article 2...
Processing headers for article 3...
Processing headers for article 4...
Processing headers for article 5...
Processing headers for article 6...
Processing headers for article 7...
Processing headers for article 8...
Processing headers for article 9...
Processing headers for article 10...
Processing headers for article 11...
Processing headers for article 12...
Processing headers for article 13...
Processing headers for article 14...
Processing headers for article 15...
Processing headers for article 16...
Processing headers for article 17...
Processing headers for article 18...
Processing headers for article 19...
Processing headers for article 20...
Processing headers for article 21...
Processing headers for article 22...
Processing headers for article 23...
Fetching article body 2... OK
Fetching article body 3... OK
Fetching article body 4... Not fetched - Spam
Fetching article body 5... OK
Fetching article body 6... OK
Fetching article body 7... OK
Fetching article body 8... OK
Fetching article body 9... OK
Fetching article body 10... OK
Fetching article body 11... OK
Fetching article body 12... OK
Fetching article body 13... OK
Fetching article body 14... OK
Fetching article body 15... OK
Fetching article body 16... OK
Fetching article body 17... OK
Fetching article body 18... OK
Fetching article body 19... OK
Fetching article body 20... OK
Fetching article body 21... OK
Fetching article body 22... Not fetched - Spam
Fetching article body 23... Not fetched - Binary post
Processing article batch...
Requested 22 messages... 3 not available or rejected.

inserting new threads from cast.testing
checking for orphans...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...

And just keeps going....

I suggest NOT looking for the Lines: ## in the NG posts... Did you know that the posts added FROM the forum dont have a Lines: ## in the header?

Can we disable the option that looks to see if Lines: exsists...

[SNA]

Its fixed now.. I dumped the ref table... But now.. why doesnt the posts from the FORUMS insert a "Lines: ##" line into the header?

[fastforward]

Originally posted by SNA
I suggest NOT looking for the Lines: ## in the NG posts... Did you know that the posts added FROM the forum dont have a Lines: ## in the header?

Can we disable the option that looks to see if Lines: exsists...
You can disable it if you really want to. But you may end up with huge useless posts and possibly binaries.

All posts have the 'Lines:' in the header regardless of their source. It is added by the news server along with all the other stuff that a client news reader can't be trusted to add.

As for the looping problem, have you made several runs and reset article numbers without clearing out the usenet load tables? I suggest you clear out your usenet_article, usenet_article_loader and usenet_ref tables and see if it happens again.

It will continue to loop as long as it finds articles that are ready for loading. You should see several of these 'loading replies' messages in sussession usually with decreasing numbers. It will then attempt to load the article into the forum. Whether it fails or succeeds to insert the post into the forum is irrelevant, it will be deleted from the loader table.

So there are only a few logical scenarios for this error to occur.

if the deletion failed for some reason. As you made no mention of an error message, the script may have been aborted at some point before the actual delete took place.
if the post it is trying to load actually originated from the forum in the first place and you have 'import existing posts' turned on, the article already exists in the post table and you reset the message counters to 0 at some point.
you changed the header information that identifies your site. The script has no idea that the post originated from you but it still found a postid and is trying to load it.

I will look at adding additional error checking to trap some of these.

[SNA]

Ok Cool...

Im using Cnews.. and it isnt adding the Lines: ## their from the posts coming off the Forums... Im guessing there is a config file somewhere to enable it...

Maybe you want to take that into condiseration...

If it doesnt find a Lines: then it should do something other consider it a binary..

Just a suggestion.. GREAT HACK :) It will be used.

[fastforward]

Originally posted by SNA
Ok Cool...

Im using Cnews.. and it isnt adding the Lines: ## their from the posts coming off the Forums... Im guessing there is a config file somewhere to enable it...

Maybe you want to take that into condiseration...

If it doesnt find a Lines: then it should do something other consider it a binary..

The only other way to determine if it's a binary post is by your own configurable spam filters. The obvious one to check for is 'begin 644'. This is actually added by default. The lines check is really only an additional check and a way for you limit the size of messages you want to download.

You say the server isn't adding the Lines field. As you mention, it is probably a configuration issue. What about other articles, or are you receiving a full IHAVE usenet feed? If it's just acting as a client to an upstream news server then you will probably see the Lines header on all other posts. If the issue is only with articles that are posted directly to your cnews server then there's not really a problem as when that message is propogated to another server, the field will be added.

The other point of course is why would we care if forum posts have it or not. They don't even get parsed by that code. We don't need to check them as we're not even downloading them. :)

[SNA]

OK I run my own newsgroups.. I am not a client.. I am a Server.

Lets say people post to the forums and I have to delete all the posts for some reason... when I go to download all the messages again it skips past all the ones created from the forums because there is no Lines: ## Line.

[Mark Hewitt]

Hey!! it works, and I didn't change anything either. Thanks fastforward for all your help. I must have got quite boring over the past couple of weeks.

Just a couple of questions on how posts appear on usenet.

I used quote to put the quoted text from the previous post in my reply and it formatted it correctly on usenet with the '>' characters in the correct place but even though I wrote my reply below the quoted text on usenet it appeared above it. On the newsgroups I posts to some people can get very upset about that.

Also
[ QUOTE ] [ /QUOTE ] (without the spaces) appeared at the top of the post.

[fastforward]

Originally posted by Mark Hewitt
Hey!! it works, and I didn't change anything either. Thanks fastforward for all your help. I must have got quite boring over the past couple of weeks.

Just a couple of questions on how posts appear on usenet.

I used quote to put the quoted text from the previous post in my reply and it formatted it correctly on usenet with the '>' characters in the correct place but even though I wrote my reply below the quoted text on usenet it appeared above it. On the newsgroups I posts to some people can get very upset about that.

Also
[ QUOTE ] [ /QUOTE ] (without the spaces) appeared at the top of the post.
I re-wrote all the outgouin post bits recently to try to tidy it up. I'll add an option for you to decide whether you want the quote above or below the message. The other thing with the '[ QUOTE ]' showing must be a bug. Does it happen all the time?

[Mark Hewitt]

Can't say if it happens all the time as I have only sent one post to usenet :)

I take it from this that it doesn't support multiple quotes within a single post?

[annams]

Not only for quote, it happens for URL too.

In case of URL it appears as [ url. The url and the closing [ /url ] are missing.

[fastforward]

It seems it depends on exactly what order the vb code appears in the message. It's not very easy to catch them all. Just taking the URL code as an example, I have to check for [ URL ], [ URL =.... ], [ URL ="..... ] etc. And then there is the problem of posts that embed the url by doing a please click HERE (http://vbulletin.com) type thing. I either replace the HERE with the underlying url which may make the sentence to appear nonsense, or I could simply strip out the URL which is probably even worse. Then there's quotes and embedded quotes, colors with the # colors without the # but with double quotes.... I could go on, but you get the picture. And if that weren't enough, you have custom bbcodes :(

Anyway, I'll re-visit the regexes that trap these and see what I can do. The safest solution, however, is to turn vbcode off for usenet forums.

[annams]

I actually tried turning off vB code for the forum, then it had USENET articles displayed as [ URL ] url goes here [ /URL ].

So I quickly turned it back on. Thanks for looking into it.

[fastforward]

Originally posted by annams
I actually tried turning off vB code for the forum, then it had USENET articles displayed as [ URL ] url goes here [ /URL ].

So I quickly turned it back on. Thanks for looking into it.
Yes, you'd need to change the options in the usenet control panel to stop urls and smilies being converted. Unfortunately, that won't fix historical posts as the bbcode is added before the post is inserted into the forum.

Not to worry, I'll at least sort out the URL, QUOTE and standard bbcode problems in the next release. Simple custom bbcodes with no parameters are easy to trap for aswell. It's just the custom codes with multiple params that will be virtually impossible to fix.

[annams]

Could you please tell me when I can expect the next release? Thanks.

[fastforward]

Originally posted by annams
Could you please tell me when I can expect the next release? Thanks.
Sunday probably. I have a fulltime time job and a family. This is already taking way too much of my time :p

Once all these little niggly bugs are fixed I'm going to slow down for a bit. I've also got to spend some time re-doing my sites when vb 2.0 is released.

But don't worry, I've still got plenty of features planned for this hack like support for binary groups and email to PM gateway so I'm not going to give up on it. Besides, I need it for my own sites.

[Mark Hewitt]

It's working great at the moment but I'll tell you how I would like to see it work, for you to consider in later versions.

The thing which concerns me at the moment is the way that the posts are appearing on usenet, namely the fact that they only appear in reply to the first post, this being a restriction of having a linear board.

However how I'd like it to work is to create a seperate template set for use in the newsgroup forums, in this the "Post Reply" button will be completely removed.

Instead underneath each individual post I would put a link "Reply to this post", as far as the forum goes this would do exactly the same thing as Quote does (thus forcing people to quote the text). However when the posts are sent out to usenet the posts appear threaded in the correct way with the posts appearing in their correct positions in the threads.

[fastforward]

I think I can do this relatively easily. It would mean a vB code edit in newreply.php and, as you mentioned, a complete new template set. Although it will still work without templates and would depend on whether you clicked the 'Reply' or the 'Quote' button.

I agree that it would be better this way. In fact I seriously considered switching to wwwThreads and write the hack for that board instead. It allows each site visitor to switch between in-line and threaded view. That way you could see the usenet posts in a threaded format too. The thing that stopped me was the lack flexible custom templates via templates.

Anyway, I'll see if I can get it working this weekend. I'll make it a global control panel option.

[annams]

Hi,

On the web server I use there are limits on the resource usage. Because of these my cron job that runs newnews.pl is getting killed quite often. My questions are:

1. When the job gets killed in the middle, would that leave either the newsgroups data or the forums data in any inconsistent state?

2. Can you modify the script to rotate through the newsgroups, so whatever newsgroup that is currently being fetched first would be fetched last in the next run. Or better yet give the option to do few newsgroups everytime.

On my web server the following are the resouce limits:

Size of Core Files - 0 MB
CPU Time Used - 30 seconds
Data Size - 3 MB
File Size Created - 1 MB
Memory Locked - 1 MB
Number of Open Files - 32
Number of Simultaneous Processes - 8

I appreciate your help (and the hack :-) )

[annams]

Does the news server need to be connected / disconnected for fetching data from every newsgroup? Can the same connection be reused for all the newsgroups currently being processed?

Please read the above question in light of resource usage problems I am having.

[fastforward]

Originally posted by annams
Does the news server need to be connected / disconnected for fetching data from every newsgroup? Can the same connection be reused for all the newsgroups currently being processed?

Please read the above question in light of resource usage problems I am having.
The script will only reconnect for each group if you have the 'Multiple Servers' option enabled. This was the the very reason I made that an option. If it's set to No, the script will make one and only one connection for the entire run.

It may be too difficult to get the script to remember where the it died on the previous run and adjust the order accordingly, but adding an option to limit the number of groups may be doable. However, the option is already there (sort of). All you need to do is adjust the batch size option. This was why the option was included. By usung this method, you can guarantee that at least some messages will be pulled from each group and that all outgoing messages will be processed.

[annams]

Yes, I had the multiple servers option, while disabled the script runs much faster. Thank you.

[jarvis]

Okay, I've searched thru most of the posts on this great hack, but I may have skipped over, or it just isn't there.

I'm not a technical expert, but would like to think I can tackle this task. Has anyone implemented this on a Win32 system, and if so, did you use the Active Perl install? Does the ActivePerl install have all of the needed mods required for this? Lastly, do I have to install an ISAPI filter thru the IIS MMC, 'cuz right now all I see is the PHP filter listed there.

Thanks, and don't laugh too hard. :)

[fastforward]

Originally posted by jarvis
Okay, I've searched thru most of the posts on this great hack, but I may have skipped over, or it just isn't there.

I'm not a technical expert, but would like to think I can tackle this task. Has anyone implemented this on a Win32 system, and if so, did you use the Active Perl install? Does the ActivePerl install have all of the needed mods required for this? Lastly, do I have to install an ISAPI filter thru the IIS MMC, 'cuz right now all I see is the PHP filter listed there.

Thanks, and don't laugh too hard. :)
I wondered how long it would be before someone wanted to run this on NT :)

The short answer is, it might work.

The latest Active state ports provide just about all the Unix functions (even fork) so there shouldn't be any problems there. The only thing I know of that Windows doesn't support is the ALRM system calls. I haven't used that so it doesn't matter.

However, you may have problems with some of the modules that are required for the script to run. I know for a fact that Active State doesn't provide binaries for all of them. If you have MSVC 5 or higher you should be able to compile all the modules just as you do for a Unix installation. You just substitute 'make' with 'nmake'.

Good luck and let me know if you have any success :)

PS. I know nuffink about the PHP/ISAPI stuff so you're on you own there :)

[n3n]

replies to a post not working..

i see the first post from usenet but the child replies do not seem to be posted on usenet.. -.-;;

[fastforward]

Originally posted by n3n
replies to a post not working..

i see the first post from usenet but the child replies do not seem to be posted on usenet.. -.-;;
My guess is this:

You started a thread locally, then you replied to it a couple of times, also locally. Then you ran the script once and only once.

Run it again. The replies can only be sent when it's known what the message id is of the post it's replying to. As the thread starter hadn't been sent to usenet it doesn't have a message id.

[fastforward]

Just a quick update on how the next release is going for those interested.

I've managed to get it working so that you can reply to individual posts within a thread and have it show up correctly threaded on usenet. It requires one single code edit in newreply.php and a hidden input field in the newreply template.

I've also fixed the stripping of bbcode problems... I think. The only way to really test it is over time.

There were also a few other little things that I fixed including SNA's request for a workaround for his CNEWS set up.

I need a bit longer to test it but I'll be putting up a new version of dBforums.com tomorrow with a complete new set of templates for the usenet forums to better accomodate usenet posts.

If you want anything e