[RELEASE] Usenet gateway

[fastforward]

THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread for the latest version and discussion

[chrispadfield]

thanks for this, i am so happy now

i am going to install and play and see if i can suggest etc things for it but probably wait to open it up to my members until the v2.0 version comes out.

Have to say this is one of the best features out there for vbulletin and bar co-branding and unlimited depth forums beats most of the additions for v2, thanks so much for making it

[chrispadfield]

ok problem...

i installed some of the modules using cpan which was ok but one of them some how decided that it wanted to re-install the whole of perl without even asking. This, scared me quite a lot and i quit it which may not have been the best thing to do i don't know.

Anyway, after that little episode i think i did everything but i get this error message:

Net::NNTP: Bad hostname 'spamkiller.newsfeeds.com
' at newnews.pl line 219
Unable to connect to spamkiller.newsfeeds.com

now is this a problem with Net::NNTP (not being installed?) or the hostname (which is correct, it works in outlook express anyway). Any suggestions?

thanks..

[fastforward]

spamkiller.newsfeeds.com is now known as spamkiller.usenet.com. They changed it around a month ago.

The Net::NNTP is part of libnet which may well have been already installed. If you're getting an error saying it can't fnd the host, it probably means the module is OK. It wouldn't have got that far otherwise.

[chrispadfield]

newsfeeds.com appears to have come back, it is working. it was a silly mistake i think the server name went on to to line in the perl script and it didn't like it...

any way, currently going through the downloads so have it working... why did you have to release it now.. it is 4.30a.m here and i can't stop (i am only kidding just so pleased to play with this!)

one question, the "rejected bad header" what is that for? i get quite a few of these, probably 20% of them so far, is it spam or is it something else they are rejected for?

[fastforward]

The bad header is usually due to strange character sets or a bad news client being used to post the article. Usually you'll see this in test and binary groups where people may be testing out there own posting programs. 20% seems high. I was getting that sort of number when I was using alt.test. But a real newsgroup only rejects a tiny percentage.

The script actually puts out the rejected header message when it can't find a message id in the header. This is usually due to the reasons above.

[chrispadfield]

ok few things:

1) get these errors once it has finished going through the articles:

Use of uninitialized value at newnews.pl line 150.
Use of uninitialized value at newnews.pl line 150.
Use of uninitialized value at newnews.pl line 150.
Use of uninitialized value at newnews.pl line 150.
Use of uninitialized value at newnews.pl line 174.

2) Posting hangs

It appears the script hangs when trying to send posts back to the newsgroups. tried twice and nothing has happened for over 10 minutes now.

3) Numbers..

is this odd:

Processing group alt.tv.stargate-sg1...2321 messages.

results:

usenet_ref : 6098
usenet_article : 1028
thread : 204
post : 774

seems a bit weird to me

i am going to try a few more newsgroups but any ideas on why posting ain't working?

[fastforward]

The uninitialized value messages are because the script is running using STRICT and with the -W switch. It's from all the regular expression testing in the article parsing bit. It basically means its trying to test an empty variable. It can be ignored and if you take of the -w switch it will disappear.

The numbers you show seem ok. The 2321 are the numbers the server reports it has. This is usually slightly high by maybe 25 to 100 messages. The usenet_ref table is the many to many link between articles and replies. Because you are doing the initial history pull you will find a lot of messages left in the usenet_article table because it contains replies to threads that have since expired from the server. Once history is collected you can set the $expire variable to a sensible value and it will gradually reduce as time goes on.

As for the posting problem, it works for me in alt.test. I'll keep looking looking and let you know if I find any problems. What message are you seeing before it hangs? Do you see the "Posting message by $poster to $newsgroup..." message?

[fastforward]

The posting problem is an SQL statement problem. I removed some unwanted fields from the usenet_outgoing table but didn't change the statement. Unfortunately I was using select * instead of listing the fields.

In the post_outgoing routine (the last function in the script) change line 281 that reads my '$q2 = db_fetch(...'
to read:
my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body FROM usenet_outgoing");
And change the line below it to:
while ( my ($poster,$newsgroup,$subject,$refs,$body) = $q2->fetchrow_array ) {

Sorry about that. The fix is in the package for download.

There's always something I mess up!

ps. There's also another little change you can make to showthread.php that will ensure the posts are ordered correctly everytime. Not essential but I've put it in the package.

[Edited by fastforward on 01-22-2001 at 12:39 AM]

[chrispadfield]

thank you, i noticed when i posted a post to a thread it actually appeared 2nd to last as opposed to last so if it sorts that out, brilliant.

thanks, will apply fixes later today and see how it goes

[chrispadfield]

ok couple of things:

I got the new download and got it working. It let me post ok but the post i did sent about 5 messages, some with the headers in which was a bit bizarre. I thought it might be corrpution in the database from old failed attempts to post so started again, and posting worked.

Now:

1) I am getting this a lot

Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.

after it finises a run of article download. Any ideas?

2) Worried about numbers of articles/posts coming through.

Can you explain what usnet_ref and usenet_article are for so i can understand. These are my stats

usnet_ref = 16520 entries
usnet_article = 3179 entries

threads = 724
posts = 1265

it just feels that i should have more threads and posts then i do.

Thanks,

I am going to continue playing.

p.s. i still get heavy heavy failed downloads form alt.tv.stargate-sg1 which is strange, it is a prefectly valid newsgroup. alt.tv.farscape however has very very few. I wonder why that is? perhaps if you have a chance you could have a play with that newsgroup.

--

Anyway, it is a pretty amazing hack i really love it just hope these few things can be ironed out

cheers

[fastforward]

Chris, the numbers you are seeing are fine. The variable messages are just warnings. See my post earlier in this thread about both these issues. If you really want to hide those warnings remove the -w after the perl line at the top of the script.

I'll look into the excessive bad header thing for you.

regards

[chrispadfield]

are thanks, i understand now. I missed that post of yours, only saw your second one and it answered my questions.

Love the hack, thanks for all the work.

[chrispadfield]

sorry to be such a complete pain here.

Ok, i ran the script again (not on cron yet)

and it went through alt.tv.farscape:

Requesting article 58127 from alt.tv.farscape...Saved
Requesting article 58128 from alt.tv.farscape...Saved
Requesting article 58129 from alt.tv.farscape...Saved
Requesting article 58130 from alt.tv.farscape...Saved
Requesting article 58131 from alt.tv.farscape...Saved
Requesting article 58132 from alt.tv.farscape...Saved
Requesting article 58133 from alt.tv.farscape...Saved
Requesting article 58134 from alt.tv.farscape...Saved
Requesting article 58135 from alt.tv.farscape...Saved
Requesting article 58136 from alt.tv.farscape...Saved
Requesting article 58137 from alt.tv.farscape...Saved
Requesting article 58138 from alt.tv.farscape...Saved
Requesting article 58139 from alt.tv.farscape...Saved
Requesting article 58140 from alt.tv.farscape...Saved
Requesting article 58141 from alt.tv.farscape...Saved
Requesting article 58142 from alt.tv.farscape...Saved
Requesting article 58143 from alt.tv.farscape...Saved
Requesting article 58144 from alt.tv.farscape...Saved

but i go to the forum, and there are only two threads now marked new which seem to have anything else in them..

weird i thought. went into myphpadmin type thing and looked at the last records under usenet_article and searched for them (ie bits of their body text) and got no results.

So it does not appear to be a date/time thing (which would explain why there were not coming up the top marked new), it just appears they are not actually getting into the database.

On this point, i understand that usenet_ref is a many to many thing so there will be loads, but what about usenet_article. Should this not be similair to the number of posts? if so, that is my problem as my usenet_article is 3 times bigger than my number of posts.

Sorry about all these questions, bet you regret releasing this now

[fastforward]

When articles are pulled down from usenet, their order within the thread is calculated and the article is placed in usenet_article. This table is just a temp holding area. At the same time the usenet_ref table is populated with the mesage ids of the articles it refers to.

The next step is to load any messages from the usenet_article table that has no refs (ie. an initial thread starter) into the thread and post tables. Then it deletes the article from the usenet_article and usenet_ref table.

Next it goes through the usenet_article table in conjuntion with the usenet_ref table and loads any messages that refer to an article already in the post table. It will only place it in to the post table if it is in order. For example, if post 1 and 2 are already in the post table and 4,5 and 6 are in the usenet_article table, the articles will NOT get loaded because article 3 is still not available. If you look at the 'ord' column, only posts at the same level or one higher will get loaded. Then it deletes the article from usenet_article and usenet_ref.

As an example for numbers you might expect; I mirror 6 newsgroups and have 2069 in thread, 4463 in post, 1288 messages still in usenet_article and 4463 in usenet_ref. My $expire is set to 14 days. This means any messages left in usenet_article that are over 14 days old will be deleted. Remember, the reason they are still in usenet_article is because the article they refer to is not available on the server or was rejected for spam or something.

The messages not showing up as new, happens due to the nature of usenet and the way messages are propogated. The modification made to showthread.php uses the 'ord' column to sort by first, then the dateline. This means makes the messages appear in the correct order. However, vBulletin uses date to determine whether the message is new. This means not all messages will show up new when they should. You will also run into problems with users not setting their clock correctly. Although the script handles time zones correctly, I often see articles with a future date because the users pc clock is wrong.

The first version I made of this script created a dummy date based on the order of the article. So each time a post was received the time was set to a few minutes after the article it refered to. This avoided the problem but meant you had no way of knowing when the post was really made. Maybe I should go back to this method. What do you think?

Addendum:
Since posting this, I just took another look at showthread.php. I believe it will be quite easy to fix the new post icon issue. I will have to create an additional dummy date field based on the actual date the article is placed into your forums. The real date will remain where it is now. I will modify showthread.php to use the dummy date field when doing anything with usenet forums but use the normal field for other forums. This will also eliminate the need for the original modification in that file. I'll try to make the fix this evening.

regards

[Edited by fastforward on 01-23-2001 at 10:48 AM]