Feed Bot Issues

[Skillz]

So I have discovered an issue with the feed bot. Not sure how long this has been going on or if it's something new.

When the feed bot gets to a URL that is currently down then it will stop at that feed and not continue. This means, out of the 30+ feeds we are currently pulling it is stops at number 5 on the list, then 6 - 30 will not be checked.

I have reached out to the author of the extension to see if it's something on my end or if it's a problem with the extension.

I thought, the "timeout" column for each URL addresses that, but apparently not.

Added in 23 minutes 10 seconds:
Crap, I just noticed it has double posted A LOT of posts while I was trying to figure out the issue. Crap, crap, crap

[StefanR5R]

I thought, the "timeout" column for each URL addresses that, but apparently not.

I had been wondering about that too. Though if a simple timeout causes the bot to quit entirely, then we should apparently increase the timeout threshold in order to decrease the likelihood of timeouts. It is currently "3", which I presume is 3 seconds. How about making it "30" in each feed?

[Skillz]

I thought about that, too; however I think the issue is the fact that when the feed is down then it doesn't matter how long the time out is since it's going to time out anyway. Correct? I meant to keep track of both the feeds I had to remove so I could add them back later, but I found two feeds that when I went to the site directly their site/server was down. So no response from them at all.

[StefanR5R]

Counter to intuition, it should be possible to disable a feed via its "feed forum" menu, top entry.

The "Feed processing has started but not completed and therefore cannot start again. If this persists you can release the process by clicking this button: Submit" condition still happens. I'll go ahead and increase the timeout to "10".

I am also disabling ODLK1's feed for now because it has got malformed XML. But since the bot was sometimes able to proceeed in the past, I suspect this particular issue does not throw off the bot.

[Skillz]

Ah that's how to disable it. I couldn't find it when I went looking to disable one that was offline.

I posted on ODLK1's site a little while ago about their feed being broken: https://boinc.multi-pool.info/latinsqua ... php?id=140

Maybe they'll fix it.

[StefanR5R]

Beefathome times out, I am disabling it too.

[StefanR5R]

This game sux. I disabled Collatz now, as the site times out today and thus kills the feed bot. Need to remember to periodically check for when Collatz returns.

[Skillz]

I need to edit the php settings to allow more than, I think, 30 seconds for it to timeout. I think since we have so many in there, by the time it gets to one that's broken/not working the 30 second timer is already almost done.

Give me a minute to get around to that here shortly.

I was also told that I need to enable something that'll help it, but I forget now. Have to go back to the phpbb forums, plugin help sub-forum to see what he told me I needed. I forgot about it.

Added in 46 minutes 24 seconds:
The extension he mentioned was curl, curl is installed on the server, but I can't get PHP to build with CURL. So I opened a support ticket to see if I can figure that out.

[StefanR5R]

Back up: Collatz, TBrada. :-)
Newly down: iThena, Gaia. :-(

[Skillz]

This is becoming a full time job for you and that wasn't my intentions. I need to figure out a better way to accomplish this so we don't have to babysit it all the time.

[StefanR5R]

Don't worry; I already tend to a different full-time job, one which actually pays. :-)

[StefanR5R]

Is it normal that the bot pulls the feeds much more frequently than configured at "Interval for automatic processing feeds (seconds)"? The Maintenance/ Admin log shows that it happens much more often, and at varying periods.

[Skillz]

I don't think it should.

Added in 3 minutes 37 seconds:
Seems when a user visits the site it triggers the bot. Not sure how to disable that. Will look into it when I get home this afternoon.

[StefanR5R]

Every now and then the feed bot is blocked by iThena. (Most of the time iThena does respond, but eventually it doesn't and then the bot needs manual intervention to get going again.) So I would rather like to leave the two iThena newsfeeds disabled for now.

[Skillz]

I'm okay with that.

[StefanR5R]

There is a new issue, perhaps because of the site migration:

"Feed processing has started but not completed and therefore cannot start again. If this persists you can release the process by clicking this button: [Submit]" --> Clicking [Submit] clears this state, as before. But:

"Fetch all feeds manually: [Submit]" --> Clicking [Submit] moves you to a blank page and does not start the bot. Or at least, the maintenance log does not show any bot activity.

[Skillz]

I will be looking for an updated version or a new feedbot for the latest version of the forum.

edit
Their might be a way to select how old of a post it searches for in the feeds and then posts any from that point forward. If we set a cronjob on the server that definitely wont ever fail w/ those lines commented out then lower how many days/hours it checks back we might be able to prevent that somehow.

[StefanR5R]

Since Sat Nov 18 at the latest, circa Wed Nov 15 at the earliest, "Fetch all feeds manually" takes no time but immediately returns with "No (new) items to fetch", which seems suspicious.

[Skillz]

Same result for me as well.

I looked around a few of the BOINC projects and didn't see any new posts. I'll check more later to see if any of them do have posts since then.

[Skillz]

Okay, its still doing the same thing and I know for sure at least one project (RakeSearch) has posted news recently. Will be looking into this tomorrow.