Weekly DC Stats scripts

[Skillz]

Does the script have to be run multiple times for a reason? Like are there instances where the script fails to do what it's supposed to do so running it every minute for an hour guarantees success at least once?

[crashtech]

No, I just want it to run once. Maybe my syntax is wrong, I need to check it. I assumed that is was failing and retrying, sounds like I am wrong about that. I only want it to run one time a week, at 3AM on Sunday morning.

I will put the whole path of the output file as well.

[Skillz]

Well the cronjob should look like this;

0 3 * * sun
or
0 3 * * 7

That's at 3am every Sunday, every week. It will only run once.

Cool site I use to check the cron scheduling is here:
https://crontab.guru/

Very useful for taking the "work" out of setting up a cronjob.

[crashtech]

Meh. Now I have something that is slightly closer to working. It runs, but it creates a blank text file instead of writing the output into it.

Code: Select all

0 3 * * 0 ./home/garage/Desktop/Weekly_Stats_Scripts/weekly_stats_dump.sh > /home/garage/Desktop/Weekly_Stats_Scripts/weekly_stats_dump_output.txt

There doesn't seem to be a problem with permissions. Running the command as a normal user from the root directory is successful.

[Skillz]

It might be better to write in the script to output the results to a file.

[crashtech]

When I run that in a terminal window, it does write the results to the file specified. So if there is a better way, I don't know how.

[Skillz]

I mean without having to use > to dump the output in the terminal to a file.

I'll check it out tonight and do some research on how to implement an output within' the script code directly. Gotta be a way.

[StefanR5R]

Remove the dot before /home in the crontab line.

[crashtech]

Remove the dot before /home in the crontab line.

Okay I will. Initially I did not have it in there, I'll take it back out.

[StefanR5R]

Opening post edited:

weekly_stats_dump.sh:
– Don't look for OGR-28 updates anymore.

fah_weekly_stats_dump.sh:
– Use firefox's user agent string in the HTTP requests which are issued via the "links" browser.
– Insert a small delay between HTTP requests, to go easy on the folding.extremeoverclocking.com web server. This script issues only 3…4 requests though, so it's not been a big deal.

[StefanR5R]

I edited the 1st post…
– Fixed fah_weekly_stats_dump.sh for ≥1 billion points/week/member.
…and the 2nd post:
– Updated the link to where I look up home teams of guest crunchers.
– Updated weekly_stats_dump_ukraine.sh with the current guests table.

[StefanR5R]

1st and 2nd post updated: added WUProp to weekly_stats_dump.sh and weekly_stats_dump_ukraine.sh

[crashtech]

I've been getting these stats scripts working again in the unlikely event that I might be called upon to be a substitute. But I'm unsure of my ability to check for changes in the Ukraine guest cruncher category.

Edit: Well, check and then properly implement changes to the script. It appears that the list gets updated in the OP as changes are discovered, so running the most recent version of the script would result in fewest errors even if a change to the list is not caught, that's what I might do if I need to sub.

[crashtech]

I got one computer working to do the stats and now I am copying the configuration to a second computer at another location. This should add enough redundancy so that I can help in the future if needed.

Ukraine stats would at worst contain the latest guest cruncher list posted to this thread.

[crashtech]

Well I seem to be doing better at the backup stats keeping, but I apparently do not have the proper character set installed to correctly render the name of whoever was 7th place in Asteroids for Team Ukraine.

[StefanR5R]

I wouldn't worry about that. FreeDC and BoincStats and the Asteroids@home site all show different characters in this user name to me. If this is not surviving the data export from Asteroids or the data imports by FreeDC and BoincStats, it would be a bit much to compensate for that in our import from FreeDC.

[crashtech]

I'm pleased to report that I have the stats scripts running as cron jobs on two different computers at two different locations, so there is now high confidence I will be able to provide the stats if needed.

[StefanR5R]

I shall take you up on that eventually. On June 2 for example, but I'll send a reminder when it's time. :-)

[crashtech]

At worst I would use a guest cruncher list for Ukraine that might possibly be a week out of date, if it's safe to assume that you edit post #2 here when you detect changes in guest crunchers.

[crashtech]

Inaccuracies have accumulated in my version of the Team Ukraine weekly stats. These users are missing their usual team affiliation in my version:

Skip Da Shu...............Guru Mountain
Giovani Avelar............none
pkasatka....................Gridcoin
Krasavetz...................TSC! Russia
Graith.........................Gridcoin

I'm not super confident about fixing this on my own; the scripts are working and I'd like to keep it that way! Perhaps an updated version of the script could be posted so that my backup versions are congruent with your main ones once more?

[StefanR5R]

I edited the 2nd post (only the CODE section) with my current version of weekly_stats_dump_ukraine.sh:
Some more guests, and minor improvements to the sed code for certain special cases of usernames.


Edit, adding more guests is most of the time just a trivial matter of adding a line of the form "username.......origin team" in the respective list. But I had to enhance the sed input code for edge cases occasionally, more recently for a name with "&" in it.

[crashtech]

Thanks, Stefan. While I imagined it was probably trivial to edit more guests/teams into the script, I am still much better at breaking scripts than fixing them. I'm glad you did it for me.

[crashtech]

I'm looking into the need to add support for Cyrillic characters in my Linux Mint installations.

[StefanR5R]

I updated the 1st post, dnet_weekly_stats_dump.sh code section:
It now discards all output lines with less than 10 credits/week, but prints the total number of lines which got discarded this way. The previous version of the script did this already with lines with 1 credit/week. (Today for example there were 71 one-credit records, 5 two-credit records, and 1 three-credit record. I think they are all due to some sort of rounding bug in distributed.net's stats keeping, and should therefore be ignored.)

I also updated the 2nd post, weekly_stats_dump_ukraine.sh code section:
I added special handling of the member names Александр, João Mota, and райдужний. These names are already corrupted in the respective project servers' stats export files. (The actual character conversion method mismatches what the XML file headers say.) They only show correctly on the various web pages of the project servers, whereas 3rd party stats sites show them in different but wrong ways. The script now converts what FreeDC is making of them back into how they look at the projects' web pages. Unfortunately, I have not figured out an easy way to do this with any arbitrary name, which is why I coded this ad hoc for these three particular names only. More names can be added later manually in the same way.

Note, I used UTF-8 encoding when I wrote the script, and when I copy-and-pasted it here into the forum post.

[crashtech]

Well, that explains why I couldn't fix it. I've incorporated your changes, thanks!