Weekly DC Stats scripts

Post by **StefanR5R** » Thu Jun 23, 2022 8:03 pm

Here are the scripts which I am currently using to create the Weekly DC Stats postings over at anandtech.com. The principle is that you start the main script in a terminal, then copy and paste the output in the forum posting editor box, and finish it off by adding some commentary of your own. (The boilerplate text and the weblinks section is already included in the script output, you don't have to copy it from anywhere else.)

Scripts? Main script? — Yep, there are three scripts involved. (Or four, since we currently have Bonus Stats.) Originally, there was just one script which fetched all of the stats data from FreeDC. But because FreeDC stopped collecting stats for non-BOINC projects in July 2020, I eventually added two more scripts:

The second script fetches Folding@home stats from several web pages at folding.extremeoverclocking.com. EOC's "last 7 days" data are somewhat hard to get, so I am using "24hr Avg" figures instead and multiply them by 7, which is equivalent apart from very small rounding errors.

The third script fetches stats from a few web pages at distributed.net. I am currently using this for the projects OGR-28 and RC5-72. Unfortunately, weekly stats are not available at distributed.net at all. Therefore I am using local cache files for all-time stats from the previous Sunday. Let's say you start this script on Sunday, August 28. The script will download up-to-date all-time stats of all current TeAm members, and of up to 1000 teams, and store them all in a files called OGR-28/28AUG2022.txt, OGR-28/28AUG2022_teams.txt, RC5-72/28AUG2022.txt, and RC5-72/28AUG2022_teams.txt. The data in there consist of name and credit, separated by tabs and newlines. Then it will look for a files called OGR-28/21AUG2022.txt and so on. If such files are there, it will match names in the respective pairs of files, subtract previous Sunday's credit from current credit, and then log the result to the text terminal in a format which looks like FreeDC's weekly stats. (The user name matching is a bit more complicated than it seems at first glance, because names are not unique. I could extract unique user IDs out of the HTML, but I implemented an easier heuristic method instead.) The script will also delete any {OGR-28,RC5-72}/{day}{month}{year}{,_teams}.txt files which don't belong to the current day or to last Sunday.

To get OGR-28 stats, the script needs to be invoked with the following parameters:
./dnet_weekly_stats_dump.sh OGR-28 28
(28 is distributed.net's numerical ID of the project called 'OGR-28'.)

To get RC5-72 stats, the script needs to be invoked such:
./dnet_weekly_stats_dump.sh RC5-72 8
or simply
./dnet_weekly_stats_dump.sh
(8 is distributed.net's numerical ID of the project called 'RC5-72'.)

Notes:

After you download these scripts, don't forget to set their executable flag.
The normal mode of operation is that you call only the main script. It will call the other two scripts and insert their output into the overall output.
The main script expects the other two scripts to be named "fah_weekly_stats_dump.sh" and "dnet_weekly_stats_dump.sh", and that these two scripts are located in the same directory as the main script.
Each of the scripts requires the text web browser "links". Most distributions should have this packaged, but it is unlikely to be installed by default.
I am relying on "links" to a) perform the web download, b) convert HTML to plaintext.
I spoke above about cache files like OGR-28/28AUG2022.txt. Actually, the dnet stats script expects these data files to reside at the path "${HOME}/Distributed_Computing/Weekly_Stats/OGR-28" and "${HOME}/Distributed_Computing/Weekly_Stats/RC5-72". You need to create these exact directories first in order to be able to run that script. If you prefer a different path, edit the top of dnet_weekly_stats_dump.sh accordingly.

weekly_stats_dump.sh (the main script)