Generating list of repositories from github using github api

[michalbundyra]

Currently https://framework.zend.com/status does not contain all repositories from zendframework org (the same on stats, documentation, issues). The current list is generated from https://github.com/zendframework/zf-mkdoc-theme/blob/gh-pages/scripts/zf-component-list.json which was created mainly to use in documentation.

I've created new script (bin/repos.php <github-token>) to generate config file with all repositories to get data from github api. It is using knplabs/github-api. I've added this as development dependency* in composer, so for now it should be generated locally and committed to the repository. We can change it and add it to the build process. Scripts requires provide github-token - because there is hourly limit on github api request (without token is 60, with token is 5000*).
Generated component list is associative array:

<?php
return [
    'zf-components' => [
        'component-name' => [
             'description' => '<github component description>',
             'docs' => <0|1>,
        ],
        ....
    ],
];

As you can see, description is fetched form github repository description, and probably it does not match description from zf-component-list.json. We can write another script to sychronize these descriptions.
docs has value 0 or 1 which means if there is documentation for the component. We check if mkdocs.yml file exists in the repository.

On documentation page we display repositories only with documentation.

When we generate statistics we use only components published on packagist, so packages like maintainers, zf3-web, ... are not gonna be on the statistics list.

* As I mentioned the script should be run locally, it's not a part of build process. This is because, not sure if you'd like to add these composer dependencies on production. Another concern is 5000 requests per hour (with github token). It seems to be a lot.
We have right now 149 repositories, so to generate the whole list of component we need to do around ~160 request to github api (1 request for each repo to check if mkdocs.yml exists, and listing all repositories). So even if we run build process multiple times on couple machines in the same hour it should be enough (shouldn't be?)

About dev dependencies... I just got an idea - if we slightly change build process:

1. install everything with dev dependencies
2. build website (repos list, stats, ... everything what we have in bin/build.php)
3. run composer install --no-dev to remove dev dependencies needed to build the website

Then we can have it on build process. What do you think?

[michalbundyra]

Forgot to mention: because we don't include all zendframework packages in statistics the total download number is slightly different.
Currently on website we have: 173,712,680
and the real number for all packages is: 187,647,968
(so only 13,935,288 difference...) 😄

[michalbundyra]

Here is PR to synchronize repos between zf-component-list.json file and github: zendframework/maintainers#22

[michalbundyra]

Ok, so... I've updated the PR. Now component list is generated on build process, and composer install runs twice: first to install all dependencies (including dev) and the second one (after build process) to remove these dev dependencies.

It is required to set GITHUB_TOKEN env variable on the server (see: creating a github token).