ci: test to check all links

[jorgeorpinel]

We have internal and external links, mainly in our MD files, which tend to change regularly and its easy to break them accidentally for different reasons. We could probably automate this check?

Per #637 (review)

[jorgeorpinel]

p.s per #637 (comment) (and biopython/biopython.github.io/issues/62)

There is also this: https://wummel.github.io/linkchecker/ But it seems to be a bit old and bloated.

[dashohoxha]

If this issue is still interesting, I may have a look at it and try to build a script.

I did not understand, what was the problem with the other script, that was deleted?

[shcheklein]

yep it's definitely useful if it's made part of the CI flow (CircleCI)

[dashohoxha]

The script is submitted here: #690

It is not completely robust and reliable, sometimes it may give false-positives, so maybe not suitable for automatic checking and CI. But it may still be useful for manual checking once in a while.

[jorgeorpinel]

Thanks. I'm a little behind on my own work but will check it out ASAP.

[shcheklein]

@jorgeorpinel @dashohoxha left my comments in the PR. I think it's a good first step, but does not resolve the issue. My major concern, no one will be using it (and someone will just do a PR do delete it like happened already) if it's not integrated into CI system.

[dashohoxha]

My major concern, no one will be using it (and someone will just do a PR do delete it like happened already)

I was actually using the deleted script each time before committing, and it always would find something to fix. But if it was only me that was using it, no problem, I can use a local copy of the script.

I can accept and merge this, but I feel it won't be used/useful w/o introducing a proper CI setup.

I think it's a good first step, but does not resolve the issue.

If it is not useful then there is no point in merging it.

Although I tried it manually and it reported a lot of issues (most of them actually are redirection problems, which are not really breaking any thing). It might be useful for manual checking, especially when we restructure the hierarchy of the docs (which we are planning to do).
But again, I can have a local copy and use it for myself.

[shcheklein]

I was actually using the deleted script each time before committing, and it always would find something to fix. But if it was only me that was using it, no problem, I can use a local copy of the script.

why not pre-commit hook? or the yarn command we have now?

If it is not useful then there is no point in merging it.

I'm not saying that it's not useful. I'm saying that there are extremely high chances that everyone will forget about it later if it's not part of the CI system and even not documented anywhere. It seems to me that it's not that big of a problem to make part of the CI system and run it on changed files? @algomaster99 could you take a look and help @dashohoxha with this?

[dashohoxha]

why not pre-commit hook?

Does this mean that I don't need to check and fix them because the git hooks will do it automatically?

or the yarn command we have now?

Which yarn command? The one mentioned here: https://dvc.org/doc/user-guide/contributing-docs#doc-style-guidelines-and-tips-for-java-script-and-markdown?
If I have to go to the docs and copy/paste a command, I usually make a small script for it.
By the way, the double star globing syntax mentioned there (**) as far as I know does not work by default, needs to be activated/enabled.

@algomaster99 could you take a look and help @dashohoxha with this?

I would prefer if @algomaster99 takes this script and makes it part of the CI system, because I don't know anything about it.

[shcheklein]

@dashohoxha

Does this mean that I don't need to check and fix them because the git hooks will do it automatically?

Yep. That's the whole point - it takes care about everything automatically before you do git commit.

Which yarn command? The one mentioned here: https://dvc.org/doc/user-guide/contributing-docs#doc-style-guidelines-and-tips-for-java-script-and-markdown?
If I have to go to the docs and copy/paste a command, I usually make a small script for it.
By the way, the double star globing syntax mentioned there (**) as far as I know does not work by default, needs to be activated/enabled.

I think, yes. @jorgeorpinel can address this question better I believe and concern with **.

makes it part of the CI system, because I don't know anything about it.

sure, np. Please, address some point in the PR and I'll merge it, will keep this ticket open. We can update the description of the ticket to mention the script.

[algomaster99]

@shcheklein sure I will look into it 👌🏼