Local folder with subfolders

[XhmikosR]

@JustinBeckwith: I'm trying to add linkinator in the nodejs.org repo.

The file structure there is like this:

build/
  ar/
    index.html
    <more folders>/
      index.html
  ca/
    index.html
    <more folders>/
      index.html
  en/
    index.html
    <more folders>/
      index.html
  <more folders>/
  static/

So, if I do linkinator build/ --recurse it fails because it doesn't see an index.html. If I do linkinator build/en --recurse then I get false positives because linkinator can't find the root.

Is there a workaround or something you could do about it?

Thanks in advance!

[JustinBeckwith]

Ahh, that's not how --recurse works :) The --recurse flag just signals that we are going to follow links from the top level scanned page, and go another level deeper. Everything in linkinator assumes the link checking is happening in the context of either a.) a url you've pointed it to or b.) a directory with an index.html that sprawls out. It doesn't really know anything about filesystem.

To get this to work today - you'd have to run linkinator for each directory. Something like:

for d in */ ; do
    npx linkinator --recurse "$d"
done

[XhmikosR]

The thing is that if I do that then linkinator cannot find the root. So I'm stuck :/

[JustinBeckwith]

I suppose I'm a little confused. What do you mean by not being able to "find the root"? Effectively - linkinator expects that it starts from a single crawling point, and can follow links to all other links it wants to be checked.

Are you saying that the subdirectories throughout your structure also have independent roots that need to be crawled?

[XhmikosR]

If I do linkinator build/en --recurse, then linkinator throws errors because /static/ isn't in the en folder, it's one folder up.

Maybe adding a root option could help in such cases.

I'm not sure how I can make linkinator work in this case. Here's a WIP branch https://github.com/XhmikosR/nodejs.org/tree/master-xmr-linkinator

[JustinBeckwith]

Thank you - I get it now! All linkinator does when you point it at a directory is naively spin up a web server from the CWD to serve requests. You could do something very similar:

npx http-server .
for d in */ ; do
    npx linkinator --recurse http://localhost:8080/$d
done

[XhmikosR]

Thanks for the reply!

The problem with this approach is that it's not cross-platform, unfortunately. Also, http-server is broken BTW. I personally use the serve package, but it seems I'm having some issues in this case.

That being said, I'll see what I can cook up. I still believe a new option for linkinator to load files from is useful though.

Also a thing I notice from a quick test is that linkinator throws for the link dns-prefetch links.

[XhmikosR]

Alright, I successfully managed to set a CLI solution but that scans only one locale.

So, I figure we need to use the API, right? Not very familiar with it but it seems linkinator returns a Promise so my first attempt doesn't work.

If you have any time, here's my WIP branch with the API solution nodejs/nodejs.org@master...XhmikosR:master-xmr-linkinator-2

First, I want to at least land the English locale CLI scanning on nodejs/nodejs.org#2565 and then see how we can improve things with using the API directly.

[JustinBeckwith]

Using the API seems like the right thing to do here for sure. Linkinator does return a promise from the check method :) There's a simple example here:
https://github.com/JustinBeckwith/linkinator/#simple-example

[JustinBeckwith]

Just wanted to check in @XhmikosR - did you end up getting this rolling with the API? I was thinking alternatively - you could create an index page at the root level (mostly for testing purposes) that has a link to all of the per-language root pages, and then scan that page.

[XhmikosR]

@JustinBeckwith I started a patch one day nodejs/nodejs.org@1cdc841 but never got to tweak it :/

Do not hesitate to PR if you have some time :) We definitely get some false positives, but my plan is to just keep a list of whitelisted domains, which will be better than nothing.

[nkuehn]

Hi, I have a similar setup with microsites under the root that link to each other. I am now throwing an index.html with nothing but <a href="/foo"></a> (or multiple of these) into the root but it's not really nice. In production the webserver does a HTTP level redirect to the actual entry page, but that's not happening in the local webserver emulation here (how should it know my custom redirect...).

It would be great if the use case would be covered by the CLI, too since it's pretty good already with colored output, grouping etc.

Conceptually the issue is about being able to separate the first entrypoint from the scope of the recursion and the root path of the local webserver.

So what about an optional flag --entrypoint that allows to e.g. call

linkinator ./output-dir --entrypoint /home

With the effect that the local webserver is started on ./output-dir but the crawler starts at http://localhost:0000/home instead of defaulting to http://localhost:0000/home?

(could be a plural form too, taking multiple entrypoints)

Does that make sense? What do you think?