Error while Installing/Compiling the requirements

[mazzespazze]

Hello,

I am encountering an error when installing your requirements. You stated that you tested with SBT 0.31. According to the same link you provide, such version of SBT is no longer available (or it even seems it never existed)

Therefore I just downloaded the most recent version from here.
Then I run the code as you suggest with the following command

sbt "runMain ch.ethz.dalab.web2text.ExtractPageFeatures index.html desired_output"

Sadly after some downloads, I encountered the following errors:

[error] web2text/cdom/Node.scala:39:33: missing parameter type

[error] val c = for (c <- children; l <- c.toString.lines) yield {"  " + l}
                            ^
[error] web2text/src/main/scala/ch/ethz/dalab/web2text/cleaneval/CleanEval.scala:140:54: value drop is not a member of java.util.stream.Stream[String]

[error] val contents = if (f.startsWith("URL:")) f.lines.drop(1).mkString("\n")
                                            ^
[error] web2text/src/main/scala/ch/ethz/dalab/web2text/features/PageFeatures.scala:29:63: type mismatch;

[error] found   : java.util.stream.Stream[String]

[error] required: Iterator[?]

[error] (blockFeatureLabels.toIterator zip blockFeatures.toString.lines)               
                                         
[error] three errors found

[error] (Compile / compileIncremental) Compilation failed

[error] Total time: 6 s, completed Dec 23, 2020, 5:07:31 PM

How else can I compile your code? Is there any docker image you made back then that it is possible to run today?

Thanks!

[tvogels]

Hi, thanks for reporting this! I think the SBT version was 1.3.3 (updated on the master branch now). Have you tried running the code in the docker image mentioned under https://github.com/dalab/web2text/blob/master/README.md#installation?

[mazzespazze]

Thanks to you for such a quick reply!
I tried installing SBT 1.3.3 and I get the exact same error message.

Now I pulled the docker image, but I'm quite a newbie when comes to docker. How do I execute the code now? Can you provide me an execution command? Thanks

This is my docker image information

hseeberger/scala-sbt 8u222_1.3.3_2.13.1 9ca10f91beae 13 months ago 721MB

[tvogels]

After creating a file result/input.html, I can run the first command from Xavier Geerinck's step-by-step guide in Docker with:

docker run -it --rm \
    --mount type=bind,source="$(pwd)",target=/root \
    hseeberger/scala-sbt:8u222_1.3.3_2.13.1

and then, inside Docker

sbt "runMain ch.ethz.dalab.web2text.ExtractPageFeatures result/input.html result/step_1_extracted_features"

or in one go with:

docker run -it --rm \
    --mount type=bind,source="$(pwd)",target=/root \
    hseeberger/scala-sbt:8u222_1.3.3_2.13.1 \
    sbt "runMain ch.ethz.dalab.web2text.ExtractPageFeatures result/input.html result/step_1_extracted_features"

Good luck. Let me know if this helps!

[mazzespazze]

The command works perfectly. I'm finally able to run the "first step".

Though now I have an error with the second step:

$python src/main/python/main.py classify result/step_1_extracted_features result/step_2_classified_labels

Traceback (most recent call last):
  File "src/main/python/main.py", line 7, in <module>
    from forward import EDGE_VARIABLES, UNARY_VARIABLES, edge, loss, unary
  File "/web2text/src/main/python/forward.py", line 2, in <module>
    from tensorflow import variable_scope, convert_to_tensor
ImportError: cannot import name 'variable_scope' from 'tensorflow' (/usr/local/lib/python3.8/dist-packages/tensorflow/__init__.py)

Perhaps an issue with the Tensorflow versioning?

[tvogels]

Hi @mazzespazze, great to hear it worked. Yes, that sounds like a Tensorflow version issue. Could you try version 1.0? Let me know if it worked.

[mazzespazze]

Hi again @tvogels, thank for your patience!

It was more sneaky than I thought. It seems Tensorflow 1.15 is currently not available for python3.8 so the complete flow for having step two running was:

python3.7 -m pip install upgrade pip
python3.7 -m pip install setuptools testresources
python3.7 -m pip install numpy==1.18.0 tensorflow==1.15.0 tensorflow-gpu==1.15.0

And then following this guide (as you mention in your ReadMe), going in the python file src/main/python/config.py and edit under FLAG such that:

import sys

FLAGS = tf.app.flags.FLAGS
FLAGS(sys.argv)

As they mention running now the code python3.7 src\main\python\main.py classify result\step_1_extracted_features result/step_2_classified_labels will end up in an error due to running the python code from the root of the project. Though they don't mention that the lines to replace are in main.py and there are several occurrences.

Regarding step three (and hopefully last) I run sbt "runMain ch.ethz.dalab.web2text.ApplyLabelsToPage result/input.html result/step_2_classified_labels result/step_3_applied_labels"that ends up in an error:

[error] (Compile / compileIncremental) java.nio.file.AccessDeniedException: /root/.cache/coursier/v1/https/repo1.maven.org/maven2/org/mongodb/casbah-core_2.10/3.1.0/casbah-core_2.10-3.1.0.jar
[error] Total time: 4 s, completed Dec 28, 2020, 12:46:56 PM

I noticed the "permission", so I tried re-running it with sudo (even though I don't think it should be the case, any idea why? :/ ), but I end up in another error:

sudo sbt "runMain ch.ethz.dalab.web2text.ApplyLabelsToPage result/input.html result/step_2_classified_labels result/step_3_applied_labels"
a lot of downloading...
[error] web2text/src/main/scala/ch/ethz/dalab/web2text/cdom/Node.scala:39:33: missing parameter type
[error]     val c = for (c <- children; l <- c.toString.lines) yield {"  " + l}
[error]                                 ^
[error] web2text/src/main/scala/ch/ethz/dalab/web2text/cleaneval/CleanEval.scala:140:54: value drop is not a member of java.util.stream.Stream[String]
[error]     val contents = if (f.startsWith("URL:")) f.lines.drop(1).mkString("\n")
[error]                                                      ^
[error] /web2text/src/main/scala/ch/ethz/dalab/web2text/features/PageFeatures.scala:29:63: type mismatch;
[error]  found   : java.util.stream.Stream[String]
[error]  required: Iterator[?]
[error]     (blockFeatureLabels.toIterator zip blockFeatures.toString.lines)
[error]                                                               ^
[error] three errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 52 s, completed Dec 28, 2020, 12:49:49 PM

So as you suggested previously I run it with the docker sbt:

docker run -it --rm --mount type=bind,source="$(pwd)",target=/root hseeberger/scala-sbt:8u222_1.3.3_2.13.1 "runMain ch.ethz.dalab.web2text.ApplyLabelsToPage result/input.html result/step_2_classified_labels result/step_3_applied_labels"

But once again, I get an error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"runMain ch.ethz.dalab.web2text.ApplyLabelsToPage result/input.html result/step_2_classified_labels result/step_3_applied_labels\": stat runMain ch.ethz.dalab.web2text.ApplyLabelsToPage result/input.html result/step_2_classified_labels result/step_3_applied_labels: no such file or directory": unknown.

[getorca]

I'm also getting the same error first reported by mazzespazze. I'm getting a similar error when running with the docker command you shared

[error] Exception in thread "main" java.io.FileNotFoundException: result/input.html (No such file or directory)
[error] 	at java.io.FileInputStream.open0(Native Method)
[error] 	at java.io.FileInputStream.open(FileInputStream.java:195)
[error] 	at java.io.FileInputStream.<init>(FileInputStream.java:138)
[error] 	at scala.io.Source$.fromFile(Source.scala:90)
[error] 	at scala.io.Source$.fromFile(Source.scala:75)
[error] 	at scala.io.Source$.fromFile(Source.scala:53)
[error] 	at ch.ethz.dalab.web2text.utilities.Util$.loadFile(Util.scala:103)
[error] 	at ch.ethz.dalab.web2text.ExtractPageFeatures$.extractPageFeatures(ExtractPageFeatures.scala:39)
[error] 	at ch.ethz.dalab.web2text.ExtractPageFeatures$.main(ExtractPageFeatures.scala:25)
[error] 	at ch.ethz.dalab.web2text.ExtractPageFeatures.main(ExtractPageFeatures.scala)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / runMain) Nonzero exit code returned from runner: 1
[error] Total time: 29 s, completed Mar 4, 2021 11:43:07 PM

[getorca]

man do I feel like an idiot, but for anyone else that finds this, you need to create result/input.html file, that's an argument you need to pass with the path of the html file you want to extract from